Hello,
I am testing an Aerospike setup in AWS. I am benchmarking with a large server and 4 clients.
When I log into the server instance and run a top command, I see the CPU is 99% idle. However, when I look at the Cloudwatch metrics, I see that the instance is using 40% CPU consistently.
I am using a c3.8xlarge instance for the server,
Any ideas why there is a discrepancy?
Thanks Chris
Seems like the problem is with Cloudwatch, an Amazon product. If you SSH into the instance and top shows you that CPU is 99% idle, that’s probably the source of truth.
Actually, I would think Cloudwatch measures the actual CPU. Especially considering that I have 4 clients sending data to the server.
Its clearly doing work, but top is not seeing it.
Just in case. I am attaching the top (1) with Shift i to turn off irix mode. (CPU shows 1090% with irix mode on)
top
top - 17:33:13 up 8 min, 1 user, load average: 2.59, 1.25, 0.50
Tasks: 297 total, 3 running, 294 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 1.9 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 3.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.6 us, 1.2 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 1.8 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.6 us, 1.2 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.6 us, 1.8 sy, 0.0 ni, 97.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.6 us, 1.8 sy, 0.0 ni, 97.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.5 us, 0.5 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.9 us, 0.5 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 : 0.5 us, 0.5 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.9 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu16 : 0.0 us, 1.4 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu17 : 0.5 us, 1.4 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu18 : 0.0 us, 0.6 sy, 0.0 ni, 98.9 id, 0.6 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu19 : 0.0 us, 1.1 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu20 : 0.0 us, 0.6 sy, 0.0 ni, 99.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu21 : 0.0 us, 1.1 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu22 : 0.6 us, 1.1 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu23 : 0.6 us, 1.1 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu24 : 0.6 us, 0.6 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu25 : 0.0 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 99.7 si, 0.0 st
%Cpu26 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu27 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu28 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu29 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu30 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu31 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu32 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu33 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu34 : 0.0 us, 0.5 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu35 : 0.0 us, 0.0 sy, 0.0 ni, 99.5 id, 0.5 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 61402988 total, 58570228 free, 2610808 used, 221952 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 58382756 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2511 root 20 0 7299624 2.004g 3096 S 30.4 3.4 44:31.11 asd
135 root 20 0 0 0 0 R 2.7 0.0 2:51.11 ksoftirqd/25
140 root 20 0 0 0 0 R 2.7 0.0 2:47.53 ksoftirqd/26
10 root 20 0 0 0 0 S 0.0 0.0 0:00.31 rcu_sched
Here is the output of mpstat 3… idle about 91%, but it is showing almost 70%.
05:37:41 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:37:44 PM all 0.19 0.00 0.65 0.06 0.00 8.35 0.00 0.00 0.00 90.75
05:37:47 PM all 0.22 0.00 0.70 0.03 0.00 8.37 0.00 0.00 0.00 90.68
05:37:50 PM all 0.21 0.00 0.51 0.04 0.00 8.32 0.00 0.00 0.00 90.92
05:37:53 PM all 0.17 0.00 0.65 0.06 0.00 8.30 0.00 0.00 0.00 90.82
05:37:56 PM all 0.22 0.00 0.66 0.07 0.00 8.38 0.00 0.00 0.00 90.67
^C
Average: all 0.20 0.00 0.64 0.05 0.00 8.34 0.00 0.00 0.00 90.77