CPU Utilization mismatch on AWS


#1

Hello,

I am testing an Aerospike setup in AWS. I am benchmarking with a large server and 4 clients.

When I log into the server instance and run a top command, I see the CPU is 99% idle. However, when I look at the Cloudwatch metrics, I see that the instance is using 40% CPU consistently.

I am using a c3.8xlarge instance for the server,

Any ideas why there is a discrepancy?

Thanks Chris


#2

Seems like the problem is with Cloudwatch, an Amazon product. If you SSH into the instance and top shows you that CPU is 99% idle, that’s probably the source of truth.


#3

Actually, I would think Cloudwatch measures the actual CPU. Especially considering that I have 4 clients sending data to the server.

Its clearly doing work, but top is not seeing it.

Just in case. I am attaching the top (1) with Shift i to turn off irix mode. (CPU shows 1090% with irix mode on)

top
top - 17:33:13 up 8 min,  1 user,  load average: 2.59, 1.25, 0.50
Tasks: 297 total,   3 running, 294 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  1.9 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  3.0 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.6 us,  1.2 sy,  0.0 ni, 98.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  1.8 sy,  0.0 ni, 98.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.6 us,  1.2 sy,  0.0 ni, 98.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.6 us,  1.8 sy,  0.0 ni, 97.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.6 us,  1.8 sy,  0.0 ni, 97.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu8  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu9  :  0.5 us,  0.5 sy,  0.0 ni, 99.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 :  0.9 us,  0.5 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu11 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu12 :  0.5 us,  0.5 sy,  0.0 ni, 99.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu13 :  0.0 us,  0.9 sy,  0.0 ni, 99.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu14 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu15 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu16 :  0.0 us,  1.4 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu17 :  0.5 us,  1.4 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu18 :  0.0 us,  0.6 sy,  0.0 ni, 98.9 id,  0.6 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu19 :  0.0 us,  1.1 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu20 :  0.0 us,  0.6 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu21 :  0.0 us,  1.1 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu22 :  0.6 us,  1.1 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu23 :  0.6 us,  1.1 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu24 :  0.6 us,  0.6 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu25 :  0.0 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 99.7 si,  0.0 st
%Cpu26 :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu27 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu28 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu29 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu30 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu31 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu32 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu33 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu34 :  0.0 us,  0.5 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu35 :  0.0 us,  0.0 sy,  0.0 ni, 99.5 id,  0.5 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 61402988 total, 58570228 free,  2610808 used,   221952 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 58382756 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  2511 root      20   0 7299624 2.004g   3096 S 30.4  3.4  44:31.11 asd
   135 root      20   0       0      0      0 R  2.7  0.0   2:51.11 ksoftirqd/25
   140 root      20   0       0      0      0 R  2.7  0.0   2:47.53 ksoftirqd/26
    10 root      20   0       0      0      0 S  0.0  0.0   0:00.31 rcu_sched

Here is the output of mpstat 3… idle about 91%, but it is showing almost 70%.

05:37:41 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
05:37:44 PM  all    0.19    0.00    0.65    0.06    0.00    8.35    0.00    0.00    0.00   90.75
05:37:47 PM  all    0.22    0.00    0.70    0.03    0.00    8.37    0.00    0.00    0.00   90.68
05:37:50 PM  all    0.21    0.00    0.51    0.04    0.00    8.32    0.00    0.00    0.00   90.92
05:37:53 PM  all    0.17    0.00    0.65    0.06    0.00    8.30    0.00    0.00    0.00   90.82
05:37:56 PM  all    0.22    0.00    0.66    0.07    0.00    8.38    0.00    0.00    0.00   90.67
^C
Average:     all    0.20    0.00    0.64    0.05    0.00    8.34    0.00    0.00    0.00   90.77