High CPU rate in suddenly


#1

Hi, Aerospike Team

We are faced high CPU usage problem in suddenly. The CPU usage doubled from 1/4 00:00 UTC. Here is a performance graph.

We didn’t add feature especially before and after. If you any advice, please teach us.

Environment and statistics

Version: Comunity Edition 3.14.0
Environment: AWS EC2 i3.xlarge * 3 (cluster size: 3, replication factor: 2)
Total Objects: 574340020
heap_efficiency_pct=92
system_free_mem_pct=59

#2

AWS rolled out security patches to the Meltdown vulnerability to the underlying hypervisor. All clouds are likely affected though I hear some much less than others - @rguo know more detail on other clouds?

See https://meltdownattack.com/.


#3

High(er) CPU is merely a symptom of current patches. That by itself is not a major issue. It will impact capacity sizing, but does not impact current operations unless you’re already CPU starved.

What is a major issue is that IO latencies were also impacted. It is up to you to determine if the increased latency presents an actual problem for your architecture. We are seeing a latency increase across the board in AWS.

Preliminary numbers shows that GCP post-patch latencies are comparable to AWS’s pre-patch numbers. However, GCP already had a large lead in IO latencies even before. A before/after analysis of GCP is still in progress.

We will have an official statement about the impact of Spectre/Meltdown patches soon.


#4

Our official statement is now released:


#5

@kporter @rguo Thank you so much ! We read the official release, then will decied actions.


#6

FYI, Amazon has made additional updates to their backend. I have seen CPU usage recover to their pre-meltdown levels. Latencies are now better, but still a marginal increase (5-10%) as compared to before Jan 4th.


#7

@rguo Thank you. We have seen CPU usage recover to pre-meltdown levels too. Also we have seen marginal increase.