Aerospike oom

[totola]

version:4.5.3.10 aerospike has set stop write 90% ,but before reach the stop write , service running out of memory ,server has 61G memory , aerospike set 58G dmesg log and configuration file in attachement

oom.gz (16.0 KB)

[pisush]

Hey @totola! Here are two articles on OOM issues and how to troubleshoot those - https://www.aerospike.com/docs/operations/troubleshoot/node/

How to troubleshoot OOM kills on systems with large amount of cache memory allocated

How to troubleshoot OOM kills on systems with large amount of cache memory allocated Context When investigating out of memory (OOM) kills on the Aerospike daemon (asd) process it is important to determine whether the issue is genuinely due to a lack of RAM or whether the system has adequate RAM but the memory is being held in cache. It is further required to understand the type of cache holding the memory as this informs the approach that should be taken to resolve the issue. This article make…

Case Study - OOM Kills on Aerospike cluster with new nodes

Case Study: OOM Kills on Aerospike cluster with new nodes Background This case presented as sudden Out Of Memory (OOM) kills of Aerospike nodes in a newly formed cluster. The OOM kills were surprising as the cluster had undergone a vertical expansion into nodes with much more power and greater storage. Data volume had not increased by any significant degree. Nodes had been restarted by the customer and the cluster was available but the question was, why had a previously stable cluster experie…

Hope this helps

[Albot]

Stop-write can take time to trigger, so it could go above 90% for some time potentially. This also doesn’t stop replica/migrations. 90% is very high and allocating all but 2G to a machine is risky too. I usually don’t set hwm over 70%… Do you have your HWM set to >60? Did your usage go over your HWM? There are other bits of overhead that use that memory in Aerospike too that don’t abide to the namespace usage. You really just need to leave more free memory for the OS, overhead, and other things to run. There’s nothing we can do to help you squeeze out another ~4G of ram out of nothing.