# Aerospike memory usage spikes up linearly **URL:** \ **Category:** Operations\ **Tags:** shared-memory\ **Created:** [September 20, 2021, 5:48am UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836 "2021-09-20T05:48:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![abhinavkulkarni](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/abhinavkulkarni/32/1726_2.png) [@abhinavkulkarni](https://discuss.aerospike.com/u/abhinavkulkarni)\ **Post date:** [September 20, 2021, 5:48am UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/1 "2021-09-20T05:48:28Z")
Hi, I have a cloud VM with a network disk attached that’s acting as a single node Aerospike server. I ingest small data in it throughout the day (sorted maps, lists, key-value, etc.). I also use the same server to serve an API response - the API traffic currently is very small. I find that the server memory usage creeps slowly and eventually have to restart the server to render it useful. Here’s my server configuration: ```auto namespace crawler { replication-factor 2 memory-size 1G default-ttl 0 #5 days, use 0 to never expire/evict. nsup-period 120 # To use file storage backing, comment out the line above and use the # following lines instead. storage-engine device { device /dev/sda write-block-size 8M # data-in-memory true # Store data in memory in addition to file. } } ``` Here’s the monitoring data from past 14 days. ![screencapture-cloud-digitalocean-droplets-263109597-graphs-2021-09-20-11_10_41 (1)](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/2X/f/fcef9abf408dba9701e6a7063975aa15fbe41e98.png) The low memory periods are when the server has crashed for a while before I was able to get to it. What can be the reason? ---
**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\ **Post date:** [September 20, 2021, 11:22pm UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/2 "2021-09-20T23:22:27Z")
Which version of Aerospike are you using? It may be useful to scrutinize the logs as well to see if this is coming from an increase in heap and if it correlates with something obvious (presence of secondary indices for example, etc…). Hard to otherwise guess. There have been occasional memory leaks of different sorts that were found and addressed (would check [the release notes](https://download.aerospike.com/download/server/notes.html) for details on known memory leaks that were addressed). ---
**Author:** ![abhinavkulkarni](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/abhinavkulkarni/32/1726_2.png) [@abhinavkulkarni](https://discuss.aerospike.com/u/abhinavkulkarni)\ **Post date:** [September 25, 2021, 5:06am UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/3 "2021-09-25T05:06:29Z")
Thanks @meher. Is there anything in particular I should be looking at in Aerospike logs? Most of the logs kind of look like the following: ```auto aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:168) NODE-ID bb9020014ac4202 CLUSTER-SIZE 1 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:250) cluster-clock: skew-ms 0 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:273) system: total-cpu-pct 14 user-cpu-pct 12 kernel-cpu-pct 2 free-mem-kbytes 1479512 free-mem-pct 72 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:296) process: cpu-pct 1 threads (9,60,51,47) heap-kbytes (1194851,1196380,1265664) heap-efficiency-pct 94.4 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:307) in-progress: info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:330) fds: proto (7,110,103) heartbeat (0,0,0) fabric (0,0,0) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:338) heartbeat-received: self 0 foreign 0 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:368) fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:418) batch-index: batches (226,0,0) delays 0 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:321) histogram dump: batch-index (226 total) msec aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:340) (00: 0000000209) (01: 0000000016) (02: 0000000001) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:431) {crawler} objects: all 3708063 master 3708063 prole 0 non-replica 0 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:493) {crawler} migrations: complete aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:522) {crawler} memory-usage: total-bytes 237316032 index-bytes 237316032 set-index-bytes 0 sindex-bytes 0 used-pct 22.10 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:594) {crawler} device-usage: used-bytes 6369981504 avail-pct 25 cache-read-pct 2.13 aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:648) {crawler} client: tsvc (0,0) proxy (0,0,0) read (4179,0,0,58,0) write (21933,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:770) {crawler} batch-sub: tsvc (0,0) proxy (0,0,0) read (0,0,0,20499,0) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:825) {crawler} scan: basic (2,0,0) aggr (0,0,0) udf-bg (1,0,0) ops-bg (0,0,0) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:881) {crawler} udf-sub: tsvc (0,0) udf (0,26,0,0) lang (0,0,0,26) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:321) histogram dump: {crawler}-read (4237 total) msec aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:331) (00: 0000000902) (01: 0000003250) (02: 0000000047) (03: 0000000017) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:340) (04: 0000000021) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:321) histogram dump: {crawler}-write (21933 total) msec aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:331) (00: 0000021789) (01: 0000000095) (02: 0000000008) (03: 0000000008) aerospikedb_1 | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:331) (04: 0000000011) (05: 0000000018) (06: 0000000003) (08: 0000000001) ``` Also, please note: 1. I haven’t really set any primary/secondary keys for my sets (I am new to Aerospike and couldn’t really figure out a way to do so) 2. I am not too sure how the data is being stored - primary storage has been specified as flash in the config, but I am not sure if memory is being utilized for caching. I was expecting to something along the lines of [Redis-on-Flash](https://redis.com/redis-enterprise/technology/redis-on-flash/) whereby hot values (recently accessed) as well as all keys are stored in memory, cold values are stored on flash. Could it be that my memory settings are wrong? That is my in-memory cache is growing and growing beyond machine memory? For reference, my machine has 1 vCPU and 2GB RAM. Thanks! ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [September 26, 2021, 3:17am UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/4 "2021-09-26T03:17:36Z")
With the default 256 post-write-queue that’s 8MiB\*256=2GiB. You need to turn that down. Do you have read-page-cache on? What’s your max-write-cache? Also Aerospike allocates ram in 1GiB slabs… 2GB is really quite small, especially if you want 8MiB objects (massive). Aerospike will need some tuning to work on machines that small. ---
**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\ **Post date:** [September 27, 2021, 11:51pm UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/5 "2021-09-27T23:51:06Z")
Good catch @Albot ! Yes, that would do it and let it increase proportionally to writes! For a small namespace that can indeed make a big difference. So, would definitely reduce the [post-write-queue](https://docs.aerospike.com/docs/reference/configuration/#post-write-queue) significantly (based on how much extra RAM is available) and your other suggestions make sense. Having `memory-size` of 1GiB does indicate this is a very small namespace. Well, never mind, this gives it away from the shared logs (1.4GiB is 72% of the system RAM): `free-mem-kbytes 1479512 free-mem-pct 72` ---
**Author:** ![abhinavkulkarni](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/abhinavkulkarni/32/1726_2.png) [@abhinavkulkarni](https://discuss.aerospike.com/u/abhinavkulkarni)\ **Post date:** [October 5, 2021, 10:27pm UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/6 "2021-10-05T22:27:01Z")
Thanks @Albot and @meher for your responses. I have a 2GB RAM (I may look into upgrading the instance to one with higher RAM in future). So, will setting `max-write-cache` to `1GB` solve all the problems? That is, will `max-write-cache` value superceed the default value of 256 of `post-write-queue`? Or should I set it explicitly to `128`? Also, as you can see in my original post, I have explicitly set `memory-size` to `1GB`. So, why does cache size blow past it? Thanks! ---
**Author:** ![meher](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/meher/32/1644_2.png) [@meher](https://discuss.aerospike.com/u/meher)\ **Post date:** [October 5, 2021, 10:51pm UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/7 "2021-10-05T22:51:08Z")
Good question. The [memory-size](https://docs.aerospike.com/docs/reference/configuration/#memory-size) only tracks the index-size, sindex-size if used, data-in-memory if enabled, set-index if used, in order to enforce evictions (if configured) and stop-writes. But there are other memory allocations that happen which are not tracked per the memory-size limit (like xdr transaction queues, write-queue cache (max-write-cache), post-write-queue, other fabric buffers… Those are typically fairly small, but in this case, with such a low memory system and a large write-block-size, it does cause this problem. ---
**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\ **Post date:** [October 5, 2022, 10:51pm UTC](https://discuss.aerospike.com/t/aerospike-memory-usage-spikes-up-linearly/8836/8 "2022-10-05T22:51:11Z")
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.