Aerospike memory usage spikes up linearly

Operations
3

Thanks @meher.

Is there anything in particular I should be looking at in Aerospike logs?

Most of the logs kind of look like the following:

aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:168) NODE-ID bb9020014ac4202 CLUSTER-SIZE 1
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:250)    cluster-clock: skew-ms 0
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:273)    system: total-cpu-pct 14 user-cpu-pct 12 kernel-cpu-pct 2 free-mem-kbytes 1479512 free-mem-pct 72
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:296)    process: cpu-pct 1 threads (9,60,51,47) heap-kbytes (1194851,1196380,1265664) heap-efficiency-pct 94.4
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:307)    in-progress: info-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:330)    fds: proto (7,110,103) heartbeat (0,0,0) fabric (0,0,0)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:338)    heartbeat-received: self 0 foreign 0
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:368)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:418)    batch-index: batches (226,0,0) delays 0
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:321) histogram dump: batch-index (226 total) msec
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:340)  (00: 0000000209) (01: 0000000016) (02: 0000000001)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:431) {crawler} objects: all 3708063 master 3708063 prole 0 non-replica 0
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:493) {crawler} migrations: complete
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:522) {crawler} memory-usage: total-bytes 237316032 index-bytes 237316032 set-index-bytes 0 sindex-bytes 0 used-pct 22.10
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:594) {crawler} device-usage: used-bytes 6369981504 avail-pct 25 cache-read-pct 2.13
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:648) {crawler} client: tsvc (0,0) proxy (0,0,0) read (4179,0,0,58,0) write (21933,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:770) {crawler} batch-sub: tsvc (0,0) proxy (0,0,0) read (0,0,0,20499,0)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:825) {crawler} scan: basic (2,0,0) aggr (0,0,0) udf-bg (1,0,0) ops-bg (0,0,0)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (ticker.c:881) {crawler} udf-sub: tsvc (0,0) udf (0,26,0,0) lang (0,0,0,26)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:321) histogram dump: {crawler}-read (4237 total) msec
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:331)  (00: 0000000902) (01: 0000003250) (02: 0000000047) (03: 0000000017)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:340)  (04: 0000000021)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:321) histogram dump: {crawler}-write (21933 total) msec
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:331)  (00: 0000021789) (01: 0000000095) (02: 0000000008) (03: 0000000008)
aerospikedb_1  | Sep 25 2021 05:00:28 GMT: INFO (info): (hist.c:331)  (04: 0000000011) (05: 0000000018) (06: 0000000003) (08: 0000000001)

Also, please note:

  1. I haven’t really set any primary/secondary keys for my sets (I am new to Aerospike and couldn’t really figure out a way to do so)
  2. I am not too sure how the data is being stored - primary storage has been specified as flash in the config, but I am not sure if memory is being utilized for caching. I was expecting to something along the lines of Redis-on-Flash whereby hot values (recently accessed) as well as all keys are stored in memory, cold values are stored on flash.

Could it be that my memory settings are wrong? That is my in-memory cache is growing and growing beyond machine memory? For reference, my machine has 1 vCPU and 2GB RAM.

Thanks!