Accumulates memory without freeing it

Operations Configuration
1

We have a cluster aerospike, consisting of 5 nodes. Aerospike is constantly increasing its memory consumption. We cleaned half of the records and data (via client.delete (key)). And at the same time, he did not free the memory in the RAM. We keep database in memory.

Seed:        [('127.0.0.1', 3000, None)]
Config_file: /home/david/.aerospike/astools.conf, /etc/aerospike/astools.conf
Cluster  (Migrations in Progress)
=================================

   1.   Cluster Name       :  dmp
   2.   Server Version     :  C-5.5.0.3
   3.   OS Version         :  Ubuntu 20.04.2 LTS (5.4.0-66-generic)
   4.   Cluster Size       :  4
   5.   Devices            :  Total 4, per-node 1
   6.   Memory             :  Total 152.000 GB, 62.00% used (94.240 GB), 38.00% available (57.760 GB)
   7.   Disk               :  Total 559.984 GB, 36.47% used (204.205 GB), 44.25% available contiguous space (247.793 GB)
   8.   Usage (Unique Data):  97.845 GB
   9.   Active Namespaces  :  1 of 1
   10.   Features           :  KVS, Query, SINDEX, Scan, UDF


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespaces~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|~~~Devices~~~~|~~~~~~~~~Memory~~~~~~~~|~~~~~~~~~~Disk~~~~~~~~~|Replication|   Master|~Usage (Unique-Data)~
         |Total|Per-Node|     Total|Used%|Avail%|     Total|Used%|Avail%|    Factors|  Objects| In-Memory|   On-Disk
dmp      |    4|       1|152.000 GB|   62|    38|559.984 GB|   36|    44|          2|656.252 M|  0.000 B | 97.845 GB
Number of rows: 1
2

Per the table you posted, it seems the data is on a device rather than in memory. It also seems this is a four node cluster rather than five.

For ~650M master records with replication factor 2, it would come to ~1.3B total records, and, at 64 bytes per record, the memory used would be: ~1.3B*64/1024/1024/1024, or ~77.5GiB.

The total memory used is 94 GiB, or 17GiB higher. It also seems you have secondary indices defined, which would also use memory.

You may want to check the log ticker entry tracking the memory before and after the deletes to check whether it is caused by the secondary index garbage collection that hasn’t run yet or if there are more records than expected or something else.