Aerospike frees memory only after restart

Hello everyone!

We have an aerospike cluster with 3 nodes. The cluster is configured this way:

service {
  user root
  group root

  paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1
  pidfile /run/aerospike/

  batch-threads 20
  service-threads 20
  transaction-threads-per-queue 20
  transaction-queues 20
  proto-fd-max 64000
  nsup-period 30
  migrate-threads 10

logging {
  file /dev/stdout {
    context any info

network {
  service {
    address any
    port 3000

  fabric {
    port 3001

  info {
    port 3003

  heartbeat {
    mode mesh
    port 3002
    interval 150
    timeout 10

    mesh-seed-address-port 3002
    mesh-seed-address-port 3002
    mesh-seed-address-port 3002

namespace cache {
  memory-size 232067M

  high-water-memory-pct 65
  ldt-enabled true
  replication-factor 2
  max-ttl 2d
  high-water-disk-pct 65
  cold-start-evict-ttl 172800
  evict-hist-buckets 1000
  evict-tenths-pct 25
  default-ttl 1d

  storage-engine device {
    write-block-size 128K
    data-in-memory false
    scheduler-mode noop

    device /dev/sdc
    device /dev/sdd
    device /dev/sde
    device /dev/sdf
    device /dev/sdg
    device /dev/sdh

The problem is that under the load memory usage is near to 50%. Delete queue size is jumping between 10k-80k but memory is not being freed.

However if I will restart aerospike node, memory will be freed and memory usage will fall down to ~35%. After few days it will return to 50-55% usage.

Why if we will restart node the memory will be freed immediately but aerospike did’nt did it by it self?

Aerospike version: 3.14.0

    Admin> show statistics like expir
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~cache Namespace Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~
    NODE                 :
    expired_objects      :   287917970          2640910366         2059579606
    non_expirable_objects:   0                  0                  0


You can speed up the delete queue processing by decreasing nsup-delete-sleep. Keep in mind that being too aggressive can plug the write paths with nsup deletes.

As for the memory, primary index arena memory isn’t freed, it will be reused if the primary index grows to need it.

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.