Cannot allocate memory but there is still memory available


Hey guys,

Server version: 3.5.8

We are facing an issue where Aerospike is unable to allocate more memory for indexing, even though there is plenty of memory available. This is the error message in the log:

Apr 15 2016 21:06:38 GMT: WARNING (index): (index.c::241) arenax alloc failed Apr 15 2016 21:06:38 GMT: WARNING (rw): (thr_rw.c::2401) {user_profile_store} write_local_pickled: fail as_record_get_create() rv = -1:0x08049bf0547dfa063248b2bd3812c3cf246db4e3 Apr 15 2016 21:06:38 GMT: INFO (rw): (thr_rw.c::2819) [NOTICE] writing pickled failed(-1)::0x08049bf0547dfa063248b2bd3812c3cf246db4e3 Apr 15 2016 21:06:38 GMT: WARNING (cf:arenax): (arenax_cold.c::45) could not allocate 1073741824-byte arena stage 24

Eventually the logs will become too full and when we log in the machine Aerospike stopped running.

We have only one namespace configured with a memory-size of 55G, but this issue is happening when the memory used is only around 30G. We are currently running on 10 nodes and the total size of ~2 billion records. We are storing the data in SSD.

Do you guys know any reasons why this is happening?

Thanks, Phu

CRITICAL (migrate): (migrate.c:migrate_tree_reduce:1872) malloc