Cannot allocate memory but there is still memory available

1

Hey guys,

Server version: 3.5.8

We are facing an issue where Aerospike is unable to allocate more memory for indexing, even though there is plenty of memory available. This is the error message in the log:

Apr 15 2016 21:06:38 GMT: WARNING (index): (index.c::241) arenax alloc failed Apr 15 2016 21:06:38 GMT: WARNING (rw): (thr_rw.c::2401) {user_profile_store} write_local_pickled: fail as_record_get_create() rv = -1:0x08049bf0547dfa063248b2bd3812c3cf246db4e3 Apr 15 2016 21:06:38 GMT: INFO (rw): (thr_rw.c::2819) [NOTICE] writing pickled failed(-1)::0x08049bf0547dfa063248b2bd3812c3cf246db4e3 Apr 15 2016 21:06:38 GMT: WARNING (cf:arenax): (arenax_cold.c::45) could not allocate 1073741824-byte arena stage 24

Eventually the logs will become too full and when we log in the machine Aerospike stopped running.

We have only one namespace configured with a memory-size of 55G, but this issue is happening when the memory used is only around 30G. We are currently running on 10 nodes and the total size of ~2 billion records. We are storing the data in SSD.

Do you guys know any reasons why this is happening?

Thanks, Phu

2

a couple years later and we’re seeing this same issue. Did you happen to find a fix, Phu? Thanks.

3

we found our issue: https://www.aerospike.com/docs/guide/limitations.html

Actual number limited by RAM and storage. Every record takes 64 bytes for the index entry. The index entry is ONLY stored in RAM. The key itself is not actually stored in the index, but the hash of the key (using RIPE-MD 160 algorithm) is. This hash with overhead takes exactly 64 bytes. The maximum number of records per namespace on a given node is limited to 4,294,967,296 on the Community Edition (2^32 due to 4 bytes used for storing references), and 34,359,738,368 on the Enterprise Edition (2^35). This represents 2TiB of RAM

4

The Known Limitations Page has been updated for Aerospike Server Enterprise Edition versions 4.2 and newer: https://www.aerospike.com/docs/guide/limitations.html

Actual number limited by RAM and storage. Every record takes 64 bytes for the index entry. The index entry is ONLY stored in RAM. The key itself is not actually stored in the index, but the hash of the key (using RIPE-MD 160 algorithm) is. This hash with overhead takes exactly 64 bytes. The maximum number of records per namespace on a given node is limited to 4,294,967,296 on the Community Edition (2^32 due to 4 bytes used for storing references), and 549,755,813,888 on the Enterprise Edition (2^39) * . This represents 32TiB of RAM.

  • The value of the 2^39 number of records can be reached using Aerospike Server Enterprise Edition version 4.2.0.2 or newer, and having index-stage-size set to 16G.