Issues on write operations

Operations
1

Hi We have a 2 node cluster (CE 3.9.1) which has suddenly begun exhibiting the behaviour outlined in the below error messages

Dec 25 2016 22:15:51 GMT: WARNING (drv_ssd): (drv_ssd.c:2739) record-add as_record_get_create() failed :0x0293870dc30053d60e4a7bdc31ac9f0b4fc061d4 Dec 25 2016 22:15:51 GMT: WARNING (cf:arenax): (arenax_ce.c:39) can’t allocate more than 256 arena stages Dec 25 2016 22:15:51 GMT: WARNING (index): (index.c:505) arenax alloc failed

Initially we saw this appear on both nodes in tandem. Long story short, we eventually rebooted node 1, at which point the errors came back on node1 post-boot sequence but node2 is free from said errors. A curios observation is that the pre for live process progressed at a slower rate for 1 of the 7 SSD’s on node 10 (59% when remaining disks were at 100%).

I am confused as to whether this issue is disk, memory or both. I seem to find conflicting info in the forums (issues with in memory index vs ssd issues).

Any inout welcomed.

Many thanks in advance, Tom

2

arena stage is 1GB - max 256GB - 256 arena stages max per namespace per node - this is RAM for storing primary indexes. This limits us to 4 billion records. 4 billion x 64 bytes for each primary index= 256 Gb RAM.
Are you trying to insert more than 4 billion records in this namespace per node including replicas?

See Known Limitations | Aerospike Documentation

“Actual number limited by RAM and storage. Every record takes 64 bytes for the index entry. The index entry is ONLY stored in RAM. The key itself is not actually stored in the index, but the hash of the key (using RIPE-MD 160 algorithm) is. This hash with overhead takes exactly 64 bytes. The maximum number of records per namespace on a given node is limited to 4,294,967,296 (2^32 due to 4 bytes used for storing references). This represents 256GiB (4,294,967,296 * 64 bytes).”

3

Thanks - that was the prime suspicion all along. We have resolved the issue now. Thanks!