Inserts fail with arena allocation error on correctly sized all-flash cluster

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Inserts fail with arena allocation error on correctly sized all-flash cluster

Problem Description

During data insertion into an all-flash cluster, the insertion fails even though the sizing calculations have been done correctly. The error shown looks as follows:

Jul 07 2021 15:08:09 GMT: WARNING (arenax): (arenax_ee.c:425) (repeated:93533) can't allocate more than 2048 arena stages
Jul 07 2021 15:08:09 GMT: WARNING (index): (index.c:737) (repeated:93533) arenax alloc failed

Explanation

When sizing an all-flash cluster it is important to make sure that the mount points are sized such that they can contain all of the sprigs required at the correct fill fraction.

Even when the overall size of the mount point is correct, there may be an issue with the data load if the size of that partition exceeds 2 TiB.

When space is allocated for an Aerospike index, it is done in discrete stages called ‘arenas’. An arena can be between 128 MiB and 16 GiB in size and is, by default 1 GiB. The maximum number of arenas that can be allocated is 2048 and this is fixed.

This implies that, unless the arena size has been increased, by changing index-stage-size, the maximum size of the index is 2048 * 1 GiB = 1 TiB.

If all 2048 arenas are allocated and more data is inserted then the insertion will fail with the aforementioned error.

Solution

Although the number of arenas themselves cannot be increased, increasing the index-stage-size should resolve the issue.

When calculating the sizing for all-flash, the index-stage-size should be considered as part of that calculation if the space requirement for the index is 2 TiB or higher.

Notes

  • Changing index-stage-size requires a cold start as the index must be rebuilt.
  • This error also applies to indexes stored in RAM but is less likely to be seen due to the generally smaller nature of RAM based indexes.
  • Index sizing details are discussed on the Capacity Planning Documentation.

Keywords

ARENA ALLOCATION FAILURE ALL FLASH INDEX ARENAX ALLOC

Timestamp

JULY 2021