During data insertion into an all-flash cluster, the insertion fails even though the sizing calculations have been done correctly. The error shown looks as follows:
Jul 07 2021 15:08:09 GMT: WARNING (arenax): (arenax_ee.c:425) (repeated:93533) can't allocate more than 2048 arena stages Jul 07 2021 15:08:09 GMT: WARNING (index): (index.c:737) (repeated:93533) arenax alloc failed
When sizing an all-flash cluster it is important to make sure that the mount points are sized such that they can contain all of the sprigs required at the correct fill fraction.
Even when the overall size of the mount point is correct, there may be an issue with the data load if the size of that partition exceeds 2 TiB.
When space is allocated for an Aerospike index, it is done in discrete stages called ‘arenas’. An arena can be between 128 MiB and 16 GiB in size and is, by default 1 GiB. The maximum number of arenas that can be allocated is 2048 and this is fixed.
This implies that, unless the arena size has been increased, by changing
index-stage-size, the maximum size of the index is 2048 * 1 GiB = 1 TiB.
If all 2048 arenas are allocated and more data is inserted then the insertion will fail with the aforementioned error.
Although the number of arenas themselves cannot be increased, increasing the
index-stage-size should resolve the issue.
When calculating the sizing for all-flash, the
index-stage-size should be considered as part of that calculation if the space requirement for the index is 2 TiB or higher.
index-stage-sizerequires a cold start as the index must be rebuilt.
- This error also applies to indexes stored in RAM but is less likely to be seen due to the generally smaller nature of RAM based indexes.
- Index sizing details are discussed on the Capacity Planning Documentation.
ARENA ALLOCATION FAILURE ALL FLASH INDEX ARENAX ALLOC