Why Aerospike evicted data?

Thank you for your submission. Let me start by explaining what “eviction” means and I hope that Google translate will do a good job.

First, let me explain “expiration.” Many databases have this today. This is set by the TTL (time to live). Any object in the database that is past its TTL will be automatically deleted from the database. For example if the TTL is set at 90 days, any data longer than 90 days will be deleted.

Now comes “eviction”. As a safety feature, Aerospike has settings that will “evict” or get rid of data once the database is getting close to full. Keep in mind that as a distributed database, you want to have extra capacity so that if a node goes down, the remaining nodes much be able to handle all the traffic. By default, we recommend that you run so that the disk not be more than 50% full (this is also for defragmentation) and memory is no more than 60% full. These are controlled by the “high-water-memory-pct” and “high-water-disk-pct”.

In addition, there is a level at which the database is considered so full that it will stop writing any new records (updates to existing records are still possible). This is called the stop-write level.

These settings are determined by the following variables in your configuration files:

high-water-memory-pct 70 high-water-disk-pct 90 stop-writes-pct 70

Your current high-water-memory-pct is 70% and according to the clmonitor output, you are at 71%. So this means that you are above the memory high water mark. In addition, you are above the stop-writes-pct, so the database is not accepting new records (the “Stop Writes” is “true” in the clmonitor output).

There are 3 ways around this:

  1. Reduce the number of records/amount of data
  2. Increase the memory allocated for the database (“memory-size 4294967296”)
  3. Increase the limits. Increasing the limits should only be seen as temporary measure.

Please refer to the documentation at: https://docs.aerospike.com/display/AS2/Managing+Storage+Capacity

2 Likes