Records TTL and Evictions - For Aerospike server version prior to 3.8


#1

For Aerospike server version beyond 3.8, please see Eviction mechanisms in Aerospike

If records TTL is set to zero (never expire) for all data in a namespace and Memory or disk HWM is reached then system will go to stop-writes.

If records TTL is greater than zero then Eviction algorithm will use a 100 bucket histogram algorithm to determine what to expire.

The bucket width or histogram base unit can be determine by dividing the MAX TTL by a hundred.

Bucket_Width = MAX_TTL/100

Note that if within a namespace records have different TTLs set and TTLs fall in different histogram buckets , Aerospike will start evicting data from the lower TTL buckets first.

If all records fall within the same bucket then eviction become random.

Info on TTL bucket can be gathered from asinfo.

asinfo -h 127.0.0.1 -p 3000 -v "hist-dump:ns=data-ns;hist=ttl"
requested value  hist-dump:ns=data-ns;hist=ttl
value is  data-ns:ttl=100,41600222,0,0,200,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5100;

Histogram format: NAMESPACE:HISTOGRAM=NUM_BUCKETS,BUCKET_WIDTH,BUCKET_0,BUCKET_1,…

Where:

  • HISTOGRAM is ttl
  • NUM_BUCKETS is 100
  • BUCKET_WIDTH varies based on data the current max Void Time in seconds. Note: Records with indefinite TTLs (TTL = 0) are not represented in ttl or evict histograms.
  • BUCKET_0 are records between 0 * BUCKET_WIDTH and 1 * BUCKET_WIDTH.
  • BUCKET_1 are records between 1 * BUCKET_WIDTH and 2 * BUCKET_WIDTH.
  • BUCKET_99 are records between 2 * BUCKET_WIDTH and 100 * BUCKET_WIDTH.

In above example

BUCKET_WIDTH = 4160022 secs

200 records are in bucket 2 and will expire in 4160022 × 3 = 12480066 secs While 5100 records are in the last bucket and will in 4160022 × 100 Algorithm will evict bucket 2 prior to bucket 99. But randomly expire records within bucket 99.

Please see our data retention doc:

http://www.aerospike.com/docs/operations/configure/namespace/retention/

and on asinfo at:

http://www.aerospike.com/docs/tools/asinfo/

Evictions Algorithm:

Verify that evictions are happening based on these eviction algorithms:

if (default-ttl == 0) || (eviction-period == 0) do not run eviction; otherwise, execute one of the following steps depending of the type of namespace

Namespace with storage engine memory:

If (memory_lwm >= data_sz + index_sz) do not run eviction algorithm; otherwise, run eviction algorithm

Namespace with storage engine ssd (data-in-memory is set to false):

if (memory_lwm >= index_sz) && (ssd_lwm >= data_sz) do not run eviction algorithm; otherwise, run eviction algorithm

Namespace with storage engine ssd (data-in-memory is set to true):

if (memory_lwm >= data_sz + index_sz) && (ssd_lwm >= data_sz) do not run eviction algorithm; otherwise, run eviction algorithm

If total size reaches HWM, data may be evicted. Exceptions are:

  1. Records with no TTL specified (in a namespace that has no default TTL) will neither be expired nor evicted. They are excluded from the TTL histograms. We sometimes refer to such records as “zero-void-time” records.
  2. Record with TTL=0 in configuration file (set to never expire) will also be excluded from eviction
  3. Records with TTL set to -1 through AS clients. (Also means record never expires)

All other records will be subject to eviction algorithms


Eviction Policy
FAQ What are Expiration, Eviction and Stop-Writes?
Changes to Default TTL Not Effecting Existing Data
Missing indexes
How does eviction work on restarts if no ttl is set
Recovering from Available Percent Zero