FAQ - What options are available to speed up cold start eviction


#1

FAQ - What options are available to speed up cold start eviction

Detail

When a node cold starts it scans the disk and reloads records into the index. It is possible, under certain conditions, that eviction will have to happen before the node can cold start. A common reason for this might be that the loading of records into the index has caused the namespace to exceed the high-water-memory-pct configured (especially if previously non durable deleted records are on the disk). When cold start eviction is taking place the following messages will be present in the aerospike.log.

Sep 29 2016 08:27:50 GMT: INFO (nsup): (thr_nsup.c:324) {test} cold-start building eviction histogram ...
Sep 29 2016 08:27:50 GMT: INFO (drv_ssd): (drv_ssd.c:3974) {test} loaded 699392 records, 0 subrecords, /opt/aerospike/data/test.dat 28%

Evictions during cold start can literally increase start up time exponentially. Indeed, everytime the high-water-memory-pct or the high-water-disk-pct threshold is breached, eviction histograms have to be recalculated for the entirety of the data already loaded to potentially only evict a few records, load a few more from disk and breach the threshold again and iterate over the sam process. What are the options to speed up the eviction process during a cold start?

Answer

There are two things that can be done to reduce the time spent evicting record during a cold start:

  • Increase the high-water-memory-pct in the aerospike.conf file.

This would have to be done prior to the cold start as it can only be done by making an entry in the namespace stanza of aerospike.conf file when the server is not up. This increases the eviction threshold for memory. This should only be used as a temporary measure to get the node running, settings should be reverted once the node has re-joined the cluster. The entry should then be removed from the aerospike.conf file so that it is not re-applied on next startup. The entry in aerospike.conf would look as follows:

high-water-memory-pct 80

The info command to revert this would be:

$asinfo -v 'set-config:context=namespace;id=test;high-water-memory-pct=60'

Do not forget to also revert in the configuration file.

  • Modify cold-start-evict-ttl

The parameter cold-start-evict-ttl sets the TTL below which records will be evicted during a cold start. This is another static parameter meaning that it must be included in aerospike.conf before start up. This parameter is measured in seconds. The value to choose for this parameter should be based on the records in the database at the time of startup.

Notes

  • Full details on high-water-memory-pct

http://www.aerospike.com/docs/reference/configuration#high-water-memory-pct

  • Full details on cold-start-evict-ttl

http://www.aerospike.com/docs/reference/configuration#cold-start-evict-ttl

Keywords

COLD START EVICT SLOW

Timestamp

9/29/16


FAQ - which is faster to complete migration - restarting an empty node vs restarting a node with data
How do I speed up cold start?
I can store 1 billion records if the cluster is new but can't reload it