How does eviction work on restarts if no ttl is set

Operations
1

Records TTL and Evictions - For Aerospike server version prior to 3.8 explains that

If records TTL is set to zero (never expire) for all data in a namespace and Memory or disk HWM is reached then system will go to stop-writes.

I have a namespace with lot of deleted records. None of them have ttls set. It is currently at 50% memory usage. I periodically delete sets and create new sets. In my case the number of records in deleted sets is slightly more that the present number of records.

I know that deleted keys will come back (because of no ttl set). In my case there is not enough memory to bring all the deleted data back into memory. If I issue a restart how does this affect the start up the process and how do keys get evicted.

I use the community version (and so cold restarts is what I do)

2

@kporter : Can someone answer this?

(I tagged you in particular because you answered a previous question of mine which is very similar)

3

If all data have TTL set to 0, nothing will be able to evict and coldstart will halt at the configured HWM (iirc). You could increase the configuration to get through the coldstart.

(using mobile, so apologize for the format :slight_smile: )

You could also attempt to reduce the amount of data that comes back. The following hasn’t been tested but should work.

If you scan your existing records and touch them, they will be rewritten to new blocks (overwriting potentially deleted data).

(You could probably stop here, continue for a more exhaustive purge of deleted data.)

Now lets fill your disks.

Halt client writes to cluster.

Next we want to remove some pesky safteys and prevent defrag from recovering space.

Dynamically set (all nodes) the min-avail-pct to 0, stop-writes to 100, defrag-sleep to 10000000.

Now create a write load (java-benchmark) into a disposable set with a small number (10000 should do) of unique keys. These records should have a single bin and short bin name containing a 10KB - 128B string.

Soon you will reach your disk capacity since defrag is set to a crawl. Once that happend stop the load.

You can now restart and there should be with very minimal (if not zero) deletes returning after coldstart.

After restarting you can delete the disposable set.

Let me know how it works out for you or if you need clarifications.

4

Thanks, I will try it out. The idea/hack looks reasonably solid!

I think everything boils down to assigning disk space as tightly as possible. In my use case, its tricky because we have a few sets with key-‘map’ data in them and a few with simple key-value data. So I had to plan disk capacity more conservatively.