Thanks for posting this question. Here are details around this topic based on the current implementation.
The general mechanism for deletes:
Records do get deleted first from the index. The data on disk does get erased through the defrag process recombining blocks in order to create new empty blocks. This is done for performance optimization.
Now when it comes to restarting the aerospike server on a node, there are 2 modes for restarting a node:
-
In case of cold start, the index will be rebuilt from persistent storage and hence, data still there will re-appear (for the part for which defrag has not processed).
-
In case of fast start (available in enterprise edition), the index is in shared memory and will be there after restart, so the deleted data will not reappear. (but a forced cold start, or full box reboot, will also make it reappear in this case).
Details on fast start can be found here: http://www.aerospike.com/docs/operations/manage/aerospike/fast_start
Now coming to your particular example, as you have data in memory, the server will go through a cold start.
-
When updating a record’s ttl to expire it (let’s say change it from 30 days to 10 seconds), the record will expire in 10 seconds and will be deleted from index when the data is accessed by the client or when the ‘nsup’ thread goes over it to expire it (which ever happens first). Now, if the defrag thread has not reclaimed the blocks containing those records, 2 records are still on disk, the one with 30 days ttl and the one with 10 seconds.
-
Upon cold start, those records will be scanned. If the 30 days ttl record is scanned first, it will be re-inserted in the index. When the 10 seconds ttl record is then scanned, it’s generation will be checked and, as it has a greater generation then the other record it will overwrite it, but as its ttl has expired, the record will be deleted. Everything is fine in this case.
-
Now if those records were scanned in the reverse order, the 10 seconds ttl record would be thrown out as expired, but then, as the 30 days ttl record is scanned, it will re-appear in the index… with a 30 days ttl.