Expired/Deleted data reappears after server is restarted

[maxim_ky]

@kporter

How would this slow down the node’s start-up time.

I mean the node will be fully operational only after the migration complete. And the migration take twice as long to finish as a cold start (with reading data from a disk) on my dataset.

@kporter

Yes, deletes only delete the 64 byte in-memory index entry. It wouldn’t help to do a write to disk unless we also tracked that delete in memory as a tombstone.

Thanks for the information. The tombstone looks like a good solution, but it’s affected by the same issues (defrag and the start-up data loading order). In general you just don’t know when is safe to remove (during defrag) the tombstone from the disk.

@kporter

Also why would you say tombstoning is an insecure workaround?

Sorry, I forgot about it because actually it’s not a delete. The whole record set can’t be reduced this way. But anyway it’s a nice workaround if you have enough memory to hold all the records (including dead ones)

@kporter

Currently the only reliable delete in Aerospike is expiration where the TTLs of all records have never been reduced. If a record expires in this way, it does not come back.

That’s a good point. So if I have a constant record TTL and I just updating my records I will never face the issue.

@kporter

Deletes have been a long running hot topic in Aerospike and there is a lot of momentum to resolve this issue, but we refuse to compromise performance, scalability, or ease of operations for deletes. Many of our users have confronted this very problem and many of those eventually found that they could use the expiration system and achieve a better workflow.

IMO the broken data integrity is the most troublesome consequence of the cold-start (with no cold-start-empty).

@kporter

Perhaps if you discussed your current architecture requiring deletes someone may be able to point out an alternative?

In some cases I really need the data integrity with deletes so probably I’ll try to use the cold-start-empty solution. User-handled tombstones are also suitable.