Sure, was just pointing out additional overhead that will occur. instead of being able to just update a record, you have to read, (first record lock), then update (second record lock) - then worry about if you are overwriting someone other client’s delete in between. To avoid that, you will have to use the Check and Set feature and check generation, retry if gen does not match. Just starts slowing the transactions down … but good thinking and deep diving! Yes, it can be handled but costs cycles and therefore latency on the transaction. And this you will have to do for every transaction in your application.
Thank you @pgupta. I appreciate the feedback.
You present some valid issues (race conditions, additional ops), for which some may have workarounds at increased complexity.
I would love to iron out the issues further, but only if we were fine-tuning a workable solution. As of now, it seems community edition users must choose between (1) limited, never-incremented ttl records or (2) a long-ttl tombstone that, even then, doesn’t close the hole 100% (and the fact that the tombstone record is indexed in expensive RAM).
Well another approach to think about … and please don’t hold my feet to the fire…I have not thought it through. The problem of deleted records resurrecting fundamentally comes from reducing the original default ttl in the namespace or ttl when first time the record is created. You reduce it by either resetting to lower value in a record update or when deleting the record, you leave the future expiration ttl on ssd with index blown in RAM. What if you start with a short ttl as default in namespace and periodically touch the records you want to keep ie extend ttl for the ones you want? Then you do your tombstone approach but don’t delete the record. Instead if it is not a tombstone bin with T, say with K, for keep, extend the ttl by touching the record. Let it expire naturally. That takes care of Kevin’s defrag issue. Then in application check for ‘T’ with Check and Set. Won’t work for Live For Ever records. Will it work??? I am just thinking out aloud.
Delete is a hard problem to solve and even with EE Durable Deletes, there is a price to pay in terms of storage space for tombstones and extra tomb raider cycles to clear the tombstones. Even then it protects you against node failure only for the tombstone duration period (one day by default) though you can set it higher. so as long as you resurrect the node within the day or that period, you are fine. Its a tradeoff between getting durable deletes and paying the storage and another background thread taking away resources from your active transaction processing.
I think your suggestion indeed works. Rephrasing what you said: (1) don’t have non-expiring records (2) expiring ones need to be touched periodically, and their ttl should never be reduced (via policy change or delete)
The above restrictions works for some use cases.
The price to pay for durable deletes–that is, a temporary backlog of yet-to-be-raided tombstones–is reasonable if deletions are not very frequent. Which I think makes the database applicable to a much wider variety of use cases.
To be super precise, it’s the void time of a record that cannot be reduced. (void time is the absolute time when a record will expire, where as ttl is a relative time, ‘from now’). So technically, if you write a record with a ttl or 200, and 150 seconds later, you set its ttl to 100, you are still OK, since you ended up pushing the void time by 50 seconds, yet you technically 'reduced the ttl of the record from 200 to 100…).
I know that is what is implied, but just to be 100% technically correct 
2 Likes
It’s my turn to face with this issue, unfortunately. After a server restart I found old “removed” records re-appeared and thanks God it happened during development phase.
Guys, you’ve made huge progress with Aerospike and you work hard on it but let me highlight one more time some of its big issue
-
When a team designs a project it should not think how to remove an item from a storage. It’s 2017 now and it’s the time for very complex systems with high level languages and data structures. I can’t recommend a product for production use if I can’t do basic operations with it. It’s too expensive to think about all these workarounds that may or may not work in some cases. I’m sure that you as a team of very talented guys do understand how horrible it is to get removed items back in a live project after you restart a server. You can have blazing fast read speed but without these basics working as expected performance doesn’t matter. “Never delete records” is not a valid approach for 2017
-
Also there is no way to remove a set from a client. It’s possible to create it (it creates automatically) but no way to remove it. Again, there are some workarounds that require servers restart but it’s not real in production. Since there is a hard limit for 1023 sets it’s a lame excuse not to have it.
I do love your product, it’s awesome from many angles but, damn, I simply have to start looking for another database not to put my project in high risk.
Thanks for submitting your feedback here. The durable delete feature was introduced in the Enterprise Edition version 3.10: http://www.aerospike.com/docs/guide/durable_deletes.html. Note that there is then a compromise on memory overhead and performance given the extra background scanning of the storage layer that would be required.
Also note that crafting your use case to effectively use expiration and eviction will have
performance and operations advantages over durable-deletes.
I agree with this. Deletion is a basic operation of any database. I’ve read multiple threads on Aerospike deletion and every proposed solution seems to have gotchas. Deletion shouldn’t have a high cognitive load. If there’s a perf impact, simply make note of it and be done with it.
3.10 was released by that doesn’t help Community Users. We need a canonical way of deleting non-expiring entries that work for everyone(That doesn’t blow away ram)
If there’s a perf impact, simply make note of it and be done with it.
This can probably be said for many of the choices the Aerospike team is faced with. You could say this about thousands of different features, histograms, query tracking, defragmentation, etc,etc, but then you’d end up not getting a lightning fast DB. Right?
I would think data deletion is a bit different than the features you listed. Hell, why even allow non-ram data stores? That way everyone ends up with a “lightning fast DB”. Really depends on how far you want to take your example.
There are clearly concessions made for speed vs utility. This doesn’t need to be an XOR situation. We can simply add another tool to the shed for people to use if they want it.
That’s fantastic advice. Thanks.
1 Like
Vorror - the only concession here is free vs enterprise. And the only issue with the free version is you can’t cold-restart nodes, or you must use expiration, or you have your app write “empty records” ( re-writing data ) in cases where expiration doesn’t work - and then write a “tomb raider” to clean them up later. I ran into a community edition user in england who is doing all of those.
For small datasets, or entry level projects, we make sure enterprise fits your budget. I’ve never seen our team say no - we want you to be on enterprise when you grow, we don’t try to make any money when you’re not making money. And you can talk to our support people instead of getting “best effort” in the forums. I think the open source we offer has a lot of features - and it’s not 6 months behind.
All of those solutions are a hassle, which is why Enterprise gets Durable deletes additionally to expunging supported in CE.
I also impressed by speed, but just can’t start using if something like that could happen…
If durable deletes are important to you, use Enterprise.
I didn’t realize from the start that a record could be set with a deleted option or something like that and then search with this option excluded. Database is great and this is not a big problem.