Multiple records after aerospike restart

After aerospike restart I have double more records in set. Before restart I have ~150M records, now ~250M. Can anybody said how can it be?

If you are storing records on disk, and the server is rebooted, the indexes are rebuilt from disk. If the location on disk of the deleted record was not overwritten prior to reboot, the record will be indexed during cold restart. The record then returns from the disk to the database as a zombie record.

Source: What happens when a client writes a record to Persisted Storage in the Aerospike database?

This is desired behaviour because AS tries to minimize writes to SSD. I dont think there is a way to turn this behaviour off. This is why I am updating records with a status bin value saying it got deleted though I never had to restart yet and therefore didn’t face the zombie records.

Does it mean what now I have incorrect data in index? For example there is a record that should not be?

Yes, that could have happened in your case. Can you reconstruct what records have been deleted? In any case, you need to manually deal with the so called “zombie records”.

All data that has been there before the shutdown is there + the data of records deleted that have not been overwritten between the point of deletion and point of shutdown/crash of the nodes.

Btw: what is your replica setting and can those 100M records all be deleted ones? It sounds a lot. Here we barely delete a record and haven’t run into any issues but so far there was no downtime.

I have no replica and it was not be deleted it once. I have a backup of namespace data, it created via asbackup utility before server shutdown. If I clear all data in namespace and restore data from backup will it solve my problem?

it was not be deleted it once

How do you mean? Did you delete records quite regularly and in large scale (atleast 100M records!)? If you didn’t, the mentioned zombie records-theory might be wrong.

If you think that the 100M records could actually be deleted records… Yes, I would restore the backup if up-to-date… However, before restoring, make sure to create a backup of the current (broken) state of the cluster data so you don’t loose anything in case we are on the wrong path here…

Data was deleted during one month. By a small chunks, about 100-200 at once. And before server shutdown I waited a time for defrag complete. I think aerospike never could not be shutdown properly, then a question: How I can update software and os include aerospike?

second server for replica… given that the performance is sufficient, this is waste of money. Or I have no choice?

I think there is no other choice than to use backups and restores in a 1-node-setup with community version and out of the box features.

But you can also implement it yourself. Just keep track of what was deleted (record-key & generation number) in a LList or create a bin called “status” and set it to “deleted” before you actually call delete… When you restart your server, simply run a script that scans through all your records and deletes those who have that bin set. I think the last approach is the most elegant. The LList-approach won’t scale and won’t allow for high throughput but might be faster to scan when you have very few deletes.

Simply waiting for defrag to complete won’t ensure that old data is overwritten for two reasons:

  • Only blocks with atleast x % unused data will be eligable for defragmentation
  • Defragmentation will ony read old blocks and create a new one. The old blocks are still there, until their space is needed for something else again.

Ok, got it! Thanks for your time and help. despite difficulties aerospike is the best key-value solution I have seen.

1 Like