Data loss when used asbackup during migrations

Hi all,

i am using aerospike version 3.15.0.1 and was trying the asbackup/asrestore feature of aerospike. One of the case that feared me is that i started asbackup when aerospike cluster was rebalancing. I have 5 node cluster and i started asbackup on each of the node with --node-list option as local. I was able to see that there was difference in the data being backed up due to migrations(One node may have large partitions then the other). However when i restored the data from those backup files i lost around 7 million keys. Is it desired behaviour? As i checked asbackup runs a basic scan with priority 2 in background and created the backup file so it is possible that due to migrations asbackup missed some partitions? @kporter @pgupta

This was a known hazard in older versions - the asbackup tool also had a flag to fail on cluster change for this reason. The issue is that scans were not partition aware in older versions - partition ownership changes mid-scan could cause entire partitions to either be missed or scanned twice. Partition-aware scans were added in 4.9.0.3 which also requires an updated client library so that it can make use of the protocol change. Support for the non-partition-based scans was dropped over a year later with 5.6.0.3.

→ Upgrade!

1 Like

@kporter @pgupta Thanks for the update.