How to Restore Aerospike from Incremental Backup file

Hi, I wanted to know how to configure Aerospike to restore the full cluster from an incremental backup file. Also, is there a way to configure retention of certain number of backup files through asbackup? Say, for example, I take a full backup of the cluster on day 1 and then only daily incremental backups for next 14 days. How can I fully restore the cluster on the 15th day? What would be the command? Do I need to cleanup the old backup files myself?

I think you would still need one of the files or directories to be the full backup, and also restore each of the individual incremental backup files/directories. (Unless all your records in the cluster gets updated in 14 days, then you would not need the full backup). This could be scripted, since both asbackup and asrestore support both a “-i” option for individual file or a -d option for directory:

I’m assuming that each incremental backup only had the changes for that day’s backup and were stored in an individual directory or file. https://www.aerospike.com/docs/tools/backup/#incremental-backup More details on asbackup commands: asbackup command-line options | Aerospike Documentation

 # asbackup --host 127.0.0.1 --namespace test --modified-after 2020-07-22_00:00:00 --directory backup_2020_07_22
2020-07-22 19:37:05 GMT [INF] [  308] Starting 100% backup of 127.0.0.1 (namespace: test, set: [all], bins: [all], after: 2020-07-22 00:00:00 UTC, before: [none], no ttl only: false, limit: 0) to backup_2020_07_22
2020-07-22 19:37:05 GMT [INF] [  308] [src/main/aerospike/as_cluster.c:195][as_cluster_add_nodes_copy] Add node BB9030011AC4202 127.0.0.1:3000
2020-07-22 19:37:05 GMT [INF] [  308] Processing 1 node(s)
2020-07-22 19:37:05 GMT [INF] [  308] Node ID             Objects        Replication    
2020-07-22 19:37:05 GMT [INF] [  308] BB9030011AC4202     1              1              
2020-07-22 19:37:05 GMT [INF] [  308] Namespace contains 1 record(s)
2020-07-22 19:37:05 GMT [INF] [  308] Directory backup_2020_07_22 does not exist, creating
2020-07-22 19:37:05 GMT [INF] [  308] Directory backup_2020_07_22 prepared for backup
2020-07-22 19:37:05 GMT [INF] [  327] Starting backup for node BB9030011AC4202
2020-07-22 19:37:05 GMT [INF] [  327] Created new backup file backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:37:05 GMT [INF] [  327] No secondary indexes
2020-07-22 19:37:05 GMT [INF] [  327] Backing up 0 UDF file(s)
2020-07-22 19:37:05 GMT [INF] [  327] Completed backup for node BB9030011AC4202, records: 1, size: 137 (~137 B/rec)
2020-07-22 19:37:06 GMT [INF] [  326] Backed up 1 record(s), 0 secondary index(es), 0 UDF file(s) from 1 node(s), 137 byte(s) in total (~137 B/rec)

and restoring (with ignoring generation on my test env)

# asrestore -d backup_2020_07_22 -g
2020-07-22 19:41:37 GMT [INF] [  373] Starting restore to 127.0.0.1 (bins: [all], sets: [all]) from backup_2020_07_22
2020-07-22 19:41:37 GMT [INF] [  373] Processing 1 node(s)
2020-07-22 19:41:37 GMT [INF] [  373] Found 1 backup file(s) in backup_2020_07_22
2020-07-22 19:41:37 GMT [INF] [  373] Opened backup file backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:41:37 GMT [INF] [  373] Restoring 0 UDF file(s)
2020-07-22 19:41:37 GMT [INF] [  373] Restoring 0 secondary index(es)
2020-07-22 19:41:37 GMT [INF] [  373] Restoring records
2020-07-22 19:41:37 GMT [INF] [  392] Restoring backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:41:37 GMT [INF] [  392] Opened backup file backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:41:38 GMT [INF] [  391] 0 UDF file(s), 0 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 137 B/rec, backed off: 0)
2020-07-22 19:41:38 GMT [INF] [  391] Expired 0 : skipped 0 : err_ignored 0 : inserted 1: failed 0 (existed 0 , fresher 0)
2020-07-22 19:41:38 GMT [INF] [  391] 100% complete, ~0s remaining

Example with script:

I have a separate directory for each incremental backup:

# ls *
backup_2020_07_20:
BB9030011AC4202_00000.asb

backup_2020_07_21:
BB9030011AC4202_00000.asb

backup_2020_07_22:
BB9030011AC4202_00000.asb

I use a for loop to restore all incremental backups:

for i in `ls * -d`; do asrestore -d  $i -g; done 
2020-07-22 19:49:23 GMT [INF] [  596] Starting restore to 127.0.0.1 (bins: [all], sets: [all]) from backup_2020_07_20
2020-07-22 19:49:23 GMT [INF] [  596] Processing 1 node(s)
2020-07-22 19:49:23 GMT [INF] [  596] Found 1 backup file(s) in backup_2020_07_20
2020-07-22 19:49:23 GMT [INF] [  596] Opened backup file backup_2020_07_20/BB9030011AC4202_00000.asb
2020-07-22 19:49:23 GMT [INF] [  596] Restoring 0 UDF file(s)
2020-07-22 19:49:23 GMT [INF] [  596] Restoring 0 secondary index(es)
2020-07-22 19:49:23 GMT [INF] [  596] Restoring records
2020-07-22 19:49:23 GMT [INF] [  615] Restoring backup_2020_07_20/BB9030011AC4202_00000.asb
2020-07-22 19:49:23 GMT [INF] [  615] Opened backup file backup_2020_07_20/BB9030011AC4202_00000.asb
2020-07-22 19:49:24 GMT [INF] [  614] 0 UDF file(s), 0 secondary index(es), 0 record(s) (0 KiB/s, 0 rec/s, 0 B/rec, backed off: 0)
2020-07-22 19:49:24 GMT [INF] [  614] Expired 0 : skipped 0 : err_ignored 0 : inserted 0: failed 0 (existed 0 , fresher 0)
2020-07-22 19:49:24 GMT [INF] [  616] Starting restore to 127.0.0.1 (bins: [all], sets: [all]) from backup_2020_07_21
2020-07-22 19:49:24 GMT [INF] [  616] Processing 1 node(s)
2020-07-22 19:49:24 GMT [INF] [  616] Found 1 backup file(s) in backup_2020_07_21
2020-07-22 19:49:24 GMT [INF] [  616] Opened backup file backup_2020_07_21/BB9030011AC4202_00000.asb
2020-07-22 19:49:24 GMT [INF] [  616] Restoring 0 UDF file(s)
2020-07-22 19:49:24 GMT [INF] [  616] Restoring 0 secondary index(es)
2020-07-22 19:49:24 GMT [INF] [  616] Restoring records
2020-07-22 19:49:24 GMT [INF] [  635] Restoring backup_2020_07_21/BB9030011AC4202_00000.asb
2020-07-22 19:49:24 GMT [INF] [  635] Opened backup file backup_2020_07_21/BB9030011AC4202_00000.asb
2020-07-22 19:49:25 GMT [INF] [  634] 0 UDF file(s), 0 secondary index(es), 0 record(s) (0 KiB/s, 0 rec/s, 0 B/rec, backed off: 0)
2020-07-22 19:49:25 GMT [INF] [  634] Expired 0 : skipped 0 : err_ignored 0 : inserted 0: failed 0 (existed 0 , fresher 0)
2020-07-22 19:49:25 GMT [INF] [  634] 100% complete, ~0s remaining
2020-07-22 19:49:25 GMT [INF] [  636] Starting restore to 127.0.0.1 (bins: [all], sets: [all]) from backup_2020_07_22
2020-07-22 19:49:25 GMT [INF] [  636] Processing 1 node(s)
2020-07-22 19:49:25 GMT [INF] [  636] Found 1 backup file(s) in backup_2020_07_22
2020-07-22 19:49:25 GMT [INF] [  636] Opened backup file backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:49:25 GMT [INF] [  636] Restoring 0 UDF file(s)
2020-07-22 19:49:25 GMT [INF] [  636] Restoring 0 secondary index(es)
2020-07-22 19:49:25 GMT [INF] [  636] Restoring records
2020-07-22 19:49:25 GMT [INF] [  655] Restoring backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:49:25 GMT [INF] [  655] Opened backup file backup_2020_07_22/BB9030011AC4202_00000.asb
2020-07-22 19:49:26 GMT [INF] [  654] 0 UDF file(s), 0 secondary index(es), 1 record(s) (0 KiB/s, 0 rec/s, 137 B/rec, backed off: 0)
2020-07-22 19:49:26 GMT [INF] [  654] Expired 0 : skipped 0 : err_ignored 0 : inserted 1: failed 0 (existed 0 , fresher 0)
2020-07-22 19:49:26 GMT [INF] [  654] 100% complete, ~0s remaining

Thank you! I will write a script to do that.

Sound good. One thing to be careful with when restoring backup in general is restoring of deleted records.

What do you recommend as the best way to deal with deleted records during restore?

There is some nice discussion on incremental backups here:

But in terms of deletes, I’d say re-running the clean up process that originally did the deletes.

1 Like

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.