The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
Synopsis:
This article describes how to clone or migrate data from one cluster to another with no to minimal downtime.
Summary:
There are situations where you may need to migrate Aerospike from an older cluster to a newer cluster with no to minimal downtime.
Use the following steps:
-
If you have created user and roles, make sure to check the details and re-create those on the new cluster.
-
Install Aerospike on the new cluster.
-
Set up XDR to ship from old data center to new data center. (Must be done on all nodes of the old cluster, can be done in a rolling fashion).
-
Note that XDR will only ship newly written data. Thus, we will have to move the existing data to the new cluster. This can be done a few different ways:
-
Leverage the
rewindfeature if using XDR 5.0 or above. Refer to the How to rewind XDR for a namespace article. -
Scan and touch all the records in the original cluster, setting the TTL to -2. Note that last-update-time of records will be changed so incremental backup/truncate features will be affected. An example UDF to do this would be
function touch(rec) record.set_ttl(rec, -2) aerospike:update(rec) endAs always, test on your system before using in production.
- Backup / Restore with the “-u” or “–unique” option (while pausing XDR shipping to avoid overwriting potential deletes). The following steps describe this method.
-
-
-
To avoid potentially overwriting delete transactions, make sure to disable xdr shipping prior to taking the backup. If the use case or setup does not allow disabling xdr shipping prior to taking the backup, any records which would be deleted on the old cluster after the backup is taken would be restored from backup/restore process.
-
For each namespace:
-
Take a backup of the old data center, using the asbackup tool.
# Example asbackup --host NODE_IP --namespace NAMESPACE_NAME --directory backups/NAMESPACE_NAMEWith server version 3.12 or above, you can do an incremental backup (“–modified-before” option) to specify only records before XDR has started shipping.
-
Using the “-u” option, restore the backed up data on the new data center using the asrestore tool.
Note: “–unique” instructs the asrestore to not overwrite existing records. If you omit this option, then by default, asrestore will check the generation, you may either get ‘generation mismatch’ error or will overwrite the records in the cluster that might be a different copy as shipped by XDR (XDR does not preserve generation when shipping). You can read more on asrestore write policies.# Example asrestore --host NODE_IP --directory backups/NAMESPACE_NAME --unique -
If intermediate files are not needed, the stdout of asbackup can be piped out to the stdin of asrestore: When running the command below, it will backup the data from the full source cluster. One of the nodes to be specified, SOURCE_NODE_IP, will be acting as the entry point to the cluster. The remaining cluster nodes will be automatically discovered. The other node to be specified, TARGET_NODE_IP, is one of the node of the target cluster, which will act as the entry point to the cluster. The remaining cluster nodes will be automatically discovered. Again, this method will not generate any backup files for future reference or use.
# Example asbackup -h SOURCE_NODE_IP -n NAMESPACE_NAME --outputfile - | asrestore -h TARGET_NODE_IP --inputfile - --unique
-
-
Re-enable XDR. (Use this Knowledge Base article for pre-5.0.) Monitor the throughput and, if necessary, limit it using the max-throughput configuration parameter (or xdr-max-ship-throughput for pre-5.0).
-
Allow XDR to synchronize completely, and verify that XDR lag is zero.
-
Redirect clients to new datacenter.
- (Option 1) Redirect all clients to the new datacenter.
- (Option 2) Redirect clients to the new datacenter one at a time. This avoid downtime but some updates during the transition will be lost.
-
You may have to add user/roles if they have not been setup at step 1.
-
To tune asbackup and restore process, please check this KB - Common asbackup and asrestore questions FAQ - Common asbackup and asrestore questions
Special case:
If the plan is to setup an active-active XDR, then before restoring the backup on the new cluster please disable xdr dynamically for the namespace you are restoring. After you are done restoring you can enable it again.
For versions prior to 3.6.1 Tools release:
Use the older backup and restore conventions -
asbackup -h NODE_IP -n NAMESPACE_NAME -d backups/NAMESPACE_NAME
asrestore -h NODE_IP -d backups/NAMESPACE_NAME -u
Notes
- It is a known behaviour that XDR will ship a small amount of records even after a
force linkdowncommand has been issued. This behaviour is documented in the FAQ - Why are a few records shipped to a remote DC when using the force linkdown commmand article.
Keywords
backup restore migrate replicate older existing
Timestamp
October 2020