This article describes how to clone or migrate data from one cluster to another with no to minimal downtime.
There are situations where you may need to migrate Aerospike from an older cluster to a newer cluster with no to minimal downtime.
Use the following steps:
If you have created user and roles, make sure to check the details and re-create those on the new cluster.
Install Aerospike on the new cluster.
Setup XDR to ship from old data center to new data center. (Can be done in a rolling fashion).
Note that XDR will only ship newly written data. Thus, we will have to move the existing data to the new cluster. This can be done a couple of different ways:
Scan and touch all the records in the original cluster. This would modify the TTL of the records (unless if TTL is set to ‘-2’ – available for server versions 3.10 and above). Note that last-update-time of records will be changed so incremental backup/truncate features (since server version 3.12) will be affected.
Backup / Restore with the “-u” or “–unique” option (while pausing XDR shipping to avoid overwriting potential deletes). The following steps describe this method.
To avoid potentially overwriting delete transactions, make sure to disable xdr shipping prior to taking the backup. Refer to the xdr-shipping-enabled configuration variable (make sure your digestlog will be able to hold the entries that would accrue during this time):
asadm> asinfo -v 'set-config:context=xdr;xdr-shipping-enabled=false'
If the use case or setup does not allow disabling xdr shipping prior to taking the backup, the records which would be deleted via XDR after the backup is taken would be restored from backup/restore process.
For each namespace:
Take a backup of the old data center, using the asbackup tool.
# Example asbackup --host NODE_IP --namespace NAMESPACE_NAME --directory backups/NAMESPACE_NAME
With server version 3.12 or above, you can do an incremental backup ("–modified-before" option) to specify only records before XDR has started shipping.
Using the “-u” option, restore the backed up data on the new data center using the asrestore tool.
Note: “–unique” instructs the asrestore to not overwrite existing records. If you omit this option, then by default, asrestore will check the generation, you may either get ‘generation mismatch’ error or will overwrite the records in the cluster that might be a different copy as shipped by XDR (XDR does not preserve generation when shipping). You can read more on asrestore write policies.
# Example asrestore --host NODE_IP --directory backups/NAMESPACE_NAME --unique
If intermediate files are not needed, the stdout of asbackup can be piped out to the stdin of asrestore: When running the command below, it will backup the data from the full source cluster. One of the nodes to be specified, SOURCE_NODE_IP, will be acting as the entry point to the cluster. The remaining cluster nodes will be automatically discovered. The other node to be specified, TARGET_NODE_IP, is one of the node of the target cluster, which will act as the entry point to the cluster. The remaining cluster nodes will be automatically discovered. Again, this method will not generate any backup files for future reference or use.
# Example asbackup -h SOURCE_NODE_IP -n NAMESPACE_NAME --outputfile - | asrestore -h TARGET_NODE_IP --inputfile - --unique
Re-enable XDR shipping (monitor the throughput and, if necessary, limit it using xdr-max-ship-throughput as digestlog entries may have accrued:
asadm> asinfo -v 'set-config:context=xdr;xdr-shipping-enabled=true'
Allow XDR to synchronize completely, and verify that XDR lag is zero.
Redirect clients to new datacenter.
- (Option 1) Redirect all clients to the new datacenter.
- (Option 2) Redirect clients to the new datacenter one at a time. This avoid downtime but some updates during the transition will be lost.
You may have to add user/roles if they have not been setup at step 1.
To tune asbackup and restore process, please check this KB - Common asbackup and asrestore questions FAQ - Common asbackup and asrestore questions
If the plan is to setup an active-active XDR, then before restoring the backup on the new cluster please disable xdr dynamically for the namespace you are restoring. You can use the following command:
asadm -e "asinfo -v 'set-config:context=namespace;id=namespace_name;enable-xdr=false'"
Verify by running this inside asadm
Admin> show config namespace like enable-xdr ~~test Namespace Configuration~ NODE : 10.0.2.15:3000 enable-xdr : false
It should show enable-xdr false on all the nodes for that namespace.
After you are done restoring you can change it to true again.
asadm -e "asinfo -v 'set-config:context=namespace;id=namespace_name;enable-xdr=true'"
Verify again using the same command. This time it should be true. This is to avoid the restored records getting shipped back to the source.
Please be informed that for the duration where it is false, xdr will not log any digests for that namespace in the digest log.
For versions prior to 3.6.1 Tools release:
Use the older backup and restore conventions -
asbackup -h NODE_IP -n NAMESPACE_NAME -d backups/NAMESPACE_NAME asrestore -h NODE_IP -d backups/NAMESPACE_NAME -u
backup restore migrate replicate older existing