How to Migrate Aerospike from one Cluster to Another


#1

Synopsis:

This article describes how to clone or migrate data from one cluster to another with no to minimal downtime.

Summary:

There are situations where you may need to migrate Aerospike from an older cluster to a newer cluster with no to minimal downtime.

Use the following steps:

  1. If you have created user and roles, make sure to check the details and re-create those on the new cluster.

  2. Install Aerospike on the new cluster.

  3. Setup XDR to ship from old data center to new data center. (Can be done in a rolling fashion).

  • Note that XDR will only ship newly written data. Thus, we will have to move the existing data to the new cluster. This can be done a couple of different ways:

    • Scan and touch all the records in the original cluster. This would modify the TTL of the records (unless if TTL is set to ‘-2’ – available for server versions 3.10 and above). Note that last-update-time of records will be changed so incremental backup/truncate features (since server version 3.12) will be affected.

    • Backup / Restore with the “-u” or “–unqiue” option (while pausing XDR shipping to avoid overwriting potential deletes). The following steps describe this method.

  1. To avoid potentially overwriting delete transactions, make sure to disable xdr shipping prior to taking the backup. Refer to the xdr-shipping-enabled configuration variable (make sure your digestlog will be able to hold the entries that would accrue during this time):

    asadm> asinfo -v 'set-config:context=xdr;xdr-shipping-enabled=false'
    

    If the use case or setup does not allow disabling xdr shipping prior to taking the backup, the records which would be deleted via XDR after the backup is taken would be restored from backup/restore process.

  2. For each namespace:

    1. Take a backup of the old data center, using the asbackup tool.

      # Example
      asbackup --host NODE_IP --namespace NAMESPACE_NAME --directory backups/NAMESPACE_NAME
      

      With server version 3.12 or above, you can do an incremental backup ("–modified-before" option) to specify only records before XDR has started shipping.

    2. Using the “-u” option, restore the backed up data on the new data center using the asrestore tool.

      Note: “–unique” instructs the asrestore to not overwrite existing records. If you omit this option, then by default, asrestore will check the generation, you may either get ‘generation mismatch’ error or will overwrite the records in the cluster that might be a different copy as shipped by XDR (XDR does not preserve generation when shipping). You can read more on asrestore write policies.

      # Example
      asrestore --host NODE_IP --directory backups/NAMESPACE_NAME --unique
      
    3. If intermediate files are not needed, the stdout of asbackup can be piped out to the stdin of asrestore: When running the command below, it will backup the data from the full source cluster. One of the nodes to be specified, SOURCE_NODE_IP, will be acting as the entry point to the cluster. The remaining cluster nodes will be automatically discovered. The other node to be specified, TARGET_NODE_IP, is one of the node of the target cluster, which will act as the entry point to the cluster. The remaining cluster nodes will be automatically discovered. Again, this method will not generate any backup files for future reference or use.

      # Example
      asbackup -h SOURCE_NODE_IP -n NAMESPACE_NAME --outputfile - | asrestore -h TARGET_NODE_IP --inputfile - --unique
      
  3. Re-enable XDR shipping (monitor the throughput and, if necessary, limit it using xdr-max-ship-throughput as digestlog entries may have accrued:

    asadm> asinfo -v 'set-config:context=xdr;xdr-shipping-enabled=true'
    
  4. Allow XDR to synchronize completely, and verify that XDR lag is zero.

  5. Redirect clients to new datacenter.

    1. (Option 1) Redirect all clients to the new datacenter.
    2. (Option 2) Redirect clients to the new datacenter one at a time. This avoid downtime but some updates during the transition will be lost.
  6. You may have to add user/roles if they have not been setup at step 1.

  7. To tune asbackup and restore process, please check this KB - Common asbackup and asrestore questions FAQ - Common asbackup and asrestore questions

Special case:

If the plan is to setup an active-active XDR, then before restoring the backup on the new cluster please disable xdr dynamically for the namespace you are restoring. You can use the following command:

asadm -e "asinfo -v 'set-config:context=namespace;id=namespace_name;enable-xdr=false'"

Verify by running this inside asadm

Admin> show config namespace like enable-xdr
~~test Namespace Configuration~
NODE : 10.0.2.15:3000
enable-xdr : false

It should show enable-xdr false on all the nodes for that namespace.

After you are done restoring you can change it to true again.

asadm -e "asinfo -v 'set-config:context=namespace;id=namespace_name;enable-xdr=true'"

Verify again using the same command. This time it should be true. This is to avoid the restored records getting shipped back to the source.

Please be informed that for the duration where it is false, xdr will not log any digests for that namespace in the digest log.

For versions prior to 3.6.1 Tools release:

Use the older backup and restore conventions -

asbackup -h NODE_IP -n NAMESPACE_NAME -d backups/NAMESPACE_NAME
asrestore -h NODE_IP -d backups/NAMESPACE_NAME -u

Keywords

backup restore migrate replicate older existing

Timestamp

11/08/2017


Migration of Aerospike cluster without downtime
What happens if the device order in a namespace is accidentally changed
Removing namespace without downtime (AER-3485)
Migration of Aerospike cluster without downtime
FAQ - Some common questions on XDR
#2

Can I ask two questions? (1) XDR is only in Enterprise version, right? (2) The NODE_IP and directory is old cluster’s, right? Even though for step 3.2 which is run in new cluster?


#3

Yes, XDR is Enterprise only.

In 3.2, NODE_IP is for the new cluster, the directory is the path the the backup taken in step 3.1, you could, optionally, move these files to the new cluster and restore from there.


#4

is it possible only to copy the differences rather than full-set backup files ? Or else, move these files by network is too heavy.


#5

Incremental backup is coming soon… stay tuned.


#6

Incremental Backup

Starting from Server and Tools version 3.12, timestamp can be specified when initiating a backup to indicate the time-period of interest. Only records whose last-update-time satisfies the specified criteria (before time T1 and/or after time T2) will be backed up. An operational routine can be established to do incremental daily backups. Use the --modified-after option.

–modified-after <YYYY-MM-DD_HH:MM:SS> - Backup data with last-update-time after the specified date-time. The system’s local timezone applies. Available in Server 3.12 and above.

See: http://www.aerospike.com/docs/tools/backup See: http://www.aerospike.com/docs/tools/backup/asbackup.html See: http://www.aerospike.com/download/tools/notes.html#3.12.0