How to add a new Datacenter (DC) in XDR

Knowledge Base XDR
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

How to add a new Datacenter (DC) in XDR (version 3.8 to pre-5.0)

Context

From version 3.8 to pre-5.0 Aerospike, adding a new datacenter requires a certain amount of changes to the aerospike.conf file of each node, followed by a rolling restart of the Aerospike server on the cluster. Seeding of nodes to the datacenter can be done dynamically, but the satisfactory configuration needs to be added to aerospike.conf so that it will remain in place across future node restarts.

If you need to add a new DC, you should add it at the end of the list. Refer to the note below for some details.
For 5.0 onwards, please see this KB article for details.

Method

Steps to configure a new DC on a cluster that already has XDR running to other DC’s. Note that this is from Aerospike server version 3.8 to pre-5.0.

Step 1:

The aerospike.conf file on each node needs to have a minimal xdr stanza, and a datacenter substanza for each DC that you intend to use.

    xdr {
        enable-xdr true 
        # The /opt/aerospike/xdr directory must exist, and be on a filesystem
        # with enough space for the full size of the digestlog.
        xdr-digestlog-path /opt/aerospike/xdr/digestlog 100G 
   
        datacenter DC1 {
        }
    }

NOTE 0: The ordering of DCs is important as the last ship time is being exchanged among the server nodes along with an ID referring to the DC according to its position in the configuration. Adding or removing a DC that is not the last one in the list could cause some records to not be shipped.

NOTE 1: If you are using a version of Aerospike between 3.8 and 13.12, you may need to add the dc-security-config-file parameter to each datacenter substanza for which you want to use security. This parameter cannot be set dynamically until version 3.8.3, and it cannot be unset (set to “none”) dynamically until version 3.12.

NOTE 2: If you are a using a version of Aerospike between 3.8 and 3.14.1.1, you will need to add the parameter dc-use-alternate-services to each datacenter stanza for which you intend to use the alternate-access-address (known as alternate-address before version 3.10). Even after 3.14.1.1, this parameter cannot be set dynamically unless the datacenter is in the CLUSTER_INACTIVE status, meaning it has no seed addresses or is not used by any namespaces.

NOTE 3: Datacenter names are case-sensitive.

Step 2:

If you need dc-security-config-file or dc-use-alternate-services but have not had to statically set them (see notes above), you can now dynamically set them:

asadm -e 'asinfo -v "set-config:context=xdr;dc=<DCNAME>;dc-security-config-file=<FILEPATH>"'
asadm -e 'asinfo -v "set-config:context=xdr;dc=<DCNAME>;dc-use-alternate-services=true"'

Step 3:

Configure seed nodes for the new DC:

asadm -e 'asinfo -v "set-config:context=xdr;dc=<DCNAME>;dc-node-address-port=<SEEDADDR1>:<SEEDPORT>;action=add"'
asadm -e 'asinfo -v "set-config:context=xdr;dc=<DCNAME>;dc-node-address-port=<SEEDADDR2>:<SEEDPORT>;action=add"'

(Technically one seed node is enough, but it is good practice to have a few, similar to the seeding of any Aerospike client library). Note that this command will return “PENDING” for the first added node. After every command, check the logs to ensure the action has completed before proceeding to the next add command:

<TIMESTAMP>: INFO (xdr): (xdr_info.c:718) For DC <DCNAME>, added new seed node <SEEDADDR1>:<SEEDPORT>
<TIMESTAMP>: INFO (info): (thr_info.c:3484) config-set command : params context=xdr;dc=<DCNAME>;dc-node-address-port=<SEEDADDR1><SEEDPORT>;action=add
<TIMESTAMP>: INFO (xdr): (xdr.c:5429) Connecting to new remote cluster '<DCNAME>'

When finished, check the different statistics and configuration for the dc:

asadm -e 'asinfo -l -v "dc/<DCNAME>"'
asadm -e 'asinfo -l -v "get-dc-config"'

All the nodes should be visible as part of the remote DC. Note that the actual connections to the remote cluster only happen when the DC definition has at least one node specified in it.

Step 4:

Enable XDR for your chosen namespaces.

Step 5:

Once XDR is working to your satisfaction, add the configuration to the configuration file aerospike.conf so it will persist across service restarts.

Notes

Keywords

XDR DYNAMIC Datacenter DC

Timestamp

Feburary 2021