How to add a new Datacenter (DC) in XDR


#1

How to add a new Datacenter (DC) in XDR, version 3.8 onwards

Context

If a skeleton of a DC configuration exists in xdr stanza, seeding a new data center can be achieved dynamically.

Sample skeleton datacenter definition in the xdr stanza

xdr {
    enable-xdr true 
    xdr-digestlog-path /opt/aerospike/xdr/digestlog 100G 
    # make sure the /opt/aerospike/xdr folder exists

    datacenter lh-trial-newdc {
    }
}

Method

Steps to configure a new DC and enable shipping for a particular namespace, version 3.8 onwards.

Step 1:

Configure the new DC using the following command (replace the DC name and node name as appropriate). Repeat this command for a few nodes in the new DC (technically one is enough, but it is good practice to have a few, similar to the seeding of any Aerospike client library). The actual connections to the remote cluster only happen when the DC definition has at least one node specified in it.

asadm -e "asinfo -v 'set-config:context=xdr;dc=lh-trial-newdc;dc-node-address-port=cf2000.trial.newdc.com:3000;action=add'"
asadm -e "asinfo -v 'set-config:context=xdr;dc=lh-trial-newdc;dc-node-address-port=cf2001.trial.newdc.com:3000;action=add'"

Note that this command will return “PENDING” for the first added node. After every command, check the logs to ensure the action has completed before proceeding to the next add command.

2016-10-07 10:11:16.474745873Z Apr 07 2017 10:11:16 GMT: INFO (xdr): (xdr_info.c:718) For DC lh-trial-newdc, added new seed node cf2000.trial.newdc.com:3000
2016-10-07 10:11:16.474760198Z Apr 07 2017 10:11:16 GMT: INFO (info): (thr_info.c:3484) config-set command : params context=xdr;dc=lh-trial-newdc;dc-node-address-port=cf2000.trial.newdc.com:3000;action=add
2016-10-07 10:11:16.474770518Z Apr 07 2017 10:11:16 GMT: INFO (xdr): (xdr.c:5429) Connecting to new remote cluster 'lh-trial-newdc'

Check the different statistics and configuration for the dc:

asadm -e "asinfo -v 'dc/lh-trial-newdc'"
asadm -e "asinfo -v 'get-dc-config'"

All the nodes should be visible as part of the remote DC.

Step 2:

This step is only necessary if the source DC cannot reach the remote DC on the regular services list. In such cases the remote DC can specify an alternate-access-address in their configuration file.

network {
    service {
        address any
        port 3000
        access-address 10.18.68.10 
        alternate-access-address 192.168.20.7
    }
}

Then, configure the source XDR to use alternate-access-address instead of address.

asadm -e "asinfo -v 'set-config:context=xdr;dc=lh-trial-newdc;dc-use-alternate-services=true'"

Please note that in versions prior to 3.10, alternate-access-address needs to be replaced with alternate-address. More details on the the networking changes can be found in the 3.10 upgrade documentation.

Step 3:

Namespaces can ship to multiple datacenters. For a particular namespace, run the command below to enable xdr shipping to a datacenter.

asadm -e "asinfo -v 'set-config:context=namespace;id=testnamespace;xdr-remote-datacenter=lh-trial-newdc;action=add'"
Step 4:

Make sure the xdr skeletal stanza in the configuration file is populated with the dynamically added nodes. In the event of any future restarts, the new configuration will then be picked from the configuration file and match what was dynamically configured.

a) Update the namespace section to include the xdr-remote-datacenter config

namespace testnamespace {

    enable-xdr true
    xdr-remote-datacenter lh-trial-newdc

    dc-use-alternate-services true # if used
    ...
    ...
}

b) In xdr section, add the nodes to the skeletal DC stanza.

xdr {
    enable-xdr true 

    xdr-digestlog-path /opt/aerospike/xdr/digestlog 100G 
    # make sure the /opt/aerospike/xdr folder exists

    datacenter lh-trial-newdc {
        dc-node-address-port cf2000.trial.newdc.com 3000
        dc-node-address-port cf2001.trial.newdc.com 3000
    }
}
Step 5:

Enable XDR shipping for the namespace (Only required if not already enabled). Enabling XDR at the namespace level will start writing to the digest log.

asinfo -v 'set-config:context=namespace;id=testnamespace;enable-xdr=true'
Step 6:

Verify if shipping has started and the throughput is increasing. XDR connects to a remote DC after shipping has started for the first time. So the Cluster up detection should be in sync with the very first instance of throughput exceeding zero.

[lh-trial-newdc]: dc-state CLUSTER_UP timelag-sec 2 lst 1468006386894 mlst 1468006389647 (2016-07-08 19:33:09.647 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 0 rsas-ms 0.004 rsas-pct 0.0
summary: throughput 3722 inflight 164 dlog-outstanding 100 dlog-delta-per-sec -10.0

Notes

XDR Configuration XDR Basics Enabling XDR functionality dynamically, prior 3.8

Keywords

XDR DYNAMIC Datacenter DC

Timestamp

2nd Aug 2016