Enable XDR Dynamically

Knowledge Base XDR
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

FAQ - Can XDR be configured dynamically?

Detail

XDR is a feature that some customers may wish to try once they have already got their first Aerospike cluster up and running. XDR 5.0 ships on a per DC basis and can have its DCs added and configured in a fully dynamic manner. Steps to do this are shown below. In earlier versions a skeleton data center entry was required for each data center that would need to be dynamically configured. In XDR 5.0 and above, only the first DC has to have a skeleton in place. Further DC entries can dynamically be added. An entry as shown below in the aerospike.conf configuration file is the minimum necessary to make DC1 a skeletal DC and to create further DCs dynamically.

dc DC1 {
 }

Answer

Aerospike Releases 5.0 and Later

The steps to create and enable shipping to a new data center (DC) are as follows:

  • Create a dc if it is not already defined using the following asinfo command:
asinfo -v "set-config:context=xdr;dc=DC1;action=create"
ok
  • Define the remote datacenter using the asinfo command shown below:
asinfo -v "set-config:context=xdr;dc=DC1;node-address-port=10.0.0.2:3000;action=add"
ok
  • Add one or many namespaces to that datacenter:
asinfo -h localhost -v "set-config:context=xdr;dc=DC1;namespace=someNewNameSpaceName;action=add
ok
  • Addtionally, with Aerospike 5.0 and later a new DC can be populated by rewinding the shipping on a namespace to the start:
asinfo -h localhost -v "set-config:context=xdr;dc=DC1;namespace=someNewNameSpaceName;action=add;rewind=all
ok
  • Datacenters can be fully removed in a similar manner:
asinfo -h localhost -v "set-config:context=xdr;dc=DC1;action=delete"
  • Individual namespaces can be removed as follows:
asinfo -h localhost -v "set-config:context=xdr;dc=DC1;namespace=someNewNameSpaceName;action=remove"

Aerospike Releases 3.8 to Aerospike 5.0

Once an XDR datacenter has been configured, enabling shipping to it for a particular namespace is a simple process which can be done without restarting any nodes, by dynamically setting the enable-xdr and xdr-remote-datacenter parameters for the namespace.

Note: The XDR stanza and datacenters must already have been configured for these steps to work.

Step 1: Associate Namespace and Datacenter

Specify the remote datacenter the namespace should ship to, across all nodes:

asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;xdr-remote-datacenter=<DCNAME>;action=add"'

A namespace can be shipped to multiple remote datacenters; run the command separately for each combination. You can see the current configuration with

asadm -e 'info dc'

Namespace Sets can also be allowlisted or blocklisted for replication dynamically:

asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;sets-enable-xdr=<true|false>"'
asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;set=<SETNAME>;set-enable-xdr=<false|true>"'

Step 2: Global XDR Shipping

Enable XDR shipping for the cluster as a whole:

asadm -e 'asinfo -v "set-config:context=xdr;enable-xdr=true"'

Note: In versions prior to 3.14.1.1, dynamically setting enable-xdr to true at the XDR context does not properly work. In those versions, enable-xdr should always be set to true at the XDR stanza level in the configuration file and controlled dynamically (or in the configuration file) at the namespace stanza level in order to enable or disable writing digest log entries into the digest log.

Step 3: XDR Shipping For Namespace

Enable XDR shipping for the namespace across all nodes:

asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;enable-xdr=true"'

Step 4: Confirmation

Verify if shipping has started and the throughput is increasing.

asadm -e 'asinfo -l -v "dc/<DCNAME>"'

XDR connects to a remote DC after shipping has started for the first time, so the CLUSTER_UP detection should be in sync with the very first instance of dc_ship_success exceeding zero.

Step 5: Persistence

Once XDR shipping has been enabled and is working satisfactorily, the best practise is to edit the aerospike.conf configuration file to add the XDR details to the namespace stanza and XDR stanza so that these persist in the event of a node restart.

Backing Out

To stop XDR shipping for a namespace:

 asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;enable-xdr=false"'

To completely disassociate the datacenter from the namespace:

asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;xdr-remote-datacenter=<DCNAME>;action=remove"'

To stop XDR shipping for the cluster as a whole (only in version 3.14.1.1 and later; see note above):

asadm -e 'asinfo -v "set-config:context=xdr;enable-xdr=false"'

For Aerospike Releases Before 3.8

In versions of Aerospike before 3.8 the XDR process was separate to the main Aerospike process. The aerospike.conf file has to be fully populated with XDR information, including xdr-remote-datacenter in the namespace stanza and a full XDR stanza including all seed nodes within the datacenter definition. If this is done, XDR can be enabled as follows:

asadm -e 'asinfo -v "set-config:context=namespace;id=<NAMESPACE>;enable-xdr=true"'
asadm -e 'asinfo -v "set-config:context=xdr;enable-xdr=true"'

The XDR service is then started as follows:

sudo /etc/init.d/aerospike_xdr resume

Notes

Keywords

ENABLE XDR DYNAMIC START CONFIGURE RESTART 3.8 REMOTE DATACENTER DC 5.0

Timestamp

November 2020

2

I believe, an entry like below in aerospike.conf

dataceneter DC1 {
 }

is good enough to make “DC1” a skeletal DC. The alternate-services and security-config file are dynamic and can be added later.