The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ - How to disable XDR
XDR version 5.x and above:
Disable XDR permanently
Perform a brute-force disabling of XDR by clearing the relevant XDR entries in aerospike.conf and proceeding with a rolling restart of the Aerospike cluster.
Disassociate a namespace to ship to a remote datacenter
In order to stop shipping to each DC you can use the following info command to remove each namespaces from those DCs
asinfo -v 'set-config:context=xdr;dc=<dcname>;namespace=<nsname>;action=remove'
or with asadm
asadm -e "asinfo -v 'set-config:context=xdr;dc=<dcname>;namespace=<nsname>;action=remove'"
Dynamically delete DC configuration
Delete a DC and remove all the associated namespace-level configs of that DC.
asinfo -v 'set-config:context=xdr;dc=<dcname>;action=delete'
or with asadm
asadm -e "asinfo -v 'set-config:context=xdr;dc=<dcname>;action=delete'"
Disable XDR temporarily
To disable XDR temporarily per namespace per DC set the configuration parameter max-throughput
to zero.
asinfo -v 'set-config:context=xdr;dc=DC1;namespace=namespaceName;max-throughput=0'
or with asadm (to update on all the nodes in the cluster):
asadm -e "asinfo -v 'set-config:context=xdr;dc=DC1;namespace=namespaceName;max-throughput=0'"
max-throughput
is the number of records that is written per second to the destination DC. Once set to zero, XDR stops writing any record to the destination dc for the namespace.
XDR less than 5.x:
The below section applied to older version of xdr which allowed for disabling of xdr at both the XDR and Namespace contexts.
Disable XDR permanently
In order to disable XDR, you can follow the steps described in the Enable XDR Dynamically knowledge base article in the reverse order.
You can always perform a brute-force disable of XDR by clearing the relevant XDR entries in aerospike.conf and proceeding with a rolling restart of the Aerospike cluster. Review the XDR window-shipping and Failed Node Processing knowledge-base article to understand how XDR reacts to node restarts within the cluster.
This knowledge-base covers how you can dynamically alter XDR configurations based on what your use-case requirements are.
Disassociate a namespace to ship to a remote datacenter.
To disassociate the namespace bar to ship to DC2 in this cluster. you can follow the example below:
Admin> info dc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~DC Information (2019-04-17 00:23:26 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node DC DC DC Namespaces Lag Records Avg Status
. . type size . (sec) Shipped Latency .
. . . . . . . (ms) .
v24-vm1.aerospike.com:3000 DC2 aerospike 2 test,bar 00:00:00 283016 4 CLUSTER_UP
v24-vm2.aerospike.com:3000 DC2 aerospike 2 test,bar 00:00:00 229964 2 CLUSTER_UP
v24-vm7.aerospike.com:3000 DC2 aerospike 2 test,bar 00:00:00 147294 3 CLUSTER_UP
v24-vm1.aerospike.com:3000 DC3 aerospike 1 bar 00:00:00 168607 1 CLUSTER_DOWN
v24-vm2.aerospike.com:3000 DC3 aerospike 1 bar 00:00:00 109937 0 CLUSTER_DOWN
v24-vm7.aerospike.com:3000 DC3 aerospike 1 bar 00:00:00 49400 0 CLUSTER_DOWN
v24-vm1.aerospike.com:3000 DC4 aerospike 0 00:00:00 0 0 CLUSTER_INACTIVE
v24-vm2.aerospike.com:3000 DC4 aerospike 0 00:00:00 0 0 CLUSTER_INACTIVE
v24-vm7.aerospike.com:3000 DC4 aerospike 0 00:00:00 0 0 CLUSTER_INACTIVE
Number of rows: 9
Admin> asinfo -v "set-config:context=namespace;id=bar;xdr-remote-datacenter=DC2;action=remove"
v24-vm2.aerospike.com:3000 (192.168.120.216) returned:
ok
v24-vm7.aerospike.com:3000 (192.168.120.188) returned:
ok
v24-vm1.aerospike.com:3000 (192.168.120.215) returned:
ok
Admin> info dc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~DC Information (2019-04-17 00:40:51 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node DC DC DC Namespaces Lag Records Avg Status
. . type size . (sec) Shipped Latency .
. . . . . . . (ms) .
v24-vm1.aerospike.com:3000 DC2 aerospike 2 test 00:00:00 323423 4 CLUSTER_UP
v24-vm2.aerospike.com:3000 DC2 aerospike 2 test 00:00:00 255323 2 CLUSTER_UP
v24-vm7.aerospike.com:3000 DC2 aerospike 2 test 00:00:00 187214 3 CLUSTER_UP
v24-vm1.aerospike.com:3000 DC3 aerospike 1 bar 00:22:27 168607 0 CLUSTER_DOWN
v24-vm2.aerospike.com:3000 DC3 aerospike 1 bar 00:22:23 109937 0 CLUSTER_DOWN
v24-vm7.aerospike.com:3000 DC3 aerospike 1 bar 00:22:19 49400 0 CLUSTER_DOWN
v24-vm1.aerospike.com:3000 DC4 aerospike 0 00:00:00 0 0 CLUSTER_INACTIVE
v24-vm2.aerospike.com:3000 DC4 aerospike 0 00:00:00 0 0 CLUSTER_INACTIVE
v24-vm7.aerospike.com:3000 DC4 aerospike 0 00:00:00 0 0 CLUSTER_INACTIVE
Number of rows: 9
Disable XDR temporarily and (optionally) stop saving digestlog entries into the digestlog
- To disable XDR for the entire source cluster:
asinfo -v 'set-config:context=xdr;enable-xdr=false'
- To disable XDR for a specific namespace only:
asinfo -v 'set-config:context=namespace;id=namespaceID;enable-xdr=false'
NOTE 1: The digestlog entries already in the digestlog will still be processed if you resume XDR at a later time. Review the knowledge-base on Impact of enable-xdr at Global vs Namespace level for further details.
NOTE 2: The connection to a remote datacenter won’t be broken unless you either dissociate the datacenter from all local namespaces, or remove all seed addresses from the datacenter definition.
Force a DC into CLUSTER_DOWN state temporarily
If for any reason, you do plan on resuming from where XDR was stopped, you can force the link down for the datacenter. This allows processing and shipping of those entries at a later time as digestlog entries will be written into the digestlog.
- To disable XDR for one datacenter:
- For Aerospike Server version 4.1 and above use the xdr-command:
asinfo -v 'xdr-command:force-linkdown=true;dc=DC1'
- For older releases, refer to this knowledge base article.
- For Aerospike Server version 4.1 and above use the xdr-command:
IMPORTANT NOTE: forcing a cluster into CLUSTER_DOWN state should always be carefully considered, as the digestlog would continue to grow in order to allow for entries to be processed when the cluster can receive records again:
- As the digest log grows, the reclaim needs to search through more pages to find the global last ship time and move the start pointer. This happens every minute and will increase disk IO and slow down the digestlog reclamation process.
- If a node at the source gets restarted while there is a lag, XDR starts processing entries from the start pointer even if it does not ship all of them, causing extra strain.
- Forcing a destination cluster in CLUSTER_DOWN could cause stop_writes if the xdr-min-digestlog-free-pct has been configured.
Suspend shipping
- To suspend shipping for the entire cluster, you can set
xdr-shipping-enabled
to false:
asinfo -v 'set-config:context=xdr;xdr-shipping-enabled=false'
- To disable incoming XDR traffic to a namespace, set ‘allow-xdr-writes’ to false on the destination cluster:
asinfo -v "set-config:context=namespace;id=namespaceID;allow-xdr-writes=true"
Notes
- Refer to the following article for details about relogging in XDR:
- Remember to run the asinfo commands within asadm to apply to all nodes across a cluster.
Keywords
XDR DISABLE DATACENTER SUSPEND SHIPPING
Timestamp
June 2018