FAQ - How to disable XDR

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

FAQ - How to disable XDR

XDR version 5.x and above:

Disable XDR permanently

Perform a brute-force disabling of XDR by clearing the relevant XDR entries in aerospike.conf and proceeding with a rolling restart of the Aerospike cluster.

Disassociate a namespace to ship to a remote datacenter

In order to stop shipping to each DC you can use the following info command to remove each namespaces from those DCs

asinfo -v 'set-config:context=xdr;dc=<dcname>;namespace=<nsname>;action=remove'

or with asadm

asadm -e "asinfo -v 'set-config:context=xdr;dc=<dcname>;namespace=<nsname>;action=remove'"

Dynamically delete DC configuration

Delete a DC and remove all the associated namespace-level configs of that DC.

asinfo -v 'set-config:context=xdr;dc=<dcname>;action=delete'

or with asadm

asadm -e "asinfo -v 'set-config:context=xdr;dc=<dcname>;action=delete'"

Disable XDR temporarily

To disable XDR temporarily per namespace per DC set the configuration parameter max-throughput to zero.

asinfo -v 'set-config:context=xdr;dc=DC1;namespace=namespaceName;max-throughput=0'

or with asadm (to update on all the nodes in the cluster):

asadm -e "asinfo -v 'set-config:context=xdr;dc=DC1;namespace=namespaceName;max-throughput=0'"

max-throughput is the number of records that is written per second to the destination DC. Once set to zero, XDR stops writing any record to the destination dc for the namespace.

XDR less than 5.x:

The below section applied to older version of xdr which allowed for disabling of xdr at both the XDR and Namespace contexts.

Disable XDR permanently

In order to disable XDR, you can follow the steps described in the Enable XDR Dynamically knowledge base article in the reverse order.

You can always perform a brute-force disable of XDR by clearing the relevant XDR entries in aerospike.conf and proceeding with a rolling restart of the Aerospike cluster. Review the XDR window-shipping and Failed Node Processing knowledge-base article to understand how XDR reacts to node restarts within the cluster.

This knowledge-base covers how you can dynamically alter XDR configurations based on what your use-case requirements are.

Disassociate a namespace to ship to a remote datacenter.

To disassociate the namespace bar to ship to DC2 in this cluster. you can follow the example below:

Admin> info dc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~DC Information (2019-04-17 00:23:26 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      Node    DC          DC     DC   Namespaces        Lag   Records       Avg             Status   
                         .     .        type   size            .      (sec)   Shipped   Latency                  .   
                         .     .           .      .            .          .         .      (ms)                  .   
v24-vm1.aerospike.com:3000   DC2   aerospike      2   test,bar     00:00:00    283016         4   CLUSTER_UP         
v24-vm2.aerospike.com:3000   DC2   aerospike      2   test,bar     00:00:00    229964         2   CLUSTER_UP         
v24-vm7.aerospike.com:3000   DC2   aerospike      2   test,bar     00:00:00    147294         3   CLUSTER_UP         
v24-vm1.aerospike.com:3000   DC3   aerospike      1   bar          00:00:00    168607         1   CLUSTER_DOWN       
v24-vm2.aerospike.com:3000   DC3   aerospike      1   bar          00:00:00    109937         0   CLUSTER_DOWN       
v24-vm7.aerospike.com:3000   DC3   aerospike      1   bar          00:00:00     49400         0   CLUSTER_DOWN       
v24-vm1.aerospike.com:3000   DC4   aerospike      0                00:00:00         0         0   CLUSTER_INACTIVE   
v24-vm2.aerospike.com:3000   DC4   aerospike      0                00:00:00         0         0   CLUSTER_INACTIVE   
v24-vm7.aerospike.com:3000   DC4   aerospike      0                00:00:00         0         0   CLUSTER_INACTIVE   
Number of rows: 9

Admin> asinfo -v "set-config:context=namespace;id=bar;xdr-remote-datacenter=DC2;action=remove"
v24-vm2.aerospike.com:3000 (192.168.120.216) returned:
ok

v24-vm7.aerospike.com:3000 (192.168.120.188) returned:
ok

v24-vm1.aerospike.com:3000 (192.168.120.215) returned:
ok

Admin> info dc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~DC Information (2019-04-17 00:40:51 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      Node    DC          DC     DC   Namespaces        Lag   Records       Avg             Status   
                         .     .        type   size            .      (sec)   Shipped   Latency                  .   
                         .     .           .      .            .          .         .      (ms)                  .   
v24-vm1.aerospike.com:3000   DC2   aerospike      2   test         00:00:00    323423         4   CLUSTER_UP         
v24-vm2.aerospike.com:3000   DC2   aerospike      2   test         00:00:00    255323         2   CLUSTER_UP         
v24-vm7.aerospike.com:3000   DC2   aerospike      2   test         00:00:00    187214         3   CLUSTER_UP         
v24-vm1.aerospike.com:3000   DC3   aerospike      1   bar          00:22:27    168607         0   CLUSTER_DOWN       
v24-vm2.aerospike.com:3000   DC3   aerospike      1   bar          00:22:23    109937         0   CLUSTER_DOWN       
v24-vm7.aerospike.com:3000   DC3   aerospike      1   bar          00:22:19     49400         0   CLUSTER_DOWN       
v24-vm1.aerospike.com:3000   DC4   aerospike      0                00:00:00         0         0   CLUSTER_INACTIVE   
v24-vm2.aerospike.com:3000   DC4   aerospike      0                00:00:00         0         0   CLUSTER_INACTIVE   
v24-vm7.aerospike.com:3000   DC4   aerospike      0                00:00:00         0         0   CLUSTER_INACTIVE   
Number of rows: 9

Disable XDR temporarily and (optionally) stop saving digestlog entries into the digestlog

  • To disable XDR for the entire source cluster:
asinfo -v 'set-config:context=xdr;enable-xdr=false'
  • To disable XDR for a specific namespace only:
asinfo -v 'set-config:context=namespace;id=namespaceID;enable-xdr=false'

NOTE 1: The digestlog entries already in the digestlog will still be processed if you resume XDR at a later time. Review the knowledge-base on Impact of enable-xdr at Global vs Namespace level for further details.

NOTE 2: The connection to a remote datacenter won’t be broken unless you either dissociate the datacenter from all local namespaces, or remove all seed addresses from the datacenter definition.

Force a DC into CLUSTER_DOWN state temporarily

If for any reason, you do plan on resuming from where XDR was stopped, you can force the link down for the datacenter. This allows processing and shipping of those entries at a later time as digestlog entries will be written into the digestlog.

  • To disable XDR for one datacenter:

IMPORTANT NOTE: forcing a cluster into CLUSTER_DOWN state should always be carefully considered, as the digestlog would continue to grow in order to allow for entries to be processed when the cluster can receive records again:

  • As the digest log grows, the reclaim needs to search through more pages to find the global last ship time and move the start pointer. This happens every minute and will increase disk IO and slow down the digestlog reclamation process.
  • If a node at the source gets restarted while there is a lag, XDR starts processing entries from the start pointer even if it does not ship all of them, causing extra strain.
  • Forcing a destination cluster in CLUSTER_DOWN could cause stop_writes if the xdr-min-digestlog-free-pct has been configured.

Suspend shipping

asinfo -v 'set-config:context=xdr;xdr-shipping-enabled=false'
  • To disable incoming XDR traffic to a namespace, set ‘allow-xdr-writes’ to false on the destination cluster:
asinfo -v "set-config:context=namespace;id=namespaceID;allow-xdr-writes=true"

Notes

  • Refer to the following article for details about relogging in XDR:
  • Remember to run the asinfo commands within asadm to apply to all nodes across a cluster.

Keywords

XDR DISABLE DATACENTER SUSPEND SHIPPING

Timestamp

June 2018

1 Like