Setup XDR to replicate from DC1 to DC2

#1

Setup XDR to replicate from DC1 to DC2

Plan is to do it in two phases.

Phase 1 :

In this phase both XDR stanza and namespace stanzas will be modified in aerospike.conf

We will first add the XDR stanza as follows:

## XDR Stanza Configurations
xdr {
    enable-xdr                   true       # static setting required to be true to start XDR
    namedpipe-path               /tmp/xdr_pipe
    digestlog-path               /etc/aerospike/digestlog 30000000000   #30GB   # circular buffer to hold records digest 
    errorlog-path                /var/log/asdxdr.log
    xdr-pidfile                  /var/run/aerospike/asxdr.pid
    local-node-port              3000
    info-port                    3004
    datacenter DC2 {
        dc-node-address-port Aerospike_Node1 3000
        dc-node-address-port Aerospike_Node2 3000
        dc-node-address-port Aerospike_Node3 3000
    }
}

NAMESPACE configuration

Then Add the following two settings to each namespace: enable-xdr false xdr-remote-datacenter DC2

namespace Namespace1 {
        enable-xdr      false
        xdr-remote-datacenter   DC2
        replication-factor 2
        memory-size 2G
        default-ttl 4d

        storage-engine device {
                file /opt/aerospike/data/file.dat
                filesize 4G
                data-in-memory true
        }

}

namespace Namespace2 {
        enable-xdr      false
        xdr-remote-datacenter   DC2
        replication-factor 2
        memory-size 2G
        default-ttl 4d

        storage-engine device {
                file /opt/aerospike/data/file2.dat
                filesize 4G
                data-in-memory true
        }

}

and do a rolling restart of asd on each nodes to load XDR stanza

This is required to load new XDR static configs into asd.

Phase 2: Enable XDR and Start XDR process from command line :

Enable XDR for replicating namespaces.

asmonitor -e "asinfo -v 'set-config:context=namespace;id=Namespace1;enable-xdr=true'"
asmonitor -e "asinfo -v 'set-config:context=namespace;id=Namespace2;enable-xdr=true'"

Verify XDR is enabled by running asmonitor to get config

asmonitor -e "asinfo -v 'get-config:'"|tr ";" "\n"|egrep ':3000|xdr'
asmonitor -e "asinfo -v 'namespace/Namespace1'"|tr ";" "\n"|egrep ':3000|xdr'
asmonitor -e "asinfo -v 'namespace/Namespace2'"|tr ";" "\n"|egrep ':3000|xdr'

Start XDR on each nodes of DC1

sudo service aerospike_xdr start

Verify XDR status through AMC or via command line

asmonitor -e 'info XDR'

Make changes persistent in aerospike.conf

enable-xdr to true for each of the namespaces being replicated in aerospike.conf for changes to persist after a restart

#2

I couldn’t start this service.

#3

This is an old article which talks about the steps when XDR and ASD were different processes. You must be using the latest aerospike build. Now XDR is an integral part of ASD. So, XDR is not started separately. It starts along with ASD. Read the deployment guide here : https://www.aerospike.com/docs/operations/configure/cross-datacenter/index.html.

#4

Thanks for the info. I have setup xdr on 2 of my clusters and I’m seeing a strange observation. I can see that when I insert a record in Cluster 1(using AQL), the record gets reflected on Cluster 2(when read using AQL). However, the records inserted using my application(c client) in Cluster 1 are not getting reflected in Cluster 2. Any idea how this might happen?

#5

No, that should not happen. This most likely means that some configuration is not done properly. Double check your configurations once. Are the namespaces used by AQL and your application same ? If not, make sure that XDR is enabled at namespace level too. If both are using the same namespace, check the set-level settings. Also check the ttl of the records written by AQL and your application. I think AQL writes with infinite TTL by default. May be your application is writing with small TTL and they are expiring on Cluster-2 by the time you are checking. There are many possibilities for this problem based on how you configured. So, please do basic checks again.