We are currently considering Aerospike for a mission critical context and have the following need :
We have 2 datacenters and each one will host a database cluster
Data stored in each cluster must be exactly the same so we plan to use XDR to sync our namespaces in an active/active topology
Data can be updated on both sides and the changes must be immediately visible on the other side
The question we have are :
Can we guarantee immediate consistency through XDR ?
How do we manage data update concurrency through XDR ?
In case of datacenter link failure we may start to have data divergence since client of each datacenter may update the same set of record while the link is down. The problem is that we cannot automatically determine which record will be the correct one because both versions may be equally valid and only a human can be able to select the correct record or even merge them. I found somewhere that such a case is supported and that we could get both records through the API to allow a human to perform the conflict resolution but I was unable to find much more information about how to do this technically from the API. Is this already implemented ?
XDR only does asynchronous replication. So, it cannot give immediate consistency in its purest sense. Strictly speaking, immediate consistency means that the client/application write should get the acknowledgement only if all the copies are updated. I guess you are not looking for this. When you say “immediate” how immediate you are talking ? Depending on the roundtrip time between the DCs, the lag between the DCs can be small, typically 1-5ms when everything else is good.
XDRs concurrency can be controlled using various config parameters. Update concurrency is controlled by “xdr-max-recs-inflight”. This will determine how many writes will be going in parallel.
You are talking about divergent versions. This is not available for XDR. It used to be available in older API for regular key-value operations. But this is not supported in our new clients as its complex to use. Please dont bank on this feature. If your usecase cannot afford to have XDR write-write conflicts (where the last shipped record wins), you can use the following technique which we recommended to other customers.
The two datacenters write in separate sets into the same namespace. So, there is no write-write conflict to start with. Each cluster updates only one set. However, at the time of read, the application may have to do two reads (from the local set and the remote set) and merge them using the necessary business logic.
“Strictly speaking, immediate consistency means that the client/application write should get the acknowledgement only if all the copies are updated.” . I am using Java client and testing XDR. But, how do i do this? Is there something that Aerospike API does ? meaning only after the data are all copied to the Remote. What i did is, i have application also in the Local Aerospike server. So, as soon as I run and it completes writing in the Local, the Application completes. But, when I query into Remote server, the records are not totally there and is still updating.
Let me refresh this question.
I have a similar situation as described in the original question, I have two Aerospike clusters in two data centers using XDR with active-active topology, our application can not write in separate sets in the same namespace.
I’ve found in the documentation a new option for conflict-resolution-policy added in 3.8.3 named “last-update-time”
Can I use this option to fix XDR conflics using the last-update timestamp? in other words Does Aerospike now supports timestamp based XDR conflict resolution?
Could you please share a link to changelog or documentation about record-level xdr conflict resolution for 5.2. All that i find are about bin level convergence from 5.4, for example:
@AndreChang but do you have information about 5.2? We have some installations with this version and i’m curious whether record-level conflict resolution was already implemented there (didn’t find proof in config reference or release notes).