XDR 5.0 has a different behaviour when adding a new dc dynamically and when adding via rolling restart.
XDR maintains a last ship time for each DC within the system metadata directory (SMD) of each node. This is necessary as from Aerospike 5.0 there is no file based logging and incoming digests are held in in-memory queues prior to shipment. At start up, the node checks the last ship time for each configured DC and ships anything later than that last ship time. When a new destination is added while the node is down, XDR just ships from the beginning. The reason that
recoveries metric increases is that the shipping described here is done in recovery mode whereby XDR reduces a partition rather than shipping from entries in an in-memory queue. This partition reduction process may occur several times within a given recovery cycle and so it is not unexpected to see an increment of greater than 4096 listed for the
recoveries statistic. XDR will only ship for the master paritions that it owns after the restart.
The most efficient and the recommended way to add a new DC is to add it dynamically. As a standard practice, when ever a dynamic change is done, we should also change the config file to reflect that change, so that if there is a restart in the future the same config takes effect.
When adding a new DC dynamically, please refer to the Enable XDR Dynamically article for detailed steps.
We only ship records from that point and not the whole namespace. If there is a need to ship from a certain point or the whole namespace, the rewind feature can be leveraged.
NOTE: the latest versions from each lineage starting from 5.0 have the same behaviour. Only some intermediate builds from 188.8.131.52 had a different behaviour which was reverted as part of AER-6365.
For 5.1, it was fixed in 184.108.40.206 For 5.2, it was fixed in 220.127.116.11 For 5.3, it was fixed in 18.104.22.168.
Everything beyond 22.214.171.124 has a similar behaviour.
- Full documentation on how to configure XDR in both static and dynamic modes can be found here.
XDR 5.0 STATIC NAMESPACE SHIPPING EVERYTHING ADDITION REWIND