The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
XDR Shipping to strongly consistent namespace returns abandon result 22
Problem Description
When XDR 5.0 has been configured to ship records from one strongly consistent namespace to another the shipping starts to return the following error:
Jul 01 2020 01:57:57 GMT-0400: WARNING (xdr): (dc.c:2356) (repeated:9) {nsStrongConsistency} DC USWEST abandon result 22
Explanation
This result will be shown when XDR has been configured to ship nsup generated delete transactions (expirations, evictions). Strong consistency enabled namespaces will not allow non-durable deletes (expunges) in order to guarantee strong consistency (expunges cannot guarantee consistency during short lived cluster splits or cold restarts for example). As a delete from a remote cluster originating via nsup would not include a tombstone it is, by definition, non-durable. For this reason the default behaviour of a strongly consistent namespace would be to reject the transaction. On the source cluster this would be reflected in the abandon result 22 message.
Solution
Care should be taken when considering how to approach resolution of this error, not least because it serves to protect the database against potentially undesirable behaviour. There are good arguments towards disallowing nsup generated deletes in any strongly consistent namespace and, certainly, eviction should be avoided. In recent Aerospike server versions 4.9 and up, nsup is disabled by default. For older versions, to shut down nsup the nsup-period should be set to 0. This means nsup will not run at all. The command to do this across all nodes is:
Admin>asinfo -v "set-config:context=namespace;id=nsStrongConsistency;nsup-period=0"
In the example above the namespace concerned is nsStrongConsistency.
If expiration is part of the use case it may be better to let nsup take care of expiration on each respective cluster as opposed to shipping non-durable deletes via XDR.
However, if there is, for some reasons, the need to ship nsup deletes via XDR, there is a parameter to control whether or not to ship those and it is ship-nsup-deletes. When this is set to false any deletes generated by nsup will not be shipped. An example is shown below:
Admin> asinfo -v "set-config:context=xdr;dc= USWEST;namespace=nsStrongConsistency;ship-nsup-deletes=false"
In the example above the DC being used is USWEST as in the original error message.
The default setting for ship-nsup-deletes is false and so unless this has been modified explicitly, XDR will not ship deletes generated by nsup.
If the particular use case requires both nsup to be on and nsup generated deletes to be shipped via XDR then there is a parameter that will allow this. The parameter strong-constency-allow-expunge will permit non-durable deletes on the destination namespace when set to true. This is set as described below:
Admin> asinfo -v "set-config:context=namespace;id=nsStrongConsistency_remote;strong-consistency-allow-expunge=true"
The major caveat here is that the destination namespace will now accept any expunge whether this comes via XDR or not. There is a further parameter that will stop the source cluster from shipping client non durable deletes when set to true, ignore-expunges, an example is listed below:
Admin> asinfo -v "expunges-config:context=xdr;dc= USWEST;namespace=nsStrongConsistency;ignore-expunges=true"
Durable deletes are always shipped.
Notes
- When running
asinfocommands it is often beneficial to run these viaasadmas this executes the command on all nodes concurrently.
Keywords
XDR NSUP ABANDON 22 EXPUNGE DELETE SHIP
Timestamp
July 2020