XDR Shipping to strongly consistent namespace returns abandon result 22

XDR Shipping to strongly consistent namespace returns abandon result 22

Problem Description

When XDR 5.0 has been configured to ship records from one strongly consistent namespace to another the shipping starts to return the following error:

Jul 01 2020 01:57:57 GMT-0400: WARNING (xdr): (dc.c:2356) (repeated:9) {nsStrongConsistency} DC USWEST abandon result 22

Explanation

This result will be shown when XDR has been configured to ship nsup generated delete transactions (expirations, evictions). Strong consistency enabled namespaces will not allow non-durable deletes (expunges) in order to guarantee strong consistency (expunges cannot guarantee consistency during short lived cluster splits or cold restarts for example). As a delete from a remote cluster originating via nsup would not include a tombstone it is, by definition, non-durable. For this reason the default behaviour of a strongly consistent namespace would be to reject the transaction. On the source cluster this would be reflected in the abandon result 22 message.

Solution

Care should be taken when considering how to approach resolution of this error, not least because it serves to protect the database against potentially undesirable behaviour. There are good arguments towards disallowing nsup generated deletes in any strongly consistent namespace and, certainly, eviction should be avoided. In recent Aerospike server versions 4.9 and up, nsup is disabled by default. For older versions, to shut down nsup the nsup-period should be set to 0. This means nsup will not run at all. The command to do this across all nodes is:

Admin>asinfo -v "set-config:context=namespace;id=nsStrongConsistency;nsup-period=0"

In the example above the namespace concerned is nsStrongConsistency.

If expiration is part of the use case it may be better to let nsup take care of expiration on each respective cluster as opposed to shipping non-durable deletes via XDR.

However, if there is, for some reasons, the need to ship nsup deletes via XDR, there is a parameter to control whether or not to ship those and it is ship-nsup-deletes. When this is set to false any deletes generated by nsup will not be shipped. An example is shown below:

Admin> asinfo -v "set-config:context=xdr;dc= USWEST;namespace=nsStrongConsistency;ship-nsup-deletes=false"

In the example above the DC being used is USWEST as in the original error message.

The default setting for ship-nsup-deletes is false and so unless this has been modified explicitly, XDR will not ship deletes generated by nsup.

If the particular use case requires both nsup to be on and nsup generated deletes to be shipped via XDR then there is a parameter that will allow this. The parameter strong-constency-allow-expunge will permit non-durable deletes on the destination namespace when set to true. This is set as described below:

Admin> asinfo -v "set-config:context=namespace;id=nsStrongConsistency_remote;strong-consistency-allow-expunge=true"

The major caveat here is that the destination namespace will now accept any expunge whether this comes via XDR or not. There is a further parameter that will stop the source cluster from shipping client non durable deletes when set to true, ignore-expunges, an example is listed below:

Admin> asinfo -v "expunges-config:context=xdr;dc= USWEST;namespace=nsStrongConsistency;ignore-expunges=true"

Durable deletes are always shipped.

Notes

  • When running asinfo commands it is often beneficial to run these via asadm as this executes the command on all nodes concurrently.

Keywords

XDR NSUP ABANDON 22 EXPUNGE DELETE SHIP

Timestamp

July 2020

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.