Please be aware of two issues that may impact your Aerospike deployment:
- HPE NVMe critical firmware update
If you use HPE NVMe SSD devices, make sure to review the following bulletin:
- Issue AER-6391 in Aerospike Database versions 184.108.40.206 - 220.127.116.11
The above mentioned versions are impacted by the following issue:
[AER-6391] - (XDR) When restarting storage-backed ‘data-in-memory’ namespaces after XDR writes have been accepted on the node, records will be improperly deleted.
This issue only manifests itself when an Aerospike cluster running such a version receives write transactions from another cluster via XDR. Here are the 2 potential symptoms:
If the cluster is both an XDR source and an XDR destination (meaning it not only receives XDR transactions but is also configured to ship via XDR to another cluster), upgrading from impacted versions 4.9 to 5.0 or above without a cold restart could lead to the following outcome:
- Any records previously written by XDR may be erroneously shipped even without explicitly enabling the XDR forward configuration parameter. For this to happen, one or more nodes must have been removed from the cluster before the rewind feature is used to reprocess records.
When running with exactly the following configuration:
- data-in-memory true (storage-engine device)
- rack-aware (rack-id differs between nodes)
- prefer-uniform-balance true (default)
- strong-consistency false (default – AP mode)
Restarting the Aerospike Database process without a cold start in multiple phases, some individually (or a rack at a time) and some where all replicas of a given partition are down, can lead to data loss. The following WARNING indicates that this issue might occur:
WARNING (drv_ssd): (drv_ssd_ee.c:630) metadata mismatch - removing <digest>
Aerospike always recommends upgrading to the latest hotfix version of a lineage. To be notified of any release (major, hotfix, database & clients) subscribe to the Announcements / Releases topic (using the notification bell at the top of the screen).