Issue impacting some 4.9 Aerospike Database versions + HPE Firmware Bulletin

Announcements Support Bulletins
1

Please be aware of two issues that may impact your Aerospike deployment:

  1. HPE NVMe critical firmware update

If you use HPE NVMe SSD devices, make sure to review the following bulletin:

Bulletin: (Revision) HPE NVMe Solid State Drives - Critical Firmware Upgrade Required for Certain HPE NVMe Solid State Drive Models to Prevent Drive Failure After 4700 Hours of Operation

  1. Issue AER-6391 in Aerospike Database versions 4.9.0.3 - 4.9.0.25

The above mentioned versions are impacted by the following issue:

[AER-6391] - (XDR) When restarting storage-backed ‘data-in-memory’ namespaces after XDR writes have been accepted on the node, records will be improperly deleted.

This issue only manifests itself when an Aerospike cluster running such a version receives write transactions from another cluster via XDR. Here are the 2 potential symptoms:

  1. If the cluster is both an XDR source and an XDR destination (meaning it not only receives XDR transactions but is also configured to ship via XDR to another cluster), upgrading from impacted versions 4.9 to 5.0 or above without a cold restart could lead to the following outcome:

    • Any records previously written by XDR may be erroneously shipped even without explicitly enabling the XDR forward configuration parameter. For this to happen, one or more nodes must have been removed from the cluster before the rewind feature is used to reprocess records.

  2. When running with exactly the following configuration:

    Restarting the Aerospike Database process without a cold start in multiple phases, some individually (or a rack at a time) and some where all replicas of a given partition are down, can lead to data loss. The following WARNING indicates that this issue might occur:

WARNING (drv_ssd): (drv_ssd_ee.c:630) metadata mismatch - removing <digest>

Both those symptoms are addressed by upgrading to version 4.9.0.26 or above and cold restarting when doing so.

Aerospike always recommends upgrading to the latest hotfix version of a lineage. To be notified of any release (major, hotfix, database & clients) subscribe to the Announcements / Releases topic (using the notification bell at the top of the screen).