Problems with Secondary Indexes and Garbage Collection

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

There is an issue in memory usage for secondary indexes with namespaces that use a storage device.

If, between garbage collection passes, a record is deleted and reinserted with a different bin value (for a bin part of a secondary index definition) before the garbage collection has processed the original bin value, the secondary digest for the first record remains in memory. It is not cleaned by garbage collection. This does not impact the records returned by a secondary index query as they are checked for before being sent back to the client. Digests for secondary indexes that ‘evade’ garbage collection are only cleaned up when secondary indexes are rebuilt, either from manually rebuilding them or restarting the node. This does not impact data in memory namespaces (as the bin values are checked upon deletion and the secondary index is updated).

Let’s walk through an example. Let’s assume we have a bin named Country and we have defined a secondary index for Country:

  PI   Country   Name  
 ---- --------- ------ 
   1   Sweden    AAA   
   2   Norway    BBB   
   3   India     CCC   
   4   USA       DDD   
   5   Sweden    EEE   
   6   Norway    FFF   
   7   USA       GGG   
   8   India     HHH   
   9   India     III   
  10   Sweden    JJJ   
  11   Norway    LLL   
  12   India     MMM

The secondary index consists of lists of digests that point to the primary indexes:

  Index:           Digests:          
 --------- ------------------------- 
  Sweden:   D(1), D(5), D(10)        
  Norway:   D(2), D(6), D(11)        
  USA:      D(4), D(7)               
  India:    D(3), D(8), D(9), D(12)

The following steps illustrate the bug:

  1. Garbage collection cleans the secondary indexes.
  2. You insert the value ‘USA’ into the Country bin. The digest for the value is updated in the secondary index for Country.
  3. You delete the value ‘USA’ from the Country bin.
  4. You insert the value ‘India’ into the Country bin. The digest for the value is updated in the secondary index for Country.

The new value for PI 4 is ‘India’:

  P    Country   Name  
 ---- --------- ------ 
   1   Sweden    AAA   
   2   Norway    BBB   
   3   India     CCC   
   4   India     DDD   
   5   Sweden    EEE   
   6   Norway    FFF   
   7   USA       GGG   
   8   India     HHH   
   9   India     III   
   10  Sweden    JJJ   
   11  Norway    LLL   
   12  India     MMM   

The record on disk is properly deleted and recreated, but the secondary index for Country is not:

  Index:              Digests:             
---------- ------------------------------- 
 Sweden:    D(1), D(5), D(10)              
 Norway:    D(2), D(6), D(11)              
 USA:       D(4), D(7)                     
 India:     D(3), D(8), D(9), D(12), D(4)
  1. Garbage collection cleans the secondary indexes. It leaves the digest for ‘USA’. It stays in memory until the secondary index is rebuilt.

When a query comes to the secondary index for Country on this node, it reads the digests for the sindex, checks the locations in the primary index, and returns data. The query checks the primary index for D(4), and it returns the correct value, ‘India’, and the query returns correct results.

The digest for USA still points to PI 4, and is never cleaned up properly.

The memory clean up works differently when data-in-memory is true and when it is not for different delete operations, client deletes, nsup deletes (expiration & eviction) and truncate.

Data-in-memory true

  • For client deletes, we do not rely on gc as we update on the fly.
  • For nsup deletes, we rely on gc and we do not reclaim the memory unless the gc is run.
  • For truncate we don’t rely on gc, we reclaim the memory on the fly as part of the truncate (may be more efficient to first destroy the index, if possible).

Data-in-memory false

  • For client deletes, we do not read the record from the disk, this is for performance, hence we rely on garbage collection.
  • For nsup deletes, we rely on gc and we do not reclaim the memory unless the gc is run.
  • For truncate we don’t rely on gc, we reclaim the memory on the fly as part of the truncate (may be more efficient to first destroy the index, if possible).

For updates on a bin which is indexed, irrespective of whether the data is on disk or in-memory we have to read the record, so we know the value of the bin and we update it correctly, hence we do not rely on garbage collection for updates.