Move Records to another Set

How Developers Are Using Aerospike Use Cases
1

Hi,

we need to move some records from the Set “item” to the set “item-archive” . Periodically.

I’m doing the job with this steps:

  • Read the records from the “item” Set with a Query.
  • Parse the records to the C# object MyItem
  • Write the object MyItem into the “item-archive” Set
  • Delete the record from the “item” Set

“item” and “item-archive” have the same Bins and key but differ for the secondary indexes.

I was wondering if there is a fast way to do this “moving”:

  1. Read and Write the record without converting it to a client language object (C# class)
  2. Given a record simply “change” its Set

Any suggestion to accomplish this task in a different way ?

2

Changing the set of a record will change its digest, hence changing the partition it belongs to, hence changing the nodes that would be owning it… so I don’t think there would be a faster way to do this…

3

So the option 2 (change Set) is not actionable.

I’m doing the job without parsing the Record to a client object (C# class). Just Read (with Query) and Put on the new Set passing the key, it is the same, and the Record.

  • Read Record from “item” Set
  • Put Record in “item-archive” Set
  • Delete the record from “item” Set

For the Put I do this to recreate the Bins from the Record:

private Bin[] CreateBins(Record record) 
    => (from bin in record.bins select new Bin(bin.Key, bin.Value)).ToArray();

Thanks

4

How are you determining what the key is for item-archive? If the key is stored as a bin, I’d imagine you could probably do this with a UDF maybe… how much data are we talking about and is it worth optimizing that far?

5

I loop through all the items ids obtained with a Query and archive the items one by one:

public void ArchiveItem(string id)
{
    var record = ReadRecord(configuration.ItemSet, id);
    WriteRecord(configuration.ItemArchiveSet, record);
    DeleteRecord(configuration.ItemSet, id);
}

private Record ReadRecord(string set, string id) => Client.Get(null, new Key(configuration.Namespace, set, id)); private void WriteRecord(string set, Record record) => Client.Put(null, CreateKey(set, record), CreateBins(record)); private void DeleteRecord(string set, string id) => Client.Delete(null, new Key(configuration.Namespace, set, id));

One table is 2M and the other is 7M (in DEV) but the items to archive are less obviously. This job is executed by a service every X hours. More frequent the job less data to archive.