Moving data from Redis to Aerospike: How to generate DIGEST string as in "Backup File Format"

How Developers Are Using Aerospike Aerospike and other Databases
1

I want to change Redis to Aerospike, and move data from Redis to Aerospike.

The easiest way is translate Redis backup file(RDB) to Aerospace backup file, and then use asrestore tool to write file into Aerospike cluster.

The digest string block me. “Backup File Format : http://www.aerospike.com/docs/tools/backup/file_format.html”

For example, for Namespace “test”, Set “demo”, Key “key1”

[step 1] use asbackup tool generate a backup file (.asb), and see the digest string is “7JEZLUt/jONdXXjTS8ply6qqyWA=”

[step 2] use Aerospike C api to generate original digest string (20 characters).

[step 3] encode original digest string to base64 digest string, it is “7JELUtjONXXjS8py6qyWA=”

[step 4] the digest strings are similar but not same.

Please help

2

Hi,

Thanks for reaching out. Aerospike actually has a CSV loader to import such data.

If you can backup your Redis data as a CSV file, then you should be able to do such migration.

Hope this helps, –meher

3

Thank you very much. I will try it

4

Hi Liu_Ld, I was trying to reproduce the digest issue using c client. I am able to get same digest both in asbackup and encoding original digest. So if I am not missing, As there is no direct API to calculate original digest, Can you point me out which API you have used for calculating original digest?

5

Use function ’ as_digest *as_key_digest(as_key *key)’ defined in as_key.h

6

Following piece of code I have used. I am getting same digest as in backup file. Please let me know if your digest creation and encoding is different from this.

char namespace[MAX_NAMESPACE_SIZE] = "test";
char set[MAX_SET_SIZE] = "demo";
char key_str[100] = "key1";

as_key key;
as_key_init_str(&key, namespace, set, key_str);

as_digest * digest = as_key_digest(&key);

printf("Key:%s, Set:%s, Digest Length %lu\n", key_str, set, sizeof(digest->value));

uint32_t digest_len = cf_b64_encoded_len(sizeof(digest->value));
printf("Encoded_digest_len %d\n", digest_len);
char* digest64 = alloca(digest_len + 1); 

cf_b64_encode(digest->value, sizeof(digest->value), digest64);
digest64[digest_len] = 0;
printf("Encoded digest:%s\n", digest64);

Output:

Key:key1, Set:demo, Digest Length 20
Encoded_digest_len 28
Encoded digest:7JEZLUt/jONdXXjTS8ply6qqyWA=
7

Done with including citrusleaf/cf_b64.h .

Thanks :relaxed:

I suggest you add the code in http://www.aerospike.com/docs/tools/backup/file_format.html.

8

I hope cf_b64.h helps you in proper encoding. That includes standard base64 encoding logic. Thanks for suggestion. Usually we recommend to use CSV loader for moving data from other nosql databases into aerospike.

9

The difference is file format.

Backup file format can be used in the whole lifetime :

  1. transfer to Aerospike,

  2. using Aerospike,

    for example, change bin-name from “bad-bin-name” to “good-bin-name” for all related keys

  3. transfer Aerospike to another db.

CSV loader format can only be used in 1st situation.

I think making efforts in backup file will get more values.

And I am pullzed why you supply two tools to do the same thing.

10

CSV formatted data dumps are more common. CSV loader is generic and can be useful both in 1st and 2nd situations you have mentioned. Aerospike data dump is different then CSV and only used by aerospike db. We may merge these tools in future.

11

I am so glad to hear this. :blush:

12

Can i parse raw key name from diggest string?

13

Hi Liu, You can’t get raw key from digest string. Let me know your exact use case so that I can suggest you alternate solution.

14

Hello, J, i want to list all keys(the key is user id) of a set.

15

Aerospike uses key and set name to generate unique digest, So it stores only digest. If your application is doing some operation after getting the list of keys of a set then you can do the same operation using digests.

But if you really want to use keys later then you can keep the key as metadata part of record using following way: While inserting one record if you set writePolicy.sendKey = true then key will be stored as metadata part of a record.

16

How to list the metadata of all records in a set?

17

You can use Scan API to get all records of a set. To get only metadata specify nobins=true in scan policy. Please find below code snippet for reference:

public void scanExample(AerospikeClient client, Parameters params) throws Exception {
    ScanPolicy policy = new ScanPolicy();
    policy.includeBinData = false;

    client.scanAll(policy, "test", "demoset", this); 
	
    System.out.println("key list:" + keyList.toString());
	
    System.out.println("digest list:" + digestList.toString());
	
}

public void scanCallback(Key key, Record record) {

        digestList.add(key.digest);
	
	//if user key stored
	keyList.add(key.userKey);
	
}
18

Thanks, J. This solution (“writePolicy.sendKey = true” + “scan only metadata”) is useful.

1 Like