Partition map logic & partitioning algorithm

How Developers Are Using Aerospike How Aerospike Works
1

Hey!!

A Key is hashed to map to partitions that is random. But how partitions are mapped to nodes?

RIPEMD-160 is used to Map keys with partitions but there is nothing mentioned about how the partition map is created. How it is decided that which is the replica and which one is the master node for a partition.

Moreover how come do you decided the number 4096 y not 1024 or anything else?

2

Hi nOOb,

partition map logic is mentioned at http://www.aerospike.com/docs/architecture/data-distribution.html . Every node as well as Aerospike client maintains latest copy of the partition map for the cluster and can look up which partition belong to which nodes (master as well as replica).

We love binary numbers, hence 4096 is 2^12 which is big enough to handle data distribution across large cluster and also small enough to keep partition map in limited size footprint.

-samir

3

Hi Samir

I went through the docs. Here it was mentioned that each partition is mapped to a master and a replica node. I didn’t understand how RIPEMD is used to map key with partition not partitions with nodes!

4

The actual partitioning algorithm is generally considered to be below the abstraction layer, but it is documented very well in the source code. Also a while back I wanted to understand the partitioning algorithm better so I wrote a python script to approximate how the partitioning works.

2 Likes
5

I wanted to understand the below scenario:

Suppose, I have 4 nodes in the cluster(A,B,C,D). Each node will be the master of 1/4 of the data. Now, if 1 node goes down(A), the partition map will be redistributed among the 3 available nodes(B,C,D). What happens when node A comes back up? Will it be the master of the same partitions/it’s not necessary?

Basically, I don’t understand how the partitions are divided among the nodes.

6

If it comes back with the same node id then it will become master for the partitions it previously owned as migrations complete.

Btw, the link I previously provided has become stale, here is the new link:

7

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.