FAQ - Why do master objects initially show 0 when restarting a node?

FAQ - Why do master_objects initially show 0 when restarting a node?

Detail

When a node restarts the number of master_objects shown on that node is 0 even though the node has returned to the cluster with data.

Answer

The Aerospike cluster protocol included in versions 3.13 and higher changed the way cluster nodes behaved with respect to migrations. When a node leaves the cluster, for whatever reason, it means that the node holding replica copies of partitions for which the absent node was master will take on the master role (assuming replication factor 2 or more). Another node within the cluster will become the replica for those partitions. Similarly, partitions for which the absent node held a replica copy will have their owner assigned to another node in the cluster.

When the original node rejoins the cluster, it will own the same partitions it previously had, assuming the rest of the cluster didn’t change (refer to the notes section below for details on this point). Nevertheless, it will not immediately reclaim masterhood for any partition since data may have changed on those partitions while the node was out. Aerospike’s Migration process will send all the latest data for the partitions this node owned (Enterprise Edition leverages the Rapid Rebalance feature to only send the records that did change). As this process progresses, partitions get updated on the node that rejoined and recover their masterhood status (if they were previously owned as master on that node).

To go in a bit more details, the partitions (and the records they own), go through 3 different ownership state:

  • non_replica_objects: this is the initial state for all the records on the node. It will typically be for a very short time (refer to the notes section below for other example situations). Indeed, as soon as the node rejoins the cluster and the partition ownership is decided across the cluster, all the records belonging to partitions the node will be owning will transition to the prole_objects ownership state.

  • prole_objects: this is the ownership state for partitions the node owns. During migrations, some of those will be switching to the master_objects ownership state.

  • master_objects: this is the ownership state for records belonging to partitions that are owned as master on the node and, when a node joins back a cluster with data, will only happen after migrations complete and the partition has the latest data.

For this reason, the returning node will show 0 master_objects on first rejoining the cluster. For a very brief period it will show non_replica_objects, then prole_objects until, at the end of migrations, it has a similar amount of master_objects and prole_objects as it did before (other than records created or durably deleted while the node was out).

Notes

  • The answer given above is predicated on the node returning to the cluster in the same state that it left. Here are some examples that would change the partition ownership across nodes in the cluster:
  • In cases where the partition ownership changes across nodes in a cluster, objects can retain the non_replica_objects ownership state throughout the migrations. Those are records that are neither master nor replica on the node but will are kept until the expected replication factor is achieved. Those records or, to be more precise, the partitions in that ownership state are then dropped.

Keywords

MASTER REPLICA 0 MIGRATION PARTITION

Timestamp

05/29/2019