FAQ - Why do
master_objects initially show 0 when restarting a node?
When a node restarts the number of
master_objects shown on that node is 0 even though the node has returned to the cluster with data.
The Aerospike cluster protocol included in versions 3.13 and higher changed the way cluster nodes behaved with respect to migrations. When a node leaves the cluster, for whatever reason, it means that the node holding replica copies of partitions for which the absent node was master will take on the master role (assuming replication factor 2 or more). Another node within the cluster will become the replica for those partitions. Similarly, partitions for which the absent node held a replica copy will have their owner assigned to another node in the cluster.
When the original node rejoins the cluster, it will own the same partitions it previously had, assuming the rest of the cluster didn’t change (refer to the notes section below for details on this point). Nevertheless, it will not immediately reclaim masterhood for any partition since data may have changed on those partitions while the node was out. Aerospike’s Migration process will send all the latest data for the partitions this node owned (Enterprise Edition leverages the Rapid Rebalance feature to only send the records that did change). As this process progresses, partitions get updated on the node that rejoined and recover their masterhood status (if they were previously owned as master on that node).
To go in a bit more details, the partitions (and the records they own), go through 3 different ownership state:
non_replica_objects: this is the initial state for all the records on the node. It will typically be for a very short time (refer to the notes section below for other example situations). Indeed, as soon as the node rejoins the cluster and the partition ownership is decided across the cluster, all the records belonging to partitions the node will be owning will transition to the
master_objects: this is the ownership state for records belonging to partitions that are owned as master on the node and, when a node joins back a cluster with data, will only happen after migrations complete and the partition has the latest data.
For this reason, the returning node will show 0
master_objects on first rejoining the cluster. For a very brief period it will show
prole_objects until, at the end of migrations, it has a similar amount of
prole_objects as it did before (other than records created or durably deleted while the node was out).
- The answer given above is predicated on the node returning to the cluster in the same state that it left. Here are some examples that would change the partition ownership across nodes in the cluster:
- In cases where the partition ownership changes across nodes in a cluster, objects can retain the
non_replica_objectsownership state throughout the migrations. Those are records that are neither master nor replica on the node but will are kept until the expected replication factor is achieved. Those records or, to be more precise, the partitions in that ownership state are then dropped.
MASTER REPLICA 0 MIGRATION PARTITION