The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ - Why do master_objects
initially show 0 when restarting a node?
Detail
When a node restarts the number of master_objects
shown on that node is 0 even though the node has returned to the cluster with data.
Answer
The Aerospike cluster protocol included in versions 3.13 and higher changed the way cluster nodes behaved with respect to migrations. When a node leaves the cluster, for whatever reason, it means that the node holding replica copies of partitions for which the absent node was master will take on the master role (assuming replication factor 2 or more). Another node within the cluster will become the replica for those partitions. Similarly, partitions for which the absent node held a replica copy will have their owner assigned to another node in the cluster.
When the original node rejoins the cluster, it will own the same partitions it previously had, assuming the rest of the cluster didn’t change (refer to the notes section below for details on this point). Nevertheless, it will not immediately reclaim masterhood for any partition since data may have changed on those partitions while the node was out. Aerospike’s Migration process will send all the latest data for the partitions this node owned (Enterprise Edition leverages the Rapid Rebalance feature to only send the records that did change). As this process progresses, partitions get updated on the node that rejoined and recover their masterhood status (if they were previously owned as master on that node).
To go in a bit more details, the partitions (and the records they own), go through 3 different ownership state:
-
non_replica_objects
: this is the initial state for all the records on the node. It will typically be for a very short time (refer to the notes section below for other example situations). Indeed, as soon as the node rejoins the cluster and the partition ownership is decided across the cluster, all the records belonging to partitions the node will be owning will transition to theprole_objects
ownership state. -
prole_objects
: this is the ownership state for partitions the node owns. During migrations, some of those will be switching to themaster_objects
ownership state. -
master_objects
: this is the ownership state for records belonging to partitions that are owned as master on the node and, when a node joins back a cluster with data, will only happen after migrations complete and the partition has the latest data.
For this reason, the returning node will show 0 master_objects
on first rejoining the cluster. For a very brief period it will show non_replica_objects
, then prole_objects
until, at the end of migrations, it has a similar amount of master_objects
and prole_objects
as it did before (other than records created or durably deleted while the node was out).
Notes
- The answer given above is predicated on the node returning to the cluster in the same state that it left. Here are some examples that would change the partition ownership across nodes in the cluster:
-
node-id
changes. -
rack-id
assignment changes. -
prefer-uniform-balance
configuration change. - In
strong-consistency
enabled namespaces, roster membership changes.
-
- In cases where the partition ownership changes across nodes in a cluster, objects can retain the
non_replica_objects
ownership state throughout the migrations. Those are records that are neither master nor replica on the node but will are kept until the expected replication factor is achieved. Those records or, to be more precise, the partitions in that ownership state are then dropped.
Keywords
MASTER REPLICA 0 MIGRATION PARTITION
Timestamp
05/29/2019