I am performing few experiments with secondary index and node migration. I have about 800k records in a 3 node cluster with replication factor of 2. There is a integer bin having secondary index. I am running on Aerospike CE 220.127.116.11. Following are my observations -
When node leaves the cluster (cluster size becomes 2) - Expected number of records are returned while executing the query on secondary index. Though from admin console I can see that data migration is ongoing while I am executing the query.
When node rejoins the cluster (cluster size becomes 3) - I wait for migration to complete and then again start the 3rd node. Here again from admin console I could see that data migration is in progress. But, when I execute the same query on secondary index, I observe that number of records matching the query criteria has reduced. I also observe that this number of matching record grows over time and finally reaches the expected value.
I found an old discussion which was explaining this over here. It talked about qnode and designated master. But it’s written in this discussion that from version 18.104.22.168 qnodes are deprecated. Can someone explain me the expected behaviour ? And is there anyway to control it? I have some specific questions -
- When node leaves or rejoins, when does master node of record change? Does this change also wait for secondary index tree building to finish?
- As query on secondary index goes to every node, does each node reply with the matching records for which it is designated master?
- How does Aerospike client discover change in the ownership? I have read about “info message”, is that what is used?