# Replica and master objects is inconsistent **URL:** \ **Category:** Migration\ **Created:** [June 7, 2017, 10:20am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243 "2017-06-07T10:20:44Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![RockyTu](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rockytu/32/785_2.png) [@RockyTu](https://discuss.aerospike.com/u/RockyTu)\ **Post date:** [June 7, 2017, 10:20am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/1 "2017-06-07T10:20:44Z")
The environment is Aerospike CE 3.9.1. the replica factor is set to 2. When the migration completes, I still find the replica and master object number is not equal. During the migration and after the migration, still have some other client write data into Aerospike. I also find new arrived data seems commit to replica object, and the diff number between replica and master is the same. I didn’t find any relevant error message in log file. So, is there any unexpected behavior ? And how can I do next ? ![](https://us1.discourse-cdn.com/flex019/uploads/aerospike/original/1X/eae96a90fdc33f1982dcedcf4b91b4faa43d3318.png) ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [June 7, 2017, 9:13pm UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/2 "2017-06-07T21:13:16Z")
AMC can show inaccurate stats sometimes. Check through asadm. You can also try restarting amc after migrations are done to see if it looks different. I have a cron job that restarts AMC every day ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [June 7, 2017, 9:25pm UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/3 "2017-06-07T21:25:00Z")
What is your AMC version? - It shows right at the bottom of the browser pane - footer. Also, on your node, what does the query below show? ``` $grep objects /var/log/aerospike/aerospike.log ``` You should see something like this: ``` INFO (info): (ticker.c:348) {ns1} objects: all 300000 master 300000 prole 0 ``` (My output is with a one node cluster - so prole is zero.) Do you see difference with AMC or is it same? ---
**Author:** ![RockyTu](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rockytu/32/785_2.png) [@RockyTu](https://discuss.aerospike.com/u/RockyTu)\ **Post date:** [June 8, 2017, 3:38am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/4 "2017-06-08T03:38:01Z")
in asadm is the same. ---
**Author:** ![RockyTu](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rockytu/32/785_2.png) [@RockyTu](https://discuss.aerospike.com/u/RockyTu)\ **Post date:** [June 8, 2017, 3:44am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/5 "2017-06-08T03:44:37Z")
I have 6 nodes cluster, replica is 2. one node: Jun 08 2017 03:24:06 GMT: INFO (info): (ticker.c:328) {production} objects: all 7693601 master 3919389 prole 3774212 another node: Jun 08 2017 03:41:55 GMT: INFO (info): (ticker.c:328) {production} objects: all 8406338 master 6005579 prole 2399283 in AMC, the number is the same, at least, the diff is around 40M records. ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [June 8, 2017, 4:09am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/6 "2017-06-08T04:09:33Z")
AMC version? ---
**Author:** ![RockyTu](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rockytu/32/785_2.png) [@RockyTu](https://discuss.aerospike.com/u/RockyTu)\ **Post date:** [June 8, 2017, 4:13am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/7 "2017-06-08T04:13:24Z")
AMC CE 4.0.12, the latest one ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [June 8, 2017, 4:18am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/8 "2017-06-08T04:18:22Z")
i would expect all = master + replica on each node. sum (all) = sum (master) + sum (replica) … sum over all nodes sum(master) = sum (replica) … sum over all nodes ---
**Author:** ![pgupta](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/pgupta/32/2351_2.png) [@pgupta](https://discuss.aerospike.com/u/pgupta)\ **Post date:** [June 8, 2017, 4:18am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/9 "2017-06-08T04:18:57Z")
since your numbers from logs and amc match, amc is not the issue then. ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [June 9, 2017, 10:48pm UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/10 "2017-06-09T22:48:02Z")
This is after migrations are finished? Anything else showing in your log? How about asadm -e “show stats like err”? ---
**Author:** ![RockyTu](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/rockytu/32/785_2.png) [@RockyTu](https://discuss.aerospike.com/u/RockyTu)\ **Post date:** [June 10, 2017, 2:57am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/11 "2017-06-10T02:57:21Z")
Yes, the migration is finished, it is strange, no relative error, and then I rolling restart all nodes, the issue disappeared, I suspect 3.9.1 is not a stable version, because the new heartbeat sub system released and some issues fixed on next version. ---
**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\ **Post date:** [June 10, 2017, 8:24pm UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/12 "2017-06-10T20:24:26Z")
I don’t see anything in release notes which addresses anything like that. I’m not really sure whats going on. At this point it may be worth opening a case with Aerospike if you have a support contract ---
**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\ **Post date:** [June 13, 2017, 1:47am UTC](https://discuss.aerospike.com/t/replica-and-master-objects-is-inconsistent/4243/13 "2017-06-13T01:47:01Z")
We did discover a way this can happen when recovering from certain split-brain conditions while working on the partition rebalance algorithm used with paxos-protocol v5. Basically it was possible for a partition recovering from a split-brain to determine there were no migrations needed. We were unable to address this issue in the old algorithm and it shouldn’t exist in the new rebalance algorithm. To switch to paxos-protocol v5, see [http://www.aerospike.com/docs/operations/upgrade/cluster\_to\_3\_13](http://www.aerospike.com/docs/operations/upgrade/cluster_to_3_13).