We have a 3 node cluster and added 2 new nodes in order to upgrade some older ones.
Everything seems fine from our application, but we are getting a large number of these warning messages in one of the node logs:
Jun 09 2015 16:19:27 GMT: WARNING (partition): (partition.c::1303) Not expected - Partition is qnode and partition 858 is in 5 state
Jun 09 2015 16:19:27 GMT: WARNING (partition): (partition.c::1303) Not expected - Partition is qnode and partition 861 is in 5 state
Jun 09 2015 16:19:27 GMT: WARNING (partition): (partition.c::1303) Not expected - Partition is qnode and partition 863 is in 5 state
They are occurring 500 times/second, so they are rapidly consuming disk. There are some migrations going on, but we’ve seen more migrations before without this issue.
I googled the error, but the one hit was the source code, which isn’t immediately useful.
Is this dangerous? what is a qnode? What is state 5?
Qnode is the node where query is served for a certain partition and state 5 is absent state. Qnode should either be node where partition is not absent. The situation arises because of interaction between new migration optimization in case of new node addition and qnode logic.
This should not cause any problem your quries shall return all the relevant data.
Thanks raj. It didn’t seem to affect anything. It did go away but not when the migrations stopped. It only went away when I shut down one of the old 3.4.1 nodes.