What is WARNING "Not expected - Partition is qnode and partition 237 is in 5 state"? (AER-3787)


#1

We have a 3 node cluster and added 2 new nodes in order to upgrade some older ones.

Everything seems fine from our application, but we are getting a large number of these warning messages in one of the node logs:

Jun 09 2015 16:19:27 GMT: WARNING (partition): (partition.c::1303) Not expected - Partition is qnode and partition 858 is in 5 state
Jun 09 2015 16:19:27 GMT: WARNING (partition): (partition.c::1303) Not expected - Partition is qnode and partition 861 is in 5 state 
Jun 09 2015 16:19:27 GMT: WARNING (partition): (partition.c::1303) Not expected - Partition is qnode and partition 863 is in 5 state

They are occurring 500 times/second, so they are rapidly consuming disk. There are some migrations going on, but we’ve seen more migrations before without this issue.

I googled the error, but the one hit was the source code, which isn’t immediately useful.

Is this dangerous? what is a qnode? What is state 5?

Thanks.


#2

Sorry, I should’ve added that we are upgrading nodes from 3.4.1 to 3.5.9 and this error seems to occur on the new 3.5.9 nodes.


#3

Hubt,

Issue identified. Will be fixed.

Qnode is the node where query is served for a certain partition and state 5 is absent state. Qnode should either be node where partition is not absent. The situation arises because of interaction between new migration optimization in case of new node addition and qnode logic.

This should not cause any problem your quries shall return all the relevant data.

This should go away once migrations finishes…

– R


#4

Thanks raj. It didn’t seem to affect anything. It did go away but not when the migrations stopped. It only went away when I shut down one of the old 3.4.1 nodes.

Thanks for the quick response.


#5

@hubt,

A JIRA has been filed to follow up on this; it’s AER-3787, just for reference. Please stay tuned for updates on our progress.

Regards,

Maud