Inconsistent reads during a node's start up

Operations
1

Hi guys,

I had an issue with 3.8.2 and a 5-node cluster. The scenario is following:

  • Restart one node
  • Write a key to the cluster while the node’s starting (before the “cake” log message)
  • Try to read (not a batch) the written key and it doesn’t exist
  • “Cake” appears and the key is here

Consistency levels (both) are default.

To me it looks like Inconsistent data after simulated failure Could you tell me if the given scenario is some kind of well-known behavior? What could I do to avoid such case (apart from “read-consistency: all”)? If you require any additional information please ask.

Regards

2

I’m sorry for the misleading information. The problem wasn’t due to non-existing keys, but operation timeouts (1000 ms). So I guess it could be explained by some migration overloads and/or Paxos lags.

3

Hm, have you tried the different keys? The reason I ask is that I am curious if it matters if the key joins a primary or replica partition on the added node. If the issue varies by key, it would suggest the partition’s state plays a role here.