Stale Data Comes Up on Node restart temporarily


#1

Scenario:

  1. In a 3 node cluster with replication factor 2, a node was stopped.
  2. Some updates were pushed to all the keys in a given set
  3. Down node was started again with cold start empty false.
  4. When down node was up, i connected to aerospike via aql and checked the keys of that set, and saw stale data for 1/3rd of keys.
  5. After some time when I queried, the data was fine which means eventually it was showing consistent data

Is this behaviour as expected? Can the cluster serve stale data for some period of time when a node is restarted? How is this thing handled?


#2

Which version of Aerospike are you using, and on which OS?


#3

We are using 3.9.1 version of aerospike OS is Linux (Amazon Linux AMI https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/ )


#7

You may want to read about read consistency levels to understand how this works and adjust the policy on your client, or server, based on your needs:

Here are the details:

http://www.aerospike.com/docs/architecture/consistency.html

And this is one article that describes how this can be overwritten at the server level: