Understanding Consistency Level Overrides


#1

Synopsis

There are two server side parameters available for tuning consistency in Aerospike.

  1. read-consistency-level-override
  2. write-commit-level-override

You may also choose the consistency levels on a per-operation basis from the clients, but the server side configurations may override the clients’ requests. The default value of both parameters is “off”, which means that the server uses the levels set by the client.

Read Consistency

If read-consistency-level-override is off and it is not set by the client, the default behavior for the server is one. The client reads the record from a single server instance.

When read-consistency-level-override is one, regardless of the client’s consistency level, the server only reads from one copy. If read-consistency-level-override is set to one, the clients may read stale data during a rolling restart or when write-commit-level-override is set to master.

Note: Aerospike may choose to read from all if the record is not found locally and the server determines there are multiple versions of the partition in the cluster. For server version 3.10.1 and above, not-found on reads do not get duplicate resolved during migrations unless if explicitly enforced (through read consistency setting on client policy or server side override).

When read-consistency-level-override is all, regardless of the client’s consistency level, the server reads all out of sync copies of the data. This setting ensures that the server does not read stale data from a node which has been out of the cluster for some time.

Note: If the partitions in the cluster are all in sync and no migrations are in progress, the server only reads from a single node. Even if "read consistency level" is set to all, the server can still read stale data if the “write commit level” is set to master.

Write Consistency

If write-commit-level-override is off and it is not specified by the client, the default behavior for the server is all. It writes to all nodes containing the partitions that hold the record.

When write-commit-level-override is all, regardless of the client’s commit level, the server commits to all nodes that have a copy of the partition that holds the record.

When write-commit-level-override is set to master, regardless of the client’s commit level, the server only commits to the master partition. If it updates and then reads the same record, the client could read a previous version of the record, even if the "read consistency level is all.

More information may be found here: http://www.aerospike.com/docs/architecture/consistency.html


Partial Data Unavailable Issue While Adding Node to Cluster
Aerospike Replica.RANDOM giving inconsistent result while migrations are in progress
Write consistencies at server and client level
Stale Data Comes Up on Node restart temporarily
Improving Read and Write Performance During Migrations
Improving Read and Write Performance During Migrations
Implications of partitions on read.consistency_level=all and write.commit_level=all and AP (re CAP)
#2

How could this be possible

could someone please elaborate

Thanks


#3

Writes in Aerospike are synchronous. The client always first writes to the node holding the master partition for the record’s key (and those are consistently hashed). If you have replication factor > 1, the master will then coordinate the replica writes to the nodes holding the replica partitions.

So, if you change the read replica policy from ‘master’ to ‘any’, you may get some of your reads going to a replica node, and reading a value that has already been modified in the master, but not yet written to this replica.

It’s your decision, based on your understanding of your business logic, whether you’re ok with reading a (potentially) slightly behind version of the record from the replica. If you keep it to the default policy setting, and always read from the master, this situation cannot happen. The benefit of reading from any of the replicas is higher read throughput.