Understanding Consistency Level Overrides

Knowledge Base Deploying
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Synopsis

There are two server side parameters available for tuning consistency in Aerospike.

  1. read-consistency-level-override
  2. write-commit-level-override

You may also choose the consistency levels on a per-operation basis from the clients, but the server side configurations may override the clients’ requests. The default value of both parameters is “off”, which means that the server uses the levels set by the client.

Read Consistency

If read-consistency-level-override is off and it is not set by the client, the default behavior for the server is one. The client reads the record from a single server instance.

When read-consistency-level-override is one, regardless of the client’s consistency level, the server only reads from one copy. If read-consistency-level-override is set to one, the clients may read stale data during a rolling restart or when write-commit-level-override is set to master.

Note: Aerospike may choose to read from all if the record is not found locally and the server determines there are multiple versions of the partition in the cluster. For server version 3.10.1 and above, not-found on reads do not get duplicate resolved during migrations unless if explicitly enforced (through read consistency setting on client policy or server side override).

When read-consistency-level-override is all, regardless of the client’s consistency level, the server reads all out of sync copies of the data. This setting ensures that the server does not read stale data from a node which has been out of the cluster for some time.

Note: If the partitions in the cluster are all in sync and no migrations are in progress, the server only reads from a single node. Even if “read consistency level” is set to all, the server can still read stale data if the “write commit level” is set to master.

Write Consistency

If write-commit-level-override is off and it is not specified by the client, the default behavior for the server is all. It writes to all nodes containing the partitions that hold the record.

When write-commit-level-override is all, regardless of the client’s commit level, the server commits to all nodes that have a copy of the partition that holds the record.

When write-commit-level-override is set to master, regardless of the client’s commit level, the server only commits to the master partition. If it updates and then reads the same record, the client could read a previous version of the record, even if the "read consistency level is all.

More information may be found here: http://www.aerospike.com/docs/architecture/consistency.html

2

How could this be possible

could someone please elaborate

Thanks

1 Like
3

Writes in Aerospike are synchronous. The client always first writes to the node holding the master partition for the record’s key (and those are consistently hashed). If you have replication factor > 1, the master will then coordinate the replica writes to the nodes holding the replica partitions.

So, if you change the read replica policy from ‘master’ to ‘any’, you may get some of your reads going to a replica node, and reading a value that has already been modified in the master, but not yet written to this replica.

It’s your decision, based on your understanding of your business logic, whether you’re ok with reading a (potentially) slightly behind version of the record from the replica. If you keep it to the default policy setting, and always read from the master, this situation cannot happen. The benefit of reading from any of the replicas is higher read throughput.

1 Like
4

I have the same question as AJVee and I don’t think the point got addressed.

Even if “read consistency level” is set to all, the server can still read stale data if the “write commit level” is set to master.

The original question is about read consistency level set to all Your answer is about read replica policy set to any

  1. It is not clear if read consistency level and read replica policy refer to the same thing
  2. It is very odd that even with read consistency level set to all, the server could still read from a single node. Isn’t the server totally defying the command?
5

@priyank you are making good points. The naming of those configurations predates the introduction of the strong-consistency configuration which makes them very confusing.

For strongly consistent namespaces (configured with the strong-consistency config set to true), the read-consistency-level-override and write-commit-level-override are not available/relevant and, with default client policies, you would always read from the node owning the master copy and, if you have write transactions that are ‘in flight’, would have to wait for them.

So, if you really need strong consistency for your namespace, you just need to use that feature and not worry about any of those other configs that are the AP mode (available under network partition).

Now if you run without the strong-consistency configured, I guess you can consider the config parameters referred to in this article as ‘best effort’ and they definitely should have been named differently :slight_smile:

Hope this helps.