Improving Read and Write Performance During Migrations


#1

There will be increased write load during migrations, which will result in increased read/write latency. This increased latency is the result of the server running “on-demand” duplicate resolution for maintaining consistency. If latency is critical and consistency requirements can be relaxed, it is possible to disable “on-demand” duplicate resolution and get enhanced read/write performance.

To disable duplicate resolution, run the following command on all the nodes:

asinfo -v "set-config:context=service;write-duplicate-resolution-disable=true" -h [host ip]

#2

Hello, guys!

Could you put more detail in the feature explanation please? And you said about reading slightly outdated records, but how can I measure this time lag?

Regards


#3

As of Aerospike 3.7.3. (AER-4542) write duplicate resolution’s impact to performance has been significantly improved.

There isn’t a way for the server to know how old its copy is without reading from other servers that may have a copy, which is what this feature disables. Disabling can result in new writes being overwritten by incoming migrations, or new writes overwritting a higher generation copy on a replica write.

Have any reservations is a strobg sign that the cost of disabling this feature is too high.


#4

Thank you for the quick reply, I got it. So a client can receive an inconsistent response until the migration has completed (with write-duplicate-resolution-disable=1).


#5

No, that is different. During migrations clients will be able to read stale data regardless of this setting. This setting prevents you from performing an update based on that stale data while a cluster is recovering.

To prevent reading stale data when there are multiple versions of the same partition see read-consistency-level-override. Also discussed here: Understanding Consistency Level Overrides.


#6

I’m sorry for the late reply.

So the algorithm looks like that: if write-duplicate-resolution-disable=0 and a migration is in progress then during any update a cluster tries to collect all the records (with the given key) from all the nodes and updates the most recent one (or all of them, it depends on write-consistency-level). Please correct me.

I used to think that if write-consistency-level=all or read-consistency-level=all (the both of them are not required) then a client can’t read a stale data in any case. Is it correct?


#7

I believe these questions are answered here: Understanding Consistency Level Overrides.