We saw data loss when we added two new nodes(same time) in 3-node cluster during live traffic.Our aerospike cluster is using shadow devices. We started these two new nodes with 8hr older backup( EBS snapshot). After migration completion, we observed that for few records there is data loss of last 8…

What version were you upgrading from and to? Which conflict-resolution-policy are you using?

Hi kporter, I was not upgrading Aerospike version. I only added two new box with 8hr old backup, that leads to loss of few new records( Actually, these new records were replaced by older records from backup). I’m using default conflict-resolution-policy that is generation

What is your server version? What generation values did you have? Take a few suspect records, read using aql, set record_print_metadata to true and read the record. Were you on a generation value rollover cusp from 65K to 0? From the conflict resolution policy link: Generation value could wrap …

We are using 3.15.1.3. I went through this link [image] Configuration Reference | Aerospike Documentation This page lists the configuration parameters that can be and it makes sense to change this field to last-update-time. Thanks for the suggestion. In ou…

[image] aanand: As per your suggestion, we may end up in getting record generation number as 1 multiple times. The record generation will continue to advance on update when using ‘last-update-time’.

[image] Data loss when added new box in cluster Operations The record generation will continue to advance on update when using ‘last-update-time’. Thanks @kporter . Then in this case, up to what max limit record generation can increase? By any chance it gets reset to …

It will wrap at 65k. Using conflict resolution policy of last update time will resolve conflicts based on that. Read-modify-write transactions use a generation equal policy which will work fine over the wrap. By the way, I overlooked the fact that the asrestore utility will still resolve based on …

@kporter I got it. ‘–unique’ will work in case of asrestore. But still I didn’t get answer asked here Data loss when added new box in cluster - #5 by aanand As per your suggestion, we may end up in getting record generation number as 1 multiple times. In this case, how should we rely on the java …

I thought this was answering that question: [image] kporter: It will wrap at 65k. Using conflict resolution policy of last update time will resolve conflicts based on that. Read-modify-write transactions use a generation equal policy which will work fine over the wrap. If not, could you try…

Data loss when added new box in cluster

Operations

kporter March 13, 2018, 8:24am 6

The record generation will continue to advance on update when using ‘last-update-time’.

Topic		Replies	Views
Conflict-resolution-policy=ttl Operations	0	1260	March 11, 2016
Why conflict-resolution-policy is on "generation" by default? Configuration	5	2382	March 3, 2016
Strange consistency problem [Resolved] Java Client	2	2952	September 16, 2015
How generation field gets updated when XDR is replicating the record	5	853	November 19, 2019
Aerospike migrations issue/ data loss Migration query	12	1865	July 15, 2019

Data loss when added new box in cluster

Related topics