# Losing records after node fails

**URL:** <https://discuss.aerospike.com/t/losing-records-after-node-fails/1305>\
**Category:** Configuration\
**Created:** [May 21, 2015, 4:34pm UTC](https://discuss.aerospike.com/t/losing-records-after-node-fails/1305 "2015-05-21T16:34:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Slader](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Slader](https://discuss.aerospike.com/u/Slader)\
**Post date:** [May 21, 2015, 4:34pm UTC](https://discuss.aerospike.com/t/losing-records-after-node-fails/1305/1 "2015-05-21T16:34:43Z")

</div>

Hi guys! We are using Aerospike in our company for quite a while now. Currently it is a cluster of 11 nodes and more than 4 000 000 000 records.

Yet we’ve encountered a strange problem: when one node fails (ssd breaks, or hangs up) we loose a portion of records. Last time we’ve lost approximately 20% of them. We evaluate the loss amount using the daily backups by comparing with backups that are made after migration is completed. We are using Aerospike 3.3.21 (replication-factor=2) So, what’s wrong with it?

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [May 22, 2015, 7:35pm UTC](https://discuss.aerospike.com/t/losing-records-after-node-fails/1305/2 "2015-05-22T19:35:22Z")

</div>

I see that you are comparing with backups after migrations complete, but I am not sure to what you are comparing them to. If you are comparing them to backups taken while migrations are in progress, it is expected that the backups will have missing and duplicated data.

Does your data have expiration set? If so did you nodes exceed a high-water threshold and start evicting?

---

<div class="post-metadata">

**Author:** ![Slader](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Slader](https://discuss.aerospike.com/u/Slader)\
**Post date:** [May 23, 2015, 9:01am UTC](https://discuss.aerospike.com/t/losing-records-after-node-fails/1305/3 "2015-05-23T09:01:53Z")

</div>

1. I am comparing them to backups taken the day before the accident (migration was not in progress)
2. Our data have no expiration. Avail pct = 40…45% in cluster

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [May 24, 2015, 2:53am UTC](https://discuss.aerospike.com/t/losing-records-after-node-fails/1305/4 "2015-05-24T02:53:27Z")

</div>

Could you provide your Aerospike.conf file?

Also which OS are you running on?

Which Aerospike features are you using?

- Key/value reads and writes?
- scan?
- query/secondary indexes?
- Complex datatypes such as list and map?
- LDTs (LList)?

Anything you can tell me about the data that was missing?