# Consistency issue with Aerospike

**URL:** <https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174>\
**Category:** Query & Indexing\
**Tags:** query\
**Created:** [November 24, 2015, 1:48pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174 "2015-11-24T13:48:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![AlexStanovsky](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@AlexStanovsky](https://discuss.aerospike.com/u/AlexStanovsky)\
**Post date:** [November 24, 2015, 1:48pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/1 "2015-11-24T13:48:05Z")

</div>

I am using in Aerospike (v3.5.12) as in memory key value store of one node.

Also using Java client(v3.1.7) to retrieve and write data.

I noticed that under certain QPS (3K) of reads(Batch) and writes. Some of the data retrieval doesn’t work.

```auto
[INFO] [24/11/2015 14:43:38.782] Write Data
[ERROR] [24/11/2015 14:43:38.937] Read Data - Not Found

```

Did someone encountered with similiar issue?

---

<div class="post-metadata">

**Author:** ![raj](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@raj](https://discuss.aerospike.com/u/raj)\
**Post date:** [December 22, 2015, 12:59pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/2 "2015-12-22T12:59:39Z")

</div>

Alex,

There is no known such issue. Could you check it with new 3.7.0.2 release where lot improvements related sindex code have been introduced.

HTH – Raj

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [December 22, 2015, 7:35pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/3 "2015-12-22T19:35:32Z")

</div>

Could it have been possible the cluster was performing migrations when the reads were not found? Prior to 3.6.x the batch system would fail with notfound during migrations if the record no longer resided on the target node.

---

<div class="post-metadata">

**Author:** ![AlexStanovsky](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@AlexStanovsky](https://discuss.aerospike.com/u/AlexStanovsky)\
**Post date:** [January 10, 2016, 1:02pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/4 "2016-01-10T13:02:41Z")

</div>

I am working on one node, also single get are not working too.

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [January 10, 2016, 8:03pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/5 "2016-01-10T20:03:19Z")

</div>

Did the server ack the write with a successful return code before the read was initiated?

---

<div class="post-metadata">

**Author:** ![AlexStanovsky](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@AlexStanovsky](https://discuss.aerospike.com/u/AlexStanovsky)\
**Post date:** [January 12, 2016, 8:59am UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/6 "2016-01-12T08:59:02Z")

</div>

I am using the async oparations in the following matter:

```
    public void store() {
        WritePolicy expirationWritePolicy = new WritePolicy();
        expirationWritePolicy.sendKey = true;
        expirationWritePolicy.priority = Priority.HIGH;
        expirationWritePolicy.expiration = 10;

        Key key = new Key(namespace, SET_NAME, requestId);
        Bin bin = new Bin(BIN_NAME, serializer.toBinary(budgetCommit));
        Bin extra = new Bin("extra", "data");

        client.put(expirationWritePolicy, new WriteListener() {
            @Override
            public void onSuccess(Key key) {
                logger.info("Succeed to store {}", requestId());
            }

            @Override
            public void onFailure(AerospikeException exception) {
                logger.error(exception, "Fail to store {}", key);
            }
        }, key, extra, bin);
    }

  public void retrieve() {
        WritePolicy defaultWritePolicy = new WritePolicy();
        defaultWritePolicy.priority = Priority.LOW;
        defaultWritePolicy.sendKey = true;

        Key key = new Key(namespace, SET_NAME, requestId);
        Bin closeExtra = new Bin("extra", "_closed");

        client.operate(defaultWritePolicy, new RecordListener() {
                    @Override
                    public void onSuccess(Key key, Record record) {
                         if (record.getValue(BIN_NAME) == null){
                               logger.error("Fail to retrieve {}", requestId);
                         }
                    }

                    @Override
                    public void onFailure(AerospikeException exception) {
                        logger.error("Fail to retrieve {} : {}", requestId, exception.getMessage());
                    }
                }, key,
                Operation.append(closeExtra), Operation.get());
    }

```

[INFO] [12/01/2016 08:37:16.732] Succeed to store 379e67dc-945d-4717-97a7-721cc8093c05 [ERROR] [12/01/2016 08:37:16.736] Fail to retrieve 379e67dc-945d-4717-97a7-721cc8093c05

The onSuccess callback is called when there is an Ack from the Aerospike.

Starting to fail around 8k QPS on master write.

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [January 12, 2016, 8:54pm UTC](https://discuss.aerospike.com/t/consistency-issue-with-aerospike/2174/7 "2016-01-12T20:54:38Z")

</div>

> [@AlexStanovsky](#):
>
> expirationWritePolicy.expiration = 10;

Ah, my suspicion is that you have breached an eviction high water mark, either [memory](http://www.aerospike.com/docs/reference/configuration/#high-water-memory-pct) or [disk](http://www.aerospike.com/docs/reference/configuration/#high-water-disk-pct). We can confirm this by running `asadm -e "info namespace"` and checking if **HWM Mem%** or **HWM Disk%** is above their respective **Used%** values.

I suspect you may have at least two TTL values that differ significantly. This use pattern will cause the lower ttl to be purged when eviction kick in. For this use case we have [set-disable-eviction](http://www.aerospike.com/docs/reference/configuration/#set-disable-eviction) which will exclude a particular set. The configuration reference page shows how to dynamically set this option for a set, and a static configuration example can be found in the [set-data-retention confiuguration documentation](http://www.aerospike.com/docs/operations/configure/namespace/retention/#set-data-retention).
