# Durability issue: lost set after 3-4 days

**URL:** <https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416>\
**Category:** General Discussion\
**Created:** [September 19, 2014, 5:23am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416 "2014-09-19T05:23:44Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [September 19, 2014, 5:23am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/1 "2014-09-19T05:23:44Z")

</div>

I had stored 10000 records in the set(in test namespace). After 3-4 days, That set is lost automcatically, without any delete operation. Secondery index is still there in the index list. Even Sometime it shows different no of objects after server restart.

Running aerospike with virtualbox on windows 8 system. Aerospike version is 3.3.12

---

<div class="post-metadata">

**Author:** ![wchu](https://avatars.discourse-cdn.com/v4/letter/w/a5b964/32.png) [@wchu](https://discuss.aerospike.com/u/wchu)\
**Post date:** [September 19, 2014, 7:21pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/2 "2014-09-19T19:21:21Z")

</div>

This sounds like there is not enough memory/storage allocated for the namespace, and automatic record eviction has kicked in to make room for newer records.

[http://www.aerospike.com/docs/operations/manage/storage/](http://www.aerospike.com/docs/operations/manage/storage/)

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [September 23, 2014, 6:33am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/3 "2014-09-23T06:33:22Z")

</div>

Thanks for the reply. Below is my AMC console. It looks storage is available. Can you verify it. Seems something wrong. How to prevent automatic record evication If size is available.

---

<div class="post-metadata">

**Author:** ![petter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/petter/32/44_2.png) [@petter](https://discuss.aerospike.com/u/petter)\
**Post date:** [September 23, 2014, 10:49am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/4 "2014-09-23T10:49:28Z")

</div>

Could you clarify. Are you saying you installed AS, added records, left the server alone for 3-4 days, came back and stuff was gone and you had done nothing?

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [September 23, 2014, 1:43pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/5 "2014-09-23T13:43:47Z")

</div>

Yes, It is like that. I had inserted 100000 records in set to test performance and performance result was Excellent. I didnt touch those records for few days. Suddenly those records disappered. Even couldnt see set in definition in list . I tried this test 3-4 times. I restart my virtual box daily. db runs on regular hdd (not ssd).

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [September 23, 2014, 11:03pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/6 "2014-09-23T23:03:59Z")

</div>

Would it be possible for you to post your config file. I’m wondering what ttl you are using on your records?

–Lucien

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [September 24, 2014, 4:39am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/7 "2014-09-24T04:39:23Z")

</div>

Yes I can provide config file. Where It would be stored in windows installation.

I am using below code to store to data ClientPolicy clientPolicy = new ClientPolicy(); AerospikeClient client = new AerospikeClient(clientPolicy, new Host(host,port)); WritePolicy policy = new WritePolicy(); //key and bin setup client.put(policy, key, bins);

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [September 27, 2014, 12:34am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/8 "2014-09-27T00:34:44Z")

</div>

I was curious on your server aerospike.conf file which should be under /etc/aerospike/aerospike.conf on the Virualbox image. Also can you confirm your default-ttl settings and storage-engine?

More info on config can be found at:

[http://www.aerospike.com/docs/operations/configure/namespace/](http://www.aerospike.com/docs/operations/configure/namespace/)

Since you are running on regular hd and not ssd. I’m assuming you are using data-in-memory similar to this stanza

```

namespace [namespace-name] {
    memory-size SIZE G # Maximum memory allocation for data and
                                    # primary index
    default-ttl 0 # Writes from client that do not provide a TTL
                               # will default to 0 or never expire
    storage-engine device { # Configure the storage-engine to use
                                    # persistence
    file /opt/aerospike/filename # Location of data file on server
    # file /opt/aerospike/another # (optional) Location of data file on server
    filesize SIZE G # Max size of each file in GB
    data-in-memory true # Indicates that all data should also be
                                    # in memory.
    }
}

```

[http://www.aerospike.com/docs/operations/configure/namespace/storage/](http://www.aerospike.com/docs/operations/configure/namespace/storage/)

If you can post the stanzas from the config file that would help troubleshoot issue.

–Lucien

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [September 29, 2014, 2:08pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/9 "2014-09-29T14:08:13Z")

</div>

Sorry Lucien, I tried to find .conf file but couldnt find it in the system

here is my server startup console log, may be it can be helfpul.

E:\aerospike\database\aerospike-vm\>vagrant up Bringing machine ‘default’ up with ‘virtualbox’ provider… ==\> default: Checking if box ‘aerospike/centos-6.5’ is up to date… ==\> default: Clearing any previously set forwarded ports… ==\> default: Clearing any previously set network interfaces… ==\> default: Preparing network interfaces based on configuration… default: Adapter 1: nat ==\> default: Forwarding ports… default: 3000 =\> 3000 (adapter 1) default: 8081 =\> 8081 (adapter 1) default: 22 =\> 2222 (adapter 1) ==\> default: Running ‘pre-boot’ VM customizations… ==\> default: Booting VM… ==\> default: Waiting for machine to boot. This may take a few minutes… default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Warning: Connection timeout. Retrying… ==\> default: Machine booted and ready! ==\> default: Checking for guest additions in VM… ==\> default: Mounting shared folders… default: /vagrant =\> E:/aerospike/database/aerospike-vm ==\> default: Machine already provisioned. Run `vagrant provision` or use the `--provision` ==\> default: to force provisioning. Provisioners marked to run always will still run.

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [September 30, 2014, 2:14am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/10 "2014-09-30T02:14:26Z")

</div>

Ah looks like you are using vagrant. If this is the vagrant image from Aerospike then your conf file will look like this.

```

namespace test {
    replication-factor 2
    memory-size 1G
    default-ttl 5d # 5days, use 0 to never expire/evict.

# storage-engine memory

    # To use file storage backing, comment out the line above and use the
    # following lines instead.
    storage-engine device {
        file /opt/aerospike/data/test.dat
        filesize 5G
        data-in-memory true # Store data in memory in addition to file.
    }
}

```

default ttl is set to 5days. Which might explain some of your data expiring. Feel free to update config to either a value of zero for never expire or increase default expiration. You can also set ttl from the client code you used to insert the data.

best, Lucien

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [September 30, 2014, 5:17am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/11 "2014-09-30T05:17:39Z")

</div>

Thank you very much Lucien. I am still finding config file ☹ . Will it be in my local sytem?

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [September 30, 2014, 6:19pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/12 "2014-09-30T18:19:13Z")

</div>

It should be in your Linux VM.

You can access the vm as follows.

vagrant ssh

then at the linux prompt run the following:

cat /etc/aerospike/aerospike.conf

Hope this helps,

More info on our configurations files can be found at:

[http://www.aerospike.com/docs/operations/configure/](http://www.aerospike.com/docs/operations/configure/)

Do you have a specific use case that you are trying to test? It may be worth trying Aerospike on Amazon EC2.

best,

Lucien

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [October 3, 2014, 4:25am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/13 "2014-10-03T04:25:33Z")

</div>

Thanks. I am testing performance with large data for my application. Like performance during search with indexd column. Looks good. Few limitation like cant filter on multiple bins though it has multiple filter in java client api. It can be better If it has UI client like other sql clients(sqlyog etc)

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [June 25, 2015, 11:09am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/14 "2015-06-25T11:09:51Z")

</div>

again faced strange problem. I setup aerospike on local linux ubuntu system. It is set with test data. Cleandup existing sets and imported new data in sets with java client. Verified imported data which was ok. restarted system due to hang. After restart it revert back to previous data and lost the new imported data.

Here is my config.

namespace test { memory-size 4G # Maximum memory allocation for data and # primary and secondary indexes.

```
storage-engine device { # Configure the storage-engine to use
                                # persistence.
file /opt/aerospike/test.dat # Location of data file on server.
# file /opt/aerospike/<another> # (optimal) Location of data file on server.
# device /dev/<device> # Optional alternative to using files.

filesize 4G # Max size of each file in GB.
data-in-memory true # Required true by data-in-index.
}

```

}

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [June 25, 2015, 8:53pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/15 "2015-06-25T20:53:10Z")

</div>

Couple of questions. Was the entire server rebooted? How did you cleanup the existing sets? I also noticed that there was no default-ttl set in your config which would default to 0 meaning never expire.

Basically what may be happening is older data that were on disk being re-indexed into memory as part of a cold-start. Please see the following article for ore info:

> [@Expired/Deleted data reappears after server is restarted](https://discuss.aerospike.com/t/expired-deleted-data-reappears-after-server-restarted/470):
>
> Aerospike 3.3.19 In-Memory + Disk Persistence, single node. With conf: default-ttl 0 # Never expire/evict fsync-max-sec 1 flush-max-ms 30000 In an empty DB, using C client to: Insert a record via aerospike\_key\_put(). Update TTL = 10 (expired in 10s) for that record via as\_operations\_add\_touch() and aerospike\_key\_operate(). asmonitor / info show 0 record after 10s. Restart server after 30s. asmonitor / info show 1 record. You can see that expired record reappear after server restarted. Is i…

For your particular case, you may need delete /opt/aerospike/test.dat prior to loading new test data.

> [@bharatkasodariya](#):
>
> Cleandup existing sets and imported new data in sets with java client

---

<div class="post-metadata">

**Author:** ![bharatkasodariya](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/bharatkasodariya/32/38_2.png) [@bharatkasodariya](https://discuss.aerospike.com/u/bharatkasodariya)\
**Post date:** [June 26, 2015, 7:51am UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/16 "2015-06-26T07:51:02Z")

</div>

rebooted entire server. cleandup sets by delete method of java client. Thanks, can set default-ttl to 0, will that solve the problem?

As per link, Seems problem remains there. Not sure but is it problem in community edition

If we delete test.data then It might remove entire data sets. Sometimes we need to clean some sets only so can lost othere data.

Is it like with hard restart(entire server reboot), chaces of data lost. It can not be 100% same data after hard restart?

---

<div class="post-metadata">

**Author:** ![lucien](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/lucien/32/2065_2.png) [@lucien](https://discuss.aerospike.com/u/lucien)\
**Post date:** [June 30, 2015, 6:28pm UTC](https://discuss.aerospike.com/t/durability-issue-lost-set-after-3-4-days/416/17 "2015-06-30T18:28:44Z")

</div>

Default-ttl is set to 0 by default if it not harcoded in the config file. Which means that records are set to not expire unless overwritten by the application.

Its not really a problem, but more of how aerospike handles deletes to ensure high performance. For more details please read [Deletes More info…](https://discuss.aerospike.com/t/how-to-ensure-that-deleted-data-from-set-does-not-come-back/1306)

In your particular case you should be able to use Method#4 to clean old data and ensure that your test sets does not return. Please see: [https://discuss.aerospike.com/t/bin-names-deletion/731](https://discuss.aerospike.com/t/bin-names-deletion/731)

You can also use Method#1 by setting a default-ttl \> 0 and wait for that data to expire prior to restarting the node. This method assumes that record was not initially set to a default-ttl of 0.