Aerospike indexes are lost / object count is lower when instance is restarted

Operations Configuration
, , , ,
1

I have configuration file that all data is stored to disk. But when restart a server all indexes are lost, how this is possible? I could not find any documentation…

I had all indexes and after restart I will just have:

2015-06-02 10:24:38 ERROR AerospikeDao:112 - Query failed: Error Code 201: Index not found

thanks a lot!

2

Hi Ripa,

Thank you for logging your issue on Aerospike forum.

Aerospike stores all the indexes in memory and when restarted all the old indexes will be not be available.

The restart will create new indexes which might take some time to establish new indexes since it goes through cold start.

For more information on cold start and indexes in memory check the below link respectively:

http://www.aerospike.com/docs/operations/manage/aerospike/#coldstart-aerospike-server

https://www.aerospike.com/docs/architecture/primary-index.html#index-persistence

3

Very good point! I had only 2000 rows in index, but I have to test that again. I think there is no special way to say if index should be stored to disk or not?

4

Hi Ripa,

Yes there is no special way to store indexes on disk.

5

Ripa,

Just to clarify in addition to what Padma has already mentioned.

Aerospike always stores indexes in memory. The data could be stored in memory/SSD/file based on your configuration setting for the namespace. Looks like you have configured your namespace to store data in a file.

Error code : 201 → Requested secondary index does not exist

When you restart Aerospike node, both primary and secondary indexes are re-built from the data that you have on disk/file. This would take a little while based on size of the data. If you look at /var/log/aerospike/aerospike.log after starting Aerospike service, you could see that these indexes are being built. Until the indexes are rebuilt, the Aerospike node is not declared available and hence cannot execute queries.

If you could share your aerospike.conf file and the aerospike.log file, we could look further into it.

-samir

6

Friends,

I am also facing this issue.

I have almost 80,000,000 data in my name space and after restart it comes around 50,000,000 and the remaining data has been lost.

One more point, it happens only if i have a huge amount of data (< 50,000,000).

Here is my aerospike.conf setting:-

namespace database {
    memory-size 16G                # Maximum memory allocation for data and
                                    # primary and secondary indexes.
   storage-engine device {         # Configure the storage-engine to use
                                    # persistence.
   file /opt/aerospike/data.bat  # Location of data file on server.
   filesize 100G               # Max size of each file in GiB. Maximum size is 2TiB
    data-in-memory true            # Required true by data-in-index.
    }
}

Please suggest …

7

Since you restarted migrations will be in progress and until migrations complete the object count will not be accurate. During migrations the object count is an under estimate.

8

please Try with below configurations.

memory-size 15G
stop-writes-pct 99
default-ttl 0 # 30 days, use 0 to never expire/evict.
storage-engine device {
	file /usr/share/aerospike/data/aerospike.dat
        filesize 300G
        data-in-memory true # Store data in memory in addition to file.
        scheduler-mode noop
        write-block-size 128K
}