**asrestore (3.9.1)** sometimes doesn't restore indexes.
In order to demonstrat…e this problem, I installed fresh aerospike on my machine (_Ubuntu 16.04 LTS_).
Package info:
```
$ dpkg -l | grep aero | awk '{print $2 " " $3}'
aerospike-server-community 3.9.1-1
aerospike-tools 3.9.1
$ md5sum Downloads/aerospike-server-community-3.9.1-ubuntu16.04.tgz
ff7fe8665f84e582ed0319b54d827c2f Downloads/aerospike-server-community-3.9.1-ubuntu16.04.tgz
```
Config info (everything else is default config):
```
namespace backuptest {
replication-factor 2
memory-size 1G
default-ttl 0
ldt-enabled true
storage-engine device {
file /opt/aerospike/data/backuptest.dat
filesize 2G
data-in-memory true
}
}
```
Inserted one record and one index:
```
$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show sets
0 rows in set (0.001 secs)
OK
aql> insert into backuptest.aaa(PK, bin) values (111 , 'val')
OK, 1 record affected.
aql> show sets
+------------------+--------------+----------------+---------+-------------------+-------+-------------------+----------+
| disable-eviction | ns | set-enable-xdr | objects | stop-writes-count | set | memory_data_bytes | deleting |
+------------------+--------------+----------------+---------+-------------------+-------+-------------------+----------+
| "false" | "backuptest" | "use-default" | 1 | 0 | "aaa" | 22 | "false" |
+------------------+--------------+----------------+---------+-------------------+-------+-------------------+----------+
1 row in set (0.001 secs)
OK
aql> show indexes backuptest
0 rows in set (0.000 secs)
OK
aql> create index idx on backuptest.aaa (PK) numeric
OK, 1 index added.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE" | "aaa" | "RW" | "idx" | "PK" | "synced" | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.001 secs)
OK
```
Created backup:
```
$ asbackup -v --host 127.0.0.1 --namespace backuptest --directory backuptest
2016-08-22 13:21:41 GMT [INF] [11298] Starting 100% backup of 127.0.0.1:3000 (namespace: backuptest, set: [all], bins: [all]) to backuptest
2016-08-22 13:21:41 GMT [VER] [11298] Connecting to cluster
2016-08-22 13:21:41 GMT [INF] [11298] [src/main/aerospike/as_cluster.c:440][as_cluster_seed_nodes] Add node BB9A31ECA9D7B50 127.0.0.1:3000
2016-08-22 13:21:41 GMT [VER] [11298] [src/main/aerospike/as_node.c:541][as_node_process_response] Node BB9A31ECA9D7B50 partition generation changed: 1
2016-08-22 13:21:41 GMT [VER] [11298] [src/main/aerospike/as_cluster.c:931][as_cluster_add_seeds] Add seed 127.0.0.1:3000
2016-08-22 13:21:41 GMT [VER] [11298] Adding node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [INF] [11298] Processing 1 node(s)
2016-08-22 13:21:41 GMT [VER] [11298] Getting cluster object count
2016-08-22 13:21:41 GMT [INF] [11298] Node ID Objects Replication
2016-08-22 13:21:41 GMT [VER] [11298] Getting object count for node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [VER] [11298] Getting info value namespace/backuptest for node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [VER] [11298] Parsing info
2016-08-22 13:21:41 GMT [INF] [11298] BB9A31ECA9D7B50 1 1
2016-08-22 13:21:41 GMT [INF] [11298] Namespace contains 1 record(s)
2016-08-22 13:21:41 GMT [VER] [11298] Preparing backup directory backuptest
2016-08-22 13:21:41 GMT [INF] [11298] Directory backuptest does not exist, creating
2016-08-22 13:21:41 GMT [INF] [11298] Directory backuptest prepared for backup
2016-08-22 13:21:41 GMT [VER] [11298] Creating counter thread
2016-08-22 13:21:41 GMT [VER] [11298] Pushing 1 job(s) to job queue
2016-08-22 13:21:41 GMT [VER] [11298] Creating 1 backup thread(s)
2016-08-22 13:21:41 GMT [VER] [11298] Waiting for 1 backup thread(s)
2016-08-22 13:21:41 GMT [VER] [11317] Entering backup thread
2016-08-22 13:21:41 GMT [INF] [11317] Starting backup for node BB9A31ECA9D7B50
2016-08-22 13:21:41 GMT [VER] [11317] 1 remaining record(s), 0 B/rec average size
2016-08-22 13:21:41 GMT [VER] [11317] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:21:41 GMT [VER] [11317] Creating backup file
2016-08-22 13:21:41 GMT [VER] [11316] Entering counter thread
2016-08-22 13:21:41 GMT [VER] [11317] Checking disk space on backuptest for 0 byte(s)
2016-08-22 13:21:41 GMT [INF] [11317] Created new backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:21:41 GMT [VER] [11317] Initializing backup file
2016-08-22 13:21:41 GMT [VER] [11317] Picked up first job, doing one shot work
2016-08-22 13:21:41 GMT [VER] [11317] Processing secondary indexes
2016-08-22 13:21:41 GMT [INF] [11317] Backing up 1 secondary index(es)
2016-08-22 13:21:41 GMT [VER] [11317] Storing index idx
2016-08-22 13:21:41 GMT [VER] [11317] Processing UDFs
2016-08-22 13:21:41 GMT [INF] [11317] Backing up 0 UDF file(s)
2016-08-22 13:21:41 GMT [VER] [11317] Signaling one shot work completion
2016-08-22 13:21:41 GMT [VER] [11317] Received scan end marker
2016-08-22 13:21:41 GMT [INF] [11317] Completed backup for node BB9A31ECA9D7B50, records: 1, size: 168 (~168 B/rec)
2016-08-22 13:21:41 GMT [VER] [11317] Closing backup file
2016-08-22 13:21:41 GMT [VER] [11317] Closing file descriptor
2016-08-22 13:21:41 GMT [VER] [11317] File size is 168
2016-08-22 13:21:41 GMT [VER] [11317] Job queue is empty
2016-08-22 13:21:41 GMT [VER] [11317] Leaving backup thread
2016-08-22 13:21:41 GMT [VER] [11298] Waiting for counter thread
2016-08-22 13:21:42 GMT [INF] [11316] Backed up 1 record(s), 1 secondary index(es), 0 UDF file(s) from 1 node(s), 168 byte(s) in total (~168 B/rec)
2016-08-22 13:21:42 GMT [VER] [11316] Leaving counter thread
2016-08-22 13:21:42 GMT [VER] [11298] Exiting with status code 0
```
Check index is still there:
```
$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE" | "aaa" | "RW" | "idx" | "PK" | "synced" | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.003 secs)
OK
```
I ran several times asrestore and index is still alive:
```
$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:23:14 GMT [INF] [11549] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:23:14 GMT [VER] [11549] Connecting to cluster
2016-08-22 13:23:14 GMT [VER] [11549] Adding node BB9A31ECA9D7B50
2016-08-22 13:23:14 GMT [INF] [11549] Processing 1 node(s)
2016-08-22 13:23:14 GMT [VER] [11549] Creating counter thread
2016-08-22 13:23:14 GMT [VER] [11549] Listing backup files in backuptest
2016-08-22 13:23:14 GMT [VER] [11567] Entering counter thread
2016-08-22 13:23:14 GMT [INF] [11549] Found 1 backup file(s) in backuptest
2016-08-22 13:23:14 GMT [VER] [11549] Triaging 1 backup file(s)
2016-08-22 13:23:14 GMT [VER] [11549] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11549] Getting file descriptor
2016-08-22 13:23:14 GMT [INF] [11549] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11549] Validating backup file version
2016-08-22 13:23:14 GMT [VER] [11549] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11549] Parsing global index and UDF data (line 4)
2016-08-22 13:23:14 GMT [VER] [11549] Parsing index in line 4
2016-08-22 13:23:14 GMT [VER] [11549] Index: idx (on set aaa)
2016-08-22 13:23:14 GMT [VER] [11549] Closing backup file
2016-08-22 13:23:14 GMT [VER] [11549] Closing file descriptor
2016-08-22 13:23:14 GMT [VER] [11549] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:23:14 GMT [INF] [11549] Restoring 0 UDF file(s)
2016-08-22 13:23:14 GMT [INF] [11549] Restoring 1 secondary index(es)
2016-08-22 13:23:14 GMT [VER] [11549] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:23:14 GMT [INF] [11549] Restoring records
2016-08-22 13:23:14 GMT [VER] [11549] Creating 1 restore thread(s)
2016-08-22 13:23:14 GMT [VER] [11549] Waiting for 1 restore thread(s)
2016-08-22 13:23:14 GMT [VER] [11568] Entering restore thread
2016-08-22 13:23:14 GMT [INF] [11568] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11568] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11568] Getting file descriptor
2016-08-22 13:23:14 GMT [INF] [11568] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:23:14 GMT [VER] [11568] Validating backup file version
2016-08-22 13:23:14 GMT [VER] [11568] Existence policy is default
2016-08-22 13:23:14 GMT [VER] [11568] Generation policy is default
2016-08-22 13:23:14 GMT [VER] [11568] Parsing index in line 4
2016-08-22 13:23:14 GMT [VER] [11568] Index: idx (on set aaa)
2016-08-22 13:23:14 GMT [VER] [11568] Ignoring index block
2016-08-22 13:23:14 GMT [VER] [11568] 1 per-thread record(s) (1 rec/s), read latency: 12 (0) us, store latency: 102 (1) us
2016-08-22 13:23:14 GMT [VER] [11568] Encountered end of file (line 12, col 1)
2016-08-22 13:23:14 GMT [VER] [11568] End of backup file reached
2016-08-22 13:23:14 GMT [VER] [11568] Closing backup file
2016-08-22 13:23:14 GMT [VER] [11568] Closing file descriptor
2016-08-22 13:23:14 GMT [VER] [11568] Job queue is empty
2016-08-22 13:23:14 GMT [VER] [11568] Leaving restore thread
2016-08-22 13:23:14 GMT [VER] [11549] Freeing 1 index(es)
2016-08-22 13:23:14 GMT [VER] [11549] Freeing 0 UDF file(s)
2016-08-22 13:23:14 GMT [VER] [11549] Waiting for counter thread
2016-08-22 13:23:15 GMT [INF] [11567] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 168 B/rec, backed off: 0)
2016-08-22 13:23:15 GMT [INF] [11567] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:23:15 GMT [INF] [11567] 100% complete, ~0s remaining
2016-08-22 13:23:15 GMT [VER] [11567] Leaving counter thread
2016-08-22 13:23:15 GMT [VER] [11549] Exiting with status code 0
$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE" | "aaa" | "RW" | "idx" | "PK" | "synced" | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.002 secs)
OK
```
But after Nth attempt to restore from backup I **lost indexes**:
```
$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:25:40 GMT [INF] [11677] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:25:40 GMT [VER] [11677] Connecting to cluster
2016-08-22 13:25:40 GMT [VER] [11677] Adding node BB9A31ECA9D7B50
2016-08-22 13:25:40 GMT [INF] [11677] Processing 1 node(s)
2016-08-22 13:25:40 GMT [VER] [11677] Creating counter thread
2016-08-22 13:25:40 GMT [VER] [11677] Listing backup files in backuptest
2016-08-22 13:25:40 GMT [VER] [11695] Entering counter thread
2016-08-22 13:25:40 GMT [INF] [11677] Found 1 backup file(s) in backuptest
2016-08-22 13:25:40 GMT [VER] [11677] Triaging 1 backup file(s)
2016-08-22 13:25:40 GMT [VER] [11677] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11677] Getting file descriptor
2016-08-22 13:25:40 GMT [INF] [11677] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11677] Validating backup file version
2016-08-22 13:25:40 GMT [VER] [11677] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11677] Parsing global index and UDF data (line 4)
2016-08-22 13:25:40 GMT [VER] [11677] Parsing index in line 4
2016-08-22 13:25:40 GMT [VER] [11677] Index: idx (on set aaa)
2016-08-22 13:25:40 GMT [VER] [11677] Closing backup file
2016-08-22 13:25:40 GMT [VER] [11677] Closing file descriptor
2016-08-22 13:25:40 GMT [VER] [11677] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:25:40 GMT [INF] [11677] Restoring 0 UDF file(s)
2016-08-22 13:25:40 GMT [INF] [11677] Restoring 1 secondary index(es)
2016-08-22 13:25:40 GMT [VER] [11677] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:25:40 GMT [INF] [11677] Restoring records
2016-08-22 13:25:40 GMT [VER] [11677] Creating 1 restore thread(s)
2016-08-22 13:25:40 GMT [VER] [11677] Waiting for 1 restore thread(s)
2016-08-22 13:25:40 GMT [VER] [11696] Entering restore thread
2016-08-22 13:25:40 GMT [INF] [11696] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11696] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11696] Getting file descriptor
2016-08-22 13:25:40 GMT [INF] [11696] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:25:40 GMT [VER] [11696] Validating backup file version
2016-08-22 13:25:40 GMT [VER] [11696] Existence policy is default
2016-08-22 13:25:40 GMT [VER] [11696] Generation policy is default
2016-08-22 13:25:40 GMT [VER] [11696] Parsing index in line 4
2016-08-22 13:25:40 GMT [VER] [11696] Index: idx (on set aaa)
2016-08-22 13:25:40 GMT [VER] [11696] Ignoring index block
2016-08-22 13:25:40 GMT [VER] [11696] 1 per-thread record(s) (1 rec/s), read latency: 10 (0) us, store latency: 45 (0) us
2016-08-22 13:25:40 GMT [VER] [11696] Encountered end of file (line 12, col 1)
2016-08-22 13:25:40 GMT [VER] [11696] End of backup file reached
2016-08-22 13:25:40 GMT [VER] [11696] Closing backup file
2016-08-22 13:25:40 GMT [VER] [11696] Closing file descriptor
2016-08-22 13:25:40 GMT [VER] [11696] Job queue is empty
2016-08-22 13:25:40 GMT [VER] [11696] Leaving restore thread
2016-08-22 13:25:40 GMT [VER] [11677] Freeing 1 index(es)
2016-08-22 13:25:40 GMT [VER] [11677] Freeing 0 UDF file(s)
2016-08-22 13:25:40 GMT [VER] [11677] Waiting for counter thread
2016-08-22 13:25:41 GMT [INF] [11695] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 168 B/rec, backed off: 0)
2016-08-22 13:25:41 GMT [INF] [11695] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:25:41 GMT [INF] [11695] 100% complete, ~0s remaining
2016-08-22 13:25:41 GMT [VER] [11695] Leaving counter thread
2016-08-22 13:25:41 GMT [VER] [11677] Exiting with status code 0
$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
0 rows in set (0.001 secs)
OK
```
When I ran asrestore one more time, indexes are restored:
```
$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:27:08 GMT [INF] [11740] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:27:08 GMT [VER] [11740] Connecting to cluster
2016-08-22 13:27:08 GMT [VER] [11740] Adding node BB9A31ECA9D7B50
2016-08-22 13:27:08 GMT [INF] [11740] Processing 1 node(s)
2016-08-22 13:27:08 GMT [VER] [11740] Creating counter thread
2016-08-22 13:27:08 GMT [VER] [11740] Listing backup files in backuptest
2016-08-22 13:27:08 GMT [VER] [11758] Entering counter thread
2016-08-22 13:27:08 GMT [INF] [11740] Found 1 backup file(s) in backuptest
2016-08-22 13:27:08 GMT [VER] [11740] Triaging 1 backup file(s)
2016-08-22 13:27:08 GMT [VER] [11740] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11740] Getting file descriptor
2016-08-22 13:27:08 GMT [INF] [11740] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11740] Validating backup file version
2016-08-22 13:27:08 GMT [VER] [11740] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11740] Parsing global index and UDF data (line 4)
2016-08-22 13:27:08 GMT [VER] [11740] Parsing index in line 4
2016-08-22 13:27:08 GMT [VER] [11740] Index: idx (on set aaa)
2016-08-22 13:27:08 GMT [VER] [11740] Closing backup file
2016-08-22 13:27:08 GMT [VER] [11740] Closing file descriptor
2016-08-22 13:27:08 GMT [VER] [11740] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:27:08 GMT [INF] [11740] Restoring 0 UDF file(s)
2016-08-22 13:27:08 GMT [INF] [11740] Restoring 1 secondary index(es)
2016-08-22 13:27:08 GMT [VER] [11740] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:27:08 GMT [INF] [11740] Restoring records
2016-08-22 13:27:08 GMT [VER] [11740] Creating 1 restore thread(s)
2016-08-22 13:27:08 GMT [VER] [11740] Waiting for 1 restore thread(s)
2016-08-22 13:27:08 GMT [VER] [11759] Entering restore thread
2016-08-22 13:27:08 GMT [INF] [11759] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11759] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11759] Getting file descriptor
2016-08-22 13:27:08 GMT [INF] [11759] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:27:08 GMT [VER] [11759] Validating backup file version
2016-08-22 13:27:08 GMT [VER] [11759] Existence policy is default
2016-08-22 13:27:08 GMT [VER] [11759] Generation policy is default
2016-08-22 13:27:08 GMT [VER] [11759] Parsing index in line 4
2016-08-22 13:27:08 GMT [VER] [11759] Index: idx (on set aaa)
2016-08-22 13:27:08 GMT [VER] [11759] Ignoring index block
2016-08-22 13:27:08 GMT [VER] [11759] 1 per-thread record(s) (1 rec/s), read latency: 14 (0) us, store latency: 79 (0) us
2016-08-22 13:27:08 GMT [VER] [11759] Encountered end of file (line 12, col 1)
2016-08-22 13:27:08 GMT [VER] [11759] End of backup file reached
2016-08-22 13:27:08 GMT [VER] [11759] Closing backup file
2016-08-22 13:27:08 GMT [VER] [11759] Closing file descriptor
2016-08-22 13:27:08 GMT [VER] [11759] Job queue is empty
2016-08-22 13:27:08 GMT [VER] [11759] Leaving restore thread
2016-08-22 13:27:08 GMT [VER] [11740] Freeing 1 index(es)
2016-08-22 13:27:08 GMT [VER] [11740] Freeing 0 UDF file(s)
2016-08-22 13:27:08 GMT [VER] [11740] Waiting for counter thread
2016-08-22 13:27:09 GMT [INF] [11758] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 1 rec/s, 168 B/rec, backed off: 0)
2016-08-22 13:27:09 GMT [INF] [11758] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:27:09 GMT [INF] [11758] 100% complete, ~0s remaining
2016-08-22 13:27:09 GMT [VER] [11758] Leaving counter thread
2016-08-22 13:27:09 GMT [VER] [11740] Exiting with status code 0
$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
| "backuptest" | "PK" | "NONE" | "aaa" | "RW" | "idx" | "PK" | "synced" | "NUMERIC" |
+--------------+------+-----------+-------+-------+-----------+------+------------+-----------+
1 row in set (0.002 secs)
OK
```
Then I did it several more times and **lost indexes again**:
```
$ asrestore -v --host 127.0.0.1 --directory backuptest -g
2016-08-22 13:29:47 GMT [INF] [12426] Starting restore to 127.0.0.1:3000 (bins: [all], sets: [all]) from backuptest
2016-08-22 13:29:47 GMT [VER] [12426] Connecting to cluster
2016-08-22 13:29:47 GMT [VER] [12426] Adding node BB9A31ECA9D7B50
2016-08-22 13:29:47 GMT [INF] [12426] Processing 1 node(s)
2016-08-22 13:29:47 GMT [VER] [12426] Creating counter thread
2016-08-22 13:29:47 GMT [VER] [12426] Listing backup files in backuptest
2016-08-22 13:29:47 GMT [INF] [12426] Found 1 backup file(s) in backuptest
2016-08-22 13:29:47 GMT [VER] [12426] Triaging 1 backup file(s)
2016-08-22 13:29:47 GMT [VER] [12426] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12426] Getting file descriptor
2016-08-22 13:29:47 GMT [INF] [12426] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12426] Validating backup file version
2016-08-22 13:29:47 GMT [VER] [12426] Found first backup file: backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12426] Parsing global index and UDF data (line 4)
2016-08-22 13:29:47 GMT [VER] [12426] Parsing index in line 4
2016-08-22 13:29:47 GMT [VER] [12426] Index: idx (on set aaa)
2016-08-22 13:29:47 GMT [VER] [12426] Closing backup file
2016-08-22 13:29:47 GMT [VER] [12426] Closing file descriptor
2016-08-22 13:29:47 GMT [VER] [12426] Pushing 1 exclusive job(s) to job queue
2016-08-22 13:29:47 GMT [INF] [12426] Restoring 0 UDF file(s)
2016-08-22 13:29:47 GMT [INF] [12426] Restoring 1 secondary index(es)
2016-08-22 13:29:47 GMT [VER] [12426] Restoring index backuptest:aaa:idx (PK)
2016-08-22 13:29:47 GMT [INF] [12426] Restoring records
2016-08-22 13:29:47 GMT [VER] [12426] Creating 1 restore thread(s)
2016-08-22 13:29:47 GMT [VER] [12426] Waiting for 1 restore thread(s)
2016-08-22 13:29:47 GMT [VER] [12445] Entering restore thread
2016-08-22 13:29:47 GMT [INF] [12445] Restoring backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12445] Opening backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12445] Getting file descriptor
2016-08-22 13:29:47 GMT [INF] [12445] Opened backup file backuptest/BB9A31ECA9D7B50_00000.asb
2016-08-22 13:29:47 GMT [VER] [12445] Validating backup file version
2016-08-22 13:29:47 GMT [VER] [12445] Existence policy is default
2016-08-22 13:29:47 GMT [VER] [12445] Generation policy is default
2016-08-22 13:29:47 GMT [VER] [12445] Parsing index in line 4
2016-08-22 13:29:47 GMT [VER] [12445] Index: idx (on set aaa)
2016-08-22 13:29:47 GMT [VER] [12445] Ignoring index block
2016-08-22 13:29:47 GMT [VER] [12444] Entering counter thread
2016-08-22 13:29:47 GMT [VER] [12445] 1 per-thread record(s) (1 rec/s), read latency: 14 (0) us, store latency: 104 (1) us
2016-08-22 13:29:47 GMT [VER] [12445] Encountered end of file (line 12, col 1)
2016-08-22 13:29:47 GMT [VER] [12445] End of backup file reached
2016-08-22 13:29:47 GMT [VER] [12445] Closing backup file
2016-08-22 13:29:47 GMT [VER] [12445] Closing file descriptor
2016-08-22 13:29:47 GMT [VER] [12445] Job queue is empty
2016-08-22 13:29:47 GMT [VER] [12445] Leaving restore thread
2016-08-22 13:29:47 GMT [VER] [12426] Freeing 1 index(es)
2016-08-22 13:29:47 GMT [VER] [12426] Freeing 0 UDF file(s)
2016-08-22 13:29:47 GMT [VER] [12426] Waiting for counter thread
2016-08-22 13:29:48 GMT [INF] [12444] 0 UDF file(s), 1 secondary index(es), 1 record(s) (0 KiB/s, 0 rec/s, 0 B/rec, backed off: 0)
2016-08-22 13:29:48 GMT [INF] [12444] Expired 0 : skipped 0 : inserted 1 : failed 0 (existed 0, fresher 0)
2016-08-22 13:29:48 GMT [VER] [12444] Leaving counter thread
2016-08-22 13:29:48 GMT [VER] [12426] Exiting with status code 0
$ aql
Aerospike Query Client
Version 3.9.1
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes backuptest
0 rows in set (0.001 secs)
OK
```
You can reproduce problem easily if you try to run asrestore 10 or more times.