Corrupted Index



We are experiencing a problem involving missing records using a secondary key. A set namely user contains the following bins: name, email, password, role and private_token- with a secondary key on the bin private_token.

Regarding the issue: As of the previous week, four nodes in Google Cloud. The secondary key of the user set was working fine. During the end of last week, we created a new node and now this specific set stopped returning any data when querying it using a secondary key. Regarding all the other set having a similar configuration, they kept returning the correct values.

The namespace n1 of the user set is using a non local SSD storage.

In order to troubleshoot the issue, we created a new namespace, namely test, that is stored in memory. Here is a script that fails in the n1 namespace but succeed in the test namespace:

INSERT INTO test.user (PK, name, email, password, role, private_token) VALUES ('', 'Test 1', '', 'password_test_1', 'role_test_1', 'private_token_1')
INSERT INTO test.user (PK, name, email, password, role, private_token) VALUES ('', 'Test 2', '', 'password_test_2', 'role_test_2', 'private_token_2')
CREATE INDEX user_private_token ON test.user(private_token) STRING

SELECT * FROM test.user WHERE private_token = 'private_token_2'

Here is the output in test namespace:

| name     | email            | password          | role          | private_token     |
| "Test 2" | "" | "password_test_2" | "role_test_2" | "private_token_2" |
1 row in set (0.001 secs)

And here in n1 namespace:

0 rows in set (0.001 secs)

Version of aerospike:

$ dpkg -l | grep aerospike
ii  aerospike-amc-community                   3.6.4                         all          AMC Community Version is a web UI based monitoring tool for Aerospike Community Edition Server releases after 2.6.3 (March 2013)
ii  aerospike-server-community                3.6.4-1                       amd64        The Aerospike distributed datastore allows fully scalable and reliable data storage with elastic server properties.
ii  aerospike-tools                           3.6.3                         amd64        Aerospike server tools.

Trying to troubleshoot in asadm:

$ asadm
Aerospike Interactive Shell, version 0.0.13
Found 6 nodes

Admin> info network
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          Node               Node                                           Fqdn                Ip   Client     Current     HB          HB
             .                 Id                                              .                 .    Conns        Time   Self     Foreign       000000000000000                            N/E         N/E    N/E         N/E
aerospike-vm-1   BB90300F00A0142    aerospike-vm-1.c.citiservi-ads.internal:3000       38   193339236      0   249014840
aerospike-vm-2   BB90500F00A0142    aerospike-vm-2.c.citiservi-ads.internal:3000       37   193339236      0   194991137
aerospike-vm-3   BB90400F00A0142    aerospike-vm-3.c.citiservi-ads.internal:3000       34   193339236      0   229395269
aerospike-vm-4   BB90200F00A0142    aerospike-vm-4.c.citiservi-ads.internal:3000       45   193339236      0   224245394
aerospike-vm-5   *BB90600F00A0142   aerospike-vm-5.c.citiservi-ads.internal:3000       37   193339236      0    17162917
Number of rows: 6

Admin> info service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          Node   Build   Cluster      Cluster     Cluster    Free   Free    Migrates        Principal    Objects       Uptime
             .       .      Size   Visibility   Integrity   Disk%   Mem%   (tx,rx,q)                .          .            .       N/E         N/E   N/E          N/E           N/E    N/E   N/E         N/E                   N/E   N/E
aerospike-vm-1   3.6.4         5   True         True           61     68   (0,0,0)     aerospike-vm-5   58.618 M   1710:56:06
aerospike-vm-2   3.6.4         5   True         True           61     69   (0,0,0)     aerospike-vm-5   57.594 M   1472:30:21
aerospike-vm-3   3.6.4         5   True         True           61     68   (0,0,0)     aerospike-vm-5   57.430 M   1638:33:39
aerospike-vm-4   3.6.4         5   True         True           61     68   (0,0,0)     aerospike-vm-5   58.242 M   1824:25:55
aerospike-vm-5   3.6.4         5   True         True           61     71   (0,0,0)     aerospike-vm-5   57.774 M   102:48:57
Number of rows: 6

Admin> info sindex
Traceback (most recent call last):
  File "/usr/lib/python2.7/", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/", line 72, in _run_code
    exec code in run_globals
  File "/usr/bin/asadm/", line 20, in <module>
  File "/usr/bin/asadm/", line 339, in main
  File "/usr/lib/python2.7/", line 141, in cmdloop
    line = self.precmd(line)
  File "/usr/bin/asadm/", line 108, in precmd
  File "/usr/bin/asadm/lib/", line 176, in execute
  File "/usr/bin/asadm/lib/", line 120, in __call__
  File "/usr/bin/asadm/lib/", line 176, in execute
  File "/usr/bin/asadm/lib/", line 158, in do_sindex
TypeError: string indices must be integers, not str

I am looking forward to hearing from you at your earliest convenience,

Best Regards