Hi,
We are experiencing a problem involving missing records using a secondary key. A set namely user contains the following bins: name, email, password, role and private_token- with a secondary key on the bin private_token.
Regarding the issue: As of the previous week, four nodes in Google Cloud. The secondary key of the user set was working fine. During the end of last week, we created a new node and now this specific set stopped returning any data when querying it using a secondary key. Regarding all the other set having a similar configuration, they kept returning the correct values.
The namespace n1 of the user set is using a non local SSD storage.
In order to troubleshoot the issue, we created a new namespace, namely test, that is stored in memory. Here is a script that fails in the n1 namespace but succeed in the test namespace:
INSERT INTO test.user (PK, name, email, password, role, private_token) VALUES ('test@test.com', 'Test 1', 'test@test.com', 'password_test_1', 'role_test_1', 'private_token_1')
INSERT INTO test.user (PK, name, email, password, role, private_token) VALUES ('test2@test.com', 'Test 2', 'test2@test.com', 'password_test_2', 'role_test_2', 'private_token_2')
CREATE INDEX user_private_token ON test.user(private_token) STRING
SELECT * FROM test.user WHERE private_token = 'private_token_2'
Here is the output in test namespace:
+----------+------------------+-------------------+---------------+-------------------+
| name | email | password | role | private_token |
+----------+------------------+-------------------+---------------+-------------------+
| "Test 2" | "test2@test.com" | "password_test_2" | "role_test_2" | "private_token_2" |
+----------+------------------+-------------------+---------------+-------------------+
1 row in set (0.001 secs)
And here in n1 namespace:
0 rows in set (0.001 secs)
Version of aerospike:
$ dpkg -l | grep aerospike
ii aerospike-amc-community 3.6.4 all AMC Community Version is a web UI based monitoring tool for Aerospike Community Edition Server releases after 2.6.3 (March 2013)
ii aerospike-server-community 3.6.4-1 amd64 The Aerospike distributed datastore allows fully scalable and reliable data storage with elastic server properties.
ii aerospike-tools 3.6.3 amd64 Aerospike server tools.
Trying to troubleshoot in asadm:
$ asadm
Aerospike Interactive Shell, version 0.0.13
Found 6 nodes
Online: 10.240.0.3:3000, 10.240.0.4:3000, 10.240.0.5:3000, 10.240.0.2:3000, 10.240.0.6:3000
Offline: 10.240.0.9:3000
Admin> info network
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Node Fqdn Ip Client Current HB HB
. Id . . Conns Time Self Foreign
10.240.0.9 000000000000000 10.240.0.9:3000 10.240.0.9:3000 N/E N/E N/E N/E
aerospike-vm-1 BB90300F00A0142 aerospike-vm-1.c.citiservi-ads.internal:3000 10.240.0.3:3000 38 193339236 0 249014840
aerospike-vm-2 BB90500F00A0142 aerospike-vm-2.c.citiservi-ads.internal:3000 10.240.0.5:3000 37 193339236 0 194991137
aerospike-vm-3 BB90400F00A0142 aerospike-vm-3.c.citiservi-ads.internal:3000 10.240.0.4:3000 34 193339236 0 229395269
aerospike-vm-4 BB90200F00A0142 aerospike-vm-4.c.citiservi-ads.internal:3000 10.240.0.2:3000 45 193339236 0 224245394
aerospike-vm-5 *BB90600F00A0142 aerospike-vm-5.c.citiservi-ads.internal:3000 10.240.0.6:3000 37 193339236 0 17162917
Number of rows: 6
Admin> info service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Node Build Cluster Cluster Cluster Free Free Migrates Principal Objects Uptime
. . Size Visibility Integrity Disk% Mem% (tx,rx,q) . . .
10.240.0.9 N/E N/E N/E N/E N/E N/E N/E N/E N/E N/E
aerospike-vm-1 3.6.4 5 True True 61 68 (0,0,0) aerospike-vm-5 58.618 M 1710:56:06
aerospike-vm-2 3.6.4 5 True True 61 69 (0,0,0) aerospike-vm-5 57.594 M 1472:30:21
aerospike-vm-3 3.6.4 5 True True 61 68 (0,0,0) aerospike-vm-5 57.430 M 1638:33:39
aerospike-vm-4 3.6.4 5 True True 61 68 (0,0,0) aerospike-vm-5 58.242 M 1824:25:55
aerospike-vm-5 3.6.4 5 True True 61 71 (0,0,0) aerospike-vm-5 57.774 M 102:48:57
Number of rows: 6
Admin> info sindex
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/bin/asadm/__main__.py", line 20, in <module>
File "/usr/bin/asadm/asadm.py", line 339, in main
File "/usr/lib/python2.7/cmd.py", line 141, in cmdloop
line = self.precmd(line)
File "/usr/bin/asadm/asadm.py", line 108, in precmd
File "/usr/bin/asadm/lib/controllerlib.py", line 176, in execute
File "/usr/bin/asadm/lib/controllerlib.py", line 120, in __call__
File "/usr/bin/asadm/lib/controllerlib.py", line 176, in execute
File "/usr/bin/asadm/lib/controller.py", line 158, in do_sindex
TypeError: string indices must be integers, not str
I am looking forward to hearing from you at your earliest convenience,
Best Regards