Aerospike showing records that aren't there


#1

Hi

Weird one, if I run a show sets in AQL on our cluster you can see the following:

+------------------+-------------+----------------+---------+-------------------+--------------------+-------------------+----------+------------+
| disable-eviction | ns          | set-enable-xdr | objects | stop-writes-count | set                | memory_data_bytes | deleting | tombstones |
+------------------+-------------+----------------+---------+-------------------+--------------------+-------------------+----------+------------+
| "false"          | "namespace" | "use-default"  | 2149131 | 0                 | "requests"         | 0                 | "false"  | 0          |
| "false"          | "namespace" | "use-default"  | 77985   | 0                 | "tracking"         | 0                 | "false"  | 0          |

however if I run a query on those sets I get nothing…

aql> select * from namespace.requests
0 rows in set (0.078 secs)

aql> select * from namespace.tracking
0 rows in set (0.078 secs)

any thoughts why this might be?

(ps: names changed to protect the innocent) :smiley:


#2

What are the versions of the Aerospike Server and Aerospike Tools packages?


#3

We’re running version 3.10 of both, i think 1 before the latest but am unable to check right now


#4

Is this on Aerospike Community?

Also could you run:

asadm -e "show stat like object"

#5

Hi @kporter

Sorry for the delay, I was away all weekend. Yes, the is Community Edition:

Servers are version 3.10.0-1 AMC is v3.6.12 Tools are version 3.10.2

I will message you the results of the asadm command privately :slight_smile:


#6

so I’ve just tried to run an asinfo command to delete the sets again (as the data is still showing as being there) and in the log I get the following over and over (edited):

Nov 21 2016 11:49:35 GMT: INFO (nsup): (thr_nsup.c:1225) {adspruce} deleting set <set_name>

Also, it is showing another set as being empty in one of the nodes, but not the other - so we have this one Node 1:

| "false"          | "adspruce" | "use-default"  | 0       | 0                 | "set_name"         | 0                 | "false"  | 0          |

and this on the other:

| "false"          | "adspruce" | "use-default"  | 77985   | 0                 | "set_name"         | 0                 | "true"   | 0          |

any further thoughts on this @rbotzer or @kporter ? :confused:


#7

The set-delete command must be executed on all nodes, did you issue the set-delete on all nodes?

You can execute on all nodes using asadm:

asadm -e "asinfo -v 'set-config:context=namespace;id=adspruce;set=set_name;set-delete=true;'"

#8

@kporter - yeah, I executed the command on all nodes. I’ve done it fine in the past with no issues, but this time something fails to work properly. The odd thing is that AMC / asadm commands show there being objects there, but if I run a select * from namespace.set_name I get 0 records back as a result?

I’ll run it through asadm and see if there is any difference.


#9

Tried reproducing your steps on Community edition ver 3.10, Initially I have users set in namespace test with 10,000 objects. Using asadm:

Admin> show stat sets
test users Set Statistics NODE : 192.168.183.146:3000
deleting : false
disable-eviction : false
memory_data_bytes: 1942349
ns : test
objects : 10000
set : users
set-enable-xdr : use-default
stop-writes-count: 0
tombstones : 0

I now delete the set. Wait for the namespace supervisor to run (I am using default setting of two minutes). Then check set stat again.

Admin> asinfo -v "set-config:context=namespace;id=test;set=users;set-delete=true;"
...
Admin> show stat sets
~~~~~~~~test users Set Statistics~~~~~~~~
NODE             :   192.168.183.146:3000   
deleting         :   true                   
disable-eviction :   false                  
memory_data_bytes:   0                      
ns               :   test                   
objects          :   0                      
set              :   users                  
set-enable-xdr   :   use-default            
stop-writes-count:   0                      
tombstones       :   0 

Further, in AQL, earlier I had all the records (see last line of previous aql command, then next), after set-delete I get:

+------------+------------------------------------------+--------+--------+-------------+-----------+------------+
10000 rows in set (0.897 secs) 

(above is from previous select * command)

aql> select * from test.users
0 rows in set (0.617 secs)