SET not replicated to other node in cluster

1

I have a simple two node cluster.

One the set contains info data and has only one row. In the aql command I see this SET on only one node, while missing on the other node. In the output info_meta is present on only one node. while other two sets are replicated to both nodes.

aql> show sets
+------------------+-----------+----------------+---------+-------------------+---------------+-------------------+----------+------------+
| disable-eviction | ns        | set-enable-xdr | objects | stop-writes-count | set           | memory_data_bytes | deleting | tombstones |
+------------------+-----------+----------------+---------+-------------------+---------------+-------------------+----------+------------+
| "false"          | "DATAA"   | "use-default"  | 14      | 0                 | "LOCALE"      | 0                 | "false"  | 0          |
| "false"          | "CACHE"   | "use-default"  | 172     | 0                 | "names"       | 0                 | "false"  | 0          |
+------------------+-----------+----------------+---------+-------------------+---------------+-------------------+----------+------------+
2 rows in set (0.000 secs)
+------------------+-----------+----------------+---------+-------------------+---------------+-------------------+----------+------------+
| disable-eviction | ns        | set-enable-xdr | objects | stop-writes-count | set           | memory_data_bytes | deleting | tombstones |
+------------------+-----------+----------------+---------+-------------------+---------------+-------------------+----------+------------+
| "false"          | "DATAA"   | "use-default"  | 18      | 0                 | "LOCALE"      | 0                 | "false"  | 0          |
| "false"          | "CACHE"   | "use-default"  | 1       | 0                 | "**info_meta**"  | 0                 | "false"  | 0          |
| "false"          | "CACHE"   | "use-default"  | 172     | 0                 | "names"       | 0                 | "false"  | 0          |
+------------------+-----------+----------------+---------+-------------------+---------------+-------------------+----------+------------+

namespace DATAA  {
        replication-factor          2
        high-water-memory-pct       70
        high-water-disk-pct         90
        stop-writes-pct             95
        # this memory is allocated for index memory
        memory-size                 20G
        default-ttl                 0

        storage-engine device {
                 file       /cacheDir/aerospike/qa_test.dat  # Location of data file on server.
                 filesize     10G                   # Max size of each file in GiB.
        }
}

namespace CACHE  {
        replication-factor      2
        high-water-memory-pct       70
        high-water-disk-pct       90
        stop-writes-pct       95
        # this memory is allocated for index memory
        memory-size         20G
        default-ttl         0

        storage-engine device {
                device        /dev/sdc
                write-block-size    1M
                defrag-lwm-pct      70
                defrag-startup-minimum  10
        }
}
2

I am wondering if AQL only shows the master objects. I see on the set LOCALE in DATAA, you have 14 objects in one node and 18 in the other. Check against total records you have inserted in other sets.

You can see replica object count in a namespace by node using AMC.

Update: Tested on three node cluster. AQL shows sum of master and replica objects on each node.

So having an odd number of objects with replication factor 2 does not add up. Also, in a 2 node cluster with replication factor 2, for Locale in dataa - you have total 32 objects ==> 16 masters - if a node has 10 masters, it should have 6 replicas and vice versa -so both should show object count of 16 each (m + r) and not 14 and 18.

$asadm Admin>info

shows master and replica counts by node.

3

There aren’t many set stats for AQL to be working with. I believe the object counts include master and replica. Are there or were there ongoing migrations when this output was collected? If so does the issue remain after migrations have finished?

Full list of set specific stats acquired using asinfo -v "sets".

ns=test:set=demo:objects=10000:tombstones=0:memory_data_bytes=0:truncate_lut=0:deleting=false:stop-writes-count=0:set-enable-xdr=use-default:disable-eviction=false
4

asinfo Output on one node $asinfo -v “sets” -U aerouser -P ns=DATAA:set=LOCALE:objects=14:tombstones=0:memory_data_bytes=0:deleting=false:stop-writes-count=0:set-enable-xdr=use-default:disable-eviction=false;ns=CACHE:set=names:objects=172:tombstones=0:memory_data_bytes=0:deleting=false:stop-writes-count=0:set-enable-xdre-default:disable-eviction=false;

It gives me error on other node though $ asinfo -v “sets” -U aerouser -P Enter Password: Authentication failed: bcrypt not installed.

5

Looks like bcrypt isn’t installed :slight_smile:.

You can run asinfo via asadm from the node with bcrypt.

asadm -e "asinfo -v sets"

Also can you answer these questions: