Nodes in different clusters on the same subnet with mesh configuration form ONE cluster

1

I am currently seeing an issue where I have two aerospike clusters running CE v3.16 with the following network configurations:

# cluster 1
network {
    service {
        address any
        port 3000
    }

    heartbeat {
        mode mesh
        address 10.34.152.53
        port 3002
        mesh-seed-address-port 10.34.151.50 3002
        mesh-seed-address-port 10.34.151.51 3002
        mesh-seed-address-port 10.34.152.53 3002

# cluster 2
network {
    service {
        address any
        port 3000
    }

    heartbeat {
        mode mesh
        address 10.34.151.52
        port 3002
        mesh-seed-address-port 10.34.151.48 3002
        mesh-seed-address-port 10.34.151.49 3002
        mesh-seed-address-port 10.34.151.52 3002

The expectation here is that because the nodes are in mesh configuration for each cluster then they should form two separate clusters; however, the nodes end up forming ONE single cluster. Is there anything that prevents two clusters in the same subnet from operating independently?

For context the namespaces in each cluster (names are examples only for illustration purposes):

Cluster 1:

a-namespace
b-namespace
c-namespace

Cluster 2:

c-namespace

The reason why both clusters have the “c-namespace” is because that namespace will eventually be migrated to Cluster 2 and deleted from Cluster 1. But because it is currently being used in production, we cannot perform that migration. We had actually set up Cluster 2 in preparation for this migration and noticed this behavior.

Is there something at the namespace level that knows that the namespace exists in both clusters and this is the reason why both clusters form a single cluster?

Through asadm I validated that Cluster 2 does not have a-namespace and b-namespace, only c-namespace as that’s what is it in its namespace configuration.

Additionally, we have clusters in our qa environment with the same exact configurations and they form two independent clusters. The only difference is that the clusters are in different subnets.

Appreciate any response in advance, thank you.

2

Theoretically, they should run as separate clusters unless you at some point you had one of the nodes of the second cluster part of the first cluster and it has those ip-addresses in its cache. You might want to use a different heartbeat port (say 4002) on the second cluster. Also see tip-clear command to clear the cache Info Command Reference | Aerospike Documentation

3

Thanks @pgupta not sure how I got into this situation but I ultimately ended up re-starting the aerospike service for the nodes in Cluster 2, watch them join Cluster 1, issue the tip-clear command (from a node in Cluster 1), stopped the aerospike service for the nodes in Cluster 2, and re-provisioned them. When they came back from re-provisioning, they correctly formed a 3 node cluster.

4

Also, use “cluster-name” config item to isolate clusters. Nodes will only join if specified cluster-name on node matches the cluster-name on the target cluster.

5

This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.