I am currently seeing an issue where I have two aerospike clusters running CE v3.16 with the following network configurations:
# cluster 1
network {
service {
address any
port 3000
}
heartbeat {
mode mesh
address 10.34.152.53
port 3002
mesh-seed-address-port 10.34.151.50 3002
mesh-seed-address-port 10.34.151.51 3002
mesh-seed-address-port 10.34.152.53 3002
# cluster 2
network {
service {
address any
port 3000
}
heartbeat {
mode mesh
address 10.34.151.52
port 3002
mesh-seed-address-port 10.34.151.48 3002
mesh-seed-address-port 10.34.151.49 3002
mesh-seed-address-port 10.34.151.52 3002
The expectation here is that because the nodes are in mesh configuration for each cluster then they should form two separate clusters; however, the nodes end up forming ONE single cluster. Is there anything that prevents two clusters in the same subnet from operating independently?
For context the namespaces in each cluster (names are examples only for illustration purposes):
Cluster 1:
a-namespace
b-namespace
c-namespace
Cluster 2:
c-namespace
The reason why both clusters have the “c-namespace” is because that namespace will eventually be migrated to Cluster 2 and deleted from Cluster 1. But because it is currently being used in production, we cannot perform that migration. We had actually set up Cluster 2 in preparation for this migration and noticed this behavior.
Is there something at the namespace level that knows that the namespace exists in both clusters and this is the reason why both clusters form a single cluster?
Through asadm I validated that Cluster 2 does not have a-namespace and b-namespace, only c-namespace as that’s what is it in its namespace configuration.
Additionally, we have clusters in our qa environment with the same exact configurations and they form two independent clusters. The only difference is that the clusters are in different subnets.
Appreciate any response in advance, thank you.