I have 2 separate aerospike clusters running in different Availability Zones in Google Compute Engine. All nodes are configured to use mesh heartbeat for clustering and the service discovery process is that on startup, each node adds other aerospike instances that are part of the SAME availability zones as peers using asinfo -v "tip:host=${hostname};port=3002".
However, I noticed that if I do a rolling update of clusters simultaneously, nodes can form clusters across availability zones even though each node is only configured to add nodes in its own AZ as peer. All the availability zones are in the same subnet 10.0.0.0/16 and all nodes use port 3002 for heartbeats.
On first glance, this looks similar to Prevent clustering between installations and I know I can prevent this using the cluster-name property. But I still don’t understand how this can happen using mesh heartbeats where each node has to explicitly add or be added as a peer to form a cluster.