Is there any limit on the cluster size when using mesh heartbeats?


#1

Hi

I am using Mesh/Unicast heartbeats with 4 nodes in my cluster and everything is just fine. But this link on Aerospike configuration (Slide number 50 in that presentation) says that:

Aerospike has found that this mechanism(Mesh) works in production with up to 20 nodes

I have not read this limitation any where else. In future, we might have cluster size which can exceed the limit of 20. Can someone please confirm if this is a known limitation ? Thanks.


#2

I believe this is old data. The currently defined maximum cluster size is 128, and I’m pretty sure that’s true of mesh as well. If not we’ll fix it … let us know.


#3

Yes, that link is an outdated presentation. There is a setting called paxos-max-cluster-size which by default is 32, the maximum permitted value is 128. The maximum size for a cluster is currently 1 less than the paxos-max-cluster-size. So by default we cluster cannot be larger than 31 nodes and 127 nodes at the maximum setting.

Prior to 3.3.26 we recommended keeping mesh clusters small due to stability issues with mesh. Mesh had traditionally been used to recover a multicast cluster when multicast networking was having problems (typically network configuration issues) and the operations team could not immediately resolve the fault. Open sourcing created a huge demand for mesh in cloud environments that did not support multicast which spurred additional engineering attention in this area. Since 3.3.26 both our mesh and multicast protocols are about par in terms of resiliency.


#4

Thanks @bbulkow and @kporter for providing this confirmation. With all the other nice features in AS, it was almost unbelievable that there could be a limit of 20 on cluster size :smile:

I don’t have required nodes to take it up to 20 and test. But in future we shall have and I will update this result with what we observe. Thanks.


#5

For Enterprise Edition, the maximum number of nodes in a cluster is 128. For versions prior to 3.14 running the heartbeat protocol v2, the limit is defined by the configuration paxos-max-cluster-size (with a default of 31). Refer to this page regarding the process to increase the default for an already running cluster: Increase Maximum Cluster Size. For Community Edition, the maximum number of nodes is limited to 31 for versions prior to 4.0 and limited to 8 for versions 4.0 and above.