[CE] Strange behaviour when third node was added

1

Hello, Community edition, 6.0.0.1 We have a high-load cluster with 3 nodes ATM, replication factor 1 NVMe drives Intel DC4510, 10G network, 128GB RAM

When we added a third node we are facing with strange behaviour of this new node only Seems what the main amount of writes is going in new node only, also after we are reaching some queue limit (on aerospike side?) “clients” is starting to open new connections after what this node becomes unreachable

10.75.1.100:3000|*BB9702F061A3DE4|10.75.1.100:3000|C-6.0.0.1|   1.310 K|   2|E90B136E71A6|True     |BB9702F061A3DE4|  1306|13177:16:56
10.75.1.101:3000| BB9525E0A5183B4|10.75.1.101:3000|C-6.0.0.1|   0.000  |   1| 295807C31AF|True     |BB9525E0A5183B4| 35809|   02:17:57
10.75.1.99:3000 | BB9201F061A3DE4|10.75.1.99:3000 |C-6.0.0.1|   1.310 K|   2|E90B136E71A6|True     |BB9702F061A3DE4|  1227|13177:24:10

We have proto-fd-max=40000, raising of this didnt helped

At this moment we have “frozen” migrations by setting migrate-max-num-incoming=0

Please help, we can provide all needed information for this

4

It looks like the node did not join the cluster properly (it is reporting a different principal node ID and is showing a cluster size of one whereas the other two nodes are showing a cluster size of two).

Looking at the logs of the node should give a clue regarding the cause preventing the node from properly joining the cluster.