How to divide a mesh cluster in to two parts


#1

FAQ How to divide a single mesh cluster in to two separate clusters.

Context

The reasons for wanting to separate a single cluster into 2 distinct clusters is two fold:

(i) Cluster C has to be divided into clusters A and B.

OR

(ii) 2 separate clusters A and B accidentally joined each other via heartbeat to attempt to form a cluster C.

Method

Note: In the section below, “nodes of cluster A/B” means the nodes intended to be part of cluster A/B respectively (for case (i)).

The steps for this involve taking down nodes of one cluster and performing a tip-clear on the other cluster. The choice of which cluster to take down can be made based on the configuration or by preference. For example one should avoid bringing down a non persisted data-in-memory cluster as this would mean data loss.

In the following example, the nodes of cluster B are going to be taken down:

Step 1: Check the config files of all nodes from both the clusters and delete a mesh-seed-host-port entry if it contains ip and port of a node from the other cluster. (Nodes from the mesh seed list of cluster A should not refer to nodes from cluster B and vice versa).

Step 2: Bring down cluster B.

Step 3a): Run the following command on all the nodes in cluster A (IP addresses provided in the example below would be the ones of cluster B):

"asinfo -v 'tip-clear:host-port-list=172.3.0.1:3002,172.3.0.2:3002, ...'" 

To verify this, run the following command on all nodes and verify that the field MeshSeedAddrPort in the output does not contain nodes from cluster B:

"asinfo -v 'dump-hb:verbose=true'"
OR

Step 3b): If the command given in step 3a does not resolve the issue, do a rolling restart of the nodes in cluster A. After rolling restart, verify this by running the following command on all nodes in cluster A and check that the field MeshSeedAddrPort in the output does not contain references to cluster B.

"asinfo -v 'dump-hb:verbose=true'" 

Step 4: Restart nodes of cluster B.

Keywords

CLUSTER, DIVIDE CLUSTER, TIP-CLEAR.

Timestamp

5/25/16