Mechanism to automate Cluster join in mesh mode

mesh

#1

Hello All,

I have a very interesting requirement here for mesh mode of aerospike config to detect other cluster nodes in network for which I would really like opinions from the experts here.

Before asking the question, the reason for not choosing multicast is that during customer end deployment we may face issues where enabling multicast is a problem for a particular client/network topology. Also mesh mode give us bit more control on which machines we want to add in cluster.

Now to the question, is there any way I can achieve this? May be there could be DNS way of figuring out IPs if by some means I can resolve a specific FQDN to set of IPs. Or I bet there is a better way of doing this.

Please share your views.

Thanks, Cheers!


#2

You have the right solution. Aerospike needs at least 1 seed node to point to when starting to join the cluster, it’ll find out the rest of the nodes automatically.

http://www.aerospike.com/docs/reference/configuration#mesh-seed-address-port

This config setting mesh-seed-address-port now accepts regular DNS entries instead of just IPs so you can setup a DNS name that you keep current with at least 1 active node in the cluster and have all the configs use that setting value to automate the joining.


#3

Thanks for the reply. But his approach needs some improvements as in case of auto-scaling, I will need to update config on 2 nodes (1 that is recently being brough UP and other one that is already a part of cluster) and then restart the services on both which has its own set of problems as during this I will be down by total capacity to handle traffic.

Any suggestion?

Also is there any way I can create 2 disjoint clusters in same network setup by somehow giving a name to cluster group.


#4

Why do you have to update the config? This is the whole point of having a DNS name…

Use something like aerospikecluster1.yourcompany.com and have that point to the IP of a seed node - which is just any node that’s currently up in your cluster. The config for every new node can use this same address as the seed node and they will join the cluster and discover all the other nodes.

If the original seed node ever goes away, just update the DNS to point to another node that’s running.


#5

I understand what you are saying. Let me explain more about the requirement.

We are going to run it inside Openstack environment and we need to take care of scaling as well as recovery as well. If one node goes down, I will have to launch a new aerospike node but to add it to the cluster, I will have to restart services on both the aerospike servers (with IP address of each other). Correct if I am wrong here. I understand from your reply that I need to just give IP/FQDN of one of the node already in cluster to this new node and it should just work.


#6

You shouldn’t need to restart, update their static configs and use the tip command to add the new node to the existing node’s seed list.

http://www.aerospike.com/docs/reference/info#tip

You can also remove the departed node from the seed list, see “tip-clear”. Various tools will also continue to report the departed node, once the new node has joined the cluster you can run the “services-alumni-reset” command to remove departed nodes from the list tools read.


#7

Ok I kind of figured out the approach but just a small question. Suppose DNS has multiple IPs for one FQDN which is configured in aerospike. Will the cluster form in that case as there will be multiple IPs in that case for same FQDN?

Also, if FQDN is resolved successfully but the instance to which it points isn’t reachable, will aerospike retry on new IP resolved by DNS or it keeps on trying on the IP it already got from DNS.