Cluster node identification

Operations Configuration
,
1

I have the following situation: An aerospike node might restart and obtain a new IP address which it uses to communicate to the cluster. The data (ssd) is persisted though. The other nodes in the cluster see the restarted aerospike node as a new node after it restarted, I assume that is because the IP changed. In the AMC: the old IP remains visible as an unavailable/down node.

How to let the restarted node identify to the rest of the cluster as ‘itself’ again after restarting?

What might be related is: I’m using mesh heartbeat and adding nodes to the cluster using the asinfo tip command.

1 Like
2

I guess the MAC address of this server also changes? If so, a drawback here will be a lot more data migrations during re-balancing after a restart.

The cluster should stop advertising the IP addresses that have left to the clients. But some of the tools request a list of all nodes that have ever been part of the cluster, and this list will include the IP addresses that have permanently departed. You can see the two listings by running the following:

# List requested by the Client Applications:
asinfo -v "services"

# List requested by diagnostic tools such as asadm:
asinfo -v "services-alumni"

Currently, the services-alumni grows monotonically through the lifetime of the server process. This state is purged on any form of restart and will be rebuilt with the currently active nodes when it rejoins.

1 Like
3

Ah, that explains. I have indeed seen a lot of migrations on a restart…

In my case: Aerospike is started in a docker container, so the mac address changes after each start. Is there any other way to set the identification key for a node manually? If not, would that be a good feature request?

4

I was able to use the docker flag --mac-address to set a custom mac-address on the Aerospike containers, thereby re-creating the same node id on restart. I still think a configuration parameter to set a custom node id could be useful though.

Information for anyone replicating this solution:

  • Use a mac address from the locally administered range.
  • This has been tested and works fine with flannel.
  • Of course, make sure each Aerospike container has a different mac or the cluster setup will fail.
  • When using fleet, you could use %i in the .service file to create a unique mac. Start instances with a zerofilled instance name; fleetctl start aerospike@01.service, that way you could simply add --mac-address 02:A5:A5:A5:A5:%i. (02 is simply the first option in the locally administered range, A5 looks like AS(AeroSpike) and %i will resolve to 01 in this example.)
2 Likes