How to remove an offline node in Aerospike? (AER-2757) [Released] [Resolved]

Operations
1

Hi,

I have lot of offline node. How to remove them from the cluster ? I see old nodes when i do “i net” in asadm.

cluster dun 192.168.0.17:3000 does not work.

Regards,

Bertrand

Admin> i net
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Node               Node                Fqdn                  Ip   Client     Current     HB          HB   
           .                 Id                   .                   .    Conns        Time   Self     Foreign   
192.168.0.17   000000000000000    192.168.0.17:3000   192.168.0.17:3000      N/E         N/E    N/E         N/E   
192.168.0.18   000000000000000    192.168.0.18:3000   192.168.0.18:3000      N/E         N/E    N/E         N/E   
192.168.0.19   BB960C588902500    192.168.0.19:3000   192.168.0.19:3000    17395   174076893      0   143043402
2

Currently you have to do a rolling restart of the cluster to reset the services-alumni state. The ticket JIRA tracking this issue is AER-2757. I see that it is code complete, pending review at the moment.

When released, you should be able to issue the following command to clear this state:

asinfo -v "services-alumni-reset"
3

We have experienced similar situation, our cluster (+10 nodes) has experienced/reported serious cluster visibility issues when one of the nodes became unavailable (offline).

We have replication set to 2 so you could expect that it should not be an issue.

Is these days any way to manually let Aerospike know about situation where one of the nodes went down and should be removed from the cluster?

4

@luk The way to do that will be the “services-alumni-reset” command mentioned above.

1 Like
5

What version is this expected to be in?

6

It will be the next release which should be 3.6.2.

1 Like
7

@bpaquet, @luk and @naoum,

Good news! We just released Aerospike Server Community Edition 3.6.2, which fixes AER-2757. You can read its release notes and download it here.

Please upgrade to this new version and let us know whether this fixes your issue.

8

5 posts were split to a new topic: How to remove offline node from Heartbeat’s seed list?

13

Where do we need to run this command. On the server where AMC is installed, On one of the nodes in the cluster, Or on all the nodes in the cluster?

14

For the purpose of not reporting departed peers to tools such as AMC and asadm you would need to run service-alumni-reset on each node.

You can run across the cluster via asadm with:

asadm -h "ip of node in cluster" -e "asinfo -v 'service-alumni-reset`"