Seeing phantom nodes in Asmonitor & Asadm


#1

Synopsis:

After removing X nodes from the cluster, I still see them when running Asmonitor or Asadm.

# asmonitor

Enter help for commands

request to 172.23.17.13 : 3000 returned error
request to 173.27.17.11 : 3000 returned error
request to 172.23.17.10 : 3000 returned error
request to 172.23.17.12 : 3000 returned error

Solution:

Reason why the phantom nodes are still showing up in Asmonitor or Asadm is due to the services-alumni list. This list keeps all the neighbor addresses (services) this server has ever know about since it’s last restart. Diagnostic tools, eg. asmonitor and asadm, uses the alumni list to enable full cluster view even during cluster disruptions.

For versions 3.6.1 and earlier, the only way to remove these phantom nodes from the services-alumni list is to restart the existing node(s), or wait for the next planned restart of the nodes.

In version 3.6.2 and above, we have a reset option that can be triggerd in such events to refresh the services-alumni list and make it equal to the services list (which keeps the list of addresses of all the current neighbour nodes which is advertised to the clients).

For more information, see below page: http://www.aerospike.com/docs/reference/info/#services-alumni-reset