Cluster Visibility Error using loopback address

We have Aerospike running on 14 nodes and they have been running for almost a year now with no issues. We have one node now that is having a very strange problem and I can’t seem to find the solution. When you try to use the asadm tool you get this

Found 14 nodes
Cluster Visibility error (Please check services list):,,,,,,,,,,,,

Notice the that is bold? I have no idea why that is the case! If you run it again it might get the correct 10.100.0.x address or it might use the

The other odd thing is that there is a to node mapping that is incorrect as well.

Also, this is what I see on the server that is having an issue. There should only be 14 nodes, not sure why the local ip is bound to the node.

< ~IP to NODE-ID Mapping~
               IP           NODE-ID   BB9ACD16E7AC40C   BB918D66E7AC40C   BB908D76E7AC40C   BB9DED56E7AC40C   BB9E2D56E7AC40C   BB9E2D26E7AC40C   BB980D76E7AC40C   BB912BFDE7AC40C   BB906BFDE7AC40C   BB9DA1BDF7AC40C   BB9F01CDF7AC40C   BB91C1BDF7AC40C   BB96E95987AC40C   BB9EE0E9C7AC40C      BB9ACD16E7AC40C
Number of rows: 15

Any help with this? I have added the access-address to the aeropsike configuration file but nothing has worked.

All help is appreciated as this is causing some issues in our production environment.


What kind of issues are you seeing? Can you post your network configure section of aerospike.conf ?

Here is the network config on the server that is having issues. It’s been suggested to try to remove the node from the cluster and re-add. Going to try that first and then see if that solves the issue.

    network {
        service {
            port 3000
    heartbeat {
        mode multicast
        port 9918

        # To use unicast-mesh heartbeats, remove the 3 lines above, and see
        # aerospike_mesh.conf for alternative.

        interval 150
        timeout 10

    fabric {
        port 3001

    info {
        port 3003

Either this is a bug in asadm or one of the nodes is advertising as an access address - it could be or a different node and you happen to be running asadm from

What version of asadm are you running?

asadm --version

What version are the servers?

asadm -h -v "asinfo -v build"

What are the access-addresses being advertised by the servers?

asadm -h -v "asinfo -v "services"
asadm -h -v "asinfo -v "peers-clear-std"

asadm version

asadm --version

build info

asinfo -v build

access address info

asinfo -v "services";;;;;;;;;;;;
asinfo -v "peers-clear-std"\

The weird thing is that on all the other nodes they work fine. There is no reference of a address anywhere. The even stranger thing is that they were all installed at roughly the same time and using the same base config.

None of the nodes are advertising (local loopback address) so this must be a bug in asadm.

Basically the host asadm uses defaults to if node is provided and in this case it remains in the output as one of the hosts.

If you were to specify the host’s access address instead of using the default, this issue should go away:

asadm -h

Well here is a weird one… If I try that I get this, but not every time, sometimes it connects as I would expect. I’m definitely puzzled on this one.

asadm -h\
Aerospike Interactive Shell, version 0.1.5\
Found 1 nodes

Not able to connect any cluster.

Config files location: /root/.aerospike/

The config files location is incorrect also. I’ve looked at the running process and it’s running the correct config.

Aerospi+ ...  Ssl  19:53  96:28 /usr/bin/asd --config-file /etc/aerospike/aerospike.conf

It is telling you where the config files for asadm are located.

The connection attempt is probably timing out, or you may have exceeded ulimit -n or proto-fd-max connections to this server.

1 Like

Default value for seed IP is asadm uses input seed node IP to get access-address for all Aerospike nodes, then it uses these access-addresses to connect actual Aerospike nodes. For seed node it uses access-address + input seed node address, so first it tries with access-address if that failed then seed address. In this case, for seed node ( asadm have two IPs [,] , when first failed to connect then only it tries second one. So it seems sometimes asadm is not able to connect, which you experienced when you provide this IP as a parameter to asadm.

As Kavin suggested this might be issue of proto-fd-max or ulimit -n exceeded. client_connections stat can give number of open connection.

asadm -e “show statistics service like client_connection” .

For proto-fd-max limit exceed, Aerospike log shows warning for dropping incoming connection.

1 Like

First of all, thanks to Hemant_Patre and kporter for all the help!! :beers: After looking info this further I found that for some reason we were running out of ports on the server which was causing the issue as it could no longer connect to the 10.x.x.x address. I have increased the ports and the issue has subsided.

Thanks again for all the help, I was tearing my hair out on this one.

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.