# Cluster Visibility Error using loopback address

**URL:** <https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539>\
**Category:** Configuration\
**Created:** [August 29, 2017, 7:50pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539 "2017-08-29T19:50:43Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![mattmnx](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mattmnx](https://discuss.aerospike.com/u/mattmnx)\
**Post date:** [August 29, 2017, 7:50pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/1 "2017-08-29T19:50:43Z")

</div>

We have Aerospike running on 14 nodes and they have been running for almost a year now with no issues. We have one node now that is having a very strange problem and I can’t seem to find the solution. When you try to use the asadm tool you get this

```auto
Found 14 nodes
Online: 10.100.0.138:3000, **127.0.0.1:3000** , 10.100.0.149:3000, 10.100.0.154:3000, 10.100.0.143:3000, 10.100.0.153:3000, 10.100.0.150:3000, 10.100.0.152:3000, 10.100.0.151:3000, 10.100.0.155:3000, 10.100.0.140:3000, 10.100.0.137:3000, 10.100.0.139:3000, 10.100.0.142:3000
Cluster Visibility error (Please check services list): 10.100.0.138:3000, 10.100.0.149:3000, 10.100.0.154:3000, 10.100.0.143:3000, 10.100.0.153:3000, 10.100.0.150:3000, 10.100.0.152:3000, 10.100.0.151:3000, 10.100.0.155:3000, 10.100.0.140:3000, 10.100.0.137:3000, 10.100.0.139:3000, 10.100.0.142:3000

```

Notice the 127.0.0.1 that is bold? I have no idea why that is the case! If you run it again it might get the correct 10.100.0.x address or it might use the 127.0.0.1.

The other odd thing is that there is a 127.0.0.1 to node mapping that is incorrect as well.

Also, this is what I see on the server that is having an issue. There should only be 14 nodes, not sure why the local ip is bound to the node.

```auto
< ~IP to NODE-ID Mapping~
               IP NODE-ID
10.100.0.136:3000 BB9ACD16E7AC40C
10.100.0.137:3000 BB918D66E7AC40C
10.100.0.138:3000 BB908D76E7AC40C
10.100.0.139:3000 BB9DED56E7AC40C
10.100.0.140:3000 BB9E2D56E7AC40C
10.100.0.142:3000 BB9E2D26E7AC40C
10.100.0.143:3000 BB980D76E7AC40C
10.100.0.149:3000 BB912BFDE7AC40C
10.100.0.150:3000 BB906BFDE7AC40C
10.100.0.151:3000 BB9DA1BDF7AC40C
10.100.0.152:3000 BB9F01CDF7AC40C
10.100.0.153:3000 BB91C1BDF7AC40C
10.100.0.154:3000 BB96E95987AC40C
10.100.0.155:3000 BB9EE0E9C7AC40C
127.0.0.1:3000 BB9ACD16E7AC40C
Number of rows: 15

```

Any help with this? I have added the access-address to the aeropsike configuration file but nothing has worked.

All help is appreciated as this is causing some issues in our production environment.

Thanks!

---

<div class="post-metadata">

**Author:** ![Albot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/albot/32/2076_2.png) [@Albot](https://discuss.aerospike.com/u/Albot)\
**Post date:** [August 30, 2017, 12:48am UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/2 "2017-08-30T00:48:56Z")

</div>

What kind of issues are you seeing? Can you post your network configure section of aerospike.conf ?

---

<div class="post-metadata">

**Author:** ![mattmnx](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mattmnx](https://discuss.aerospike.com/u/mattmnx)\
**Post date:** [August 30, 2017, 3:12pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/3 "2017-08-30T15:12:34Z")

</div>

Here is the network config on the server that is having issues. It’s been suggested to try to remove the node from the cluster and re-add. Going to try that first and then see if that solves the issue.

```auto
    network {
        service {
            address 10.100.0.136
            access-address 10.100.0.136
            port 3000
    }
    
    heartbeat {
        mode multicast
        multicast-group 239.1.99.222
        port 9918

        # To use unicast-mesh heartbeats, remove the 3 lines above, and see
        # aerospike_mesh.conf for alternative.

        interval 150
        timeout 10
    }

    fabric {
        port 3001
    }

    info {
        port 3003
    }
}

```

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [August 30, 2017, 4:40pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/4 "2017-08-30T16:40:03Z")

</div>

Either this is a bug in asadm or one of the nodes is advertising 127.0.0.1 as an access address - it could be 10.100.0.136 or a different node and you happen to be running asadm from 10.100.0.136.

What version of asadm are you running?

```auto
asadm --version

```

What version are the servers?

```auto
asadm -h 10.100.0.136 -v "asinfo -v build"

```

What are the access-addresses being advertised by the servers?

```auto
asadm -h 10.100.0.136 -v "asinfo -v "services"
asadm -h 10.100.0.136 -v "asinfo -v "peers-clear-std"

```

---

<div class="post-metadata">

**Author:** ![mattmnx](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mattmnx](https://discuss.aerospike.com/u/mattmnx)\
**Post date:** [August 30, 2017, 6:27pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/5 "2017-08-30T18:27:49Z")

</div>

asadm version

```
asadm --version
0.1.5

```

build info

```
asinfo -v build
3.10.0-.3

```

access address info

```auto
asinfo -v "services" 10.100.0.155:3000;10.100.0.152:3000;10.100.0.140:3000;10.100.0.154:3000;10.100.0.139:3000;10.100.0.142:3000;10.100.0.137:3000;10.100.0.151:3000;10.100.0.138:3000;10.100.0.153:3000;10.100.0.143:3000;10.100.0.149:3000;10.100.0.150:3000

```

```auto
asinfo -v "peers-clear-std"\
27,3000,[[BB9EE0E9C7AC40C,,[10.100.0.155]],[BB9F01CDF7AC40C,,[10.100.0.152]],[BB9E2D56E7AC40C,,[10.100.0.140]],[BB96E95987AC40C,,[10.100.0.154]],[BB9DED56E7AC40C,,[10.100.0.139]],[BB9E2D26E7AC40C,,[10.100.0.142]],[BB918D66E7AC40C,,[10.100.0.137]],[BB9DA1BDF7AC40C,,[10.100.0.151]],[BB908D76E7AC40C,,[10.100.0.138]],[BB91C1BDF7AC40C,,[10.100.0.153]],[BB980D76E7AC40C,,[10.100.0.143]],[BB912BFDE7AC40C,,[10.100.0.149]],[BB906BFDE7AC40C,,[10.100.0.150]]]

```

---

<div class="post-metadata">

**Author:** ![mattmnx](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mattmnx](https://discuss.aerospike.com/u/mattmnx)\
**Post date:** [August 30, 2017, 6:29pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/6 "2017-08-30T18:29:31Z")

</div>

The weird thing is that on all the other nodes they work fine. There is no reference of a 127.0.0.1 address anywhere. The even stranger thing is that they were all installed at roughly the same time and using the same base config.

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [August 30, 2017, 8:53pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/7 "2017-08-30T20:53:27Z")

</div>

None of the nodes are advertising 127.0.0.1 (local loopback address) so this must be a bug in asadm.

Basically the host asadm uses defaults to 127.0.0.1 if node is provided and in this case it remains in the output as one of the hosts.

If you were to specify the host’s access address instead of using the default, this issue should go away:

```auto
asadm -h 10.100.0.136

```

---

<div class="post-metadata">

**Author:** ![mattmnx](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mattmnx](https://discuss.aerospike.com/u/mattmnx)\
**Post date:** [August 31, 2017, 12:08am UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/8 "2017-08-31T00:08:55Z")

</div>

Well here is a weird one… If I try that I get this, but not every time, sometimes it connects as I would expect. I’m definitely puzzled on this one.

```
asadm -h 10.100.0.136\
Aerospike Interactive Shell, version 0.1.5\
Found 1 nodes
Offline: 10.100.0.136:3000

Not able to connect any cluster.

Config files location: /root/.aerospike/

```

The config files location is incorrect also. I’ve looked at the running process and it’s running the correct config.

```
Aerospi+ ... Ssl 19:53 96:28 /usr/bin/asd --config-file /etc/aerospike/aerospike.conf

```

---

<div class="post-metadata">

**Author:** ![kporter](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/kporter/32/515_2.png) [@kporter](https://discuss.aerospike.com/u/kporter)\
**Post date:** [August 31, 2017, 1:18am UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/9 "2017-08-31T01:18:43Z")

</div>

> [@mattmnx](#):
>
> The config files location is incorrect also. I’ve looked at the running process and it’s running the correct config.

It is telling you where the config files for asadm are located.

> [@mattmnx](#):
>
> Well here is a weird one… If I try that I get this, but not every time, sometimes it connects as I would expect. I’m definitely puzzled on this one.

The connection attempt is probably timing out, or you may have exceeded `ulimit -n` or [proto-fd-max](http://www.aerospike.com/docs/reference/configuration#proto-fd-max) connections to this server.

---

<div class="post-metadata">

**Author:** ![Hemant\_Patre](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/hemant_patre/32/1121_2.png) [@Hemant\_Patre](https://discuss.aerospike.com/u/Hemant_Patre)\
**Post date:** [August 31, 2017, 10:38am UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/10 "2017-08-31T10:38:18Z")

</div>

Default value for seed IP is 127.0.0.1. asadm uses input seed node IP to get access-address for all Aerospike nodes, then it uses these access-addresses to connect actual Aerospike nodes. For seed node it uses access-address + input seed node address, so first it tries with access-address if that failed then seed address. In this case, for seed node (10.100.0.136) asadm have two IPs [10.100.0.136, 127.0.0.1] , when first failed to connect then only it tries second one. So it seems sometimes asadm is not able to connect 10.100.0.136, which you experienced when you provide this IP as a parameter to asadm.

As Kavin suggested this might be issue of proto-fd-max or ulimit -n exceeded. client\_connections stat can give number of open connection.

asadm -e “show statistics service like client\_connection” .

For proto-fd-max limit exceed, Aerospike log shows warning for dropping incoming connection.

---

<div class="post-metadata">

**Author:** ![mattmnx](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mattmnx](https://discuss.aerospike.com/u/mattmnx)\
**Post date:** [August 31, 2017, 2:02pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/11 "2017-08-31T14:02:37Z")

</div>

First of all, thanks to Hemant\_Patre and kporter for all the help!! 🍻 After looking info this further I found that for some reason we were running out of ports on the server which was causing the 127.0.0.1 issue as it could no longer connect to the 10.x.x.x address. I have increased the ports and the issue has subsided.

Thanks again for all the help, I was tearing my hair out on this one.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.aerospike.com/system/32/2274_2.png) [@system](https://discuss.aerospike.com/u/system)\
**Post date:** [September 6, 2017, 2:02pm UTC](https://discuss.aerospike.com/t/cluster-visibility-error-using-loopback-address/4539/12 "2017-09-06T14:02:45Z")

</div>

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.