Aerospike Cluster with multicast would be broken

cocoa_dev_maemae · November 10, 2017, 10:41am

Summary

I’d like to construct the aerospike cluster with 3 servers(=aerospike101,102,103), however, it does not go well.

Could you take an advice?

Conditions

aerospike version: aerospike server community edition v3.15.01
OS: CentOS 7.4.1708

Symptom

When the 3 aerospike servers(=aerospike101,102,103) are started initially, asadm shows that the cluster is normal. IP addresses are hidden with * because of the privacy.

[cocoamaemae@aerospike101 ~]$ asadm

Aerospike Interactive Shell, version 0.1.13

Found 3 nodes Online: *******(node1 ip address):3000, *******(node2 ip address):3000, *******(node3 ip address):3000

Admin> info network

Node Node Ip Build Cluster Cluster Cluster Principal Client Uptime . Id . . Size Key Integrity . Conns .

aerospike101:3000 *BB9CEDA634C9A50 *******:3000 C-3.15.0.1 3 7158AB0DB916 True BB9CEDA634C9A50 3 00:00:09

aerospike102:3000 BB9C2DA634C9A50 1*******:3000 C-3.15.0.1 3 7158AB0DB916 True BB9CEDA634C9A50 3 00:00:10

aerospike103:3000 BB902A5634C9A50 *******:3000 C-3.15.0.1 3 7158AB0DB916 True BB9CEDA634C9A50 4 00:00:11 Number of rows: 3

Once the construction is succeeded, however, the cluster would be broken after a few minutes passed. IP addresses are hidden with * because of the privacy.

[cocoamaemae@aerospike101 ~]$ asadm
Aerospike Interactive Shell, version 0.1.13

Found 1 nodes
Online:  *******:3000
Extra nodes in alumni list: ********(node2 ip address):3000, ********(node3 ip address):3000

Admin> info network
~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~
Node               Node                  Ip        Build   Cluster        Cluster     Cluster         Principal   Client     Uptime
                                    .                 Id                   .            .      Size            Key   Integrity                 .    Conns          .
aerospike101:3000   *BB9CEDA634C9A50   ******:3000   C-3.15.0.1         1   DEC994B610E8   True        BB9CEDA634C9A50        2   00:18:16
    Number of rows: 1

After the cluster was broken, the below message was confirmed in /etc/aerospike/aerospike.log

WARNING (as): (as.c:337) will not write PID file in new-style daemon mode
Nov 10 2017 06:45:05 GMT: INFO (fabric): (fabric.c:2520) fabric: node bb9ceda634c9a50 departed
Nov 10 2017 06:45:05 GMT: INFO (fabric): (fabric.c:935) fabric_node_disconnect(bb9ceda634c9a50)
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5520d848 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef57a0db88 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef55a0dac8 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef58a0d948 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef57e0d548 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef54a0d208 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5960db88 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef58e0dd08 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5560d588 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5740d148 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef56c0d948 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef53a0de48 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef54e0d5c8 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5420d508 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef55e0dac8 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef59a0d388 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5860d588 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5660de08 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5920df88 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5820d0c8 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5460d8c8 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5620db48 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef53e0da48 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1599) fc 0x7fef5700da08 is not in (node 0x7fef5994f008)->fc_hash
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:1159) fabric_node_destructor(0x7fef5994f008)
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:2219) 20: epoll : error, will close: fc 0x7fef4b80ddc8 fd 80 errno 11 signal {err:0, hup:0, rdhup:1}
Nov 10 2017 06:45:05 GMT: DETAIL (fabric): (fabric.c:2219) 25: epoll : error, will close: fc 0x7fef4b40d408 fd 81 errno 11 signal {err:0, hup:0, rdhup:1}

The aerospike.conf is like the blow. The only network setting is showed, the other settings are hidden

network {
    service {
        address any
        port 3000
        reuse-address
        access-address *******(node ip address)
        alternate-access-address *******(ip address entering into node)
    }

heartbeat {
    mode multicast
    multicast-group 239.1.1.1
    port 9918
    address *******(node ip address)

    interval 150
    timeout 10
}

fabric {
    address *******(node ip address)
    port 3001
}

info {
    address *******(node ip address)
    port 3003
}

}

The multicast connection surely has been already checked, and it is normal.

Why the aerospike cluster is broken?

FYI When using mesh, the aerospike cluster was not broken. It was normal.

kporter · November 10, 2017, 4:30pm

There are some common switch issues associated with multicast, see the multicast troubleshooting guide:

https://www.aerospike.com/docs/operations/troubleshoot/cluster#validate-multicast

pgupta · November 10, 2017, 9:27pm

Turn IGMP snooping off or turn on Querier on your core router. The initial multicast membership established lasts for 300 seconds. If IGMP snooping is on to prune the traffic away from unneeded switch segments, the member ship has to be renewed. Aerospike establishes first membership and per the mutlicast protocol respond to renewal requests. The core router’s querier has to generate the renewal requests if you want IGMP snooping on.

cocoa_dev_maemae · November 15, 2017, 9:01am

aerospike_nw

The current network configure is like the above. I’m not using any routers. There are just two L2 switches.

Currently, IP snooping is turning on to avoid the extra packets. If I turned off IP snooping, a persuasive influence occurred, so I hesitate to turn off it.

Could you tell me why IP snooping should be turned off?

pgupta · November 15, 2017, 2:07pm

This is interesting. You are running heartbeat over two network interfaces - so if one switch goes out, you don’t lose the cluster.

On each eth1 or eth2 by itself, you really don’t need IGMP snooping because there is nothing to prune here, since you are not going to a router, layer 2 does not need to snoop into layer 3 to build a local multicast routing table at layer 2 level.

However, I don’t know how to configure heartbeat to run in parallel over two network interfaces - and whether it can be done in the aerospike configuration file in the first place?

I will wait to see if someone else comments.

In the meantime, I see the two interfaces are bonded to bond0. You should turn igmp snooping off and see if the cluster stays intact.

lucien · November 20, 2017, 5:10pm

What’s the netwok bonding mode being used?

Are some of teh interfaces having more than 1 hop?

You could try bumping up mcast-ttl in aerospike config.

cocoa_dev_maemae · November 30, 2017, 9:10am

Finally, I turned the switch’s IGMP snooping off. And then, the cluster was not broken.

Thanks a lot.

system · December 6, 2017, 9:10am

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Aerospike cluster setup where multicast too powerfull Configuration	2	1926	January 5, 2015
Aerospike not clustering using multicast or mesh Installation	17	4030	November 27, 2017
Servers split into two clusters when deploying one cluster with multicast mode Planning	1	1740	February 1, 2018
Aerospike replica err and asadm strange output on cluster status	3	1497	May 19, 2017
Cluster Visibility Error using loopback address Configuration	11	2624	September 6, 2017

Aerospike Cluster with multicast would be broken

Related Topics