Cluster Will not Form Due to Multiple Network Interfaces

Cluster Will not Form Due to Multiple Network Interfaces

Problem Description

  • Nodes unable to join cluster
  • Some nodes seem to be able to form a sub-cluster
  • Depending on the order in which nodes are started, different ones join and others do not

Errors may be seen in the Aerospike logs, for the nodes which are not joining the cluster:

INFO (clustering): (clustering.c:5286) paxos round timed out for proposal id e4434b820bb8:0
INFO (clustering): (clustering.c:5988) sent cluster join request to e4434b815ddc
WARNING (clustering): (clustering.c:4313) ignoring paxos accepted from node e4434b820bb8 - it is not in acceptor list

Aerospike admin tool (asadm) info may show errors like this:

$ asadm --port 8080

Found 2 nodes
Online: 10.0.0.1:8080, 10.0.0.3:8080
Cluster Visibility error (Please check services list): 10.0.0.3:8080
Extra nodes in alumni list: 10.0.0.1:8080

Explanation

When dealing with clustering issues, and Aerospike nodes not consistently joining a cluster, one place to initially check would be the network interfaces binding configurations. When a node has multiple interfaces configured and Aerospike Fabric or Heartbeat address binding are set to “any” (default value), these connections would span accross these interfaces. Issues can arise if one of these interfaces is a special purpose interface like “iDRAC” or is connecting to different networks (ie: different routing or acl rules, etc). The Aerospike Heartbeat/Fabric traffic could in such cases take undesired routes and fail or timeout.

Let us look at a specific example with an iDRAC interface on a bare metal server.

From a shell, on Aerospike nodes, check all IP addresses associated on all network devices, using (ip):

$ ip addr

Which, in this example, returned (partial output):

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:81:58:88 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    4286398039 41839366 0       0       0       237313  
    TX: bytes  packets  errors  dropped carrier collsns 
    5393030297 41988059 0       0       0       0       
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:43:4b:81:58:8a brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    0          0        0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    0          0        0       0       0       0       
4: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 54:48:10:fc:e7:17 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    30355040   531978   0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    47303962   634811   0       0       0       0     

There are multiple network interfaces shown here:

  • eth0
  • eth1
  • idrac

Typical default naming convention for LAN connections are eth0, eth1, et. al.

One thing that stands out in this case is that there is an additional network interface, (iDRAC) (the Dell Remote Access Controller).

The following log samples also confirms an address binding of “any” is in use. We can see that fabric, mesh heartbeat, and service network are bound to all active interfaces ( ipv4 : 0.0.0.0 and ipv6 : [::] )

Mar 24 2021 13:18:09 GMT: INFO (hb): (hb.c:6972) initializing mesh heartbeat socket: 0.0.0.0:3002
Mar 24 2021 13:18:09 GMT: INFO (fabric): (socket.c:815) Started fabric endpoint 0.0.0.0:3001
Mar 24 2021 13:18:09 GMT: INFO (fabric): (socket.c:815) Started fabric endpoint [::]:3001
Mar 24 2021 13:18:09 GMT: INFO (hb): (hb.c:7001) mtu of the network is 1500
Mar 24 2021 13:18:09 GMT: INFO (hb): (socket.c:815) Started mesh heartbeat endpoint 0.0.0.0:3002
Mar 24 2021 13:18:09 GMT: INFO (hb): (socket.c:815) Started mesh heartbeat endpoint [::]:3002
Mar 24 2021 13:18:09 GMT: INFO (nsup): (nsup.c:187) starting namespace supervisor threads
Mar 24 2021 13:18:09 GMT: INFO (service): (service.c:936) starting reaper thread
Mar 24 2021 13:18:09 GMT: INFO (service): (socket.c:815) Started client endpoint 0.0.0.0:3000
Mar 24 2021 13:18:09 GMT: INFO (service): (socket.c:815) Started client endpoint [::]:3000
Mar 24 2021 13:18:09 GMT: INFO (service): (service.c:197) starting accept thread
Mar 24 2021 13:18:09 GMT: INFO (info-port): (thr_info_port.c:298) starting info port thread
Mar 24 2021 13:18:09 GMT: INFO (info-port): (socket.c:815) Started info endpoint 0.0.0.0:3003
Mar 24 2021 13:18:09 GMT: INFO (info-port): (socket.c:815) Started info endpoint [::]:3003

Solution

  • Check networking stanzas in the (Aerospike configuration file).
  • Ensure that the network service, fabric, info, and heartbeat bindings are configured such that Aerospike will use a particular network interface.

In the previous example, these stanzas specified address any:

network {
	service {
		address any
    ...
	}
	fabric {
		address any
    ...
	}
	info {
		address any
	}
	heartbeat {
    address any
		...
	}
}

To resolve this issue, set address eth0:

network {
	service {
		address eth0
    ...
	}
	fabric {
		address eth0
    ...
	}
	info {
		address eth0
	}
	heartbeat {
    address eth0
		...
	}
}

Alternatively we can name specific IPs to bind to, which are particular to the network interface(s) we wish to use. In case of tls network, one could alternatively use tls-address for the interface binding.

Notes

It is a good practice to understand the network being used by each interface on an active node. This will help avoid unpredictable behaviors and clustering or netwok latency.

Keywords

CLUSTERING HEARTBEAT PAXOS IDRAC UNSTABLE

Timestamp

June 2021

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.