The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
FAQ - Aerospike and Network Bonding
Detail
In many cases customers may reach a network bottleneck when using only one network card. Network bandwidth for a particular node may be extended through network bonding and use of multiple network cards.
Answer
Aerospike server will broadcast only a single IP to an aerospike client. The network interface used by the client can be controlled by specifying an access-address in the aerospike.conf. In certain cases, a customer may need to increase the network bandwidth provided by a single card and configure network bonding to use multiple network cards.
Here are the different modes of bonding available in Linux:
There can be 7 possible “modes” for bonded interfaces. These modes determine the way in which traffic sent out of the bonded interface is actually dispersed over the real interfaces. Modes 0, 1, and 2 are by far the most commonly used among them.
Mode 0 (balance-rr)
This mode transmits packets in a sequential order from the first available slave through the last. If two real interfaces are slaves in the bond and two packets arrive destined out of the bonded interface the first will be transmitted on the first slave and the second frame will be transmitted on the second slave. The third packet will be sent on the first and so on. This provides load balancing and fault tolerance.
Mode 1 (active-backup)
This mode places one of the interfaces into a backup state and will only make it active if the link is lost by the active interface. Only one slave in the bond is active at an instance of time. A different slave becomes active only when the active slave fails. This mode provides fault tolerance.
Mode 2 (balance-xor)
Transmits based on XOR formula. (Source MAC address is XOR’d with destination MAC address) modula slave count. This selects the same slave for each destination MAC address and provides load balancing and fault tolerance. Please also see mode 4 for addtional info on XOR policy and the xmit_hash_policy.
Mode 3 (broadcast)
This mode transmits everything on all slave interfaces. This mode is least used (only for specific purpose) and provides only fault tolerance.
Mode 4 (802.3ad)
This mode is known as Dynamic Link Aggregation mode. It creates aggregation groups that share the same speed and duplex settings. This mode requires a switch that supports IEEE 802.3ad Dynamic link. Slave selection for outgoing traffic is done according to the transmit hash policy, which may be changed from the default simple XOR policy via the xmit_hash_policy option. Note that not all transmit policies may be 802.3ad compliant, particularly inregards to the packet mis-ordering requirements of section 43.2.4 of the 802.3ad standard. Differing peer implementations will have varying tolerances for noncompliance.
This mode requires additional considerations:
"Linux, by default, uses MAC address (layer 2) xmit policy for LACP. Since network traffic could be on different subnets, the hashing may be incorrect (only using 1 interface to transmit). We need to change this to layer3+4. This can be achieved by editing the xmit_hash_policy :
echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy and restart the network interface."
Additional info on xmit_hash_policy:
xmit_hash_policy — Selects the transmit hash policy used for slave selection in balance-xor and 802.3ad modes. 1 or layer3+4 — Uses upper layer protocol information (when available) to generate the hash. This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves.
The formula for unfragmented TCP and UDP packets used is:
((<source> <port> <XOR> <destination>) <XOR>
((<source> <IP> <XOR> <destination> <IP>) AND 0xffff)
<modulo slave count>
For fragmented TCP or UDP packets and all other IP protocol traffic, the source and destination port information is omitted. For non-IP traffic, the formula is the same as the layer2 transmit hash policy.
This policy intends to mimic the behavior of certain switches; particularly, Cisco switches with PFC2 as well as some Foundry and IBM products.
The algorithm used by this policy is not 802.3ad compliant.
Mode 5 (balance-tlb)
This is called as Adaptive transmit load balancing. The outgoing traffic is distributed according to the current load and queue on each slave interface. Incoming traffic is received by the current slave.
Mode 6 (balance-alb)
This is Adaptive load balancing mode. This includes balance-tlb + receive load balancing (rlb) for IPV4 traffic. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the server on their way out and overwrites the src hw address with the unique hw address of one of the slaves in the bond such that different clients use different hw addresses for the server.
Notes
References:
Keywords
network bonding round-robin balance-xor
Timestamp
12/19/2016