Using iptables to simulate network issues in an Aerospike cluster

Using iptables to simulate network issues in an Aerospike cluster

Context

During your testing of how Aerospike reacts during fault conditions, you may wish to simulate network issues, either between your clients and your Aerospsike cluster, or the intra-cluster communication between one or many of the nodes in your Aerospike cluster.

Before setting up this test scenario, it is wise to take a moment to consider what a machine (whether that be client or server) will see during a network failure, and trying to replicate that the best we can by using a local firewall (in this case iptables) on the machines in question

Ports

The best way to simulate a network failure is to block incoming/outgoing traffic based on the ports Aerospike uses. This allows us to limit only Aerospike traffic, and keep other services like SSH available. By default, Aerospike makes use of the following ports:

Name Port (TCP) TLS Port (TCP) Description
Service 3000 4333 Used on Aerospike cluster nodes for incomming traffic from client machines. It is also used for incoming XDR connections from other clusters
Fabric 3001 3011 Used for the Aerospike cluster nodes to share data between themselves. This carries things such as replication traffic and migrations in case of cluster changes
Heartbeat (mesh) 3002 3012 Used between Aerospike cluster nodes to check that other nodes are still there. By default this is every 150ms

Note: For the rest of this guide we are going to assume our setup is not using TLS. If you are using TLS for any/all of your communications, you will need to subtitute the above port numbers for the ones we use here

Based on the above, we can see that in order to simulate client-server network issues, we need to block port 3000, and to simulate intra-cluster network issues we need to block ports 3001/3002 between server nodes

DROP or REJECT?

iptables allow us the option to DROP or REJECT a packet. Both of these options will prevent the packet from reaching the destination specified in the rule, but they differ in what is sent back to the source. In the case of a DROP rule, the packet is silently discarded and no reply is sent back to the source. With a REJECT rule, the packet is still discarded, but an error is also returned to the source, in the form on an ICMP error packet. These packets are designed to allow networked devices to understand what is happening on the network, and receiving an ICMP error back is what we would expect to see. We can ask iptables to return various different ICMP error packets such as icmp-net-unreachable, icmp-host-unreachable, icmp-port-unreachable or icmp-proto-unreachable. For our testing it does not matter which ICMP error packet is returned (as long as we receive one), so we can leave it as the default, which is icmp-port-unreachable.

Dealing with Established connections

It is not uncommon for iptables to already be configured to allow previously established connections through, especially on the INPUT chain. This means that iptables rules only need to dictate whether a new connection is allowed to be established or not, and then once it is estabished, it is trusted and allowed through. Because the vast majority of packets will relate to a previously established connection, to be as efficient as possible, the rule to allow through these packets is often the very first rule.

When implementing our new rules, we need to consider that Aerospike will re-use existing connections wherever possible, so it is likely that we will have existing connections already open and being reused. This means we need to make sure that our blocks are before any rules that allow established connections through. Because this is less likely to be a configured on the OUTPUT chain of the source machine, where possible it is easier to do the block here rather than on the INPUT chain of the destination machine

Examples

Blocking Client to Cluster communication

On the client machine(s), add the following rule

iptables -A OUTPUT -p tcp --dport 3000 -J REJECT

Blocking Client to Single Cluster Node communication

On the client machine(s), add the following rule

iptables -A OUTPUT -p tcp -d 10.10.10.10 --dport 3000 -J REJECT

Blocking Intra-cluster communication to a Single Cluster Node

On the cluster node you wish to block communication to, add the following rule

iptables -A INPUT -p tcp --dport 3001:3002 -J REJECT

Note: If this machine already has a rule to allow ESTABLISHED connections on it’s INPUT chain, the above rule will need to be added before the rule to allow ESTABLISHED connections. Explaining iptables configuration is beyond the scope of this guide, but in this particular instance you will most likely want something similar to this rule instead

iptables -I INPUT 1 -p tcp --dport 3001:3002 -J REJECT

Removing rules afterwards

Once you have finished your testing, you can remove the rules you added by running the same command, but with the -D option instead of the -A option. For example, to remove our Cluster to Cluster communication block you would run the following

iptables -D OUTPUT -p tcp --dport 3000 -J REJECT

Notes

  • WARNING It is strongly recommended that the above is only run in a test environment. Also, adding iptables rules without fully understanding them could result in either exposing more of your server than intended, or blocking more than you intended (including the SSH session you are using to connect to the server). If in any doubt, please check with your security/network teams first

Keywords

IPTABLES CLIENT SERVER BLOCK

Timestamp

March 2021

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.