FAQ - Is it possible for tcpdump to impact Aerospike transactions?

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

FAQ - Is it possible for tcp dump to impact Aerospike transactions?

Detail

When using certain packet-tracing or system monitoring tools, the interfaces may be put into promiscuous mode. In this mode, all packets are subject to filtering. One of the popular tools that has to put interfaces into the promiscuous mode to function is tcpdump.

During tcpdump, keep in mind that you may experience increased network latencies or dropped max throughput on the network interfaces.

Outside of running tcpdump, there are other monitoring tools which may be putting interfaces in promiscuous mode from time to time while gathering interface stats. Such monitoring tools must be avoided in order to ensure that the interfaces won’t exhibit spikes in latency/throughput issues. The below answer discusses how to check if interfaces are and whether they have been put in promiscuous mode.

Answer

The easiest way to check if an interface is currently in promisc mode is by running the following command:

[root@localhost ~]# netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0     2075      0      0      0     1370      0      0      0 BPRU
lo        16436   0     1985      0      0      0     1985      0      0      0 LRU

The flags section contains the P flag, which means the interface is currently in promiscuous mode.

Alternatively, this can be checked using ifconfig, see the flags section contains PROMISC flag printed:

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:1.2.3.4  Bcast:255.255.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1                     
          RX packets:22685771 errors:0 dropped:83424 overruns:0 frame:0
          TX packets:13461 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1604651517 (1.4 GiB)  TX bytes:1475694 (1.4 MiB)

To check for historical states, have a look at dmesg:

$ dmesg |grep promisc
Nov 14 07:53:49 test kernel: device eth0 entered promiscuous mode
Nov 14 07:54:03 test kernel: device eth0 left promiscuous mode
Nov 14 07:54:49 test kernel: device eth0 entered promiscuous mode
Nov 14 07:55:03 test kernel: device eth0 left promiscuous mode
Nov 14 07:55:49 test kernel: device eth0 entered promiscuous mode
Nov 14 07:56:03 test kernel: device eth0 left promiscuous mode

As can be seen, the device eth0 has entered and exited promiscuous mode. This line is seen repeating every minute, indicating this is done by either a cron job or a monitoring software. Cron and any monitoring processes must be checked to ensure the interface is NOT put in promiscuous mode. In this example, every time the interface enters promiscuous mode, our machine has max throughput decreased by about 30%.

Promiscuous mode may be forced to the ‘off’ position (although it will go back to ‘on’ if requested to do so):

# ip link set [interface] promisc off

Keywords

TCPDUMP PROMISCUOUS MODE PROMISC LATENCY THROUGHPUT

Timestamp

September 2020