The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.
How to handle net.netfilter.nf_conntrack_count max exceeded
Problem
If you use netfilter / iptables, you may encounter dropped connections and this error message in syslog:
CRITICAL net.netfilter.nf_conntrack_count = 212912 (greater than 209715.2 [ net.netfilter.nf_conntrack_max = 262144 ])
Reason
Netfliter has got a set limit of connections it can handle. These connections also have set long-term timeouts. On rapid connection open-close or open-drop, netfilter can run out of connection limits in its conntrack table.
Limits set by default:
net.netfilter.nf_conntrack_max = 262144
net.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
Fix
You can increase the max conntrack table size and decrease the fin_wait(), close_wait() and time_wait() to 60 seconds to see if that helps. If the connections timeout as opposed to being closed, you would need to adjust the other timeouts to a lower value as well.
Please note that conntrack in itself is not designed to be fast and growing its tables will mean more lookups and possible slowdowns in connections tracking.
To make the change, making a dynamic change with sysctl first on one node and proceeding to others once no adverse effects are observed is a good practice. Once confirmed stable, the change should be made permanent in /etc/sysctl.conf.
And as always, check in a dev environment before making changes to production.
Dynamic changes:
$ for i in close_wait fin_wait time_wait ; do sysctl -w net.netfilter.nf_conntrack_tcp_timeout_${i}=60 ; done
$ sysctl -w net.netfilter.nf_conntrack_max=524288
Revert change:
$ for i in close_wait fin_wait time_wait ; do sysctl -w net.netfilter.nf_conntrack_tcp_timeout_${i}=120 ; done
$ sysctl -w net.netfilter.nf_conntrack_max=262144
Make change permanent:
$ cat <<EOF > /etc/sysctl.conf
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_max = 524288
EOF
Keywords
netfilter, conntrack, limit
Timestamp
03/22/2018