How to handle net.netfilter.nf_conntrack_count max exceeded


#1

How to handle net.netfilter.nf_conntrack_count max exceeded

Problem

If you use netfilter / iptables, you may encounter dropped connections and this error message in syslog:

CRITICAL net.netfilter.nf_conntrack_count = 212912 (greater than 209715.2 [ net.netfilter.nf_conntrack_max = 262144 ])

Reason

Netfliter has got a set limit of connections it can handle. These connections also have set long-term timeouts. On rapid connection open-close or open-drop, netfilter can run out of connection limits in it’s conntrack table.

Limits set by default:

net.netfilter.nf_conntrack_max = 262144
net.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300

Fix

You can increase the max conntrack table size and decrease the fin_wait(), close_wait() and time_wait() to 60 seconds to see if that helps. If the connections timeout as opposed to being closed, you would need to adjust the other timeouts to a lower value as well.

Please note that conntrack in itself is not designed to be fast and growing it’s tables will mean more lookups and possible slowdowns in connections tracking.

To make the change, making a dynamic change with sysctl first on one node and proceeding to others once no adverse effects are observed is a good practice. Once confirmed stable, the change should be made permanent in /etc/sysctl.conf.

And as always, check in a dev environment before making changes to production.

Dynamic changes:

$ for i in close_wait fin_wait time_wait ; do sysctl -w net.netfilter.nf_conntrack_tcp_timeout_${i}=60 ; done
$ sysctl -w net.netfilter.nf_conntrack_max=524288

Revert change:

$ for i in close_wait fin_wait time_wait ; do sysctl -w net.netfilter.nf_conntrack_tcp_timeout_${i}=120 ; done
$ sysctl -w net.netfilter.nf_conntrack_max=262144

Make change permanent:

$ cat <<EOF > /etc/sysctl.conf
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_max = 524288
EOF

Keywords

netfilter, conntrack, limit

Timestamp

03/22/2018