Connection Tracking conntrack

Connection tracking (the nf_conntrack subsystem, historically ip_conntrack) is the part of netfilter that makes the firewall stateful. Without it, every packet is judged in isolation; with it, the kernel remembers flows — it records the first packet of every connection in a hash table keyed by a tuple (source/destination address, source/destination port, and protocol), then recognizes every subsequent packet as belonging to that flow and labels it with a state (NEW, ESTABLISHED, RELATED, or — as a non-stored match bit — INVALID). This is the basis of both stateful firewalling (“allow replies to connections I started, without an explicit rule for the return traffic”) and NAT (which stores its address rewrite in the conntrack entry so both directions are translated consistently). Everything below pins to Linux 6.12 LTS source (net/netfilter/nf_conntrack_core.c, as of 2026-06).

Conntrack registers as a netfilter hook function (nf_conntrack_in, at priority -200 on PRE_ROUTING/LOCAL_OUT) — it is a consumer of the hook framework, so read The Netfilter Framework and Hooks first for where it sits in the pipeline. The address-rewriting layer that builds on it is Network Address Translation NAT.

Mental Model: One Entry, Two Tuples, Both Directions

A conntrack entry is a single struct nf_conn object that represents one bidirectional flow but stores it as two tuples — the original direction (the way the connection was opened) and the reply direction (the inverse). Both tuples are inserted into the same hash table, so a packet arriving from either side finds the same nf_conn. The genius of the design is the inverse tuple: for a TCP connection 10.0.0.1:5000 → 1.1.1.1:443, the original tuple is {src 10.0.0.1:5000, dst 1.1.1.1:443, TCP} and the reply tuple is its inverse {src 1.1.1.1:443, dst 10.0.0.1:5000, TCP}. When the SYN-ACK comes back, its tuple equals the reply tuple of the existing entry, so the kernel instantly knows it is the return traffic of a known flow — and labels it ESTABLISHED rather than treating it as a brand-new connection that a firewall would have to explicitly permit.

flowchart TB
  P1["First packet:<br/>SYN 10.0.0.1:5000 -> 1.1.1.1:443"] --> LOOK{"tuple in<br/>hash table?"}
  LOOK -->|"no match"| NEW["init_conntrack():<br/>allocate nf_conn<br/>ctinfo = IP_CT_NEW"]
  NEW --> STORE["store ORIGINAL tuple<br/>+ inverted REPLY tuple<br/>(unconfirmed)"]
  STORE --> CONF["packet survives rules<br/>confirm hook inserts<br/>both tuples into table"]
  P2["Reply packet:<br/>SYN-ACK 1.1.1.1:443 -> 10.0.0.1:5000"] --> LOOK2{"tuple in<br/>hash table?"}
  LOOK2 -->|"matches REPLY tuple"| EST["set IPS_SEEN_REPLY<br/>ctinfo = ESTABLISHED_REPLY<br/>future packets = ESTABLISHED"]

The lifecycle of one connection in conntrack. What it shows: the first packet finds no matching tuple, so a new nf_conn is allocated with the original tuple and its computed inverse (reply) tuple, marked IP_CT_NEW, and only confirmed (inserted into the live hash) after it survives the ruleset; the reply packet’s tuple matches the stored reply tuple, flipping the connection to seen-reply / ESTABLISHED. The insight to take: conntrack stores a flow as two tuples precisely so a single hash lookup on any packet — from either side — lands on the same entry, which is what makes stateful matching and consistent two-way NAT possible.

The Organising Fact — A Fixed-Size Hash Table With Exactly Two Failure Modes

Strip away the protocol trackers, the helpers and the netlink plumbing and what remains is small enough to state in one sentence: conntrack is a hash table of tuples with a hard maximum, and every operational problem anyone has ever had with it is either (a) the table filling up, or (b) two CPUs racing to insert the same flow. Learning those two failure modes, and the four counters that distinguish them, is most of what operating conntrack consists of.

	Failure mode (a): the table fills	Failure mode (b): the insert race
What happens	nf_conntrack_count reaches nf_conntrack_max; a new flow cannot be allocated	Two CPUs build unconfirmed entries for the same tuple; only one can be inserted
Where in the code	__nf_conntrack_alloc() → early_drop() → -ENOMEM	__nf_conntrack_confirm() finds an equal key → nf_ct_resolve_clash()
Who it hits	Any high-flow-count box: NAT gateways, busy Kubernetes nodes, load balancers	Anything that sends two datagrams of one tuple at once — canonically a stub resolver’s back-to-back A and AAAA queries
Symptom	Intermittent connection failures; nf_conntrack: table full, dropping packet in dmesg	A single dropped packet, and therefore a retransmit delay — in the famous case, a 5-second DNS stall
Counter that rises	drop, early_drop	insert_failed and drop, if unresolved; clashres if resolved
The fix	nf_conntrack_max / nf_conntrack_buckets, and shorter timeouts	Nothing to tune — the kernel absorbs it, if the protocol sets allow_clash and the kernel is new enough
The tell	nf_conntrack_count near nf_conntrack_max	High clashres with insert_failed == 0 (healthy) vs non-zero insert_failed (not)

The two failure modes, side by side. What it shows: they are diagnosed from completely different counters and fixed by completely different means, yet both present to a user as “the network is intermittently broken”. The insight to take: before tuning anything, read /proc/net/stat/nf_conntrack and nf_conntrack_count. If drop/early_drop are climbing, you have a capacity problem. If insert_failed is climbing, you have a race problem — and raising nf_conntrack_max will not help it by one packet.

Almost everything else in this note exists to make one of those two rows comprehensible: the timeouts, because they decide how fast the table drains; the states and tuples, because they decide what “the same flow” means; the confirm deferral, because it is what creates the race window in the first place.

Boundary with Network Address Translation NAT

These two notes meet at nf_ct_resolve_clash() and it is worth being explicit about who owns what, because the same function is discussed in both. Network Address Translation NAT owns the NAT-level story: why DNAT rules that pick different backends turn a benign clash into an unmergeable one, how --random-fully and port allocation interact with it, and the source-port-exhaustion mechanisms (nf_nat_used_tuple_harder(), nf_nat_used_tuple_new()) that are not about clashes at all. This note owns the conntrack-level story: why the race window exists, exactly what the confirm path does when it loses, the IPS_NAT_CLASH hidden-entry trick, which protocols opt in, and what each /proc/net/stat/nf_conntrack column actually means. Read the NAT note for “why does my DNAT setup cause this”; read this one for “what is the kernel doing about it”. The clash race predates NAT and happens without it — NAT merely makes it far more likely to be unmergeable.

The Tuple — What Identifies a Flow

The key is struct nf_conntrack_tuple (include/net/netfilter/nf_conntrack_tuple.h), whose source comment states it plainly: *“A tuple' is a structure containing the information to uniquely identify a connection. ie. if two packets have the same tuple, they are in the same connection; if not, they are not."* It is split into a **manipulable** part (src — what NAT may rewrite) and a **fixed** part (dst`):

struct nf_conntrack_tuple {
	struct nf_conntrack_man src;   /* manipulable: src addr (u3), src port (u), l3num */
	struct {
		union nf_inet_addr u3;     /* dst address */
		union { __be16 all; struct { __be16 port; } tcp; /* ...udp/icmp/sctp/gre... */ } u;
		u_int8_t protonum;         /* L4 protocol: TCP/UDP/ICMP/... */
		u_int8_t dir;              /* direction (original vs reply) */
	} dst;
};

So the identifying fields are the classic 5-tuple — source IP, source port, destination IP, destination port, and the Layer-4 protocol number — plus the Layer-3 protocol (l3num, e.g. AF_INET/AF_INET6) and a direction byte. For protocols without ports (ICMP, GRE) the u union holds a protocol-specific identifier instead: ICMP uses {type, code, id}, GRE uses a key. The split into src (manipulable) and dst (fixed) exists for NAT: SNAT rewrites src, DNAT rewrites dst, and conntrack records what the rewrite was so it can be reversed on the reply.

The tuple, field by field

Field	C type	Part	Carries	Rewritten by
src.u3	union nf_inet_addr	manipulable	Source address (IPv4 in, IPv6 in6)	SNAT / masquerade
src.u.all	__be16	manipulable	Source port (TCP/UDP/SCTP/DCCP/UDP-Lite), or ICMP id, or GRE key	SNAT source-port allocation
src.l3num	u_int16_t	manipulable struct, but never rewritten	Layer-3 protocol: AF_INET (2) or AF_INET6 (10)	—
dst.u3	union nf_inet_addr	fixed	Destination address	DNAT / redirect
dst.u.all	__be16	fixed	Destination port, or ICMP {type, code}, or GRE key	DNAT destination-port rewrite
dst.protonum	u_int8_t	fixed	Layer-4 protocol number — 6 TCP, 17 UDP, 1 ICMP, 47 GRE, 132 SCTP	—
dst.dir	u_int8_t	fixed	IP_CT_DIR_ORIGINAL (0) or IP_CT_DIR_REPLY (1)	—

The tuple’s fields and the src/dst split. What it shows: the “5-tuple” is really a 7-field structure — the classic five plus the L3 family and a direction byte — and it is deliberately partitioned into a half NAT may rewrite and a half it may not. The insight to take: the split is not cosmetic. struct nf_conntrack_man (the src half) is a separate type precisely so the NAT code can be handed “the part you are allowed to change” as a typed object. Note that dst being “fixed” refers to the original-direction tuple: DNAT rewrites the destination of the original direction, but it does so by altering the reply tuple’s src — which is the same field, seen from the other side. The two-tuple design means every rewrite is expressible as a change to a manipulable half.

Two consequences fall straight out of this layout. First, ICMP is trackable at all because src.u/dst.u are unions rather than ports: an ICMP echo’s id field plays the role of a source port, and the {type, code} pair the role of a destination port, so an echo request and its reply invert into each other exactly like a TCP flow. Second, GRE needs a call key — which is why PPTP requires a conntrack helper to learn the key from the control channel before the data tunnel can be tracked.

The States

The state a packet is given is the enum ip_conntrack_info value (ctinfo) stored on the skb via nf_ct_set(skb, ct, ctinfo). The enum (include/uapi/linux/netfilter/nf_conntrack_common.h) is:

enum ip_conntrack_info {
	IP_CT_ESTABLISHED,       /* 0: part of an established connection */
	IP_CT_RELATED,           /* 1: related to an existing connection */
	IP_CT_NEW,               /* 2: started a new connection to track */
	IP_CT_IS_REPLY,          /* 3: >= this indicates reply direction */
	IP_CT_ESTABLISHED_REPLY = IP_CT_ESTABLISHED + IP_CT_IS_REPLY,
	IP_CT_RELATED_REPLY = IP_CT_RELATED + IP_CT_IS_REPLY,
	IP_CT_UNTRACKED = 7,     /* (in-kernel) deliberately not tracked */
};

How ctinfo is computed lives in resolve_normal_ct() (nf_conntrack_core.c), and reading it is the clearest possible explanation of the states:

if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY) {
	ctinfo = IP_CT_ESTABLISHED_REPLY;     /* matched the reply tuple */
} else {
	unsigned long status = READ_ONCE(ct->status);
	if (likely(status & IPS_SEEN_REPLY))
		ctinfo = IP_CT_ESTABLISHED;       /* we've seen both directions */
	else if (status & IPS_EXPECTED)
		ctinfo = IP_CT_RELATED;           /* expected by a helper */
	else
		ctinfo = IP_CT_NEW;               /* original dir, no reply yet */
}

• NEW — the packet matches a conntrack entry that has only ever seen traffic in the original direction (no reply yet). In practice this is the first packet of a flow: resolve_normal_ct finds no existing tuple, calls init_conntrack() to allocate the nf_conn, and the entry has not yet seen a reply, so ctinfo is IP_CT_NEW.
• ESTABLISHED — the connection has seen traffic both ways. The IPS_SEEN_REPLY status bit is set the first time a packet matches the reply tuple (in nf_conntrack_in, test_and_set_bit(IPS_SEEN_REPLY_BIT, ...)). After that, both directions are ESTABLISHED. The comment in the source is exact: “Once we’ve had two way comms, always ESTABLISHED.”
• RELATED — the packet belongs to a different flow that is logically connected to an existing one. The classic case is an ICMP “destination unreachable” error carrying the headers of a tracked flow, or an FTP data connection on a separate port that the FTP control connection’s helper predicted. RELATED is set when the matched entry has IPS_EXPECTED (it was created from an expectation — see Helpers below).
• INVALID — not a stored state. This is the trap. INVALID is not a value in enum ip_conntrack_info; it is a match bit (NF_CT_STATE_INVALID_BIT = 1 << 0, in nf_conntrack_common.h) that the iptables/nftables ctstate match reports when a packet could not be associated with any tracked connection. In nf_conntrack_in(), the paths that bump the per-namespace invalid statistic and return NF_ACCEPT without attaching a conntrack to the skb (e.g. a packet whose L4 header cannot be parsed, a checksum failure when nf_conntrack_checksum is on, or a TCP segment that the TCP tracker judges out-of-window) are what userspace rules see as INVALID. So NEW/ESTABLISHED/RELATED are ctinfo values carried on the skb; INVALID and UNTRACKED are ctstate match bits the firewall tests — a distinction the kernel headers make crisp but secondary docs routinely blur.

A typical stateful firewall ruleset is therefore: ct state established,related accept (let replies and related traffic through with no per-service rule), ct state new rules to decide what new connections are allowed, and ct state invalid drop to discard packets that fit no flow.

The five states as a table

State as a rule sees it	Kernel representation	ctstate match bit	What puts a packet here
new	ctinfo = IP_CT_NEW (2)	1 << 3 = 8	Original direction, entry exists but IPS_SEEN_REPLY is clear. In practice the first packet of a flow — resolve_normal_ct() missed the hash and called init_conntrack(). Also any later original-direction packet while the reply is still outstanding
established	IP_CT_ESTABLISHED (0) or IP_CT_ESTABLISHED_REPLY (3)	1 << 1 = 2 for both	Reply direction always; original direction once IPS_SEEN_REPLY is set. The source comment is exact: “Once we’ve had two way comms, always ESTABLISHED”
related	IP_CT_RELATED (1) or IP_CT_RELATED_REPLY (4)	1 << 2 = 4 for both	The matched entry carries IPS_EXPECTED — it was created from a helper’s expectation (FTP data channel, SIP media) — or it is an ICMP error that nf_conntrack_handle_icmp() associated with a tracked flow
invalid	no ctinfo at all; skb->_nfct is NULL	1 << 0 = 1	Three code paths in nf_conntrack_in(), each bumping the invalid stat and returning NF_ACCEPT untracked (see below)
untracked	ctinfo = IP_CT_UNTRACKED (7), ct pointer NULL	1 << 6 = 64	A NOTRACK target / ct state untracked rule in the raw table set it before conntrack ran; nf_conntrack_in() returns NF_ACCEPT immediately without touching the table

The state table. What it shows: only three of the five are ctinfo values stored on the packet. invalid is the absence of a conntrack and untracked is a deliberate opt-out — both are conditions the rule engine tests, not states the kernel stores. The insight to take: this is the single most-blurred distinction in conntrack documentation, and it has a practical edge. Because an INVALID packet has no nf_conn attached, it also gets no NAT applied — so a packet your ruleset lets through as INVALID on a masquerading gateway leaves with an un-rewritten source address. That is why ct state invalid drop is standard practice rather than mere tidiness.

The match-bit column repays a second look. NF_CT_STATE_BIT(ctinfo) is defined as 1 << ((ctinfo) % IP_CT_IS_REPLY + 1) (nf_conntrack_common.h:38, and IP_CT_IS_REPLY is 3). Taking ctinfo modulo 3 is what folds IP_CT_ESTABLISHED_REPLY (3) back onto IP_CT_ESTABLISHED (0) and IP_CT_RELATED_REPLY (4) back onto IP_CT_RELATED (1). That one modulo is why ct state established accept matches traffic in both directions with a single rule — the kernel keeps the direction in ctinfo for its own use, and erases it before the firewall sees it.

The three roads to INVALID

INVALID is a bucket, and knowing which of the three paths produced it changes the diagnosis entirely. All three live in nf_conntrack_in() (nf_conntrack_core.c:2008 onward, v6.12), and all three do the same two things: NF_CT_STAT_INC_ATOMIC(state->net, invalid) and return NF_ACCEPT with nothing attached to the skb.

1. get_l4proto() returned dataoff <= 0. The packet could not be parsed down to a transport header — a truncated header, an unreassembled fragment on a path where defragmentation did not run, an IPv6 extension-header chain that does not terminate. Diagnosis: malformed or fragmented traffic.
2. resolve_normal_ct() returned without attaching a conntrack. nf_ct_get(skb, &ctinfo) comes back NULL. Diagnosis: this is the “could not be associated with any connection” case proper.
3. The protocol tracker rejected the packet — nf_conntrack_handle_packet() returned <= 0. For TCP this is the big one: the segment fell outside the tracked window, so nf_conntrack_tcp_packet() refused it. Note the special case immediately above it: a return of -NF_REPEAT means the TCP tracker saw a SYN reopening a closed connection, and the code goto repeats to build a fresh entry instead of marking anything invalid. Diagnosis: window tracking — reach for nf_conntrack_tcp_be_liberal.

Only path 3 responds to nf_conntrack_tcp_be_liberal=1, which restricts INVALID to out-of-window RST segments only (nf_conntrack-sysctl.rst: “If it’s non-zero, we mark only out of window RST segments as INVALID”), and its narrower sibling nf_conntrack_tcp_ignore_invalid_rst=1, which stops marking out-of-window RSTs invalid at all. Both read 0 — the documented defaults — on the revision host. A fourth relevant knob, nf_conntrack_tcp_loose, defaults to 1 and controls something different: whether conntrack will adopt a mid-stream connection it never saw the handshake of. Set it to 0 and a flow that was established before the firewall came up becomes INVALID on its next packet rather than being picked up as NEW.

stateDiagram-v2
    direction LR
    [*] --> Untracked: raw table NOTRACK —<br/>nf_conntrack_in returns<br/>NF_ACCEPT immediately
    Untracked --> [*]

    [*] --> Invalid: parse failed / no ct attached /<br/>protocol tracker rejected —<br/>no nf_conn, so NO NAT applied
    Invalid --> [*]

    [*] --> New: first original-direction packet;<br/>init_conntrack allocates nf_conn;<br/>IPS_SEEN_REPLY clear
    New --> New: more original-direction<br/>packets, still no reply
    New --> Unconfirmed: entry exists but is<br/>NOT yet in the hash table
    Unconfirmed --> Confirmed: survived the ruleset;<br/>__nf_conntrack_confirm<br/>at POST_ROUTING / LOCAL_IN
    Unconfirmed --> [*]: ruleset dropped the packet —<br/>entry freed, table unpolluted
    Confirmed --> Established: reply tuple matched;<br/>test_and_set_bit IPS_SEEN_REPLY
    Established --> Established: both directions;<br/>timeout refreshed per packet
    Established --> Assured: protocol tracker sets<br/>IPS_ASSURED — now immune<br/>to early_drop
    Assured --> [*]: timeout expires;<br/>gc_worker reaps
    Established --> [*]: timeout expires
    New --> [*]: short timeout expires —<br/>tcp syn_sent 120s, udp 30s —<br/>or early_drop evicts it

    state Related {
        [*] --> Expectation: helper parsed the control<br/>stream, registered a tuple
        Expectation --> Matched: packet hit the expectation;<br/>new nf_conn gets IPS_EXPECTED
        Matched --> [*]: ctinfo = IP_CT_RELATED
    }

The conntrack state machine, including the states a firewall rule never sees. What it shows: the NEW → ESTABLISHED → ASSURED progression is the visible part, but two invisible transitions do the real work — unconfirmed → confirmed, which is where an entry becomes globally visible and where the insert race happens, and established → assured, which is what makes an entry immune to eviction under pressure. The insight to take: the Unconfirmed state is the origin of both of conntrack’s virtues and its most famous bug. Deferring insertion until the packet has survived the ruleset is why a dropped first packet leaves no garbage in the table; it is also why two CPUs can independently build entries for the same tuple and only discover the conflict at the very end. IPS_ASSURED matters for the opposite reason: early_drop_list() skips assured entries, so under table pressure a half-open flow is sacrificed to preserve a working one.

Mechanical Walk-through: nf_conntrack_in

The hook entry point is nf_conntrack_in() (nf_conntrack_core.c), registered at priority -200. Step by step for a packet:

1. Already tracked? nf_ct_get(skb, &ctinfo) checks if a conntrack is already attached (e.g. loopback, or an UNTRACKED packet from the raw table’s NOTRACK); if so it returns NF_ACCEPT immediately.
2. Find the L4 protocol. get_l4proto() parses down to the transport header; if it cannot, the invalid counter is bumped and the packet is accepted untracked.
3. ICMP special-case. ICMP/ICMPv6 errors are handled by nf_conntrack_handle_icmp, which can associate the error with the flow it references (this is how an ICMP error becomes RELATED).
4. Resolve the tuple. resolve_normal_ct() computes the packet’s tuple, hashes it (hash_conntrack_raw), and looks it up with __nf_conntrack_find_get. On a miss it calls init_conntrack() to allocate a new nf_conn (with the original tuple and its computed inverse reply tuple). It then sets ctinfo via the logic shown above and attaches the conntrack to the skb with nf_ct_set.
5. Protocol state machine. nf_conntrack_handle_packet() dispatches to the per-protocol tracker (nf_conntrack_tcp_packet, nf_conntrack_udp_packet, etc.), which validates the packet against the protocol’s state machine (for TCP: is this a valid segment for the current TCP state?) and refreshes the timeout via nf_ct_refresh_acct. A protocol tracker can return -NF_REPEAT (TCP reopen of a closed connection) which loops back to step 4 to build a fresh entry, or a drop.
6. Mark seen-reply. If this packet matched the reply tuple, IPS_SEEN_REPLY is set and an IPCT_REPLY event is cached.

Crucially, a new entry is not yet in the live hash table — it is unconfirmed. It is only confirmed (__nf_conntrack_confirm, registered at priority INT_MAX on POST_ROUTING/LOCAL_IN) once the packet has survived the entire ruleset and is about to leave the box. This deferral is deliberate: if a firewall rule drops the first packet, no half-formed entry pollutes the table, and only the original direction triggers confirmation (if (CTINFO2DIR(ctinfo) != IP_CT_DIR_ORIGINAL) return NF_ACCEPT;).

flowchart TD
  PKT["Packet arrives at PRE_ROUTING<br/>or LOCAL_OUT, priority -200"] --> A{"nf_ct_get: already<br/>tracked, or UNTRACKED?"}
  A -->|"yes"| ACC1["NF_ACCEPT immediately<br/>(loopback, raw-table NOTRACK)"]
  A -->|"no"| B{"get_l4proto:<br/>dataoff greater than 0?"}
  B -->|"no"| INV1["bump invalid<br/>NF_ACCEPT, untracked<br/>-> rules see ctstate INVALID"]
  B -->|"yes"| C{"ICMP or ICMPv6?"}
  C -->|"yes"| ICMP["nf_conntrack_handle_icmp:<br/>can attach the error to the<br/>flow it references -> RELATED"]
  C -->|"no"| D["resolve_normal_ct"]
  ICMP --> D
  D --> E{"hash_conntrack_raw +<br/>__nf_conntrack_find_get:<br/>tuple present?"}
  E -->|"hit, REPLY dir"| F1["ctinfo = ESTABLISHED_REPLY"]
  E -->|"hit, ORIGINAL dir"| F2{"IPS_SEEN_REPLY set?"}
  F2 -->|"yes"| F3["ctinfo = ESTABLISHED"]
  F2 -->|"no, IPS_EXPECTED"| F4["ctinfo = RELATED"]
  F2 -->|"no"| F5["ctinfo = NEW"]
  E -->|"miss"| G["init_conntrack:<br/>__nf_conntrack_alloc"]
  G --> G1{"count greater than<br/>nf_conntrack_max?"}
  G1 -->|"yes"| G2["early_drop: evict a<br/>non-ASSURED entry"]
  G2 -->|"nothing evictable"| G3["bump drop<br/>'table full, dropping packet'<br/>NF_DROP"]
  G2 -->|"evicted"| G4["bump early_drop"]
  G1 -->|"no"| G4
  G4 --> G5["allocate nf_conn,<br/>store ORIGINAL + inverted REPLY<br/>UNCONFIRMED — not in the hash"]
  G5 --> F5
  F1 --> H["nf_conntrack_handle_packet:<br/>per-protocol state machine<br/>+ nf_ct_refresh_acct"]
  F3 --> H
  F4 --> H
  F5 --> H
  H -->|"returns 0 or less"| INV2{"-NF_REPEAT?"}
  INV2 -->|"yes (TCP reopen)"| D
  INV2 -->|"no"| INV3["bump invalid<br/>(+ drop if NF_DROP)<br/>-> rules see ctstate INVALID"]
  H -->|"ok"| I["NF_ACCEPT — the ruleset runs<br/>filter, nat, mangle; NAT sets<br/>the final reply tuple"]
  I --> J{"packet survived<br/>the ruleset?"}
  J -->|"no"| K["unconfirmed entry freed<br/>— table stays clean"]
  J -->|"yes"| L["__nf_conntrack_confirm<br/>POST_ROUTING / LOCAL_IN,<br/>priority INT_MAX"]
  L --> M{"CTINFO2DIR is<br/>ORIGINAL?"}
  M -->|"no"| ACC2["NF_ACCEPT — only the original<br/>direction confirms"]
  M -->|"yes"| N{"scan both buckets:<br/>equal key present?"}
  N -->|"chain exceeded 50-79"| O["bump chaintoolong<br/>+ insert_failed<br/>NF_DROP"]
  N -->|"no"| P["set IPS_CONFIRMED,<br/>__nf_conntrack_hash_insert,<br/>NF_ACCEPT"]
  N -->|"yes — CLASH"| Q["nf_ct_resolve_clash<br/>(see the race section)"]

The complete nf_conntrack_in → ruleset → __nf_conntrack_confirm path, annotated with every statistics counter it can bump. What it shows: the packet crosses conntrack twice — once near the start of the pipeline to find or create the entry, once at the very end to insert it — and almost every counter in /proc/net/stat/nf_conntrack corresponds to one specific edge in this graph. The insight to take: trace the J -->|no| edge. An entry created at the top and killed by a DROP rule in the middle never reaches the hash table, which is the whole payoff of deferred confirmation. Now trace it again and notice that the gap between G5 and L is the entire ruleset — nftables chains, NAT, mangle, possibly an NFQUEUE round-trip to userspace. That gap is the race window, and it is as wide as your firewall is complicated.

The Insert Race and nf_ct_resolve_clash()

This is failure mode (b), and it is the part of conntrack most worth understanding in detail, because it is the mechanism behind one of the most-reported bugs in container networking — and because the kernel’s answer to it is unusually elegant.

Where the window comes from

The previous section’s last paragraph is the cause. An entry created by init_conntrack() is unconfirmed: it exists, it is attached to the skb, but it is not in the hash table. It only goes in at __nf_conntrack_confirm(), registered at priority INT_MAX on POST_ROUTING/LOCAL_IN — after the entire ruleset, after NAT has chosen its rewrite. That deferral buys two real things: a packet the firewall drops leaves no entry behind, and NAT gets to finalise the reply tuple before it is hashed.

It also opens a window. Between init_conntrack() on one CPU and __nf_conntrack_confirm() on that same CPU, another CPU can run the whole sequence for a packet with the identical tuple. Both CPUs hold private, unconfirmed nf_conn objects describing the same flow. Both eventually reach confirm. One wins.

The canonical trigger is a stub resolver. glibc’s getaddrinfo() sends the A and AAAA queries back to back from the same socket — same source IP, same source port, same destination, same protocol. Two UDP datagrams, one tuple, two CPUs (RSS/RPS will happily steer them to different queues). That is the race, and it is why the symptom has a DNS flavour.

Before v5.6 (and before v4.7 for the simplest case) the loser’s packet was simply dropped. For TCP a drop costs a fast retransmit. For a UDP DNS query it costs the resolver’s full retry timeout — which in glibc’s default resolv.conf is five seconds. Hence kubernetes/kubernetes#56903, “DNS intermittent delays of 5s”, opened 6 December 2017: the reporter’s minimal reproducer is a curl -w %{time_namelookup} loop that occasionally prints a value starting with a digit other than zero.

What the kernel does about it

sequenceDiagram
    autonumber
    participant A as CPU A — A query
    participant H as nf_conntrack_hash<br/>global table
    participant B as CPU B — AAAA query
    Note over A,B: glibc sends both queries from<br/>10.2.3.4 port 42 to 10.8.8.8 port 53,<br/>back to back on one socket

    A->>H: lookup tuple {10.2.3.4:42, 10.8.8.8:53, UDP}
    H-->>A: miss
    A->>A: init_conntrack — private, UNCONFIRMED
    B->>H: lookup the same tuple
    H-->>B: still a miss, because A has not confirmed yet
    B->>B: init_conntrack — a second private, UNCONFIRMED entry
    Note over A,B: THE WINDOW. Two nf_conn objects,<br/>same original tuple, neither in the table.

    A->>A: ruleset runs — DNAT picks backend 10.0.0.6,<br/>so A's reply tuple is {10.0.0.6:5353, 10.2.3.4:42}
    B->>B: ruleset runs — DNAT picks backend 10.0.0.7,<br/>so B's reply tuple is {10.0.0.7:5353, 10.2.3.4:42}

    A->>H: __nf_conntrack_confirm, both bucket locks held
    H-->>A: no equal key — insert ORIGINAL and REPLY. A WINS.

    B->>H: __nf_conntrack_confirm, both bucket locks held
    H-->>B: ORIGINAL key already present, goto out
    B->>B: nf_ct_resolve_clash

    alt l4proto allow_clash is false — TCP, ICMP, SCTP
        B->>B: bump drop AND insert_failed
        B-->>B: NF_DROP — packet lost, retransmit delay
    else entries are mergeable — __nf_ct_resolve_clash
        B->>B: nf_ct_can_merge — identical, or exact reverse
        B->>B: nf_ct_acct_merge counters, nf_ct_put loser,<br/>nf_ct_set skb to A's entry
        B->>B: bump clash_resolve
        B-->>B: NF_ACCEPT — packet proceeds on A's entry
    else reply tuples differ — nf_ct_resolve_clash_harder
        B->>B: scan reply bucket — is B's REPLY tuple unique?
        B->>B: timeout set to 1 second — status gains<br/>IPS_FIXED_TIMEOUT and IPS_NAT_CLASH
        B->>H: hlist_nulls_add_fake for ORIGINAL — a dead node,<br/>so lookups and ctnetlink never see it
        B->>H: real insert of the REPLY tuple only
        B->>B: bump clash_resolve
        B-->>B: NF_ACCEPT — packet proceeds on B's own entry
    end

    Note over H: The reply from 10.0.0.6 matches A's reply tuple.<br/>The reply from 10.0.0.7 matches B's reply-only entry,<br/>is un-DNATed correctly, and B's entry then deletes<br/>itself because IPS_NAT_CLASH is set.

The insert race between two CPUs, and all three of its outcomes. What it shows: the race is not resolved by preventing it — the window is deliberate — but by teaching the loser three progressively more desperate ways to survive. The insight to take: the third branch is the clever one. nf_ct_resolve_clash_harder() inserts the loser’s entry into the reply bucket only, and puts a deliberately dead hlist_nulls node in the original bucket via hlist_nulls_add_fake(). The result is an entry that no forward lookup can find (so original-direction packets keep matching the winner) but that the reply will find (so the second DNS answer is un-DNATed correctly and delivered). It is given a one-second IPS_FIXED_TIMEOUT and tagged IPS_NAT_CLASH so it deletes itself on that first reply, can never reach ASSURED, and is always evictable. A temporary half-entry, visible from one side only.

The three paths, in the code

The dispatcher is nf_ct_resolve_clash() (nf_conntrack_core.c:1163, v6.12), reached from the out: label of __nf_conntrack_confirm() when a bucket scan finds an equal key. Its own docstring names the symptom without naming DNS:

The major problem with such packet drop is the extra delay added by the packet loss — it will take some time for a retransmit to occur (or the sender to time out when waiting for a reply). This function attempts to handle the situation without packet drop.

Path 1 — refuse. if (!l4proto->allow_clash) goto drop; Straight to NF_CT_STAT_INC(net, drop); NF_CT_STAT_INC(net, insert_failed); return NF_DROP;. Note that both counters rise together on this path, which is what makes them interpretable in the aggregate.

Path 2 — merge (__nf_ct_resolve_clash). Calls nf_ct_can_merge(ct, loser_ct), which is nf_ct_match(ct, loser_ct) || nf_ct_match_reverse(ct, loser_ct). If the two entries describe the same flow, the loser is simply discarded: take a reference on the winner, fold the loser’s packet/byte counters into it with nf_ct_acct_merge(), nf_ct_put() the loser, re-point skb->_nfct at the winner with nf_ct_set(), bump clash_resolve, return NF_ACCEPT. The packet then proceeds as if the race had never happened.

Path 3 — insert sideways (nf_ct_resolve_clash_harder). Only reachable when the entries are not mergeable, which in practice means NAT gave them different reply tuples. It first rescans the reply bucket: “Reply direction must never result in a clash, unless both origin and reply tuples are identical.” If the loser’s reply tuple is unique, it does the half-insert described in the diagram caption. If the reply tuple also clashes, it falls back to __nf_ct_resolve_clash() on that entry.

The nf_ct_match_reverse half of path 2 is new at v6.12 and handles a genuinely different race — two peers that begin talking to each other simultaneously, so both directions are picked up as NEW rather than one being NEW and the other ESTABLISHED. Florian Westphal’s commit message calls it “esoteric”: “ruleset must permit NEW connections in either direction and both peers must already have a bidirectional traffic flow at the time conntrack gets enabled” (commit a4e6a103, 10 September 2024).

Which protocols opt in

Clash resolution is not global. It is gated per Layer-4 protocol by a single bool in struct nf_conntrack_l4proto (include/net/netfilter/nf_conntrack_l4proto.h), commented “Resolve clashes on insertion races.” Checked file by file at v6.12:

Protocol	File	allow_clash	Why
UDP	nf_conntrack_proto_udp.c:285	true	Two datagrams of one tuple are genuinely indistinguishable; merging them loses nothing
UDP-Lite	nf_conntrack_proto_udp.c:307	true	Same argument
GRE	nf_conntrack_proto_gre.c:307	true	Keyed tunnel sessions, same argument
TCP	nf_conntrack_proto_tcp.c	not set → false	Silently merging two TCP flows would be a correctness bug — sequence state cannot be merged
SCTP	nf_conntrack_proto_sctp.c	not set → false	Association state, same argument as TCP
ICMP / ICMPv6	nf_conntrack_proto_icmp.c, _icmpv6.c	not set → false	—
DCCP	nf_conntrack_proto_dccp.c	not set → false	—
generic (unknown L4)	nf_conntrack_proto_generic.c	not set → false	—

Which protocols allow clash resolution at v6.12. What it shows: exactly three, and they are the three connectionless or key-identified ones. The insight to take: this is why the famous bug is a DNS bug. DNS is UDP, so allow_clash is true and the race is absorbable; had getaddrinfo() used TCP, the same race would be an unavoidable insert_failed drop. It also means a TCP workload showing non-zero insert_failed has no kernel-side mitigation available — the only fixes are in the ruleset or the traffic steering.

A correction worth recording

Network Address Translation NAT states that “exactly two protocols set it: UDP and UDP-Lite … TCP, ICMP and ICMPv6 do not — checked file by file.” The TCP/ICMP half is right, but GRE also sets allow_clash = true at v6.12 (nf_conntrack_proto_gre.c:307), so the count is three, not two. The conclusion drawn there — that DNS benefits because DNS is UDP — is unaffected.

Dating the mitigation

Every row below was established by existence-checking net/netfilter/nf_conntrack_core.c at consecutive tags, counting occurrences of the named symbol — not from release notes or changelogs:

Mechanism	First release	Evidence	Commit
nf_ct_resolve_clash() — merge identical entries instead of dropping	v4.7	0 occurrences at v4.6, 2 at v4.7	71d8c47f, Pablo Neira Ayuso, 1 May 2016
nf_ct_resolve_clash_harder() + IPS_NAT_CLASH — reply-only insert	v5.6	0 at v5.5, 3 at v5.6; IPS_NAT_CLASH 0 at v5.5, 2 at v5.6	6a757c07, Florian Westphal, 3 Feb 2020
clash_resolve statistic — the clashres column	v5.10	0 occurrences at v5.9, 2 at v5.10	bc924704, Florian Westphal, 26 Aug 2020
nf_ct_match_reverse() — clash resolution for reverse collisions	v6.12	0 occurrences at v6.11, 3 at v6.12	a4e6a103, Florian Westphal, 10 Sep 2024

The clash-resolution timeline. What it shows: eight years of incremental work, and the newest piece landed in the release this note pins to. The insight to take: the clash_resolve counter arriving four and a half years after the mechanism it counts is the single most important row here, and the commit that added it says why: “There is a misconception about what insert_failed means. We increment this even when a clash got resolved, so it might not indicate a problem.” Before v5.10, insert_failed counted successes as well as failures. Any runbook, dashboard alert or blog post that reads insert_failed as “races that cost a packet” is describing a v5.10-or-later kernel, and is simply wrong about anything older. On a pre-5.10 box the two are indistinguishable.

The Conntrack Table: Size, Hashing, and Limits

The flow store is a hash table of struct nf_conntrack_tuple_hash nodes (two per nf_conn — one per direction), with these tunables (Documentation/networking/nf_conntrack-sysctl.rst):

• nf_conntrack_buckets — number of hash buckets. Default is total RAM / 16384, clamped to [1024, 262144]. Writable only in the initial namespace.
• nf_conntrack_max — the hard cap on tracked entries. Defaults to nf_conntrack_buckets. The doc is explicit and worth internalizing: because each connection occupies two tuple slots (original + reply), a maxed-out table with max == buckets has an average hash chain length of 2, not 1.
• nf_conntrack_count — read-only current count.

When a new entry would exceed nf_conntrack_max, __nf_conntrack_alloc() does not simply fail — it first calls early_drop(), which scans a hash bucket for an entry that has not yet seen a reply (!IPS_ASSURED) and evicts it to make room. Only if early-drop finds nothing does allocation fail with -ENOMEM, the packet is dropped, the drop stat increments, and the kernel logs the infamous nf_conntrack: table full, dropping packet (net_warn_ratelimited). A background garbage-collection worker (gc_worker) also walks the table periodically, expiring timed-out entries and, when pressure is high (above 95% of max), early-dropping non-assured flows.

The sizing defaults — where the documentation and the code disagree

The tunables above are documented in Documentation/networking/nf_conntrack-sysctl.rst, and for two of them the documentation is a simplification that gives the wrong number on real machines. The authority is nf_conntrack_init_start() (nf_conntrack_core.c:2619, v6.12):

int max_factor = 8;                                   /* ← NOT the value used by default */
...
if (!nf_conntrack_htable_size) {                      /* i.e. no hashsize= module param */
        nf_conntrack_htable_size
                = (((nr_pages << PAGE_SHIFT) / 16384) / sizeof(struct hlist_head));
        if (BITS_PER_LONG >= 64 &&
            nr_pages > (4 * (1024 * 1024 * 1024 / PAGE_SIZE)))
                nf_conntrack_htable_size = 262144;    /* >4 GiB on 64-bit: flat override */
        else if (nr_pages > (1024 * 1024 * 1024 / PAGE_SIZE))
                nf_conntrack_htable_size = 65536;     /* >1 GiB: flat override */
        if (nf_conntrack_htable_size < 1024)
                nf_conntrack_htable_size = 1024;
        max_factor = 1;                               /* ← the assignment that matters */
}
nf_conntrack_max = max_factor * nf_conntrack_htable_size;

Three things to take from reading it rather than the prose:

1. max_factor is 8 at its declaration and 1 in the default path. Stopping at the first definition gives nf_conntrack_max = 8 × buckets. The = 8 only survives when you booted with an explicit nf_conntrack.hashsize= — the comment says so: “When a table size is given we use the old value of 8 to avoid implicit reduction of the max entries setting.” So the documented “nf_conntrack_max is set to nf_conntrack_buckets by default” is correct for the default, and silently becomes 8× if you ever set hashsize by hand.
2. The “divide total memory by 16384” formula is not what runs on most machines. It divides by 16384 and then by sizeof(struct hlist_head) (8 bytes on 64-bit), and — decisively — is overwritten wholesale by a flat 262144 above 4 GiB or 65536 above 1 GiB. On a 2 GiB 64-bit box the documented formula predicts 16384 buckets; the code produces 65536. The two only coincide above 4 GiB, where the flat override happens to equal the documented ceiling.
3. nf_conntrack_expect_max is documented as buckets / 256 and is actually four times that. nf_conntrack_expect.c:719 sets nf_ct_expect_hsize = nf_conntrack_htable_size / 256 and then nf_ct_expect_max = nf_ct_expect_hsize * 4. The documentation describes the hash size, not the maximum.

All three are checkable against a running kernel:

Tunable	Documented default	Code default	Read on the revision host (128 GiB, 64-bit)	Verdict
nf_conntrack_buckets	total RAM ÷ 16384, clamped [1024, 262144]	262144 flat, because nr_pages > 4 GiB on 64-bit	262144	Doc and code agree here, by coincidence
nf_conntrack_max	“set to nf_conntrack_buckets”	max_factor × buckets, max_factor = 1 unless hashsize= was given	262144	Doc right for the default, silent 8× otherwise
nf_conntrack_expect_max	nf_conntrack_buckets / 256 → would be 1024	(buckets / 256) × 4 → 4096	4096	Documentation is wrong by 4×; measurement settles it
nf_conntrack_count	read-only current count	atomic_read(&cnet->count), per network namespace	297–461 across readings	—

Documented defaults versus the code, checked against a live kernel. What it shows: two of four documented values do not describe what the kernel computes, and one of them is off by a clean factor of four. The insight to take: the 4,096 reading is the proof, and it is why measurement matters even when a primary source exists — Documentation/ is a primary source that can still be stale. The general lesson is the max_factor one: a default read from the first assignment in a function is a hypothesis, not a value. Read to the end of the block.

Note the second-order consequence of max == buckets: since each connection occupies two hash slots (original and reply), a table filled to nf_conntrack_max has an average chain length of 2, not 1 — the documentation states this explicitly and it is the reason bumping nf_conntrack_max without bumping nf_conntrack_buckets degrades lookup time. There is a hard ceiling on that degradation: __nf_conntrack_confirm() abandons an insert once a bucket chain exceeds MIN_CHAINLEN + get_random_u32_below(MAX_CHAINLEN) — that is 50 + rand(0..29), so between 50 and 79 entries, randomised per insert so an attacker cannot aim at a precise threshold. Exceeding it bumps chaintoolong and insert_failed and drops the packet. A randomised limit in the 50–79 range is an anti-hash-flooding measure, not a capacity knob.

How the table drains

Two mechanisms free entries, and they are frequently conflated.

early_drop(), synchronous, on the allocation path. When atomic_inc_return(&cnet->count) exceeds nf_conntrack_max, __nf_conntrack_alloc() calls early_drop() before failing. early_drop_list() walks a hash bucket and evicts the first entry that is not IPS_ASSURED, not dying, and in the right namespace. Only if that finds nothing does allocation return -ENOMEM, the drop stat rises and the kernel emits the ratelimited nf_conntrack: table full, dropping packet. This is why a table at 100% can still accept new flows: half-open connections are sacrificed to make room.

gc_worker(), asynchronous, on a workqueue. A delayed work item that reaps expired entries. It is adaptive rather than fixed-interval: it tracks an avg_timeout, bounded by GC_SCAN_INTERVAL_MIN (1 s) and GC_SCAN_INTERVAL_MAX (60 s), caps each individual scan at GC_SCAN_MAX_DURATION (10 ms) so it never monopolises a CPU, and stops early once it has expired GC_SCAN_EXPIRED_MAX (64000 / HZ) entries in one pass. Crucially, when occupancy exceeds 95% of nf_conntrack_max (nf_conntrack_max95 = nf_conntrack_max / 100u * 95u) it switches from “reap the expired” to “also early-drop the droppable”, using gc_worker_can_early_drop() — which returns true for anything not IPS_ASSURED, plus anything the protocol’s own can_early_drop() callback nominates (for TCP, entries in closing states).

That 95% threshold is the number to watch. A box sitting below it has a garbage collector that only removes genuinely dead flows; a box that crosses it has a garbage collector actively culling live-but-unassured ones. The behavioural change is invisible in any single counter, which is why nf_conntrack_count / nf_conntrack_max belongs on a dashboard with a line drawn at 0.95.

Timeouts per Protocol

Every flow has a timeout that is refreshed on each packet (nf_ct_refresh_acct); when it expires, the gc worker reaps the entry. Timeouts are per-protocol and, for TCP, per-state (nf_conntrack-sysctl.rst):

• TCP established: nf_conntrack_tcp_timeout_established — default 432000 s (5 days). This is the famous one: a long-idle TCP connection stays tracked for five days by default, which is why connection-heavy workloads exhaust the table.
• TCP syn_sent 120 s, syn_recv 60 s, fin_wait 120 s, time_wait 120 s, close 10 s, close_wait 60 s, last_ack 30 s.
• UDP: nf_conntrack_udp_timeout 30 s, rising to nf_conntrack_udp_timeout_stream 120 s once a stream is detected (UDP has no states, so conntrack synthesizes “is this a stream” from seeing replies).
• ICMP: 30 s. GRE: 30 s (180 s for a stream). Generic (unknown L4): 600 s.

These are the levers operators reach for: dropping tcp_timeout_established from 5 days to, say, an hour dramatically cuts table occupancy on a busy box, at the cost of breaking genuinely long-idle connections that resume after the timeout.

Every timeout, read from a live kernel

The complete set, read from /proc/sys/net/netfilter/ on the revision host (Fedora 44, kernel 7.1.13, defaults untouched). A sysctl only appears once its protocol tracker module is loaded, which is why DCCP and UDP-Lite are absent here:

sysctl (net.netfilter. prefix)	Value	In human terms	What refreshes it
nf_conntrack_tcp_timeout_established	432000	5 days	Any in-window segment on an established flow
nf_conntrack_tcp_timeout_syn_sent	120	2 min	SYN sent, no SYN-ACK yet
nf_conntrack_tcp_timeout_syn_recv	60	1 min	SYN-ACK seen, handshake incomplete
nf_conntrack_tcp_timeout_fin_wait	120	2 min	FIN sent
nf_conntrack_tcp_timeout_close_wait	60	1 min	FIN received, local side not closed
nf_conntrack_tcp_timeout_last_ack	30	30 s	Waiting for the final ACK
nf_conntrack_tcp_timeout_time_wait	120	2 min	Both sides closed — the window NAT port reuse reclaims from
nf_conntrack_tcp_timeout_close	10	10 s	RST or full close
nf_conntrack_tcp_timeout_max_retrans	300	5 min	Applied after tcp_max_retrans retransmissions
nf_conntrack_tcp_timeout_unacknowledged	300	5 min	Data sent but never acknowledged
nf_conntrack_tcp_max_retrans	3	—	Retransmit count that switches to the max_retrans timeout
nf_conntrack_udp_timeout	30	30 s	Any datagram, before a reply has been seen
nf_conntrack_udp_timeout_stream	120	2 min	Any datagram once the flow is treated as a stream
nf_conntrack_icmp_timeout	30	30 s	An echo request/reply pair
nf_conntrack_icmpv6_timeout	30	30 s	Same, IPv6
nf_conntrack_gre_timeout	30	30 s	Keyed GRE session
nf_conntrack_gre_timeout_stream	180	3 min	GRE once replies are flowing
nf_conntrack_generic_timeout	600	10 min	Any L4 protocol with no dedicated tracker
nf_conntrack_sctp_timeout_established	210	3.5 min	SCTP association
nf_conntrack_sctp_timeout_closed	10	10 s	—
nf_conntrack_sctp_timeout_cookie_wait / _cookie_echoed	3 / 3	3 s	SCTP four-way handshake stages
nf_conntrack_sctp_timeout_shutdown_sent / _recd / _ack_sent	3 / 3 / 3	3 s	SCTP teardown stages
nf_conntrack_sctp_timeout_heartbeat_sent	30	30 s	—
nf_conntrack_frag6_timeout	60	1 min	IPv6 fragment reassembly hold time
nf_flowtable_tcp_timeout	30	30 s	Offloaded (flowtable) TCP flow — not the conntrack timeout
nf_flowtable_udp_timeout	30	30 s	Offloaded UDP flow

Every conntrack timeout on a default Fedora 44 kernel. What it shows: a spread of more than four orders of magnitude, from 3 seconds (SCTP handshake stages) to 432,000 (established TCP). The insight to take: two rows dominate table occupancy in practice and they are the two extremes. Established TCP at five days means an idle SSH session, a database pool connection or a leaked socket keeps a table slot for most of a working week — this is the single biggest lever for a box approaching nf_conntrack_max. And generic_timeout at 600 s is the quiet one: any protocol without a dedicated tracker (an unusual IP protocol number, an encapsulation the kernel does not parse) gets ten minutes per flow, twenty times UDP’s thirty seconds. Note also the last two rows: nf_flowtable_* timeouts govern the offload fast path, are not conntrack timeouts, and are a common source of confusion when tuning a forwarding box.

Two structural points the table cannot show. First, timeouts are refreshed on every packet via nf_ct_refresh_acct(), so they are idle timers, not lifetimes — a flow sending one packet a minute survives indefinitely under a 30-second UDP timeout. Second, UDP has no states, so conntrack synthesises the stream/non-stream distinction from whether replies have been seen; that is the entire difference between the 30 s and 120 s UDP rows.

Dropping tcp_timeout_established from five days to, say, an hour cuts occupancy dramatically on a busy box, at the cost of silently breaking genuinely long-idle connections that try to resume — which then fail in the confusing way, as INVALID rather than as a clean reset. The usual mitigation is application-level keepalives shorter than whatever you set.

Confirmed against v6.12 source, not just read from a running 7.1 kernel. The live values above were cross-checked field by field against the initialiser arrays in the pinned tree, and every one matches:

• tcp_timeouts[] in nf_conntrack_proto_tcp.c — SYN_SENT 2 MINS, SYN_RECV 60 SECS, ESTABLISHED 5 DAYS, FIN_WAIT 2 MINS, CLOSE_WAIT 60 SECS, LAST_ACK 30 SECS, TIME_WAIT 2 MINS, CLOSE 10 SECS, RETRANS 5 MINS, UNACK 5 MINS, and tn->tcp_max_retrans = 3.
• udp_timeouts[] in nf_conntrack_proto_udp.c:27 — UDP_CT_UNREPLIED 30*HZ, UDP_CT_REPLIED 120*HZ.
• sctp_timeouts[] in nf_conntrack_proto_sctp.c — all eight values as listed.
• nf_ct_icmp_timeout = 30*HZ and nf_ct_generic_timeout = 600*HZ in their respective files.

Two things the sysctl listing hides. tcp_timeouts[] has an eleventh entry, TCP_CONNTRACK_SYN_SENT2 = 2 MINS, for a simultaneous open — a state with no exposed sysctl, so you cannot tune it. And the UDP “stream” promotion is not simply “a reply was seen”: nf_conntrack_udp_packet() stamps ct->proto.udp.stream_ts = 2 * HZ + jiffies when the entry is first created, and only once time_after(jiffies, stream_ts) and IPS_SEEN_REPLY is set does it apply the 120-second timeout and set IPS_ASSURED. So a UDP flow must be both bidirectional and at least two seconds old before it is treated as a stream — a single request/response pair completing in under two seconds never gets promoted, and never becomes assured. That two-second gate is what keeps a DNS query from occupying an un-evictable table slot for two minutes. The same function carries the explicit guard if (unlikely((status & IPS_NAT_CLASH))) return NF_ACCEPT; — clash entries are barred from ever reaching ASSURED, exactly as nf_ct_resolve_clash_harder()’s comment promises.

Helpers (Application-Layer Gateways)

Some protocols negotiate secondary connections on ports chosen at runtime — File Transfer Protocol (FTP) in active mode opens a data connection on a port announced inside the control stream; Session Initiation Protocol (SIP) negotiates RTP media ports the same way. A stateful firewall that only knew the control connection would block the data connection as NEW/unexpected. Conntrack helpers (Application-Layer Gateways, ALGs) parse the control stream, extract the negotiated address/port, and register an expectation — a pending tuple that, when matched by an incoming packet, creates a conntrack entry flagged IPS_EXPECTED, which resolve_normal_ct then labels RELATED. That is the whole mechanism behind “ct state related accept lets FTP data through.” Helpers run at priority +300 (NF_IP_PRI_CONNTRACK_HELPER), after the main lookup but before confirm. Helpers are also a historic security risk (a malicious payload can spoof an expectation, e.g. CVE-2017-style helper abuses), which is why the recommended modern practice is to attach helpers explicitly via a CT target / ct helper rule rather than relying on automatic port-based assignment.

The nf_conntrack_helper sysctl no longer exists

A very widely repeated hardening recommendation is sysctl -w net.netfilter.nf_conntrack_helper=0, to stop helpers attaching themselves by destination port. That knob was removed in v6.0 and the advice now describes a vanished setting.

Established by existence-check against net/netfilter/nf_conntrack_standalone.c, counting NF_SYSCTL_CT_HELPER and the "nf_conntrack_helper" sysctl name:

Tag	Occurrences	Verdict
v5.18	4	Sysctl present
v5.19	4	Sysctl present
v6.0	0	Removed
v6.1	0	Still gone
v6.12	0	Still gone (this note’s pin)

Corroborated by measurement: on the revision host, ls /proc/sys/net/netfilter/ | grep helper returns nothing, while forty-one other nf_conntrack_* entries are present. Writing to it fails because the path does not exist, and a sysctl -w in a hardening script will exit non-zero — which is at least a loud failure rather than a silent one.

The behaviour it used to gate is now the only behaviour: helpers must be attached explicitly, with a CT --helper target in iptables or a ct helper object in nftables. There is no automatic port-based assignment left to disable. If your ruleset relied on automatic assignment — FTP data connections working without a rule, say — it stopped working at v6.0, and the fix is to add the explicit attachment, not to restore a sysctl.

Uncertain

Verify: the exact release in which automatic helper assignment (as distinct from the sysctl that gated it) stopped being the default, and whether any distribution back-ported the removal. Reason: this note establishes that the sysctl disappeared between v5.19 and v6.0 by existence-check, but has not traced the nf_ct_auto_assign_helper code path itself across releases — the sysctl had already defaulted to “off” for several releases before its removal, so the behavioural change and the symbol’s disappearance are probably different dates. To resolve: bisect sysctl_auto_assign_helper / nf_ct_auto_assign_helper in net/netfilter/nf_conntrack_helper.c across the v4.x–v6.0 range. uncertain

Reading /proc/net/stat/nf_conntrack

This file is the whole operational story in seventeen columns, it is world-readable, and — unlike /proc/net/nf_conntrack or the conntrack CLI — it needs no privilege at all. It is also mildly booby-trapped, in three separate ways, and getting them wrong is how people arrive at conclusions their kernel does not support.

The header lies about three columns

ct_cpu_seq_show() (net/netfilter/nf_conntrack_standalone.c, v6.12) prints a fixed header and then a seq_printf whose arguments do not all correspond to it. Reading the two side by side is the only way to know what you are looking at:

Column in the header	Value actually printed	Reality at v6.12
entries	nf_conntrack_count(net)	Not per-CPU. A single global count, re-printed identically on every row
clashres	st->clash_resolve	Insert races absorbed without losing a packet. Added in v5.10
found	st->found	Only bumped in nf_conntrack_tuple_taken() — i.e. during a NAT source-tuple search, not on ordinary lookups
new	literal 0	Dead. The counter was removed; the column is a placeholder
invalid	st->invalid	Packets conntrack refused to associate with a flow (the three roads above)
ignore	literal 0	Dead placeholder
delete	literal 0	Dead placeholder
chainlength	st->chaintoolong	Misnamed. Not a chain length — a count of inserts abandoned because a bucket chain exceeded 50–79
insert	st->insert	Only bumped in nf_conntrack_hash_check_insert() — the ctnetlink path. Normal packet-driven confirms do not increment it
insert_failed	st->insert_failed	Since v5.10: races that could not be resolved, plus chaintoolong and invalid-extension failures
drop	st->drop	Packets conntrack dropped — table full, unresolvable clash, or a tracker returning NF_DROP
early_drop	st->early_drop	Entries evicted by early_drop() to make room
icmp_error	st->error	Misnamed. The generic error counter, not ICMP-specific
expect_new	st->expect_new	Packets that matched a helper expectation
expect_create	st->expect_create	Expectations registered by a helper
expect_delete	st->expect_delete	Expectations removed
search_restart	st->search_restart	Lockless lookup had to restart because the nulls value changed under it — a hash resize or entry move raced the read

The column header versus the printed value. What it shows: four columns are outright dead (new, ignore, delete print a literal zero), two are misnamed (chainlength is chaintoolong, icmp_error is error), and entries is global rather than per-CPU. The insight to take: chainlength is the worst of these. It reads like a gauge you would compare against nf_conntrack_buckets, and it is actually a failure counter — a non-zero value means inserts were abandoned and packets dropped. Likewise, a zero in insert says nothing about whether the box is tracking connections; it only says nothing created an entry through netlink. The header is a stable ABI that outlived its own fields.

The entries trap

entries is printed as nf_conntrack_count(net) — the current, global count — on every row. It is not a per-CPU value, so summing it is wrong by a factor of nproc. Measured on the revision host at 2026-09-12T14:01:55+02:00, 32 CPUs:

$ awk 'NR>1{for(i=1;i<=NF;i++)s[i]+=strtonum("0x"$i)}END{print s[1]}' /proc/net/stat/nf_conntrack
9504
$ cat /proc/sys/net/netfilter/nf_conntrack_count
297

9504 = 297 × 32 exactly. The correct reading is 297. Every other column is a genuine per-CPU accumulator and must be summed. A one-line check distinguishes them without reading the source — a column whose value is byte-identical on every row is either global or zero:

$ awk 'NR==1{for(i=1;i<=NF;i++)h[i]=$i;n=NF;next}
       {r++; for(i=1;i<=NF;i++){s[i]+=strtonum("0x"$i);
        if(r>1 && $i!=f[i]) v[i]=1; f[i]=$i}}
       END{for(i=1;i<=n;i++) printf "%-16s sum=%-8d %s\n", h[i], s[i],
            (v[i] ? "per-CPU (varies)" : "identical on every row")}' \
  /proc/net/stat/nf_conntrack
 
entries          sum=9504     identical on every row     ← global; real value is 297
clashres         sum=4674     per-CPU (varies)
found            sum=0        identical on every row
new              sum=0        identical on every row     ← dead column
invalid          sum=190      per-CPU (varies)
ignore           sum=0        identical on every row     ← dead column
delete           sum=0        identical on every row     ← dead column
chainlength      sum=0        identical on every row
insert           sum=0        identical on every row
insert_failed    sum=0        identical on every row
drop             sum=0        identical on every row
early_drop       sum=0        identical on every row
icmp_error       sum=0        identical on every row
expect_new       sum=0        identical on every row
expect_create    sum=0        identical on every row
expect_delete    sum=0        identical on every row
search_restart   sum=0        identical on every row

What this reading says

The host is an ordinary developer workstation running twenty-two Docker bridge networks, each with a MASQUERADE rule, so it is genuinely doing NAT for container egress. clashres = 4,674 with insert_failed = 0 and drop = 0.

That is the clash-resolution machinery working, in production, on an idle desktop. Nearly five thousand insert races have occurred on this box and not one of them cost a packet. Two observations sharpen it:

• It is a live counter, and it moves. Successive readings seventeen minutes apart gave 4,612 and then 4,674 — sixty-two clashes in seventeen minutes of near-idle use, about 3.6 per minute. An earlier reading during a previous session recorded 3,294. Insert races are not an exotic event on a NAT-heavy box; they are routine background weather that used to be dropped packets.
• insert_failed = 0 is only meaningful because this kernel is ≥ v5.10. Before the clash_resolve counter existed, all 4,674 of these would have been counted as insert_failed, on a box where nothing was wrong.

Counter	Reading	Interpretation
entries (global)	297	Live entries; agrees with nf_conntrack_count
nf_conntrack_max / _buckets	262144 / 262144	0.11% occupancy — failure mode (a) is nowhere in sight
clashres	4,674	Races absorbed. Healthy, and evidence the mechanism is exercised
insert_failed	0	No race cost a packet
drop	0	Conntrack dropped nothing
early_drop	0	Never had to evict to make room
chainlength (chaintoolong)	0	No bucket ever exceeded the 50–79 limit
invalid	190	Packets with no associated flow — note these get no NAT applied
search_restart	0	No lockless lookup raced a table change
expect_*	0	No helper expectations, despite nf_conntrack_netbios_ns being a loaded module

Live conntrack counters. What it shows: the clash mechanism is firing thousands of times on a machine where nothing is wrong. The insight to take: the diagnostic is the ratio, not either number alone. clashres high and insert_failed zero is the healthy signature — the kernel is absorbing races. insert_failed climbing at all, on UDP-heavy traffic, is the five-second-DNS signature. And insert_failed climbing on TCP traffic has no kernel-side fix available, because TCP does not set allow_clash. Watch clashres and insert_failed together, or you cannot tell a working mitigation from an absent one.

Inspecting Conntrack

The table is exposed two ways. The /proc/net/nf_conntrack file (when CONFIG_NF_CONNTRACK_PROCFS is set) dumps one line per entry; far more usefully, the conntrack userspace tool (from conntrack-tools, over the ctnetlink interface) gives structured access:

# conntrack -L
tcp  6 431999 ESTABLISHED src=10.0.0.1 dst=1.1.1.1 sport=5000 dport=443 \
        src=1.1.1.1 dst=10.0.0.1 sport=443 dport=5000 [ASSURED] mark=0 use=1

Reading the line: protocol tcp (L4 number 6), the remaining-timeout 431999 seconds (≈5 days, an established TCP flow), the state ESTABLISHED, then the two tuples — the first src=/dst=/sport=/dport= set is the original direction, the second is the reply direction (note it is the inverse). [ASSURED] means the flow has the IPS_ASSURED status bit set (it has seen reply traffic) and is therefore skipped by early_drop_list — verified in nf_conntrack_core.c, early_drop_list does if (test_bit(IPS_ASSURED_BIT, &tmp->status) ... ) continue;, so only non-assured entries are evictable under pressure; [UNREPLIED] would mark a NEW-only flow eligible for eviction.

Other invocations: conntrack -L -p tcp --dport 443 filters, conntrack -E streams live events, conntrack -D ... deletes entries, and conntrack -C (or reading nf_conntrack_count) gives the current count for capacity monitoring.

Uncertain

Verify: the exact textual format of the conntrack -L line above — field order, the bracketed [ASSURED]/[UNREPLIED] tokens, the mark=/use= trailers. Reason: conntrack-tools is not installed on the revision host (command -v conntrack → not found) and is a separate userspace project outside the kernel tree, so neither the tool nor its output could be exercised during this revision. The semantics are verified against v6.12 headers and early_drop_list() ([ASSURED] ⇔ IPS_ASSURED, two-tuple original/reply layout), and nf_ct_resolve_clash_harder()’s own comment confirms that clash entries are hidden from this listing: “This also hides the clashing entry from ctnetlink iteration, i.e. conntrack -L won’t show them.” To resolve: install conntrack-tools and compare, or read src/conntrack.c from the netfilter project. uncertain

What could not be inspected here, and why

Attempt	Result	Privilege or package required
cat /proc/net/nf_conntrack	Permission denied — the file is -r--r----- root root	CAP_NET_ADMIN (or root). This is the per-entry dump; the aggregate stats file is world-readable and was used instead
conntrack -L / -E / -C	command -v conntrack → not installed	The conntrack-tools package. It speaks ctnetlink, which itself needs CAP_NET_ADMIN
Reading /proc/net/stat/nf_conntrack	Succeeded, world-readable	Nothing — this is the whole basis of the measurements above
Reading /proc/sys/net/netfilter/nf_conntrack_*	Succeeded, all 41 entries readable	Nothing. Writing any of them needs root, so no tunable was changed
Provoking a clash deliberately (parallel getaddrinfo under a DNAT rule)	Not attempted	Adding an nftables/iptables DNAT rule needs CAP_NET_ADMIN; no passwordless sudo on this host. The clashres counter was read as it stood instead
bpftool, perf, strace on the conntrack path	Not installed	Would have given per-call tracing of nf_conntrack_in / __nf_conntrack_confirm
/sys/kernel/debug/tracing conntrack tracepoints	/sys/kernel/debug is mode 0700, root-owned	Root
tcpdump to observe a DNS retry	Installed, but fails without CAP_NET_RAW	CAP_NET_RAW

Everything quantitative in this note therefore comes from two world-readable sources — /proc/net/stat/nf_conntrack and /proc/sys/net/netfilter/ — plus v6.12 source. Nothing was inferred from a tool that was not run.

Failure Modes

The two headline ones — a full table and the insert race — are covered in depth above (“The Organising Fact”, “The Insert Race and nf_ct_resolve_clash()”, “The sizing defaults”). What follows is the rest of the catalogue, plus the operational summary of those two.

Table full → packet drops. The single most common conntrack production failure. Symptom: intermittent connection failures under load and nf_conntrack: table full, dropping packet in dmesg. Diagnosis: compare nf_conntrack_count to nf_conntrack_max. Fix: raise nf_conntrack_max (and nf_conntrack_buckets proportionally — leaving buckets small makes hash chains long and lookups slow), and/or shorten the established-TCP timeout. On a Kubernetes node this manifests as flaky pod networking; it is why CNIs and node tuning guides bump these sysctls.

Asymmetric routing breaks state. Conntrack assumes it sees both directions of a flow. If the reply path bypasses the box (asymmetric routing, ECMP without flow affinity), the reply tuple is never matched, the flow stays NEW/UNREPLIED, and a ct state invalid drop rule (or a NAT that needs the reply) silently breaks the connection. This is a frequent cause of “works in one direction only” bugs in multi-homed setups.

The per-packet cost at scale. Conntrack does a hash lookup, refcount, and timeout-refresh on every packet of every flow, plus an insert on the first packet — under local_bh_disable()/RCU. At millions of flows this is real CPU and cache pressure; the table itself consumes memory (each nf_conn is hundreds of bytes). This is exactly why high-performance paths bypass it: XDP-based load balancers (Katran) and Cilium’s eBPF datapath implement their own flow tables to avoid nf_conntrack, and the kernel’s flowtable offload (IPS_OFFLOAD) lets established flows skip the full conntrack/rule traversal after the first few packets.

Five-second DNS delays. The signature of failure mode (b). Symptom: getaddrinfo() intermittently takes almost exactly 5.000 s (the glibc resolver’s default retry timeout) while the network is otherwise healthy. Diagnosis: insert_failed in /proc/net/stat/nf_conntrack is non-zero and climbing, ideally alongside a non-zero clashres. Fix: on a kernel ≥ v5.6 most of these are already absorbed; what remains is usually a ruleset that gives the A and AAAA queries different DNAT targets, which is addressed in Network Address Translation NAT. Application-side workarounds — options single-request-reopen or single-request in resolv.conf, or a node-local DNS cache — stop the two queries from sharing a tuple in the first place. On a kernel older than v5.10 this diagnosis is unavailable, because insert_failed there counts resolved clashes too.

INVALID from out-of-window TCP. Strict TCP window tracking can mark legitimately reordered or retransmitted segments INVALID, causing drops. The nf_conntrack_tcp_be_liberal sysctl loosens this (only out-of-window RSTs are INVALID), a common fix for “some TCP connections randomly stall behind the firewall.”

Alternatives and When to Choose Them

• No conntrack (stateless rules). You can firewall statelessly (match on addresses/ports per packet) and pay zero conntrack cost — but you lose established,related and must hand-write return-traffic rules, and you cannot do NAT (which requires conntrack to store the mapping). Choose stateless only for pure high-rate drop/allow with no NAT, ideally in XDP.
• eBPF flow tables (Cilium, Katran). For service-mesh / L4-LB scale, an eBPF map-based flow table in XDP or tc-BPF replaces nf_conntrack with a purpose-built, cheaper structure. Choose this when conntrack is the bottleneck and you control the datapath.
• Flowtable offload. Keep conntrack but offload established flows to the nf_flowtable fast path (optionally to hardware, IPS_HW_OFFLOAD) so they bypass full rule evaluation. Best of both worlds for a forwarding box with steady long-lived flows.

See Also

Where this note stops. Conntrack is the flow table: tuples, states, timeouts, the hash, and the insert race. Two neighbours own what sits on top of it:

Question	Owned by
“Why does my DNAT-to-multiple-backends setup trigger clashes? What does --random-fully change? Why am I running out of source ports?”	Network Address Translation NAT
“Where exactly does nf_conntrack_in run relative to my rules, and at what priority?”	The Netfilter Framework and Hooks

The dividing line at nf_ct_resolve_clash() is stated in full under “The Organising Fact” above.

• The Netfilter Framework and Hooks — the hook framework conntrack registers on (priority -200 / INT_MAX)
• Network Address Translation NAT — built directly on conntrack; the address mapping lives in the nf_conn
• iptables and x_tables / nftables and the nf_tables Subsystem — the rule engines whose ctstate match reads conntrack
• The Network Receive Path — where nf_conntrack_in runs at PRE_ROUTING
• The Routing Subsystem and FIB — the routing decision conntrack/NAT straddle
• NetworkPolicy — Kubernetes stateful policy ultimately backed by conntrack
• XDP Express Data Path — the eBPF fast path that bypasses conntrack at scale
• MOC: Linux Networking Stack MOC (§7 Netfilter)