The neighbour subsystem is the protocol-independent core in net/core/neighbour.c that answers one question for every outgoing packet on a multi-access link: “I know the next-hop IP address — what link-layer (MAC) address do I write into the Ethernet header?” It maintains a neighbour table of struct neighbour entries, each keyed by an L3 address and carrying the resolved L2 address plus a small state machine (NUD_INCOMPLETE, NUD_REACHABLE, NUD_STALE, NUD_DELAY, NUD_PROBE, NUD_FAILED) that tracks how trustworthy that mapping is. Both IPv4 ARP (the arp_tbl) and IPv6 Neighbour Discovery (the nd_tbl) are just two clients of this one generic engine: they supply a constructor, a solicit callback, and a hash, and the generic code does the queueing, timing, garbage collection, and reachability bookkeeping (neighbour.c, v6.12). Two things about it are worth carrying away before anything else. The state machine is the subsystem — nine states, a timer, and a rule for what a packet does in each — and it is RFC 4861’s IPv6 machine, retrofitted onto IPv4 ARP, which is why ip neigh shows IPv4 entries in states RFC 826 never defined. And the table is a cache with a hard capacity: gc_thresh3 defaults to 1,024 entries, and exceeding it does not evict, it fails the allocation and drops the packet with arp_cache: neighbor table overflow! in dmesg. That is the production failure every operator of a container host or a flat-L2 router eventually meets, and it is the note’s centre of gravity. This note covers the generic engine; the IPv4/IPv6 protocol specifics live in ARP and Neighbour Discovery.
All code and constants below are pinned to Linux 6.12, read from raw.githubusercontent.com/torvalds/linux/v6.12/ and re-verified line by line on 2026-09-12. Per kernel.org/releases.json read the same day, 6.12 is one of five longterm branches (currently 6.12.109, alongside 6.18.51, 6.6.156, 6.1.187 and 5.15.220); it is the vault’s pin, not the only LTS. The neighbour core has been architecturally stable for many years, and every mechanism described here was also observed running live on Linux 7.1.13 (Fedora 44) — the box the measurements in Demonstrated on This Box come from — so the claims hold well outside the pinned tag. Where a number was read from this machine rather than from source, it says so.
What this note owns, and what it does not. This note owns the generic neighbour engine: struct neigh_table, struct neighbour, the NUD state machine, the arp_queue, the timers, and the garbage collector — everything in net/core/neighbour.c. It stops at three boundaries. The Routing Subsystem and FIB owns the longest-prefix-match lookup that produces the next-hop address this subsystem resolves; the handoff is exact — the FIB answers “which nexthop”, this note answers “how to address that nexthop on the wire”, and neither knows anything about the other’s data structures. ARP and Neighbour Discovery owns the two protocol clients’ wire formats and receive-side parsing (arp_process(), ndisc_recv_ns()); this note covers only the generic hooks they plug into. The Network Transmit Path owns everything after dev_queue_xmit(). Network Namespaces owns the namespace machinery that makes the demonstration below possible; this note only uses it.
Mental Model
Think of the neighbour subsystem as a per-protocol cache with a confidence rating attached to each entry. The cache maps “next-hop IP” to “MAC address.” But a MAC mapping is not a fact you learn once and keep forever — the machine on the other end can be powered off, replaced, or move to a different switch port, and its IP can be reassigned. So each cached entry carries a state describing how recently the kernel had positive evidence that the mapping is still live, and a timer that drives the entry through that state as time passes and traffic flows.
The decisive design choice is that this confidence machinery is shared across address families. ARP for IPv4 and Neighbour Discovery for IPv6 differ only in how they solicit (ARP broadcasts an ARP_REQUEST; ND multicasts an ICMPv6 Neighbour Solicitation) and how they parse a reply. Everything else — the table, the hashing, the per-entry state machine, the queue of packets waiting on resolution, the garbage collector, the reachability confirmation from TCP — is identical, so it is written once in neighbour.c and parameterized by a struct neigh_table.
The Neighbour Unreachability Detection (NUD) state machine, taken directly from the transitions in __neigh_event_send() and neigh_timer_handler() (neighbour.c). What it shows: a new entry begins NUD_NONE, goes NUD_INCOMPLETE while resolving, and lands in NUD_REACHABLE once a reply arrives; thereafter it ages to NUD_STALE, is re-probed via NUD_DELAY→NUD_PROBE when traffic resumes, and is declared NUD_FAILED if probes go unanswered. The insight: the kernel never blindly trusts an old MAC mapping forever — NUD_REACHABLE is “confirmed recently”; NUD_STALE is “probably still right but unverified”; and the DELAY/PROBE cycle is a lazy, traffic-triggered re-verification that costs nothing while a flow is idle. The flag bits (NUD_REACHABLE = 0x02, NUD_STALE = 0x04, etc.) are a bitmask so the timer code can test families of states at once with state & NUD_IN_TIMER.
Read those three masks as three different questions. NUD_VALID asks “is there a usable MAC address in ha[]?” — six of the eight states say yes. NUD_CONNECTED asks the stronger “is the MAC address trusted enough to skip the state-machine re-check on the transmit fast path?” — only three states say yes, which is why neigh_connect() swaps output to neigh_connected_output() exactly on entry to those. NUD_IN_TIMER asks “does this entry have a running per-entry timer?” — the four transient states do; STALE, FAILED, NOARP, PERMANENT and NONE sit still until something happens to them.
State
Bit
VALID?
CONNECTED?
IN_TIMER?
Entered when
Left when
What a transmitted packet does
NUD_NONE
0x00
no
no
no
neigh_alloc() zeroes the entry
first packet arrives for it
triggers the transition to INCOMPLETE; the packet itself is queued
NUD_INCOMPLETE
0x01
no
no
yes
first transmit to an unresolved address
a reply arrives (→REACHABLE) or probes >= neigh_max_probes() (→FAILED)
queued on arp_queue; __neigh_event_send() returns 1 so the caller does not transmit
NUD_REACHABLE
0x02
yes
yes
yes
solicitation answered, or an upper layer confirmed a DELAY entry
now > confirmed + reachable_time
sent immediately via neigh_connected_output() — no state check, no branch
NUD_STALE
0x04
yes
no
no
reachable_time expired and the entry has been idle longer than delay_first_probe_time
a packet is sent (→DELAY), or the GC reaps it
sent immediately using the stale MAC, and the entry moves to DELAY
NUD_DELAY
0x08
yes
no
yes
a packet was sent on a STALE entry, orreachable_time expired while the entry was recently used
confirmed is fresh (→REACHABLE) or delay_first_probe_time elapses (→PROBE)
sent immediately; the entry is already usable, re-verification runs in the background
NUD_PROBE
0x10
yes
no
yes
DELAY expired without confirmation
probe answered (→REACHABLE) or probes exhausted (→FAILED)
sent immediately using the old MAC while unicast probes go out in parallel
NUD_FAILED
0x20
no
no
no
probes exhausted in INCOMPLETE or PROBE
garbage collected, or a reply arrives and neigh_update() revives it
dropped (SKB_DROP_REASON_NEIGH_FAILED); arp_queue was already purged by neigh_invalidate()
NUD_NOARP
0x40
yes
yes
no
the constructor saw a device with IFF_NOARP, or a point-to-point/loopback/multicast destination
never (pseudo-state)
sent immediately; there is nothing to resolve
NUD_PERMANENT
0x80
yes
yes
no
ip neigh add ... nud permanent
only by explicit administrative deletion
sent immediately; never aged, never garbage collected
The complete NUD state table. What it shows: the bit value, which of the three composite masks each state belongs to, and — the column that actually matters operationally — what happens to a packet handed to an entry in that state. The insight to take: only two of the nine states ever stop a packet. INCOMPLETE queues it and FAILED drops it; the other seven all transmit immediately. The STALE→DELAY→PROBE cycle is deliberately designed so that re-verification never costs the data path a single delayed packet — Linux sends on a possibly-stale MAC and checks afterwards, because a wrong MAC costs one round trip of loss while a blocked send costs the application latency on every refresh.
Two of these are pseudo-states in the strict sense the UAPI header uses: NUD_NOARP and NUD_PERMANENT “never change and make no address resolution or NUD” (neighbour.h L74–L79). They are in NUD_CONNECTED — so they take the fast output path — but not in NUD_IN_TIMER, so no timer ever fires for them and no probe is ever sent.
The Data Structures
The table itself is a struct neigh_table (neighbour.h). The fields that matter conceptually:
struct neigh_table { int family; /* AF_INET, AF_INET6, AF_DECnet ... */ unsigned int key_len; /* 4 for IPv4, 16 for IPv6 */ __u32 (*hash)(const void *pkey, ...); /* family hash */ int (*constructor)(struct neighbour *); /* arp_constructor */ struct neigh_parms parms; /* tunables: reachable_time, probe counts */ int gc_interval; int gc_thresh1, gc_thresh2, gc_thresh3; /* GC water marks */ struct delayed_work gc_work; /* the periodic garbage collector */ atomic_t entries; /* total entries */ atomic_t gc_entries; /* entries subject to GC */ struct neigh_hash_table __rcu *nht; /* the RCU-protected hash table */ struct pneigh_entry **phash_buckets; /* "proxy" neighbour table */};
nht is an RCU-protected hash table: the read path (every transmitted packet does a lookup) walks the bucket chains under rcu_read_lock() with no write lock at all, which is why the neighbour table is one of the canonical RCU read-mostly structures the networking stack relies on. Writers (insert, delete, resize) take tbl->lock and use rcu_assign_pointer / call_rcu so in-flight readers see a consistent snapshot.
Each entry is a struct neighbour:
struct neighbour { struct neighbour __rcu *next; /* hash chain */ struct neigh_table *tbl; struct neigh_parms *parms; unsigned long confirmed; /* jiffies of last reachability proof */ unsigned long updated; /* jiffies of last state change */ refcount_t refcnt; unsigned int arp_queue_len_bytes; struct sk_buff_head arp_queue; /* packets waiting on resolution */ struct timer_list timer; /* drives the NUD state machine */ unsigned long used; /* jiffies of last use (for GC) */ atomic_t probes; /* solicitations sent so far */ u8 nud_state; /* NUD_REACHABLE etc. */ seqlock_t ha_lock; /* protects ha[] (the MAC) */ unsigned char ha[...]; /* the resolved hardware address */ struct hh_cache hh; /* cached L2 header */ int (*output)(struct neighbour *, struct sk_buff *); const struct neigh_ops *ops; struct net_device *dev; u8 primary_key[];/* the L3 address (variable length) */};
Three fields carry most of the meaning. confirmed is the timestamp (in jiffies, the kernel’s tick counter) of the last positive proof that this neighbour is reachable — set when a solicitation reply arrives, and crucially also updated by upper layers (see Reachability confirmation below). arp_queue is the queue of packets parked while the entry is NUD_INCOMPLETE; the misleading name (it is used by IPv6 ND too, not just ARP) is historical. output is a function pointer that the state machine swaps to change behaviour without a branch in the fast path: when an entry is connected (NUD_REACHABLE), output points at neigh_connected_output (fast — the header can be prepended directly); when suspect (NUD_STALE and friends), it points at neigh_resolve_output (which re-checks state). The ha[] MAC address is protected by a seqlock_t so the read path can copy it racing with a writer without locking, retrying only if the writer was concurrent.
How a TX Packet Drives Resolution
The single most important code path is what happens when the IP layer hands a packet down to a neighbour that is not yet resolved. The packet does not block and does not get dropped — it is queued, a solicitation is fired, and the packet is sent later when the reply arrives. This is the whole reason the subsystem exists.
sequenceDiagram
autonumber
participant App as Application
participant IP as IP layer (dst_output)
participant N as neighbour entry
participant Q as arp_queue
participant Wire as Ethernet segment
participant Peer as 10.99.0.2
App->>IP: sendto(10.99.0.2)
IP->>N: neigh_resolve_output(neigh, skb)
Note over N: state NUD_NONE
N->>N: __neigh_event_send() sets NUD_INCOMPLETE<br/>probes = ucast_solicit (3)
N->>Q: __skb_queue_tail(skb) — packet parked, rc = 1
N->>Wire: neigh_probe() calls ops->solicit = arp_solicit()<br/>broadcast ARP request "who has 10.99.0.2"
Wire->>Peer: ff:ff:ff:ff:ff:ff ARP request
Peer-->>Wire: unicast ARP reply, lladdr 76:95:cd:53:e8:62
Wire->>N: arp_process() calls neigh_update(NUD_REACHABLE, lladdr)
N->>N: __neigh_update() writes ha[] under ha_lock<br/>state NUD_INCOMPLETE to NUD_REACHABLE
N->>Q: drain: while (skb = __skb_dequeue(arp_queue))
Q->>N: re-inject each parked skb through n1->output
N->>Wire: neigh_connected_output() prepends header, dev_queue_xmit()
Wire->>Peer: the original IP packet, finally on the wire
Cold resolution end to end, with the message contents taken from a run captured on this machine (see Demonstrated on this box below — the peer MAC 76:95:cd:53:e8:62 is the real one from that run). What it shows: the application’s sendto() returns long before the packet is on the wire. Between step 5 and step 12 the packet is sitting in kernel memory on arp_queue while a broadcast ARP request goes out and a unicast reply comes back. The insight to take: resolution is asynchronous and invisible to the application. There is no blocking system call and no error — a send() to an unresolvable address succeeds, and the loss only shows up later as SKB_DROP_REASON_NEIGH_FAILED once the entry reaches NUD_FAILED. This is why “the connection just silently does nothing” is the classic symptom of a broken L2 path, and why ip neigh show is the first command to run.
The solicit callback in step 6 is one of only two protocol-specific hooks on the whole path. arp_tbl installs arp_solicit via one of three neigh_ops vectors chosen by arp_constructor() at entry-creation time: arp_generic_ops (solicit + neigh_resolve_output), arp_hh_ops (same, but connected_output also goes through neigh_resolve_output because the device caches a hardware header), and arp_direct_ops, which has no solicit callback at all and points both outputs at neigh_direct_output — that is the vector used for NUD_NOARP devices (net/ipv4/arp.c L130–L150, v6.12).
Note the detail in neigh_probe() (neighbour.c L1068): it peeks the tail of arp_queue, clones that skb, and hands the clone to solicit. The clone exists so that the queue can overflow and drop the original without the in-flight solicitation losing the packet it needed to derive the source IP address from — ARP requests carry a sender protocol address, and it must be the address the queued packet would have used.
The entry point is neigh_resolve_output() (neighbour.c), which the routing layer installs as the dst->output for a route whose next-hop is unresolved:
int neigh_resolve_output(struct neighbour *neigh, struct sk_buff *skb){ int rc = 0; if (!neigh_event_send(neigh, skb)) { /* (1) try to resolve / confirm */ struct net_device *dev = neigh->dev; unsigned int seq; if (dev->header_ops->cache && !READ_ONCE(neigh->hh.hh_len)) neigh_hh_init(neigh); /* (2) build cached L2 header */ do { __skb_pull(skb, skb_network_offset(skb)); seq = read_seqbegin(&neigh->ha_lock); err = dev_hard_header(skb, dev, ntohs(skb->protocol), neigh->ha, NULL, skb->len); /* (3) prepend MAC hdr */ } while (read_seqretry(&neigh->ha_lock, seq)); /* (4) seqlock retry */ if (err >= 0) rc = dev_queue_xmit(skb); /* (5) hand to the device queue */ else goto out_kfree_skb; } return rc; /* if (1) returned non-zero, skb was queued */}
Line (1): neigh_event_send() is a thin wrapper around __neigh_event_send(). It returns 0 if the neighbour is already resolved and the packet may proceed immediately, and non-zero if the packet has been queued pending resolution (in which case neigh_resolve_output returns without sending — the queued copy will be sent later). Lines (2)–(4): once resolved, the code builds (or reuses a cached) link-layer header. dev_hard_header writes the Ethernet header using neigh->ha, read under the ha_lock seqlock — read_seqbegin/read_seqretry form a retry loop so that if a writer updated ha[] concurrently, the read is redone. Line (5): dev_queue_xmit pushes the now-complete frame down to the qdisc/driver — this is the boundary into The Network Transmit Path.
The interesting logic is inside __neigh_event_send(), which is the heart of the queueing-and-soliciting behaviour:
int __neigh_event_send(struct neighbour *neigh, struct sk_buff *skb, const bool immediate_ok){ int rc = 0; write_lock_bh(&neigh->lock); if (neigh->nud_state & (NUD_CONNECTED | NUD_DELAY | NUD_PROBE)) goto out_unlock_bh; /* (A) already usable: rc=0, send now */ if (!(neigh->nud_state & (NUD_STALE | NUD_INCOMPLETE))) { /* state is NUD_NONE: start resolving */ if (NEIGH_VAR(neigh->parms, MCAST_PROBES) + NEIGH_VAR(neigh->parms, APP_PROBES)) { atomic_set(&neigh->probes, NEIGH_VAR(neigh->parms, UCAST_PROBES)); WRITE_ONCE(neigh->nud_state, NUD_INCOMPLETE); /* (B) -> INCOMPLETE */ neigh->updated = jiffies; ... neigh_add_timer(neigh, next); /* arm retransmit timer */ } else { WRITE_ONCE(neigh->nud_state, NUD_FAILED); /* no probing configured */ kfree_skb_reason(skb, SKB_DROP_REASON_NEIGH_FAILED); return 1; } } else if (neigh->nud_state & NUD_STALE) { WRITE_ONCE(neigh->nud_state, NUD_DELAY); /* (C) stale entry, used: -> DELAY */ neigh_add_timer(neigh, jiffies + NEIGH_VAR(neigh->parms, DELAY_PROBE_TIME)); } if (neigh->nud_state == NUD_INCOMPLETE) { if (skb) { while (neigh->arp_queue_len_bytes + skb->truesize > NEIGH_VAR(neigh->parms, QUEUE_LEN_BYTES)) { /* (D) bound the queue */ struct sk_buff *buff = __skb_dequeue(&neigh->arp_queue); if (!buff) break; neigh->arp_queue_len_bytes -= buff->truesize; kfree_skb_reason(buff, SKB_DROP_REASON_NEIGH_QUEUEFULL); } skb_dst_force(skb); __skb_queue_tail(&neigh->arp_queue, skb); /* (E) park the packet */ neigh->arp_queue_len_bytes += skb->truesize; } rc = 1; /* tell caller: queued, do not send */ }out_unlock_bh: if (immediate_probe) neigh_probe(neigh); /* (F) fire the solicitation now */ ... return rc;}
The decision tree: (A) if the entry is already NUD_CONNECTED/DELAY/PROBE, it is usable — return 0 and let the caller transmit immediately. (B) if it is fresh (NUD_NONE), transition to NUD_INCOMPLETE, seed the probe counter, and arm the retransmit timer. (C) if it is NUD_STALE (probably-valid-but-unverified), the packet is sent using the stale MAC, but the entry transitions to NUD_DELAY to trigger lazy re-verification — this is the optimization that avoids re-soliciting on every idle-then-active flow. (D)–(E): while NUD_INCOMPLETE, the packet is appended to arp_queue, but the queue is bounded by QUEUE_LEN_BYTES (the unres_qlen_bytes sysctl): if it would overflow, the oldest queued packet is dropped with SKB_DROP_REASON_NEIGH_QUEUEFULL. (F)neigh_probe() actually calls the family solicit callback (ARP request / ND solicitation). Returning rc = 1 tells neigh_resolve_output not to transmit — the packet is now the kernel’s responsibility, to be flushed when the reply lands.
Resolution Completes — Flushing the Queue
When the ARP reply or ND advertisement arrives, the protocol code calls neigh_update(), which transitions the entry to a valid state and drains arp_queue. The relevant fragment of __neigh_update():
if (!(old & NUD_VALID)) { /* we were INCOMPLETE, now becoming VALID */ struct sk_buff *skb; while (neigh->nud_state & NUD_VALID && (skb = __skb_dequeue(&neigh->arp_queue)) != NULL) { struct dst_entry *dst = skb_dst(skb); struct neighbour *n2, *n1 = neigh; write_unlock_bh(&neigh->lock); rcu_read_lock(); /* re-lookup the top-level neigh in the path (shaper/eql/teql may differ) */ n2 = (dst && dst->obsolete != DST_OBSOLETE_DEAD) ? dst_neigh_lookup_skb(dst, skb) : NULL; if (n2) n1 = n2; READ_ONCE(n1->output)(n1, skb); /* re-inject each queued packet */ if (n2) neigh_release(n2); rcu_read_unlock(); write_lock_bh(&neigh->lock); } __skb_queue_purge(&neigh->arp_queue); neigh->arp_queue_len_bytes = 0;}
Each parked packet is dequeued and pushed back through n1->output — which is now neigh_connected_output (the entry is valid), so the L2 header is prepended and dev_queue_xmit sends it for real. The comment about re-looking-up the neighbour exists because intermediate qdiscs like sch_teql or the bonding/eql layer can route the packet through a different neighbour than the one that just resolved, so the code re-derives the correct output neighbour per packet rather than assuming the resolved entry. This re-injection is the moment a send() that “blocked” on resolution finally reaches the wire.
The Timer: Ageing and Re-Probing
neigh_timer_handler() is the engine that ages entries. It runs per-entry whenever the entry is in a state with the NUD_IN_TIMER bit set (INCOMPLETE, REACHABLE, DELAY, PROBE). Its logic, condensed:
NUD_REACHABLE: if now <= confirmed + reachable_time, the entry is still fresh; re-arm for the remaining time. If it has gone idle past DELAY_PROBE_TIME, drop to NUD_STALE (call neigh_suspect(), which swaps output back to the slow path). The default reachable_time for ARP is 30 seconds (arp_tbl.parms.reachable_time = 30 * HZ, arp.c), randomized per-entry between ½× and 1½× to avoid synchronized refreshes.
NUD_DELAY: if the entry was confirmed within DELAY_PROBE_TIME, promote back to NUD_REACHABLE (an upper-layer confirmation arrived — no probe needed). Otherwise transition to NUD_PROBE and start sending unicast solicitations.
NUD_PROBE / NUD_INCOMPLETE: keep re-transmitting solicitations every RETRANS_TIME. Once probes >= neigh_max_probes(), give up: transition to NUD_FAILED, call neigh_invalidate(), and notify userspace via netlink.
The confirmed-vs-used distinction is what makes NUD_DELAY cheap. used is updated on every transmit; confirmed is updated only by proof of reachability. A flow that keeps sending updates used constantly, but if confirmed is stale the entry still drops to NUD_STALE/DELAY — Linux does not assume that the act of sending proves the neighbour received anything. Only a reply (or an upper-layer confirmation) updates confirmed.
neigh_max_probes() — how many solicitations before FAILED
With the stock values measured on this machine (ucast_solicit = 3, app_solicit = 0, mcast_solicit = 3, mcast_resolicit = 0) this evaluates to 6 for an INCOMPLETE entry (3 + 0 + 3) and 3 for a PROBE entry (3 + 0 + 0, because mcast_resolicit defaults to 0). A never-seen address therefore burns six solicitations before being declared FAILED, while a previously-known neighbour that stops answering is declared dead after only three — the kernel is less patient with something that used to work than with something it has never met. The measured run below records exactly probes 6 on the failed entry, which is the arithmetic confirming itself.
The counter is reset on the DELAY→PROBE transition (atomic_set(&neigh->probes, 0) in neigh_timer_handler()), which is why the PROBE budget is independent of anything spent earlier.
Tunables — /proc/sys/net/{ipv4,ipv6}/neigh/
Every tunable is per-interface with a default/ template; a new interface inherits default/ at creation time, so changing default/ does not retroactively affect interfaces that already exist. The values below were read directly from this machine (Fedora 44, Linux 7.1.13, USER_HZ = 100) on 2026-09-12 and cross-checked against the arp_tbl / nd_tbl initializers in the v6.12 source, which match.
Tunable (neigh/default/)
IPv4
IPv6
Source constant (v6.12)
What it governs
gc_thresh1
128
128
arp_tbl.gc_thresh1
Floor. Periodic GC returns immediately while tbl->entries is below it — entries are never reaped at all under this mark.
gc_thresh2
512
512
arp_tbl.gc_thresh2
Soft pressure. Above it, neigh_alloc() runs a forced GC if the last flush was >5 s ago, and forced GC’s clean target is gc_entries − gc_thresh2.
gc_thresh3
1024
1024
arp_tbl.gc_thresh3
Hard ceiling on non-PERMANENT entries. Allocation past it fails and logs neighbor table overflow!. This is the number that causes production outages.
base_reachable_time_ms
30000
30000
30 * HZ / ND_REACHABLE_TIME
Nominal REACHABLE lifetime. Not used directly — see reachable_time below.
base_reachable_time
30
30
same value in seconds
Deprecated seconds-granularity alias of the above. Writing either updates both.
delay_first_probe_time
5
5
NEIGH_VAR_DELAY_PROBE_TIME = 5 * HZ
Seconds spent in DELAY before escalating to PROBE; also the idle window that decides REACHABLE→DELAY versus REACHABLE→STALE.
retrans_time_ms
1000
1000
1 * HZ / ND_RETRANS_TIMER = HZ
Interval between successive solicitations while INCOMPLETE or PROBE. Floored at HZ/100 (10 ms) by the timer handler.
retrans_time
100
1000
—
Trap. The IPv4 file is in USER_HZ ticks (100 ticks = 1 s); the IPv6 file is in milliseconds. Same underlying value, two different units — always read retrans_time_ms.
mcast_solicit
3
3
NEIGH_VAR_MCAST_PROBES = 3
Multicast/broadcast probes sent in INCOMPLETE.
ucast_solicit
3
3
NEIGH_VAR_UCAST_PROBES = 3
Unicast probes sent in PROBE to reconfirm a known address.
mcast_resolicit
0
0
not in the initializer → 0
Extra multicast probes after unicast+app probes in PROBE. Zero by default, so a PROBE entry never falls back to broadcasting.
app_solicit
0
0
not in the initializer → 0
Probes handed to a userspace ARP daemon over netlink before falling back. Zero means no daemon.
gc_stale_time
60
60
NEIGH_VAR_GC_STALETIME = 60 * HZ
Idle seconds after which periodic GC may reap an entry (only once above gc_thresh1).
gc_interval
30
30
arp_tbl.gc_interval = 30 * HZ
Nominal GC period. Note: the periodic worker actually re-arms itself at base_reachable_time >> 1 (15 s), not at gc_interval.
unres_qlen_bytes
212992
212992
SK_WMEM_MAX
Byte cap on arp_queue per entry. 212,992 B = 208 KiB, sized to hold “256 packets of medium size”.
unres_qlen
101
101
derived, deprecated since 3.3
Read-only-in-spirit packet-count view of the same limit.
locktime
100
0
NEIGH_VAR_LOCKTIME = 1 * HZ (IPv4 only)
Minimum ticks before an ARP reply may overwrite an existing ha[]. An anti-flap and weak anti-spoof measure. nd_tbl omits it entirely — hence the 0.
anycast_delay
100
100
1 * HZ
Max random delay before answering a solicitation for an anycast address.
proxy_delay
80
80
(8 * HZ) / 10
Max random delay before answering a proxy ARP/NDP request (0.8 s).
proxy_qlen
64
64
NEIGH_VAR_PROXY_QLEN = 64
Depth of the proxy-request backlog.
interval_probe_time_ms
5000
5000
5 * HZ
Probe interval for NTF_MANAGED entries only — entries userspace asked the kernel to keep permanently resolved. Minimum 1.
Every neigh tunable, with the value read from this box and the kernel constant it comes from. What it shows: the IPv4 and IPv6 columns are identical everywhere except retrans_time (a units difference, not a value difference) and locktime (genuinely IPv4-only). The insight to take: two entries in this table are unit traps and one is a documentation trap. retrans_time means ticks on IPv4 and milliseconds on IPv6. base_reachable_time and base_reachable_time_ms are the same knob in different units. And gc_interval is not the GC interval — neigh_periodic_work() re-queues itself with NEIGH_VAR(&tbl->parms, BASE_REACHABLE_TIME) >> 1 (neighbour.c L1030–L1031), so on stock settings the sweep runs every 15 seconds regardless of what gc_interval says.
reachable_time is randomized, and not the value you wrote
base_reachable_time_ms is a base, never the actual timeout. neigh_periodic_work() recomputes each neigh_parms’ effective reachable_time every 300 seconds:
The kernel’s own comment states the range: “ARP entry timeouts range from 1/2 BASE_REACHABLE_TIME to 3/2 BASE_REACHABLE_TIME.” So with the stock 30 s base, a given entry’s REACHABLE lifetime is somewhere in 15–45 seconds, redrawn every five minutes. This is deliberate de-synchronization, the same idea RFC 4861 §6.3.2 mandates for IPv6 so that a segment full of hosts does not re-probe in lockstep after a router advertisement. It also means you cannot predict when a specific entry will age out, and a test that sleeps base_reachable_time and asserts STALE is flaky by construction.
Reachability Confirmation From Upper Layers
A subtle and important optimization: the neighbour subsystem does not have to re-ARP a busy neighbour, because TCP tells it the path is alive. When a TCP connection receives an acknowledgement for data it sent, the dst/neighbour was demonstrably reachable, so the stack calls neigh_confirm():
static inline void neigh_confirm(struct neighbour *n){ if (n) { unsigned long now = jiffies; if (READ_ONCE(n->confirmed) != now) /* avoid dirtying the cache line */ WRITE_ONCE(n->confirmed, now); }}
This sets confirmed = now (carefully avoiding a redundant write to keep the cache line clean under contention). The next time the timer runs, a NUD_DELAY entry sees a recent confirmed and is promoted straight back to NUD_REACHABLEwithout ever sending a probe. This is the kernel exploiting transport-layer evidence to avoid redundant link-layer chatter — a heavily-used TCP connection never triggers an ARP/ND re-solicitation as long as data keeps flowing both ways. UDP and other protocols that lack acknowledgements do not provide this signal, so their neighbours fall back to the active probe cycle.
Garbage Collection and the gc_thresh Trio
Neighbour entries consume kernel memory and must be reaped. The table is a cache, and this is the part of the note that matters most in production: every other mechanism here degrades gracefully, but the capacity limit does not — it drops packets.
flowchart TD
A["neigh_alloc() — a packet needs<br/>an entry that does not exist"] --> B{"exempt_from_gc?"}
B -->|yes| Z["allocate<br/>(does not count toward gc_entries)"]
B -->|no| C["entries = atomic_inc_return(&tbl->gc_entries) - 1"]
C --> D{"entries >= gc_thresh3<br/>(1024)?"}
D -->|yes| F["neigh_forced_gc()"]
D -->|no| E{"entries >= gc_thresh2 (512)<br/>AND last_flush older than 5s?"}
E -->|yes| F
E -->|no| Z
F --> G{"freed anything<br/>OR now below gc_thresh3?"}
G -->|yes| Z
G -->|no| X["net_info_ratelimited<br/>'arp_cache: neighbor table overflow!'<br/>table_fulls++ — ALLOCATION FAILS<br/>packet dropped"]
Z --> Y["entry born NUD_NONE,<br/>output = neigh_blackhole"]
W["neigh_periodic_work()<br/>every base_reachable_time/2 = 15s"] --> V{"tbl->entries < gc_thresh1<br/>(128)?"}
V -->|yes| U["return — do nothing at all"]
V -->|no| T["walk every hash bucket<br/>reap refcnt==1 entries that are<br/>NUD_FAILED or idle > gc_stale_time"]
The allocation path’s water-mark check on the left, the periodic collector on the right. What it shows: three thresholds acting at two completely different times — gc_thresh1 gates a background sweep on tbl->entries, while gc_thresh2 and gc_thresh3 gate a synchronous emergency collection on tbl->gc_entries, inline on the transmit path. The insight to take: follow the no edge out of the last decision. When forced GC cannot free anything and the table is still at gc_thresh3, there is no queue, no retry and no backpressure — neigh_alloc() returns NULL, the packet is dropped, and the only trace is a rate-limited log line and a counter. The neighbour table is the one cache in the transmit path that fails closed.
Two mechanisms exist, governed by three water marks set in arp_tbl/nd_tbl (defaults from arp.c): gc_thresh1 = 128, gc_thresh2 = 512, gc_thresh3 = 1024 (tunable via /proc/sys/net/ipv4/neigh/default/gc_thresh{1,2,3}).
Periodic GC (neigh_periodic_work, a deferrable workqueue item re-armed every BASE_REACHABLE_TIME/2): if entries < gc_thresh1 it does nothing — below the first water mark, entries are left alone. Above it, it walks the whole hash table and reaps entries that are NUD_FAILEDor have been idle past GC_STALETIME, but only if their refcount is 1 (nobody else references them) and they are not NUD_PERMANENT or externally learned (NTF_EXT_LEARNED).
Forced GC (neigh_forced_gc), invoked synchronously from neigh_alloc() when allocating a new entry would push gc_entries >= gc_thresh3 (or >= gc_thresh2 and the last flush was more than 5 seconds ago): it aggressively walks the gc_list and removes reclaimable entries (NUD_FAILED, NUD_NOARP, stale), bounded to ~1 ms of work. If even forced GC cannot free enough and entries >= gc_thresh3, the allocation fails with the famous neighbour table overflow! log message and the packet that triggered it is dropped:
The water marks form a hysteresis band: gc_thresh1 is the floor below which GC is disabled, gc_thresh2 is a “soft” pressure point where forced GC kicks in if it has been a while, and gc_thresh3 is the hard ceiling that caps the table size. The interview-favourite gotcha is that gc_thresh3 is the true maximum number of entries — a router talking to thousands of hosts on a flat L2 segment will hit it and start dropping, which is exactly the failure mode below.
The two counters that the water marks are compared against are different
This is the subtlety that makes gc_thresh tuning confusing in practice. struct neigh_table carries two atomics, and the three water marks are not all compared against the same one:
tbl->entries — every entry in the table, including PERMANENT and NOARP ones.
tbl->gc_entries — only entries subject to garbage collection. neigh_alloc() takes an exempt_from_gc argument and, when it is set, jumps straight to do_alloc without touching gc_entries at all.
neigh_periodic_work() gates on atomic_read(&tbl->entries) < READ_ONCE(tbl->gc_thresh1), but neigh_alloc() compares gc_entries against gc_thresh2 and gc_thresh3 (neighbour.c L469–L492, v6.12). So gc_thresh3 is, precisely, the maximum number of garbage-collectable entries — which is why the ip-sysctl.rst documentation words it as “Maximum number of non-PERMANENT neighbor entries allowed” (Documentation/networking/ip-sysctl.rst L189–L194). Statically pinned entries do not consume the budget, which is a real mitigation: on a host with a handful of known peers, ip neigh add ... nud permanent removes them from the pressure calculation entirely.
The eviction criteria also differ between the two collectors, and the difference matters when you are reasoning about which entries survive an overflow:
Periodic GC (neigh_periodic_work)
Forced GC (neigh_forced_gc)
Trigger
Workqueue, re-armed every base_reachable_time / 2 (15 s stock)
Synchronously inside neigh_alloc(), on the allocation that crosses a water mark
Gate
tbl->entries >= gc_thresh1
gc_entries >= gc_thresh3, orgc_entries >= gc_thresh2 and last_flush older than 5 s
Walks
the whole hash table, bucket by bucket, with cond_resched()
tbl->gc_list only — the list of GC-eligible entries
Skips
NUD_PERMANENT, anything in NUD_IN_TIMER, NTF_EXT_LEARNED
anything with refcnt != 1
Reaps when
refcnt == 1and (NUD_FAILED or idle longer than gc_stale_time, 60 s)
refcnt == 1and (NUD_FAILED or NUD_NOARP or multicast or updated older than 5 s)
Bounded by
nothing — it is a workqueue item and may resched
1 ms of wall time (ktime_get_ns() + NSEC_PER_MSEC, checked every 16 entries) and a max_clean of gc_entries − gc_thresh2
The two garbage collectors compared. What it shows: they gate on different counters, walk different structures, and use different eviction criteria. The insight to take: forced GC is the one that runs when you are in trouble, and it is hard-capped at one millisecond of work. Under real overflow pressure it often cannot free gc_entries − gc_thresh2 entries in 1 ms, returns a shrunk short of the target, and the allocation fails anyway. That is the mechanism behind the observation that once a box starts logging overflow it tends to keep logging it — the emergency collector is deliberately too polite to dig you out, because it runs in softirq context on the transmit path and a long walk there would be worse than the drop.
Note the exact spelling of the message: the kernel logs neighbor table overflow! — American spelling, with an exclamation mark, prefixed by the table id (arp_cache or ndisc_cache), through net_info_ratelimited() (neighbour.c L486–L487). Grepping dmesg for “neighbour” finds nothing. The rate limiting also means the log line count bears no relation to the number of packets dropped.
Instrumenting It: /proc/net/stat/arp_cache
Alongside the per-entry view from ip neigh, the kernel exports table-wide counters at /proc/net/stat/arp_cache and /proc/net/stat/ndisc_cache. Read from this machine (32 logical CPUs), the header and first rows look like this:
All values are hexadecimal and there is one row per CPU. Two things about this file trip people up.
entries is a global value printed on every row — never sum it
Twelve of the thirteen columns are genuine per-CPU counters pulled from struct neigh_statistics, and summing them across rows is correct. The first column is not. neigh_stat_seq_show() prints atomic_read(&tbl->entries) — a single table-wide atomic — identically on every row (neighbour.c L3468–L3500, v6.12):
seq_printf(seq, "%08x %08lx %08lx %08lx ...", atomic_read(&tbl->entries), /* GLOBAL — same on every row */ st->allocs, st->destroys, st->hash_grows, ...);
On this 32-CPU box every row reads 00000064 = 100 entries, the true table size. A monitoring agent that sums the column reports 3,200 — a 32× inflation, and one that then alerts on approaching gc_thresh3 = 1024 fires a false alarm on a table that is 10 % full. Verify the column you are aggregating is per-CPU before you aggregate it; awk over this file is a common and wrong idiom.
The columns worth alerting on are the last few. table_fulls is incremented on exactly the same line that logs neighbor table overflow!, so it is the machine-readable form of that event and, unlike the log message, it is not rate-limited — it counts every failed allocation. unresolved_discards counts packets dropped from a full arp_queue. forced_gc_runs rising is the early warning that precedes table_fulls: it means allocations are already crossing gc_thresh2. res_failed counts entries that reached NUD_FAILED (incremented in neigh_invalidate()), which on this box reads a handful per CPU from ordinary WiFi churn.
A second trap: this file is per network namespace only in the sense that /proc/net follows the reader’s netns. A fresh namespace created with unshare -n that has not yet had /proc remounted will show the host’s/proc/net through an inherited mount, or nothing at all — the run below hit exactly that and cat /proc/net/stat/arp_cache returned No such file or directory.
Demonstrated on This Box
Everything above was checked against a live run on the machine this note was written on: AMD Ryzen AI MAX+ 395, Fedora 44, Linux 7.1.13, 32 logical CPUs, 2026-09-12. The technique is unshare -rn, which maps the invoking uid to 0 inside a new user namespace and gives that mapped root CAP_NET_ADMINinside the new network namespace only — enough to create interfaces, assign addresses and write /proc/sys/net/ipv4/neigh/* without any real privilege on the host.
unshare -rn bash -c ' unshare -n sleep 600 & # a second netns to be the peer PEER=$! ip link add veth0 type veth peer name veth1 ip link set veth1 netns $PEER # move one end away, so ARP is really needed ip addr add 10.99.0.1/24 dev veth0 && ip link set veth0 up nsenter -t $PEER -n ip addr add 10.99.0.2/24 dev veth1 nsenter -t $PEER -n ip link set veth1 up echo 3000 > /proc/sys/net/ipv4/neigh/veth0/base_reachable_time_ms ip monitor neigh dev veth0 & ping -c1 10.99.0.2'
The second namespace is the part that matters. A veth pair with both ends in the same namespace does not exercise the neighbour subsystem at all: 10.99.0.2 is then a local address, the route lookup resolves it through the local table, and no entry is ever created. The first attempt at this demonstration made that mistake and produced an empty ip neigh show — a useful negative result, and a reminder that ip neigh being empty can mean “nothing to resolve” rather than “resolution is broken”.
Result 1 — the full REACHABLE/DELAY/PROBE cycle, with base_reachable_time_ms cut to 3000 so the ageing is observable inside a short run:
First, the entry went REACHABLE → DELAY, never touching STALE. That is not a bug and it is the transition the mental-model diagram above was corrected for. neigh_timer_handler() splits the exit from REACHABLE on used, not on confirmed:
if (state & NUD_REACHABLE) { if (time_before_eq(now, neigh->confirmed + neigh->parms->reachable_time)) { next = neigh->confirmed + neigh->parms->reachable_time; /* still alive */ } else if (time_before_eq(now, neigh->used + NEIGH_VAR(neigh->parms, DELAY_PROBE_TIME))) { WRITE_ONCE(neigh->nud_state, NUD_DELAY); /* used recently */ ... } else { WRITE_ONCE(neigh->nud_state, NUD_STALE); /* genuinely idle */ notify = 1; }}
The ping at t=0 set used, so when the (randomized) reachable_time expired around t≈3 s the entry was still within delay_first_probe_time = 5 s of its last use and went straight to DELAY. An entry you stop using entirely goes to STALE instead. STALE is the resting state of an idle neighbour; DELAY is the resting state of an actively-used one whose confirmation has expired.
Second, DELAY resolved back to REACHABLE by t≈8 s, because the unicast probe was answered. The cycle then repeated — REACHABLE again at t≈12–16 s, DELAY again at t≈20 s. This is the steady state of a neighbour that is alive but not carrying acknowledged traffic: a single unicast probe roughly every reachable_time, forever.
Third, the ip monitor neigh transcript for the same run reads:
The DELAY transitions are missing from the netlink stream even though polling saw them. That is by design, not a race: in neigh_timer_handler() the REACHABLE→DELAY branch is the only transition that does not set notify = 1, so no RTM_NEWNEIGH is emitted. REACHABLE→STALE, DELAY→REACHABLE, DELAY→PROBE and anything→FAILED all do notify. The practical consequence is that a tool built on ip monitor neigh cannot see an entry enter DELAY and will report a neighbour as REACHABLE for the whole delay window. Polling ip neigh show and streaming ip monitor neigh genuinely disagree here, and the stream is the incomplete one.
Result 2 — forcing FAILED by ARPing an address that does not exist:
+ ping -c1 -W1 10.99.0.77
--- t+0s: 10.99.0.77 INCOMPLETE
--- t+2s: 10.99.0.77 INCOMPLETE
--- t+5s: 10.99.0.77 FAILED
--- t+10s: 10.99.0.77 FAILED
$ ip -s neigh show dev veth0
10.99.0.77 used 11/71/7 probes 6 FAILED
The entry stayed INCOMPLETE for between two and five seconds, then flipped to FAILED — consistent with retrans_time_ms = 1000 and neigh_max_probes() = ucast_solicit + app_solicit + mcast_solicit = 3 + 0 + 3 = 6. And ip -s neigh reports probes 6, the arithmetic landing exactly on the predicted value. Note also that the FAILED entry has no lladdr at all: neigh_invalidate() purged arp_queue and the entry carries no hardware address, which is why every subsequent packet to it is dropped rather than sent somewhere wrong.
The three numbers in used 11/71/7 are, in order, used, confirmed and updated, each printed as jiffies elapsed since, not as an absolute. confirmed = 71 on an entry that was never confirmed is neigh_add_timer()’s clamp at work: it pushes n->confirmed to a safe distance from the jiffies wraparound point rather than leaving it at zero, “still show[ing] to user space large times in the past”.
Result 3 — NOARP interfaces bypass all of this. This box runs tailscale0:
$ ip -br link show tailscale0
tailscale0 UNKNOWN <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP>
$ ip neigh show dev tailscale0
(empty)
tailscale0 is a POINTOPOINT TUN device carrying IFF_NOARP. A point-to-point link has exactly one possible destination, so there is nothing to resolve: “which MAC do I address this to” has no meaning when the device has no link-layer addressing at all.
arp_constructor() (net/ipv4/arp.c, v6.12) has four separate routes into NUD_NOARP, and it is worth reading them precisely because the obvious guess is wrong:
Case (1) is the one tailscale0 takes, and the discriminator is !dev->header_ops — “this device has no link-layer header at all” — not IFF_NOARP. That is the only case that gets arp_direct_ops, whose .solicit pointer is NULL, so neigh_probe()’s if (neigh->ops->solicit) guard simply does nothing. Cases (2)–(4) are Ethernet-class devices that do have a header: multicast, where the MAC is derived arithmetically from the IP by arp_mc_map(); explicitly IFF_NOARP or loopback interfaces, which use the device’s own MAC; and broadcast destinations or point-to-point devices, which use dev->broadcast. All four land in NUD_NOARP, which is in NUD_CONNECTED (so packets take the fast output path) but not in NUD_IN_TIMER (so no timer ever fires and no probe is ever sent). The constructor also notes at the top that for IFF_LOOPBACK | IFF_POINTOPOINT it overwrites primary_key with INADDR_ANY — every destination on a point-to-point link collapses to one entry, because they are all the same wire. It also means gc_thresh3 pressure from a tunnel-heavy host comes from the underlay interface, not the tunnels. And it means a NOARP interface can never produce the FAILED state, which is worth remembering when a WireGuard or Tailscale path is black-holing: ip neigh will tell you nothing, because the neighbour subsystem was never in the path.
By contrast the box’s real L2 interfaces do keep entries. enp191s0 is up but carries no neighbours (no traffic on that segment at the time of reading), while the WiFi interface wlp192s0 shows the ordinary mix:
$ ip neigh show
192.168.1.21 dev wlp192s0 lladdr ec:91:61:8c:69:57 STALE
192.168.1.1 dev wlp192s0 lladdr 2c:70:4f:54:9e:8f REACHABLE
192.168.1.37 dev wlp192s0 lladdr de:5d:69:ab:18:c5 REACHABLE
fe80::2e70:4fff:fe54:9e8f dev wlp192s0 lladdr 2c:70:4f:54:9e:8f router STALE
The default gateway is REACHABLE because traffic is flowing through it and TCP keeps confirming it; the idle host at .21 has aged to STALE. Note that the IPv6 link-local entry for the same router is a separate entry in a separate table (nd_tbl, exported as ndisc_cache) resolving to the same MAC — the two protocol tables share no state whatsoever, which is why an IPv4 neighbour table overflow leaves IPv6 connectivity intact and vice versa. Note also that ip neigh show with no arguments hidesNOARP and FAILED entries. Measured on this box:
Command
Entries
ip neigh show
4
ip neigh show nud all
167 — of which 163 are NOARP, 2 REACHABLE, 2 STALE
ip -4 neigh show nud all
97
/proc/net/stat/arp_cache column 1
0x64 = 100
/proc/net/stat/ndisc_cache column 1
0x48 = 72
The gap between four and 167 is the default filter, not a discrepancy — this host has 24 Docker bridges and six veth interfaces, and every one of them contributes NOARP entries for broadcast and multicast destinations. The 97-versus-100 and 70-versus-72 differences are ordinary churn between two commands run a second apart. The operational lesson: on a container host, the entries that count against gc_thresh3 are overwhelmingly ones ip neigh show does not print. Always diagnose an overflow with ip neigh show nud all | wc -l, or better, read the table counter out of /proc/net/stat/arp_cache — which, being the global atomic described above, is the one number that is exactly right.
Pseudo-States, Proxy, and the ip neigh View
Two states are pseudo-states that never participate in the timer machine: NUD_PERMANENT (a static entry added by an administrator with ip neigh add ... nud permanent, never garbage-collected, never aged) and NUD_NOARP (an entry on an interface that does not use neighbour resolution at all — e.g. point-to-point links, or addresses where the MAC is implied). NUD_NONE is the just-allocated, not-yet-resolved zero state.
There is also a separate proxy neighbour table (phash_buckets) used for Proxy ARP — answering ARP requests on behalf of another host (e.g. so a router can answer for hosts behind it). This is a distinct mechanism from the main table and is keyed in pneigh_entry structures.
Userspace observes and manipulates all of this through the ip neighbour command and the NETLINK_ROUTE socket. ip neigh show prints each entry’s L3 address, device, MAC, and NUD state in lowercase (reachable, stale, delay, probe, failed, permanent). This is the same RTM_NEWNEIGH/RTM_DELNEIGH netlink interface the kernel uses to notify userspace of state transitions (neigh_update_notify), which is how tools like arpwatch and routing daemons learn about reachability changes.
ARP versus Neighbour Discovery — What Actually Differs
The generic engine is shared, so the differences between the two clients are narrower than people expect: they are almost entirely about how a solicitation is carried on the wire and what the protocol lets you carry alongside it. The state machine, the timers, the queue, and the GC are identical code.
Its own L2 protocol, EtherType 0x0806 — not carried in IP at all
ICMPv6 inside IPv6, so it inherits IPsec, extension headers and the IPv6 checksum
Solicitation
ARPOP_REQUEST (opcode 1) broadcast to ff:ff:ff:ff:ff:ff — every host on the segment takes the interrupt
Neighbour Solicitation, ICMPv6 type 135, multicast to the target’s solicited-node address ff02::1:ffXX:XXXX — NIC hardware multicast filters drop it for uninterested hosts
Reply
ARPOP_REPLY (opcode 2), unicast
Neighbour Advertisement, ICMPv6 type 136, with Solicited / Override / Router flags
MAC carried in
fixed header fields (ar_sha / ar_tha)
TLV options: Source Link-Layer Address (ND_OPT_SOURCE_LL_ADDR, type 1) and Target Link-Layer Address (type 2) — optional, so an NA can legitimately carry no MAC
Anti-spoof at receive
none in the protocol; Linux adds locktime (1 s) to refuse rapid ha[] overwrites
hop-limit must be exactly 255, checked in ndisc_rcv() for all five message types before dispatch — an off-link attacker cannot forge one, because any router would have decremented it
Unsolicited update
gratuitous ARP — a request or reply for your own address
ARP against Neighbour Discovery, field by field. What it shows: the two protocols differ in encapsulation, addressing discipline, and security posture — but not in cache semantics. The insight to take: ND’s two real wins are the solicited-node multicast address (which lets the NIC filter out solicitations for other hosts, instead of ARP’s broadcast that wakes every CPU on the segment) and the hop-limit-255 rule, which is a free, exact off-link filter that ARP has no equivalent of.
The state machine is RFC 4861’s, retrofitted onto ARP
This is the historical fact that explains the whole design, and it is easy to miss because the note’s subject is net/core/. The NUD state machine is an IPv6 invention. RFC 826 — two and a half pages of ARP, from 1982 — specifies no timeouts, no reachability states and no probe counts at all; it says only that an implementation may age entries. RFC 4861 §7.3.2 defines INCOMPLETE, REACHABLE, STALE, DELAY and PROBE by name, with those exact semantics, and §10 fixes the constants.
Put the RFC’s constants next to what this machine actually reports and the provenance is unmistakable:
RFC 4861 §10 constant
RFC value
Linux neigh/default/ file
IPv4 value read here
REACHABLE_TIME
30,000 ms
base_reachable_time_ms
30000
RETRANS_TIMER
1,000 ms
retrans_time_ms
1000
DELAY_FIRST_PROBE_TIME
5 seconds
delay_first_probe_time
5
MAX_MULTICAST_SOLICIT
3 transmissions
mcast_solicit
3
MAX_UNICAST_SOLICIT
3 transmissions
ucast_solicit
3
MAX_ANYCAST_DELAY_TIME
1 second
anycast_delay
100 ticks = 1 s
MIN_RANDOM_FACTOR / MAX_RANDOM_FACTOR
0.5 / 1.5
the neigh_rand_reach_time() range
½× – 1½×
RFC 4861’s protocol constants against this box’s IPv4 ARP defaults. What it shows: every single one matches — including the randomization factors. The insight to take: Linux did not design these numbers, and it did not design them for ARP. arp_tbl in net/ipv4/arp.c is populated with the IPv6 Neighbour Discovery constants, because the generic engine in net/core/neighbour.c is an implementation of RFC 4861’s neighbour cache that ARP was subsequently made a client of. This is why ip neigh shows IPv4 entries in states RFC 826 never defined, and why the IPv4 and IPv6 tunable trees are near-identical — they are the same spec, read twice.
Failure Modes
neighbor table overflow! in dmesg, intermittent connectivity — the failure everyone eventually hits. This is the centre of gravity of the whole subsystem. The neighbour table is a cache, and like every cache it has a hard capacity; unlike most caches, exceeding it does not evict, it fails the allocation and drops the packet.
The symptom set is distinctive and worth memorising, because the surface presentation looks nothing like “a table is full”:
Bursts of packet loss only to destinations not already in the table. Established flows keep working perfectly, because their entries already exist. This is why the problem presents as “new connections randomly fail while existing ones are fine” — the single most misleading symptom in the set, and the reason people chase DNS, conntrack or MTU first.
net_info_ratelimited() output in dmesg, spelled arp_cache: neighbor table overflow! — American spelling, so dmesg | grep neighbour finds nothing.
table_fulls in /proc/net/stat/arp_cache climbing. This is the reliable signal: unlike the log line it is not rate-limited, so it counts every failure.
ip neigh show nud all | wc -l approaching gc_thresh3. Note nud all — as measured above, the default listing hid 163 of this box’s 167 entries.
It bites routers, Kubernetes nodes and dense virtualization hosts on a large flat L2 network. The kernel’s own documentation states the trigger and the remedy in one sentence: gc_thresh3 is the “Maximum number of non-PERMANENT neighbor entries allowed. Increase this when using large numbers of interfaces and when communicating with large numbers of directly-connected peers” (ip-sysctl.rst L189–L194, v6.12). A container host satisfies both clauses at once — this box alone has 30 network interfaces contributing 167 entries at idle.
The fix is to raise all three water marks together, keeping the hysteresis band, e.g. net.ipv4.neigh.default.gc_thresh1=4096, gc_thresh2=8192, gc_thresh3=16384, and the matching net.ipv6.neigh.default.* (the two tables are independent and are sized independently). Raise gc_thresh1 along with the others: leaving it at 128 while raising gc_thresh3 means the periodic collector runs constantly and sweeps the entire — now much larger — hash table every 15 seconds for nothing. Two caveats that follow from the mechanism rather than from folklore: the knobs are read through READ_ONCE() on every allocation, so a sysctl -w takes effect immediately with no restart; but neigh/default/ is only a template copied at interface creation, so interfaces that already exist keep the old values and must be set individually or the interfaces recreated.
Uncertain
Verify: the claim, present in an earlier revision of this note, that “the Cilium and Calico documentation both call out gc_thresh3 tuning for large clusters.” Reason: checked and not substantiated. On 2026-09-12 the string gc_thresh appeared zero times in each of docs.cilium.io/en/stable/operations/performance/tuning/ (HTTP 200, 102,580 B), the same page’s reStructuredText source in cilium/cilium@main, docs.tigera.io/calico/latest/operations/troubleshoot/troubleshooting, the amazon-vpc-cni-k8s README, kubernetes/kopscluster_spec.md, the Kubernetes website’s sysctl-cluster.md, the GKE troubleshooting page, the AKS custom-node-configuration page, and the OpenShift recommended-host-practices page. The advice itself is sound and is stated by the kernel’s own ip-sysctl.rst, cited above — but the vendor-documentation attribution is withdrawn as unverified. To resolve: find a specific vendor page, issue or runbook that names the tunable, or leave the kernel documentation as the sole citation. uncertain
Stale entry pointing at a dead/replaced host (silent black-holing). If a host changes its MAC (NIC swap, VM migration, failover that does not send a gratuitous ARP) while an entry is NUD_STALE, the first packets are sent to the old MAC and silently lost until the DELAY→PROBE cycle re-solicits and neigh_update() corrects ha[]. The window is roughly DELAY_PROBE_TIME (default 5 s). This is why HA failover schemes broadcast a gratuitous ARP — it forces every neighbour to update ha[] immediately rather than waiting for the lazy probe.
arp_queue overflow under a resolution stall. If resolution is slow (the target is briefly unreachable) while an application sends hard, packets pile up in arp_queue until unres_qlen_bytes is hit, then the oldest queued packets are dropped (SKB_DROP_REASON_NEIGH_QUEUEFULL). A burst of UDP to an unresolved next-hop can lose its leading packets this way while later ones survive — counter-intuitive head-drop behaviour.
NUD_FAILED storms. A neighbour that genuinely cannot be resolved (no host at that IP) cycles to NUD_FAILED after mcast_probes solicitations, and every subsequent packet to it re-enters the machine and re-fails. Misconfigured routes pointing at non-existent next-hops generate continuous ARP traffic and failed-entry churn visible in ip -s neigh.
Alternatives and Boundaries
The neighbour subsystem is not the routing table. Routing (the FIB) decides which interface and which next-hop IP a packet takes; the neighbour subsystem only resolves that next-hop IP to a MAC. They are two distinct lookups: first the longest-prefix-match route lookup yields a dst_entry with a next-hop, then the neighbour lookup resolves the next-hop. On a directly-connected destination the next-hop is the destination itself; on a routed destination it is the gateway, which is why a host on a remote network resolves the router’s MAC, not the destination’s.
flowchart LR
P["skb for 93.184.216.34"] --> R["FIB lookup<br/>fib_lookup() / LPC-trie<br/>OWNED BY: The Routing Subsystem and FIB"]
R --> D["dst_entry<br/>oif = enp191s0<br/>nexthop = 192.168.1.1"]
D --> N["neighbour lookup<br/>__neigh_lookup on arp_tbl<br/>key = 192.168.1.1<br/>OWNED BY: this note"]
N --> H{"nud_state?"}
H -->|"NUD_CONNECTED"| C["neigh_connected_output()<br/>prepend cached L2 header"]
H -->|"NUD_STALE / DELAY / PROBE"| S["neigh_resolve_output()<br/>re-check state, then send"]
H -->|"NUD_NONE / INCOMPLETE"| Q["queue on arp_queue,<br/>solicit, return"]
C --> X["dev_queue_xmit()<br/>OWNED BY: The Network Transmit Path"]
S --> X
The two-lookup pipeline and where the ownership boundaries fall. What it shows: the routing lookup is keyed on the destination93.184.216.34 and answers with an interface plus a nexthop; the neighbour lookup is keyed on the nexthop192.168.1.1 and answers with a MAC. The insight to take: these are two different keys, and that is why a traceroute-visible routing problem and an ip neigh-visible resolution problem produce identical symptoms at the application. ip route get <dst> tells you what the first lookup decided; ip neigh show <that nexthop> tells you what the second one did. Diagnose in that order — resolving the wrong one wastes the outage.
For interfaces that genuinely have no link-layer addressing to resolve — loopback, point-to-point tunnels, some IP-over-IP — entries are created NUD_NOARP and the whole solicit machinery is bypassed. For static environments where MACs are known and fixed, administrators install NUD_PERMANENT entries to skip resolution and ageing entirely (common in security-sensitive setups to defeat ARP spoofing, paired with arptables/nftables filtering).
See Also
The Routing Subsystem and FIB — the direct upstream. Its LPC-trie longest-prefix-match answers which nexthop; this note answers how to address that nexthop on the wire. The handoff is the dst_entry’s nexthop address becoming this table’s lookup key. It owns route selection, nexthop groups and ECMP; this note owns none of that.
ARP and Neighbour Discovery — the IPv4 ARP and IPv6 ND protocol clients that plug into this generic engine (arp_tbl, nd_tbl, the solicit/parse callbacks, the wire formats). This note owns the engine, that note owns the protocols.
The Network Transmit Path — where neigh_resolve_output → dev_queue_xmit hands the completed frame to the qdisc/driver. Everything past dev_queue_xmit() belongs there.
The Network Receive Path — the inbound counterpart; ARP replies and Neighbour Advertisements arrive through it before reaching neigh_update().
struct sk_buff — the packet buffer queued in arp_queue and re-injected on resolution; skb->truesize is what unres_qlen_bytes is accounted against.
Network Namespaces — each netns has its own arp_tbl and its own neigh/ sysctl tree, which is what makes the unshare -rn demonstration above safe and reproducible.