vault

The Network Receive Path

16 min read

The Network Receive Path

The receive (RX) path is the chain of kernel code that turns a frame sitting in a Network Interface Card’s (NIC’s) memory ring into bytes a reader can pull out of a socket with recv(). Its defining shape is interrupt-then-poll: a hardware interrupt tells the kernel “frames have arrived,” but the real per-packet work is deferred to a software interrupt (softirq) where the driver polls its ring under a budget. That deferred work climbs the stack one layer at a time — driver → NAPI poll → optional Generic Receive Offload (GRO) coalescing → __netif_receive_skb_core (taps, eXpress Data Path (XDP), the rx_handler, and protocol dispatch through the ptype tables) → the protocol handler ip_rcv, which is where the netfilter PREROUTING hook fires → the transport layer, which finally drops the data onto a struct sock’s receive queue and wakes any sleeping reader. This note traces that path function by function against Linux 6.12 LTS (net/core/dev.c, net/core/gro.c, net/ipv4/ip_input.c; verified 2026-06-05). The sibling NAPI and Polled Receive zooms into the poll loop itself; The IP Layer takes over where this note hands off to ip_rcv.

Mental Model

Think of the RX path as a bucket brigade with one deliberate hand-off from the interrupt world to the softirq world. The NIC’s hard interrupt is loud but does almost nothing — it just rings a bell (napi_schedule) and goes quiet. The actual hauling of packets up the stack happens later, in NET_RX_SOFTIRQ context, where the driver’s poll() function drains its receive ring in batches. This decoupling is the entire performance story of modern Linux networking: under a flood, the kernel stops taking one interrupt per packet and instead polls, amortizing the cost of crossing the interrupt boundary across many frames.

flowchart TB
  WIRE["Frame on the wire"] --> DMA["NIC DMA: write frame<br/>into RX ring descriptor"]
  DMA --> IRQ["Hard IRQ handler (driver)<br/>mask IRQ, napi_schedule()"]
  IRQ -->|"raises NET_RX_SOFTIRQ"| SOFTIRQ["net_rx_action()<br/>per-CPU softirq, budget=300"]
  SOFTIRQ -->|"calls driver"| POLL["driver poll(): drain RX ring<br/>build sk_buff per frame"]
  POLL --> GRO["napi_gro_receive()<br/>coalesce same-flow segments"]
  GRO -->|"flush"| NRSKB["netif_receive_skb (core)<br/>__netif_receive_skb_core()"]
  NRSKB --> TAPS["taps (AF_PACKET / tcpdump)<br/>generic XDP, tc ingress, rx_handler"]
  TAPS -->|"ptype_base dispatch"| IPRCV["ip_rcv()<br/>NF_INET_PRE_ROUTING hook"]
  IPRCV --> ROUTE["ip_rcv_finish: route lookup<br/>local deliver vs forward"]
  ROUTE -->|"local"| L4["ip_local_deliver -> tcp_v4_rcv / udp_rcv"]
  L4 --> RECVQ["enqueue on sk->sk_receive_queue<br/>sk_data_ready() wakes reader"]
  RECVQ -->|"recv() returns"| APP["Application"]

The receive path from wire to socket in Linux 6.12. What it shows: the single hand-off point (the dashed transition from the hard IRQ to net_rx_action) splits the path into an “interrupt half” that does almost nothing and a “softirq half” that does all the real work — allocating struct sk_buffs, coalescing them with GRO, running every inspection hook, and dispatching to the protocol handler. The insight to take: every box from __netif_receive_skb_core downward runs in softirq context on one CPU, holds rcu_read_lock(), and is where packet-steering, firewalling, and protocol demultiplexing all live — so latency and CPU cost on RX are dominated by what happens after the poll, not by the interrupt itself.

Mechanical Walk-through

1. DMA and the hard interrupt — the driver does almost nothing

When a frame arrives, the NIC uses Direct Memory Access (DMA) to copy it straight into a buffer the driver pre-posted in its receive ring (a circular array of descriptors, each pointing at a DMA-mapped page). The NIC then raises a hardware interrupt. The driver’s hard-IRQ handler is deliberately tiny: it acknowledges and masks further RX interrupts for that queue, then calls napi_schedule() (which expands to __napi_schedule). That, in turn, calls ____napi_schedule to add the driver’s struct napi_struct to the current CPU’s softnet_data.poll_list and raises NET_RX_SOFTIRQ (net/core/dev.c). Then the hard IRQ returns. No sk_buff has been allocated yet; no protocol code has run. This is the core of the NAPI (“New API”) design — see NAPI and Polled Receive for the full state machine.

There is a legacy fallback for non-NAPI drivers and for software devices: netif_rx(). It calls netif_rx_internal(), which (optionally consulting Receive Packet Steering, RPS) calls enqueue_to_backlog() to push the skb onto a per-CPU backlog queue (softnet_data.input_pkt_queue) and schedules the special per-CPU backlog NAPI (sd->backlog, whose poll is process_backlog). The kernel comment on netif_rx is blunt: “Modern NIC driver should use NAPI and GRO” (net/core/dev.c, v6.12). So netif_rx is the slow door; real NICs go through napi_gro_receive.

2. net_rx_action — the softirq poll loop with a budget

NET_RX_SOFTIRQ runs net_rx_action() (net/core/dev.c). It splices the CPU’s poll_list into a local list and loops, calling napi_poll(n, &repoll) on each scheduled NAPI instance. Two limits bound how long it runs: a packet budget (net_hotdata.netdev_budget, default 300) decremented by the work each poll reports, and a time limit of netdev_budget_usecs (default 2000 µs) converted to jiffies. When either is exhausted, the loop bumps sd->time_squeeze, re-raises NET_RX_SOFTIRQ, and bails — deferring the rest so the CPU is not monopolized:

if (unlikely(budget <= 0 ||
             time_after_eq(jiffies, time_limit))) {
        sd->time_squeeze++;
        break;
}

That time_squeeze counter, visible in /proc/net/softnet_stat (column 3), is the canonical signal that RX softirq processing is being throttled — a classic symptom under heavy load. Each NAPI gets a per-poll weight (NAPI_POLL_WEIGHT = 64, include/linux/netdevice.h): the driver’s poll() must process at most weight packets and return the count. Returning less than the weight means “ring drained” and lets NAPI re-enable the NIC’s interrupt (via napi_complete_done); returning exactly the weight means “more to do,” so the instance stays on the poll list and is polled again. This is the heart of interrupt-vs-poll switching.

3. The driver poll() builds skbs and calls napi_gro_receive

Inside its poll(), the driver walks completed RX descriptors, and for each one allocates a struct sk_buff (typically via napi_build_skb/napi_alloc_skb), sets skb->protocol from the L2 header with eth_type_trans(), and hands the skb up by calling napi_gro_receive(napi, skb) (net/core/gro.c). This is the single most common entry into the generic stack from a modern driver. napi_gro_receive does:

gro_result_t napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
{
        skb_mark_napi_id(skb, napi);
        skb_gro_reset_offset(skb, 0);
        ret = napi_skb_finish(napi, skb, dev_gro_receive(napi, skb));
        return ret;
}

dev_gro_receive() is Generic Receive Offload: it tries to merge this skb into an existing skb on the NAPI’s GRO list if they belong to the same flow (same 4-tuple, consecutive TCP sequence numbers, compatible options). Merging means appending the new payload as a page fragment to the held skb, so the upper stack later processes one large skb instead of many small ones — the per-packet cost of routing, netfilter, and socket lookup is paid once for a coalesced super-segment. GRO is the receive-side inverse of TSO; its own leaf is Generic Receive Offload. When a merged skb is complete (or napi_gro_flush is called at the end of the poll because it is too old), napi_gro_complete → gro_normal_one queues it onto napi->rx_list, and gro_normal_list flushes that batch into netif_receive_skb_list_internal() — the list-based variant of the next stage. If GRO decides not to hold the skb (GRO_NORMAL), napi_skb_finish sends it on immediately the same way.

4. __netif_receive_skb_core — the grand central station of RX

Everything below the driver funnels into __netif_receive_skb_core() (net/core/dev.c), the function that decides who gets to see the packet and in what order. It runs under rcu_read_lock() and proceeds, in order:

  1. Timestamp and reset headers. It resets the network header offset (skb_reset_network_header) so L3 parsing starts at the right place.
  2. Generic XDP. If a generic-XDP program is attached (generic_xdp_needed_key), it runs do_xdp_generic(). A non-XDP_PASS verdict drops the packet here, before any tap sees it. (Native XDP runs earlier still, inside the driver before the skb exists — this generic path is the slow fallback.)
  3. VLAN untag for hardware-accelerated VLAN tags.
  4. Taps (ptype_all). It walks two lists of packet_type entries registered for all protocols: the global net_hotdata.ptype_all and the device’s skb->dev->ptype_all. This is where AF_PACKET sockets and tcpdump/libpcap receive their copy — every sniffer hooks ptype_all. The deliver_skb helper clones the skb to each tap.
  5. tc ingress and netfilter ingress. If ingress_needed_key is set, sch_handle_ingress() runs the clsact/ingress qdisc and any tc-BPF programs (Queueing Disciplines qdisc, Linux eBPF MOC), then nf_ingress() runs the netfilter ingress hook.
  6. rx_handler. rcu_dereference(skb->dev->rx_handler) — if the device has a receive handler registered (a Linux bridge, a bond/team, macvlan, or Open vSwitch datapath all install one), it is called. A handler can return RX_HANDLER_CONSUMED (it took the packet — e.g. the bridge forwarded it), RX_HANDLER_ANOTHER (re-loop, e.g. the packet was redirected to a different skb->dev), RX_HANDLER_EXACT, or RX_HANDLER_PASS. This is the kernel primitive that makes software bridging work transparently.
  7. Protocol dispatch via ptype_base. Finally, the exact protocol handler is chosen:
type = skb->protocol;
if (likely(!deliver_exact)) {
        deliver_ptype_list_skb(skb, &pt_prev, orig_dev, type,
                               &ptype_base[ntohs(type) & PTYPE_HASH_MASK]);
}

ptype_base is a hash table of packet_type handlers keyed by EtherType. ETH_P_IP (0x0800) resolves to the packet_type whose .func is ip_rcv; ETH_P_IPV6 (0x86DD) resolves to ipv6_rcv; ETH_P_ARP to arp_rcv. Each protocol registers its handler at init with dev_add_pack(). The dispatch defers the last matched handler into *ppt_prev and the one-core wrapper __netif_receive_skb_one_core calls it via an indirect-call-wrapper that the compiler can devirtualize to ip_rcv/ipv6_rcv for speed:

ret = INDIRECT_CALL_INET(pt_prev->func, ipv6_rcv, ip_rcv, skb,
                         skb->dev, pt_prev, orig_dev);

If no handler matches, the skb is freed with reason SKB_DROP_REASON_UNHANDLED_PROTO and rx_dropped is bumped.

5. Where RPS/RFS steers — netif_receive_skb_internal

There are two ways into the core function. The list path used by GRO is netif_receive_skb_list_internal; the single-skb public API is netif_receive_skb() → netif_receive_skb_internal(). The latter is where Receive Packet Steering (RPS) and Receive Flow Steering (RFS) intervene:

static int netif_receive_skb_internal(struct sk_buff *skb)
{
        ...
        if (static_branch_unlikely(&rps_needed)) {
                int cpu = get_rps_cpu(skb->dev, skb, &rflow);
                if (cpu >= 0) {
                        ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);
                        ...
                }
        }
        ret = __netif_receive_skb(skb);
        ...
}

get_rps_cpu() hashes the packet’s flow and, if RPS is configured, returns a different CPU to process it on. The skb is then enqueued on that CPU’s backlog queue (enqueue_to_backlog), which schedules the remote CPU’s backlog NAPI; process_backlog() there eventually calls __netif_receive_skb. This is how a single hardware RX queue’s interrupts can be fanned out across cores in software. The full mechanism — RPS, RFS, and the hardware Receive Side Scaling (RSS) that distributes interrupts across queues at the NIC — is the subject of Receive Side Scaling and Packet Steering. The key point for this note is where it hooks: between the GRO/driver layer and the protocol handler, by redirecting onto another CPU’s backlog.

6. ip_rcv — the protocol handler and the PREROUTING hook

ip_rcv() (net/ipv4/ip_input.c) is the registered ptype_base handler for IPv4. It first calls ip_rcv_core(), which validates the IP header — version is 4, header length ≥ 5 words, the IP header checksum via ip_fast_csum, and that skb->len is at least the declared total length — and trims the skb to the true IP length, orphaning any stale socket reference. Then comes the pivotal line:

return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
               net, NULL, skb, dev, NULL,
               ip_rcv_finish);

This is where the netfilter PREROUTING hook fires. NF_HOOK runs every registered hook function at the NF_INET_PRE_ROUTING chain point — this is where Destination NAT (DNAT) rewrites a packet’s destination, where connection tracking (conntrack) first sees an inbound flow, and where iptables -t nat -A PREROUTING / nft prerouting rules execute (see The Netfilter Framework and Hooks, Connection Tracking conntrack, Network Address Translation NAT). Only if the hooks return NF_ACCEPT does ip_rcv_finish run. The crucial architectural fact is that PREROUTING runs before the routing decision, so DNAT can change where the packet is routed.

ip_rcv_finish calls ip_rcv_finish_core (which does the FIB route lookup, caching the result in skb_dst) and then dst_input(skb), an indirect call to the route’s input function — ip_local_deliver for packets destined to this host, or ip_forward for packets that must be routed elsewhere. That local-deliver-versus-forward branch is the subject of IP Routing Decision and Forwarding; the L3 mechanics belong to The IP Layer.

7. Local delivery to L4 and the socket receive queue

For a locally destined packet, ip_local_deliver() runs the NF_INET_LOCAL_IN (INPUT) netfilter hook, reassembles IP fragments if needed, then hands the skb to the transport handler registered in the inet_protos array — tcp_v4_rcv for TCP, udp_rcv for UDP. The transport layer performs the socket lookup: it hashes the 4-tuple (source IP, source port, destination IP, destination port) to find the matching struct sock. For TCP, an established connection is found in the ehash; a SYN to a listener is found in the lhash and may go through SYN-cookie / accept-queue logic. Once the owning sock is found, the payload skb is appended to sk->sk_receive_queue (out-of-order data first lands in the out-of-order queue), accounted against the socket’s receive buffer (sk_rmem_alloc vs sk_rcvbuf), and the socket’s data-ready callback sk->sk_data_ready — normally sock_def_readable — is invoked. That wakes any thread blocked in recv() and fires the wait-queue callback that makes the socket readable for epoll. The reader’s subsequent recv() copies bytes out of sk_receive_queue into userspace, and the journey is complete. The transport-layer detail lives in The TCP Protocol in Linux and The UDP Protocol in Linux; the wakeup machinery in Socket Wait Queues and Wakeups.

The ptype_base table — protocol demultiplexing in detail

The kernel cannot hard-code “if EtherType is IPv4 call ip_rcv” because protocols are pluggable (IPv6, ARP, MPLS, AF_PACKET taps, and even IPv4 itself are all registered modules/subsystems). Instead each protocol calls dev_add_pack() at init with a struct packet_type:

static struct packet_type ip_packet_type __read_mostly = {
        .type = cpu_to_be16(ETH_P_IP),
        .func = ip_rcv,
        .list_func = ip_list_rcv,
};

dev_add_pack files this into one of three places depending on .type: ptype_all (if .type == 0, i.e. a tap that wants everything — AF_PACKET/pcap), the per-device ptype_specific list, or the global ptype_base[] hash bucket ntohs(type) & PTYPE_HASH_MASK for a specific EtherType. At RX time, __netif_receive_skb_core walks ptype_all first (so sniffers see frames before protocol processing, including frames that will later be dropped), then indexes ptype_base by EtherType to find the protocol owner. The list_func (ip_list_rcv) is the batched variant invoked when GRO delivered a list of skbs, amortizing the route lookup across the batch. This table is the concrete mechanism behind the abstract phrase “the stack demultiplexes by protocol.”

Failure Modes and How to Diagnose Them

Softirq starvation / time_squeeze. Under a packet flood, net_rx_action repeatedly hits its budget/time limit, bumping time_squeeze (column 3 of /proc/net/softnet_stat, per CPU). Symptoms: high %softirq CPU on a few cores (mpstat -P ALL), ksoftirqd threads pegged, latency spikes, and RX drops. Mitigations are exactly the RPS/RFS/RSS levers in Receive Side Scaling and Packet Steering (spread the load across cores), raising net.core.netdev_budget / netdev_budget_usecs, or enabling threaded NAPI (/sys/class/net/<dev>/threaded) so poll runs in a dedicated kthread the scheduler can balance, rather than in softirq context.

Backlog drops. When RPS or netif_rx enqueues to a remote CPU’s backlog and that queue exceeds net.core.netdev_max_backlog (net_hotdata.max_backlog), enqueue_to_backlog drops with reason SKB_DROP_REASON_CPU_BACKLOG and increments sd->dropped (column 2 of /proc/net/softnet_stat). A nonzero column-2 count means a CPU’s backlog overflowed — raise netdev_max_backlog or rebalance RPS.

Ring overruns (rx_missed/rx_fifo). If the driver cannot drain its RX ring fast enough (softirq throttled, or too few ring descriptors), the NIC drops frames at the hardware ring and reports them in ethtool -S <dev> counters like rx_missed_errors/rx_no_buffer. This is upstream of the kernel path entirely — the fix is more ring descriptors (ethtool -G), more RX queues, or interrupt coalescing tuning. Distinguish these from netif_receive_skb-level drops, which show as SKB_DROP_REASON_* in dropwatch / perf trace/tracepoints.

Unhandled-protocol drops. Frames whose EtherType has no ptype_base handler are freed with SKB_DROP_REASON_UNHANDLED_PROTO and bump rx_nohandler. Seeing this for an expected protocol usually means the relevant module is not loaded.

Diagnosing the path. The receive path is densely instrumented with tracepoints — trace_netif_receive_skb, trace_napi_gro_receive_entry/exit, trace_napi_poll, and per-skb kfree_skb with a drop reason. perf trace, bpftrace, and the dropwatch tool turn these into a precise picture of where a packet died.

Alternatives and When to Choose Them

The standard path described here allocates an sk_buff per (coalesced) frame and pays for the full protocol stack. Three escape hatches trade generality for speed:

  • Native XDP runs an eBPF program in the driver before __netif_receive_skb_core, before an skb is even allocated. It returns XDP_DROP (line-rate DDoS scrubbing), XDP_TX (bounce back out), XDP_REDIRECT (to another NIC or an AF_XDP socket), or XDP_PASS (fall through to the normal path above). Choose XDP when you must make a verdict at line rate and can express it in eBPF.
  • AF_XDP delivers raw frames into a userspace-shared memory ring (UMEM), bypassing the entire stack from __netif_receive_skb_core onward. Choose it for a userspace data plane (DPDK-style) that wants kernel driver support but not the kernel’s protocol processing.
  • The standard napi_gro_receive path (this note) is the right default for everything that needs the kernel’s TCP/IP stack, netfilter, routing, and socket semantics — i.e. ~all normal applications. GRO already recovers most of the per-packet overhead for streaming workloads.

Production Notes

The interrupt-then-poll design and the softnet_stat counters are the bread and butter of high-throughput tuning. Netflix, Cloudflare, and others have documented that on busy edge servers the dominant RX cost is not the interrupt but the per-packet work in and above __netif_receive_skb_core, which is precisely why GRO (fewer trips up the stack) and RPS/RFS (spread those trips across cores) matter so much. The kernel’s own scaling guide (docs.kernel.org/networking/scaling.html) is the primary reference for RSS/RPS/RFS tuning, and the NAPI document (docs.kernel.org/networking/napi.html) for poll/threaded-NAPI/busy-poll. A recurring real-world gotcha: a single hardware RX queue (common on cheap or virtualized NICs) funnels all interrupts to one CPU, so one core saturates while others idle — the fix is enabling RPS to fan the post-poll work out, since you cannot add hardware queues you do not have. Another: forgetting that tcpdump attaches at ptype_all, before netfilter and routing, so a packet captured by tcpdump may still be dropped by an iptables PREROUTING rule a microsecond later — the capture proves the frame reached the host, not that it was accepted.

The numeric defaults quoted here are confirmed against net/core/hotdata.c at v6.12, which initializes net_hotdata with .netdev_budget = 300, .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ (= 2000 µs at the common HZ = 1000), .max_backlog = 1000, and .dev_rx_weight = 64; NAPI_POLL_WEIGHT = 64 is in include/linux/netdevice.h. All are runtime-tunable via sysctl net.core.netdev_budget, netdev_budget_usecs, netdev_max_backlog, and dev_weight (the last scaled by dev_weight_rx_bias into dev_rx_weight), so a running system may show different values.

See Also

Graph View

Backlinks