vault

The IP Layer

16 min read

The IP Layer

The Internet Protocol (IP) layer is Linux’s implementation of the network layer (Layer 3, “L3”) — the connectionless, best-effort datagram service that gets a packet from one host to another across a routed internetwork. It owns three jobs: on receive, validate the IP header and decide whether a packet is for us (local delivery up to TCP/UDP/ICMP) or for someone else (forwarding); on transmit, prepend the IP header, pick a route, and hand the datagram to the link layer; and on forward, decrement the time-to-live and route the packet onward. In the Linux kernel this is ip_rcv → ip_rcv_finish → a routing decision that branches to ip_local_deliver (up the stack) or ip_forward (router), with ip_output/__ip_queue_xmit carrying locally-generated packets the other way (net/ipv4/ip_input.c, ip_output.c, v6.12 LTS, verified 2026-06-06). IPv4 follows RFC 791; IPv6 follows RFC 8200 and differs in ways this note makes precise — extension headers instead of fixed options, no header checksum, and routers that never fragment. This note picks up where The Network Receive Path hands off to ip_rcv, and hands off in turn to IP Routing Decision and Forwarding for the route lookup and IP Fragmentation and Reassembly for the MTU machinery.

Mental Model

Think of the IP layer as a sorting office with one decisive question at its center: “is this packet addressed to this building, or does it need to go back out for delivery elsewhere?” Everything upstream of that question is validation (is this a well-formed IP datagram?); everything downstream is disposition (deliver locally, forward, or — for outbound packets — route and ship). The pivotal mechanism is the routing decision, which on the receive path is wedged between the netfilter PREROUTING hook and the actual delivery, so that Destination Network Address Translation (DNAT) can change where a packet is routed before the routing lookup runs.

The single most important structural fact is that IP is a connectionless datagram service: each packet is routed independently, there is no per-packet ordering or retransmission guarantee, and the IP layer itself keeps almost no state about a “flow.” Reliability and ordering are the transport layer’s job (see The TCP Protocol in Linux); IP’s contract is only “make a best effort to deliver this single datagram to the destination address, or drop it and optionally complain via ICMP.”

flowchart TB
  subgraph RX["Receive — bottom-up"]
    RCV["ip_rcv()<br/>validate header, checksum, length"] -->|"NF_INET_PRE_ROUTING<br/>(PREROUTING: DNAT, conntrack)"| RFIN["ip_rcv_finish()<br/>FIB route lookup"]
    RFIN -->|"dst_input()"| DEC{"route decision:<br/>for this host?"}
    DEC -->|"local"| LD["ip_local_deliver()<br/>NF_INET_LOCAL_IN (INPUT)<br/>reassemble fragments"]
    DEC -->|"not local"| FWD["ip_forward()<br/>NF_INET_FORWARD<br/>ttl--, re-emit"]
    LD --> DISP["ip_local_deliver_finish()<br/>inet_protos[proto] dispatch"]
    DISP -->|"IPPROTO_TCP"| TCP["tcp_v4_rcv"]
    DISP -->|"IPPROTO_UDP"| UDP["udp_rcv"]
    DISP -->|"IPPROTO_ICMP"| ICMP["icmp_rcv"]
  end
  subgraph TX["Transmit — top-down"]
    QX["__ip_queue_xmit()<br/>route, push IP header, set TTL/ID"] -->|"ip_local_out:<br/>NF_INET_LOCAL_OUT (OUTPUT)"| OUT["ip_output()<br/>NF_INET_POST_ROUTING (SNAT)"]
    OUT --> FIN["ip_finish_output()<br/>MTU check -> fragment? -> ip_finish_output2"]
    FIN --> NEIGH["neighbour resolves L2 (MAC),<br/>hand to driver"]
  end
  FWD -.->|"dst_output"| FIN

The IP layer’s receive, transmit, and forward paths in Linux 6.12. What it shows: the receive path is a validate-then-route-then-dispatch pipeline, with the routing decision (dst_input) splitting local delivery from forwarding; the transmit path mirrors it, pushing a header, routing, and fragmenting if needed. The insight to take: the netfilter hooks straddle the routing decision deliberately — PREROUTING runs before the route lookup (so DNAT can redirect the packet) and POSTROUTING runs after it (so Source NAT/masquerade can rewrite the source once the outgoing interface is chosen). A forwarded packet (dotted line) re-enters the transmit machinery at ip_finish_output, which is why a router fragments forwarded traffic at exactly the same place a host fragments its own.

The IPv4 Header — What Every Field Means

Every IPv4 datagram begins with a 20-byte fixed header (struct iphdr), optionally followed by up to 40 bytes of options. The fields the IP layer actually acts on, walked in order:

  • Version (4 bits) — must be 4. ip_rcv_core checks iph->version != 4 and drops otherwise (ip_input.c:496).
  • IHL — Internet Header Length (4 bits) — header length in 32-bit words, so a value of 5 means 20 bytes (no options) and the maximum 15 means 60 bytes. The kernel rejects iph->ihl < 5.
  • TOS / DSCP+ECN (8 bits) — Differentiated Services Code Point plus the 2-bit Explicit Congestion Notification field. The kernel reads ECN bits to maintain per-type statistics (IPSTATS_MIB_NOECTPKTS + (iph->tos & INET_ECN_MASK)).
  • Total Length (16 bits) — the entire datagram (header + data) in bytes, capping a single IPv4 datagram at 65,535 bytes. The kernel reads it (iph_totlen), checks skb->len is at least this long, and trims the skb to the true IP length with pskb_trim_rcsum, because the link layer may have padded the frame.
  • Identification (16 bits), Flags (3 bits), Fragment Offset (13 bits) — the fragmentation control fields, the entire subject of IP Fragmentation and Reassembly. The IP layer here only tests them: ip_is_fragment(iph) returns true if the More-Fragments flag or a nonzero offset is set, which routes the packet into reassembly.
  • Time To Live, TTL (8 bits) — decremented by every forwarding router; when it would reach zero the router drops the packet and sends an ICMP “Time Exceeded.” This is what traceroute exploits. On the forward path, ip_forward checks ip_hdr(skb)->ttl <= 1 before decrementing.
  • Protocol (8 bits) — the L4 payload type: IPPROTO_TCP (6), IPPROTO_UDP (17), IPPROTO_ICMP (1). This is the index into the inet_protos[] dispatch table on local delivery.
  • Header Checksum (16 bits) — a ones-complement checksum over the header only (not the payload). ip_rcv_core verifies it with ip_fast_csum; a mismatch drops the packet with SKB_DROP_REASON_IP_CSUM. Because TTL changes on every hop, every forwarding router must recompute this checksum.
  • Source and Destination Address (32 bits each) — the IPv4 addresses that drive the routing lookup.

Mechanical Walk-through — The Receive Path

1. ip_rcv and ip_rcv_core — validation

ip_rcv() is the packet_type handler the protocol stack dispatched to from __netif_receive_skb_core when the EtherType was ETH_P_IP (0x0800) — see The Network Receive Path for how that dispatch works. Its first act is ip_rcv_core(skb, net), which performs the RFC-mandated sanity checks (ip_input.c:456): drop frames received in promiscuous mode that were not addressed to us (PACKET_OTHERHOST); ensure the linear skb area holds a full struct iphdr (pskb_may_pull); check version 4 and ihl >= 5; verify the header checksum with ip_fast_csum; validate that skb->len >= iph_totlen; and finally trim the skb down to the true total length. The kernel comment cites the rule directly: “RFC1122: 3.2.1.2 MUST silently discard any IP frame that fails the checksum.” If anything fails the skb is freed with a specific SKB_DROP_REASON_* and a SNMP counter (IPSTATS_MIB_INHDRERRORS, IPSTATS_MIB_CSUMERRORS) is bumped — these are the counters behind nstat//proc/net/snmp.

2. The PREROUTING hook fires before routing

Having validated the header, ip_rcv does not route directly. It calls:

return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
               net, NULL, skb, dev, NULL,
               ip_rcv_finish);

This runs every registered NF_INET_PRE_ROUTING netfilter hook, and only if they all return NF_ACCEPT does control continue into ip_rcv_finish. This ordering is architecturally load-bearing: because PREROUTING runs before the routing lookup, a DNAT rule (iptables -t nat -A PREROUTING -j DNAT, or an nft prerouting chain) can rewrite the destination address, and the subsequent route lookup will then route the packet to the new destination. Connection tracking (Connection Tracking conntrack) also first observes an inbound flow here. The netfilter framework itself — its five hook points and how rules attach — is The Netfilter Framework and Hooks.

3. ip_rcv_finish — the routing lookup

ip_rcv_finish calls ip_rcv_finish_core, whose central act is the routing input lookup: ip_route_input_noref(skb, iph->daddr, iph->saddr, iph->tos, dev) consults the Forwarding Information Base (FIB) and caches the result as a dst_entry attached to the skb (skb_dst). The dst_entry carries an input function pointer that is the routing decision in concrete form. An optimization called early demux (tcp_v4_early_demux/udp_v4_early_demux, gated on sysctl_ip_early_demux) can short-circuit this for already-established connections by finding the owning socket directly and reusing its cached route. After the lookup, ip_rcv_finish calls:

ret = dst_input(skb);

dst_input is an indirect call through the route’s input pointer. For a packet destined to a local address that pointer is ip_local_deliver; for a packet that must be routed onward it is ip_forward; for multicast it is ip_mr_input. This single indirect call is the local-deliver-versus-forward branch — the heart of the IP layer — and the full mechanics of how the FIB chooses are IP Routing Decision and Forwarding.

4. ip_local_deliver — reassemble, INPUT hook, and L4 dispatch

For a locally-destined packet, ip_local_deliver does three things in order (ip_input.c:242):

int ip_local_deliver(struct sk_buff *skb)
{
        struct net *net = dev_net(skb->dev);
 
        if (ip_is_fragment(ip_hdr(skb))) {
                if (ip_defrag(net, skb, IP_DEFRAG_LOCAL_DELIVER))
                        return 0;
        }
 
        return NF_HOOK(NFPROTO_IPV4, NF_INET_LOCAL_IN,
                       net, NULL, skb, skb->dev, NULL,
                       ip_local_deliver_finish);
}

First, if the packet is a fragment, reassemble it — ip_defrag queues the fragment and returns nonzero (stopping delivery) until the whole datagram is reconstructed, at which point the reassembled skb proceeds (see IP Fragmentation and Reassembly). Second, fire the NF_INET_LOCAL_IN (INPUT) hook — this is where a host firewall’s input policy lives (iptables -A INPUT, nft ... hook input). Third, on accept, call ip_local_deliver_finish.

ip_local_deliver_finish strips the IP header off the skb (__skb_pull(skb, skb_network_header_len(skb)), so the skb now starts at the L4 header) and calls ip_protocol_deliver_rcu(net, skb, ip_hdr(skb)->protocol) under rcu_read_lock(). That function is the protocol demultiplexer:

ipprot = rcu_dereference(inet_protos[protocol]);
if (ipprot) {
        ...
        ret = INDIRECT_CALL_2(ipprot->handler, tcp_v4_rcv, udp_rcv, skb);
        ...
}

inet_protos[] is a sparse array indexed by IP protocol number (net/ipv4/protocol.c). Each transport registers its handler at init with inet_add_protocol (net/ipv4/af_inet.c): inet_protos[IPPROTO_TCP].handler = tcp_v4_rcv, [IPPROTO_UDP].handler = udp_rcv, [IPPROTO_ICMP].handler = icmp_rcv. The INDIRECT_CALL_2 wrapper lets the compiler devirtualize the common TCP/UDP cases for speed (mitigating retpoline cost). If no handler is registered for the protocol number, the kernel sends an ICMP “Protocol Unreachable” and drops with SKB_DROP_REASON_IP_NOPROTO. Raw sockets (AF_INET/SOCK_RAW) get a copy first via raw_local_deliver. The transport handlers then perform the socket lookup and queue data onto a struct sock — picked up in The TCP Protocol in Linux and The UDP Protocol in Linux.

5. ip_forward — the router path

If the route’s input pointer was ip_forward, this host is acting as a router for the packet (ip_forward.c:83). Forwarding requires net.ipv4.ip_forward = 1; otherwise the FIB lookup never selects ip_forward as the input function. The sequence is RFC-mandated:

  1. Sanity: drop if the packet was not addressed to this host at L2 (pkt_type != PACKET_HOST) or if it carries a socket reference (a locally-generated packet should never be forwarded).
  2. TTL check before decrement: if (ip_hdr(skb)->ttl <= 1) goto too_many_hops; — and too_many_hops sends ICMP_TIME_EXCEEDED / ICMP_EXC_TTL. This is the exact mechanism traceroute relies on: send packets with increasing TTL, collect the “Time Exceeded” from each successive router.
  3. MTU check: mtu = ip_dst_mtu_maybe_forward(...); if the packet exceeds it and the Don’t-Fragment bit is set, send ICMP_DEST_UNREACH / ICMP_FRAG_NEEDED carrying the MTU (htonl(mtu)) and drop — this is the ICMP message that drives Path MTU Discovery. (If DF is clear, the packet falls through to ip_fragment on the output side; see IP Fragmentation and Reassembly.)
  4. skb_cow to get a writable copy, then ip_decrease_ttl(iph) — which decrements TTL and incrementally updates the header checksum.
  5. Fire the NF_INET_FORWARD hook (where iptables -A FORWARD / a Kubernetes pod-to-pod policy lives), then ip_forward_finish → dst_output, which sends the packet back out through the transmit machinery (ip_output → ip_finish_output).

That a forwarded packet re-enters ip_output/ip_finish_output is why routers fragment forwarded traffic at the same code as a host fragments its own outbound traffic — there is one egress path.

Mechanical Walk-through — The Transmit Path

A locally-generated packet (from TCP/UDP, see The Network Transmit Path) enters L3 at __ip_queue_xmit (for connected TCP) or ip_output (for UDP via ip_send_skb). __ip_queue_xmit (ip_output.c:457) performs the route output lookup if the skb is not already routed, then pushes the 20-byte IPv4 header into the skb’s headroom (skb_push), fills version/IHL/TTL/protocol/addresses, and selects the Identification field (used by fragmentation; for TCP it is randomized to avoid an information leak). It then calls ip_local_out → __ip_local_out, which sets the total length, computes the header checksum (ip_send_check), and fires the NF_INET_LOCAL_OUT (OUTPUT) hook. On accept, dst_output calls ip_output:

int ip_output(struct net *net, struct sock *sk, struct sk_buff *skb)
{
        ...
        return NF_HOOK_COND(NFPROTO_IPV4, NF_INET_POST_ROUTING,
                            net, sk, skb, NULL, dev,
                            ip_finish_output,
                            !(IPCB(skb)->flags & IPSKB_REROUTED));
}

ip_output fires the NF_INET_POST_ROUTING (POSTROUTING) hook — the last IP-layer hook, where SNAT/masquerade rewrites the source address now that the outgoing interface is fixed — and then ip_finish_output → __ip_finish_output makes the MTU decision: if the skb is a Generic Segmentation Offload super-packet hand it to ip_finish_output_gso (see Segmentation Offloads GSO TSO); else if skb->len > mtu, hand it to ip_fragment; otherwise the fast path ip_finish_output2, which resolves the next-hop MAC via the neighbour subsystem and hands the frame to the driver. That MTU branch is the transmit-side entry point of IP Fragmentation and Reassembly.

How IPv6 Differs

IPv6 (RFC 8200) is the same datagram service with a deliberately simplified and extended header, and the kernel mirrors the IPv4 path almost function-for-function (net/ipv6/ip6_input.c, ip6_output.c) — ipv6_rcv → ip6_rcv_finish → ip6_input/ip6_forward, dispatching through inet6_protos[] to tcp_v6_rcv/udpv6_rcv. The differences that matter:

  • No header checksum. The IPv6 header has no checksum field at all. IPv6 relies on the link layer and the L4 (TCP/UDP) checksums to catch corruption, which means a forwarding IPv6 router does not recompute any IP-level checksum on each hop — a small but real per-hop saving and a real simplification of the forwarding fast path. (ip6_input.c has no ip_fast_csum equivalent.)
  • Extension headers replace options. Instead of cramming options into a variable-length region of a single header, IPv6 chains extension headers (Hop-by-Hop Options, Routing, Fragment, Destination Options, etc.), each linked by a “Next Header” field. The kernel walks this chain in ip6_protocol_deliver_rcu with a resubmit loop: it reads inet6_protos[nexthdr], and a handler that is not marked INET6_PROTO_FINAL (i.e. an extension header rather than a real transport) processes its header and loops back to dispatch the next one (ip6_input.c:361). The Hop-by-Hop Options header (Next Header value 0) is handled specially and early, in ipv6_rcv itself, because it must be examined by every node on the path.
  • The fixed header is larger but simpler. 40 bytes (vs. IPv4’s 20) — the addresses alone are 16 bytes each. But there is no IHL, no fragmentation field in the base header (it moves to the Fragment extension header), and no checksum, so the base header is fixed-size and faster to parse.
  • Routers never fragment. This is the single biggest behavioral difference and it is enforced in code: ip6_fragment’s too-big path (fail_toobig) sends ICMPV6_PKT_TOOBIG and drops, and a forwarding router never fragments at all — only the source may fragment, by inserting a Fragment extension header. The full treatment is in IP Fragmentation and Reassembly; the consequence here is that IPv6 makes Path MTU Discovery effectively mandatory.
  • Minimum MTU is 1280 bytes. IPV6_MIN_MTU = 1280 (include/uapi/linux/in6.h) — every IPv6 link must support at least this, so a source can always fall back to 1280-byte packets and never need to fragment below that.

Failure Modes and How to Diagnose Them

Packets validated but not delivered (“tcpdump sees it, the app doesn’t”). A frame can pass ip_rcv’s checksum/length checks (so tcpdump, which taps before the IP layer at ptype_all, captures it) and then be dropped by an INPUT firewall rule, fail the route lookup (reverse-path filtering, rp_filter, drops with SKB_DROP_REASON_IP_RPFILTER), or be addressed to a port with no listening socket. Diagnose by reading the per-reason drop counters: perf trace -e skb:kfree_skb, dropwatch, or the eBPF pwru tool show the exact SKB_DROP_REASON_* and the function that dropped it. The IP-level SNMP counters (nstat -z, /proc/net/snmp) — InHdrErrors, InAddrErrors, InDiscards — tell you which class of failure.

Forwarding silently does nothing. If ip_forward never runs, check sysctl net.ipv4.ip_forward (and the per-interface forwarding knob) — with it off, the FIB lookup classifies non-local packets as “not for us, no route to forward” and they are dropped, never reaching ip_forward. This is the single most common “my Linux router doesn’t route” cause.

TTL exhaustion in a loop. A routing loop manifests as a burst of ICMP_TIME_EXCEEDED and InHdrErrors climbing; the TTL field is what bounds the damage to ~255 hops rather than an infinite loop.

Checksum offload confusion. On modern NICs the IP/L4 checksum may be validated/computed by hardware (CHECKSUM_UNNECESSARY/CHECKSUM_PARTIAL on the skb). A packet captured by tcpdump on the sending host can show a “bad” checksum that is actually correct on the wire because the NIC fills it in after the capture point — a classic false alarm.

Alternatives and Boundaries

The IP layer described here is the standard L3 path, and it is what almost everything uses. The escape hatches bypass it entirely rather than replacing it:

  • XDP runs in the driver before ip_rcv (indeed before an skb exists) and can drop/redirect a packet without the IP layer ever seeing it — the basis of line-rate firewalls and load balancers that do their own L3 logic in eBPF.
  • AF_XDP delivers raw frames to userspace, so a DPDK-style userspace TCP/IP stack implements its own IP layer and the kernel’s is bypassed for that traffic.
  • Tunneling/encapsulation (IP-in-IP, GRE, VXLAN, WireGuard) re-enters the IP layer with an outer header, so the same ip_rcv/ip_output code runs twice per packet — once for the outer datagram and once for the decapsulated inner one. See Tunnel and Overlay Interfaces.

For everything that needs the kernel’s routing, netfilter, and socket semantics — i.e. ~all normal applications — the path in this note is the only game in town.

Production Notes

The IP layer’s hook placement is the foundation of essentially all Linux packet manipulation in production. The PREROUTING-before-routing / POSTROUTING-after-routing ordering is precisely what makes kube-proxy’s Service implementation work: a ClusterIP is DNAT’d at PREROUTING to a real pod IP before the route lookup, and return traffic is SNAT’d at POSTROUTING — the kernel mechanism underneath Service (Kubernetes) and Network Address Translation NAT. The inet_protos[] dispatch table is also why a protocol module (SCTP, DCCP) that is not loaded causes “Protocol Unreachable” rather than a silent drop — a useful diagnostic signal.

A recurring real-world subtlety: the IP layer trims every skb to the header’s declared total length (pskb_trim_rcsum), so any trailing link-layer padding is discarded before L4 sees the packet — relevant when debugging short Ethernet frames padded to the 60-byte minimum, where the padding is not part of the IP datagram. And for IPv6, the absence of a header checksum means that a single-bit corruption in an IPv6 header that the link-layer CRC happened to miss will not be caught at L3 — it will surface as a routing anomaly or an L4 checksum failure instead, which can make IPv6 header corruption subtler to diagnose than the IPv4 equivalent.

See Also

Graph View

Backlinks