vault

sk_buff Memory Layout and Headroom

13 min read

sk_buff Memory Layout and Headroom

Every skb points at a single contiguous head buffer that is carved into four regions by four markers — head, data, tail, and end. The space between head and data is the headroom; the space between data and tail is the data (the packet bytes currently in play); the space between tail and end is the tailroom; and sitting exactly at end is the skb_shared_info trailer. The genius of this layout is that adding or removing a header is just moving a marker — no bytes are copied. As an outgoing packet descends the stack, each layer calls skb_push() to retreat data leftward into the pre-reserved headroom and writes its header there; as an incoming packet climbs, each layer calls skb_pull() to advance data rightward past the header it just consumed. The kernel documentation calls this the “Basic sk_buff geometry,” and it is verified here against include/linux/skbuff.h at the v6.12 and v6.18 long-term-support tags (the geometry definitions and the skb_push/skb_pull/skb_put/skb_reserve helpers are byte-identical between the two LTS branches, diffed 2026-06-05).

This note owns the linear-buffer geometry and the pointer operations. Its sibling struct sk_buff owns the field anatomy of the metadata struct; skb_shared_info and Paged Fragments owns the skb_shared_info trailer that lives at end and the paged/scatter-gather data beyond the linear buffer. Read those for their respective depths.

Mental Model — Four Markers, One Buffer

Picture the head buffer as a single allocation. The kernel’s own ASCII diagram, lifted from the “Basic sk_buff geometry” documentation block in skbuff.h (per the source, v6.12), is:

                                ---------------
                               | sk_buff       |
                                ---------------
   ,---------------------------  + head
  /          ,-----------------  + data
 /          /      ,-----------  + tail
|          |      |            , + end
|          |      |           |
v          v      v           v
 -----------------------------------------------
| headroom | data |  tailroom | skb_shared_info |
 -----------------------------------------------

flowchart LR
  subgraph BUF["head buffer (one allocation)"]
    direction LR
    HR["headroom<br/>(free, before data)"]
    DATA["data<br/>(the packet bytes)"]
    TR["tailroom<br/>(free, after data)"]
    SI["skb_shared_info<br/>(trailer at end)"]
  end
  HEAD(["head →"]) -.-> HR
  D(["data →"]) -.-> DATA
  TAIL(["tail →"]) -.-> TR
  END(["end →"]) -.-> SI
  HR -. "skb_push: data moves LEFT<br/>(prepend header)" .-> DATA
  DATA -. "skb_pull: data moves RIGHT<br/>(strip header)" .-> TR

The four markers and the three regions they delimit. What it shows: head is fixed at the buffer start and end is fixed at the buffer end (where skb_shared_info sits); data and tail are the two movable markers that bound the live packet bytes. skb_push retreats data to the left, eating into headroom to make room for a prepended header; skb_pull advances data to the right, returning consumed-header space to the headroom; skb_put advances tail to the right, eating into tailroom to append data. The insight to take: prepending and stripping headers — the most frequent operation in the whole stack — costs a pointer adjustment and a few writes, never a memcpy of the payload.

The Four Markers

Two of the markers are stored as raw pointers and two as offsets, all referenced from the struct sk_buff:

  • head (unsigned char *) — the start of the head buffer. Fixed for the buffer’s life (until a reallocation by pskb_expand_head). All offsets in the skb are measured relative to it.
  • data (unsigned char *) — the start of the current packet data. This is the marker layers move. skb->data is where “the packet, as this layer sees it,” begins.
  • tail (sk_buff_data_t) — the end of the current data. On 64-bit builds sk_buff_data_t is a 32-bit offset from head (because NET_SKBUFF_DATA_USES_OFFSET is defined when BITS_PER_LONG > 32), so tail is fetched via skb_tail_pointer(skb) which returns skb->head + skb->tail (per the helper, v6.12). Storing it as an offset both saves 4 bytes and lets it survive a head reallocation.
  • end (sk_buff_data_t) — the end of the usable buffer, i.e. where the skb_shared_info trailer begins. Also an offset on 64-bit; skb_end_pointer(skb) returns skb->head + skb->end.

From these, three derived quantities define the regions:

  • Headroom = data - head — free bytes before the data, available for prepending headers. Returned by skb_headroom(skb), whose entire body is return skb->data - skb->head;.
  • Data length (linear) = tail - data — the live packet bytes (this is skb_headlen(), the linear portion of the total len).
  • Tailroom = end - tail — free bytes after the data, available for appending. Returned by skb_tailroom(skb), which is return skb_is_nonlinear(skb) ? 0 : skb->end - skb->tail; — note it returns 0 for a non-linear skb, because once data has spilled into paged fragments you cannot meaningfully append to the linear tail (per the helpers, v6.12).

The Four Operations — All Pointer Moves, No Copies

The four canonical operations are tiny inline functions. The point of reproducing them is that they are almost nothing — which is exactly the lesson.

skb_put(skb, len) — append len bytes at the tail

Grows the data region toward end. The __skb_put inline is literally:

static inline void *__skb_put(struct sk_buff *skb, unsigned int len)
{
    void *tmp = skb_tail_pointer(skb);   /* remember the old tail */
    SKB_LINEAR_ASSERT(skb);              /* must be linear */
    skb->tail += len;                    /* advance tail right */
    skb->len  += len;                    /* total length grows */
    return tmp;                          /* pointer to the new space */
}

The checked variant skb_put() (in skbuff.c) adds a guard: if (unlikely(skb->tail > skb->end)) skb_over_panic(...) — overrunning the tail past end is a kernel BUG, because it would scribble over the skb_shared_info trailer (per skb_put in net/core/skbuff.c, v6.12). skb_put returns a pointer to the start of the appended region so the caller can write into it — the idiom is hdr = skb_put(skb, sizeof(*hdr)); then fill hdr. This is how a layer appends (used on the build side and for trailers).

skb_push(skb, len) — prepend len bytes at the head

Grows the data region toward head by eating into headroom — the workhorse of header prepending on transmit:

void *skb_push(struct sk_buff *skb, unsigned int len)
{
    skb->data -= len;                    /* retreat data left */
    skb->len  += len;                    /* total length grows */
    if (unlikely(skb->data < skb->head)) /* ran out of headroom? */
        skb_under_panic(skb, len, __builtin_return_address(0));
    return skb->data;                    /* pointer to the new header space */
}

data moves left into the headroom; the new bytes are now part of the packet, and skb->data points at the first of them so the caller writes its header there. If there is not enough headroom (data would go below head), the kernel panics via skb_under_panic — which is precisely why headroom is pre-reserved (below). On transmit, TCP skb_pushes its header, then IP skb_pushes the IP header, then the link layer skb_pushes the Ethernet header, each prepend retreating data a little further into the reserved headroom.

skb_pull(skb, len) — strip len bytes from the head

The inverse of push, the workhorse of header consumption on receive:

static inline void *__skb_pull(struct sk_buff *skb, unsigned int len)
{
    skb->len -= len;                     /* total length shrinks */
    /* (debug check that len <= linear len) */
    return skb->data += len;             /* advance data right, return new data */
}

data moves right, past the header the current layer just finished with; that header’s bytes are now back in the (logical) headroom and skb->data points at the next layer’s header. As a received frame climbs the stack — Ethernet, then IP, then TCP — each skb_pull peels off one header. Crucially, the header-offset bookmarks (network_header, etc., on struct sk_buff) are not disturbed by the pull, so a higher layer can still locate a lower header it has already pulled past. The checked pskb_may_pull(skb, len) variant is what defensive code uses: it ensures len bytes are actually present in the linear region, dragging paged data into the linear part if necessary, and returns false (so the caller drops the packet) if the packet is too short — this is the standard guard before reading any header field off a possibly-non-linear skb.

skb_reserve(skb, len) — create headroom up front

static inline void skb_reserve(struct sk_buff *skb, int len)
{
    skb->data += len;                    /* move BOTH data ...      */
    skb->tail += len;                    /* ... and tail to the right */
}

skb_reserve is called on a fresh, empty skb (one with no data yet, where data == tail) to shift the empty data region rightward, opening up headroom before it. Because it moves data and tail together by the same amount, the data length (tail - data) stays zero while the headroom (data - head) grows by len. The documentation is explicit that this is “only allowed for an empty buffer.” This is the call that establishes the headroom so that subsequent skb_pushes have room to prepend headers without panicking.

Why Headroom Is Pre-Reserved — NET_SKB_PAD

Here is the crux. If a freshly allocated skb had data == head (zero headroom), the first skb_push of any header would immediately panic — there would be nowhere to prepend. To avoid that, every skb is allocated with headroom already reserved, so each layer can prepend its header cheaply, and only an unusually deep header stack ever forces a reallocation.

The standard amount is NET_SKB_PAD, defined as max(32, L1_CACHE_BYTES) (per include/linux/skbuff.h, v6.12 — identical in v6.18). The source comment explains the two reasons for the value precisely:

  1. Avoid reallocation on header growth. “The networking layer reserves some headroom in skb data … used to avoid having to reallocate skb data when the header has to grow. In the default case, if the header has to grow 32 bytes or less we avoid the reallocation.” A typical received Ethernet+IPv4+TCP packet, once delivered, may need a few bytes of headroom for retransmit or tunnel encapsulation; 32+ bytes covers the common cases.
  2. Cache-line packing. “Using max(32, L1_CACHE_BYTES) makes sense (especially with RPS) to reduce average number of cache lines per packet.” Aligning the headroom to a cache line keeps the hot header bytes (NET_IP_ALIGN(2) + ethernet(14) + IP(20) + ports(8)) within one 64-byte block that get_rps_cpu() and the flow dissector touch.

The allocator wires this in: __netdev_alloc_skb does len += NET_SKB_PAD and then skb_reserve(skb, NET_SKB_PAD), so a driver that asks for an N-byte buffer transparently gets NET_SKB_PAD bytes of headroom plus its N bytes — “Users should allocate the headroom they think they need without accounting for the built-in space,” as the __netdev_alloc_skb documentation puts it. The NAPI-context allocator napi_alloc_skb reserves a little more, NET_SKB_PAD + NET_IP_ALIGN, folding in the IP-alignment pad as well (all per net/core/skbuff.c, v6.12).

There is a related constant, NET_IP_ALIGN (default 2). Because the Ethernet header is 14 bytes, reserving 2 extra bytes of headroom makes the IP header land on a 4-byte boundary, which matters on architectures that fault or slow down on unaligned multi-byte loads. So a receive skb’s data typically starts at head + NET_SKB_PAD + NET_IP_ALIGN. The two constants serve different goals — NET_SKB_PAD is room for prepending and cache packing, NET_IP_ALIGN is alignment of the IP header — and on most x86 builds NET_IP_ALIGN is 2 while some architectures override it to 0 because their hardware tolerates unaligned access at no cost.

For the transmit side, the device’s dev->needed_headroom lets a stacked/tunneling device advertise extra headroom it will need (for an encapsulation header it intends to prepend), and the higher layers honor it when allocating, so the eventual encapsulation skb_push finds room without reallocating.

The Trailer at end — skb_shared_info

The bytes from end to the true end of the allocation are not tailroom — they hold the struct skb_shared_info trailer, reachable via skb_shinfo(skb), which is defined as ((struct skb_shared_info *)(skb_end_pointer(skb))) — i.e. the shared-info struct sits exactly at end (per the macro, v6.12). This trailer holds the array of paged fragments (frags[]), the frag_list chain pointer, Generic Segmentation Offload sizing, and the dataref count that tracks sharing of the data buffer. The allocator deliberately places it “exactly at the end of the allocated zone, to allow max possible filling before reallocation” — meaning the linear data region can grow toward end as far as possible before the shared-info trailer is hit. The whole reason skb_tailroom() is end - tail (and not the literal end of the allocation) is to stop skb_put from overwriting this trailer. The trailer’s full contents and the paged-fragment model are the subject of skb_shared_info and Paged Fragments.

The macro SKB_WITH_OVERHEAD(X) captures the relationship: for a head allocation of X bytes, the data area available is X - SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) — the allocation minus the aligned trailer.

Worked Example — Building a Packet on Transmit

Following the source’s own description of the build sequence and the helpers above, a transmit skb’s geometry evolves like this:

  1. Allocate a buffer with NET_SKB_PAD (and, for tunnels, dev->needed_headroom) bytes of headroom reserved via skb_reserve. Now data == tail, headroom = the reserved amount, data length = 0.
  2. skb_put the payload — copy the user’s bytes in, advancing tail. Now data length = payload size.
  3. skb_push the TCP header — data retreats left into headroom; write the TCP header there; skb_reset_transport_header() bookmarks it.
  4. skb_push the IP header — data retreats further; write the IP header; skb_reset_network_header() bookmarks it.
  5. skb_push the Ethernet header — data retreats again; write the MAC header; skb_reset_mac_header() bookmarks it.

Each step that prepended a header was a pointer subtraction and a memory write into the pre-reserved headroom — the payload, sitting between data and tail, was never touched or copied. On receive the mirror runs with skb_pull peeling headers off in the opposite order. This is the concrete payoff of the headroom design.

Failure Modes

  • skb_under_panic — ran out of headroom. A skb_push of more bytes than skb_headroom() provides. Symptom: a kernel panic/oops with skb_under_panic in the trace. Cause: a device or tunnel that prepends a header the allocated headroom did not account for — the fix is to advertise dev->needed_headroom or call skb_cow_head()/pskb_expand_head() to grow the headroom (which does reallocate and copy) before pushing.
  • skb_over_panic — ran out of tailroom. A skb_put past end, which would clobber skb_shared_info. Same class of bug on the append side.
  • Reading paged bytes as linear. Dereferencing skb->data + offset for a field that lives past skb_headlen() on a non-linear skb reads garbage (or faults). The discipline is to call pskb_may_pull(skb, needed) first; it returns false (→ drop) if the bytes are not present and pulls them linear if they are.
  • Assuming tailroom on a non-linear skb. skb_tailroom() returns 0 once data_len > 0; code that ignores this and tries to skb_put will not find the space it expects.
  • Headroom destroyed by pskb_expand_head. Reallocating the head buffer (to grow head/tailroom) copies the data and resets the markers; raw pointers into the old buffer dangle. This is why the kernel stores header positions as offsets and re-derives pointers through the accessors after any operation that might expand the head.

See Also

Graph View

Backlinks