vault

BTF and vmlinux.h

14 min read

BTF and vmlinux.h

vmlinux.h is a single, machine-generated C header that contains every type the running kernel uses — every struct, union, enum, and typedef, with their exact field layouts — extracted from the kernel’s own BTF and dumped back into compilable C. It is produced by one command, bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h, and a BPF program that #includes it gains access to the full kernel type universe without installing a single kernel header package (Nakryiko, BPF CO-RE). This is the workflow that severed eBPF development from the kernel-devel/linux-headers build dependency: instead of matching scattered #include <linux/sched.h> against the target’s source tree, you build against one self-consistent header derived from the kernel’s own type metadata. Because the generated header wraps every type in a Clang attribute that makes ordinary field accesses CO-RE-relocatable, the resulting object also runs across kernel versions. This note is the practical companion to BTF (BPF Type Format) — that note explains the format; this one explains how you actually use it to build a portable BPF program.

Mental Model

The right way to think about vmlinux.h is as a round-trip through BTF: the kernel’s C source was compiled to DWARF, pahole distilled DWARF into deduplicated BTF and embedded it in the kernel, and bpftool now reverses the last step, turning that BTF back into C type declarations. The output is not the kernel’s original headers — it is a reconstruction of the kernel’s types from their BTF descriptions, faithful to layout but stripped of everything BTF doesn’t record (notably preprocessor macros). One header, generated from the running kernel, replaces the entire linux/*.h include tree for BPF purposes.

flowchart LR
  KSRC["kernel C source"] -->|"compile -g"| DWARF["DWARF debug info"]
  DWARF -->|"pahole (build time)"| KBTF["kernel BTF<br/>/sys/kernel/btf/vmlinux"]
  KBTF -->|"bpftool btf dump<br/>format c"| VH["vmlinux.h<br/>(one header, all kernel types)"]
  VH -->|#include| BPFC["prog.bpf.c"]
  HELP["bpf_helpers.h<br/>bpf_tracing.h<br/>bpf_core_read.h"] -->|#include| BPFC
  BPFC -->|"clang -g -target bpf"| OBJ["prog.bpf.o<br/>(+ .BTF, CO-RE relocs)"]
  OBJ -->|libbpf load| KERN["running kernel<br/>(CO-RE relocates vs its BTF)"]

The vmlinux.h round-trip and build pipeline. What it shows: kernel types make a full loop — source → DWARF → BTF (in the kernel) → back to C as vmlinux.h via bpftool. The BPF program includes that one header plus libbpf’s helper headers, compiles to an object carrying its own BTF and CO-RE relocations, and is relocated against the target kernel’s BTF at load. The insight to take: vmlinux.h removes the compile-time dependency on kernel headers, while CO-RE removes the run-time dependency on a specific kernel layout. Together they are why a modern eBPF tool ships as one small static binary that runs everywhere — neither alone is sufficient; you need both.

The Generation Command, Line by Line

The canonical incantation, documented in the libbpf overview, is (docs.kernel.org libbpf overview):

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Walking it piece by piece. bpftool btf dump is the BTF inspection subcommand; its synopsis is bpftool btf dump BTF_SRC [format FORMAT] (bpftool-btf docs). The file /sys/kernel/btf/vmlinux argument names the BTF source as a file on disk — here the kernel’s own BTF blob, which the kernel exposes in sysfs when built with CONFIG_DEBUG_INFO_BTF=y (a file source can be “an ELF file or raw BTF file (e.g., from /sys/kernel/btf/)”). The format c option selects C-syntax output instead of the default raw BTF dump; with format c “the output is sorted by default” so the type declarations come out in a compilable dependency order. Redirecting to vmlinux.h captures the generated header. The whole thing runs in well under a second and needs no compiler, no kernel source, and no headers — only bpftool and a BTF-enabled kernel.

The libbpf-bootstrap project wraps exactly this in a helper script whose single functional line is (gen_vmlinux_h.sh):

$(dirname "$0")/bpftool btf dump file ${1:-/sys/kernel/btf/vmlinux} format c

The ${1:-/sys/kernel/btf/vmlinux} parameter expansion means “use the path passed as $1, or default to the live kernel’s sysfs BTF.” That fallback parameter is how you generate a header for a different kernel than the one you are running — point it at a raw BTF blob extracted from a target kernel image (for example, one pulled from BTFHub for a kernel that lacks its own BTF). The script does no architecture handling and no error recovery; it assumes the BTF source exists.

Why It Replaces Kernel Headers

Before vmlinux.h, a BPF program that touched kernel internals had to include the actual kernel headers — #include <linux/sched.h> for task_struct, #include <linux/fs.h> for file, and so on. That required the kernel-devel/linux-headers package matching the exact running kernel to be installed on every build (and, in the BCC model, every target) machine, and it dragged in a fragile, deeply-nested include tree riddled with #ifdef CONFIG_* conditionals and architecture-specific paths that frequently failed to compile under Clang’s BPF target.

vmlinux.h collapses all of that into one self-consistent file. Because it is generated from the running kernel’s BTF, it contains precisely the types that kernel actually uses, already resolved for the configuration and architecture in play — “no guesswork about version compatibility during initial compilation” (grant.pizza). It even contains “some more internal kernel types not available anywhere else” — types that never appear in any shipped header because they are private to a .c file, yet are visible in BTF and therefore in vmlinux.h (Nakryiko, BPF CO-RE). For BPF development this is strictly more than kernel headers expose.

The one thing it does not contain is macros. BTF records types, not preprocessor #defines, so any constant or function-like macro from the kernel headers (TASK_RUNNING, S_IFMT, container_of, …) is absent from vmlinux.h (grant.pizza; Nakryiko, BPF CO-RE). The program must either define the constants it needs itself or pull them from libbpf’s helper headers, which redefine the commonly-needed ones. This is the chief practical sharp edge of dropping kernel headers, and it is worth internalizing: a missing-symbol compile error after switching to vmlinux.h almost always means a macro that BTF could not carry.

The preserve_access_index Trick — Why Plain p->pid Becomes Portable

The single most important thing bpftool ... format c does beyond emitting type declarations is to wrap the entire set of records in a Clang attribute push/pop. At the top of the generated header it emits a pragma equivalent to:

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif

and a matching #pragma clang attribute pop at the bottom (guarded by the same #ifndef). This was a deliberate bpftool design decision — the patch is literally titled “apply preserve_access_index attribute to all types in BTF dump” (patch by Nakryiko, 2020). __attribute__((preserve_access_index)) is a Clang feature that tells the compiler to record a CO-RE relocation for every field access on the annotated types. Those recorded relocations are later resolved by libbpf at program load time, which rewrites each access to the offset where the field actually lives in the target kernel’s BTF (Nakryiko, BPF CO-RE) — see the precise load-time-vs-compile-time distinction below.

Uncertain

Verify: the exact literal text of the pragma lines emitted by bpftool btf dump ... format c (the #pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record) / pop wording and the BPF_NO_PRESERVE_ACCESS_INDEX guard) on the bpftool version shipping with 6.12. Reason: the patchwork page stating it verbatim was blocked (Anubis); the wording above is reconstructed from a web-search excerpt of that patch plus secondary docs, not read from the patch or from generated output at the 6.12 tag. To resolve: run bpftool btf dump file /sys/kernel/btf/vmlinux format c | head on a 6.12 host, or read tools/bpf/bpftool/btf.c at the v6.12 tag. uncertain

The payoff is enormous for ergonomics. Without the attribute, every portable field read had to be written with an explicit builtin — bpf_core_read(&dst, sizeof(dst), &src->field) or __builtin_preserve_access_index(...). With every type in vmlinux.h carrying preserve_access_index, you write the natural C access pid = task->pid; and Clang automatically emits the CO-RE relocation behind it, because task_struct is a “record” the pushed attribute applies to. The escape hatch is the guard: defining #define BPF_NO_PRESERVE_ACCESS_INDEX before including vmlinux.h disables the auto-relocation (occasionally needed when a type must be accessed at fixed offsets). The CO-RE relocation machinery itself — how those recorded relocations are resolved against the target kernel — is the subject of CO-RE Relocations and Field Access; here the point is only that vmlinux.h is what arms every access for relocation.

A Realistic Program Skeleton

Here is the include structure of a real *.bpf.c from libbpf-bootstrap, with commentary (Nakryiko, libbpf-bootstrap):

#include "vmlinux.h"            /* (1) all kernel types; MUST come first */
#include <bpf/bpf_helpers.h>    /* (2) SEC(), bpf_printk(), map macros, helper protos */
#include <bpf/bpf_tracing.h>    /* (3) BPF_PROG / BPF_KPROBE arg-unpacking macros */
#include <bpf/bpf_core_read.h>  /* (4) BPF_CORE_READ() and CO-RE read helpers */
#include "bootstrap.h"          /* (5) types shared between BPF and userspace */

Line (1) — vmlinux.h must be included first and alone among kernel headers. Lines (2)-(4) are libbpf’s own helper headers (shipped with libbpf, not with the kernel): bpf_helpers.h provides the SEC("...") section macro, the BTF-defined map declaration macros, and helper-function prototypes; bpf_tracing.h provides the architecture-aware macros that unpack probe arguments; bpf_core_read.h provides BPF_CORE_READ() for the cases where you want an explicit CO-RE read. Line (5) is the project’s own shared header carrying structs used on both the BPF and userspace sides (e.g. the event struct sent over a ring buffer).

The corresponding Makefile rule makes vmlinux.h a build dependency so the object recompiles when the header changes (Nakryiko, libbpf-bootstrap):

$(OUTPUT)/%.bpf.o: %.bpf.c $(LIBBPF_OBJ) $(wildcard %.h) vmlinux.h
	$(CLANG) -g -O2 -target bpf -c $< -o $@

The -g flag is not optional here: it tells Clang to emit .BTF/.BTF.ext debug info into the object, which is what carries the program’s own BTF and the CO-RE relocation records that preserve_access_index generated. Omit -g and the object loads with no BTF and no relocations — CO-RE silently does nothing and the program breaks on any kernel whose layout differs. -target bpf selects the BPF backend; -O2 is required because the verifier rejects un-optimized BPF.

Checked-In vs. Regenerated; the Per-Architecture Question

A frequent design decision is whether to commit a vmlinux.h to your repo or regenerate it at build time. The libbpf-bootstrap project leans toward committing pre-generated headers, organized per architecture, via the libbpf/vmlinux.h submodule (Nakryiko, libbpf-bootstrap):

vmlinux.h/include/
├── aarch64/
│   ├── vmlinux_6.6.h
│   └── vmlinux.h -> vmlinux_6.6.h
└── x86_64/
    ├── vmlinux_6.6.h
    └── vmlinux.h -> vmlinux_6.6.h

The reason architecture matters is that struct layouts differ by ABI — pointer width, alignment, and conditionally-compiled fields are not identical between x86_64 and aarch64 — so the header is per-architecture. It is, importantly, not per-target-kernel: thanks to CO-RE, a vmlinux.h generated from one kernel (here 6.6) produces an object that relocates correctly onto other kernel versions at load time. You generate the header once per build architecture and rely on CO-RE for the version portability, rather than regenerating against every deployment kernel. Committing the header also makes builds reproducible and independent of whatever kernel the CI machine happens to run.

Failure Modes and Common Misunderstandings

The defining gotcha is that vmlinux.h cannot coexist with system kernel headers. Including both vmlinux.h and, say, <linux/sched.h> “will inevitably run into type redefinitions and conflicts” because both define struct task_struct, with no include guards bridging them (Nakryiko, libbpf-bootstrap). The rule is absolute: in a *.bpf.c, include vmlinux.h and libbpf’s bpf/*.h helper headers, and nothing from /usr/include/linux. A flood of “redefinition of ‘struct …’” errors is the unmistakable symptom of breaking this rule.

The second gotcha is missing macros, covered above: a 'TASK_RUNNING' undeclared error after dropping kernel headers means a constant that BTF could not carry; define it yourself.

The third is the missing-BTF failure at generation time. If the kernel was built without CONFIG_DEBUG_INFO_BTF, /sys/kernel/btf/vmlinux does not exist and the bpftool btf dump command fails outright. The recovery is to obtain that kernel’s BTF from an external archive (BTFHub) and pass its path as the script argument, or to generate the header on a different machine running a BTF-enabled kernel of the same architecture — CO-RE then relocates onto the BTF-less target at load time (libbpf can be fed an external BTF for the target via its BTF-override path). The relevant config gate is DEBUG_INFO_BTF in lib/Kconfig.debug, which requires pahole v1.16+ (v6.12 Kconfig.debug); see BTF (BPF Type Format) for the full config story.

A fourth, milder issue is file size: vmlinux.h is large — on the order of tens of thousands of lines — because it contains every kernel type (grant.pizza). This is harmless (it compiles fast and the unused types cost nothing in the object) but surprises people the first time they open it.

Module BTF and the Relationship to CO-RE

vmlinux.h as generated above covers the core kernel types only. Types defined in a kernel module live in that module’s split BTF at /sys/kernel/btf/<module-name>, built incrementally on the vmlinux base (see BTF (BPF Type Format) for split BTF). To get a module’s types into a header you dump the module BTF with the vmlinux BTF as base — bpftool btf dump file /sys/kernel/btf/<mod> -B /sys/kernel/btf/vmlinux format c — because a split blob is meaningless without its base (bpftool-btf docs). In practice most tools only need core-kernel types and a single vmlinux.h suffices; module-type access is the exception.

The connection to CO-RE is worth stating precisely to avoid conflation. vmlinux.h is a compile-time artifact: it gives the compiler the type declarations it needs and, through preserve_access_index, arms each field access for relocation. CO-RE is a load-time process: libbpf reads the relocation records the compiler emitted and rewrites each access to the offset where the field actually lives in the BTF of the kernel the program is loading onto (Nakryiko, BPF CO-RE). The vmlinux.h you compiled against and the kernel BTF you relocate against need not be the same kernel — that decoupling is the entire point. The mechanics of that relocation are out of scope here; they belong to CO-RE (Compile Once Run Everywhere) and CO-RE Relocations and Field Access.

Alternatives and When to Choose Them

The alternative to vmlinux.h is the BCC model: ship the BPF C source plus Clang/LLVM and kernel headers to the target, and compile on the host at runtime so the local headers supply the correct offsets (Nakryiko, BPF CO-RE). Choose BCC-style runtime compilation only when you genuinely cannot predict the types you’ll touch (highly dynamic instrumentation, e.g. ad-hoc bpftrace one-liners), or when you must read macros and constants that BTF cannot carry without redefining them. For anything shipped as a product — an agent, a CLI, a daemon — the vmlinux.h + libbpf + CO-RE path wins decisively: one small static binary, no compiler or headers on the target, no runtime compilation failures. The trade-off you accept is defining a handful of macros yourself and respecting the “no system kernel headers” rule.

Production Notes

The vmlinux.h workflow is the default for essentially every modern libbpf-based tool — bpftrace’s AOT mode, Cilium’s BPF, Parca/Pyroscope profilers, and the libbpf-bootstrap template all build this way. The two operational lessons from the field are (1) commit the header (per architecture) rather than regenerating in CI, for reproducible builds independent of the CI kernel; and (2) always compile with -g, since the silent failure mode of a missing -g (no BTF, no relocations, works-on-the-build-kernel-only) is one of the most confusing CO-RE bugs to diagnose — the program loads fine and reads garbage on a different kernel. When a CO-RE tool misbehaves only on certain kernels, the first checks are: was the object built with -g, and does the target have /sys/kernel/btf/vmlinux (or an external BTF supplied to libbpf).

See Also

Graph View

Backlinks