vault

CO-RE (Compile Once Run Everywhere)

16 min read

CO-RE (Compile Once Run Everywhere)

CO-RE (Compile Once – Run Everywhere) is the mechanism that lets a single, pre-compiled BPF object file load and run correctly across many different Linux kernel versions, even though those kernels lay out their internal data structures differently. The core problem is brutally simple: a BPF program that reads task->pid must, at the machine-code level, dereference a fixed byte offset into a struct task_struct — but that offset changes when the struct gains, loses, or reorders fields between kernel releases, so a binary compiled against one kernel reads garbage (or crashes the verifier) on another (Nakryiko, BPF Portability and CO-RE). CO-RE solves this by compiling the program once against a description of some kernel’s types, having Clang emit relocation records that say “here I accessed field pid of struct task_struct,” and then having the loader (libbpf) rewrite those accesses at load time to the offsets the target kernel actually uses — using the target kernel’s own self-description, BTF (BPF Type Format), as ground truth. The result is the change that turned eBPF from a kernel-developer toy into an industry technology: modern tools ship as small static binaries instead of dragging a C compiler and kernel headers onto every host. This note is the why and the shape; the deep mechanism — the relocation kinds, the macros, the instruction patching — lives in CO-RE Relocations and Field Access.

Version context

Kernel-side facts here are pinned to Linux 6.12 LTS (released 2024-11-17). Userspace tooling — libbpf and Clang/LLVM — is versioned separately and dated where it matters; the bpf_core_read.h macros referenced are as of libbpf 1.5.0 (2024). CO-RE was first introduced around the Linux 5.2–5.4 era (2019); by 6.12 it is the default, mature path for every libbpf-based tool.

The Portability Problem

A BPF program is, at bottom, native-ish bytecode that reads kernel memory directly. When you write task->pid in BPF C, the compiler does not emit “look up the field named pid” — there is no runtime field lookup in compiled C. It emits a load from a constant byte offset: something like “read 4 bytes at task + 0x4d8.” That 0x4d8 is baked in at compile time from the layout of struct task_struct in whatever headers the compiler saw.

The trouble is that kernel data structures are internal implementation detail with no stability guarantee whatsoever. Between any two kernel versions — even minor ones, even between two distro builds of the same version with different CONFIG_* options — fields get added, removed, renamed, reordered, moved into nested sub-structs, or have their types changed (Nakryiko, BPF Portability and CO-RE). In Nakryiko’s words: “Different kernel versions will have struct fields shuffled around inside a struct, or even moved into a new inner struct. Fields can be renamed or removed, their types changed.” Any one of those changes shifts the byte offset of every field after it.

So a BPF program compiled against kernel 5.10’s task_struct that hard-codes offset 0x4d8 for pid will, on a 5.15 kernel where pid actually lives at 0x4e0, silently read four bytes from the wrong place — perhaps the middle of some unrelated field. This is not a theoretical edge case; it is the normal situation. A task_struct on a modern kernel is well over a kilobyte of frequently-rearranged fields, and CONFIG-gated members (debug fields, lockdep state, cgroup pointers) move everything below them. There is no version of “compile against the kernel headers and ship the binary” that survives contact with a second kernel.

flowchart LR
  subgraph A["Compile against kernel A"]
    SA["struct task_struct<br/>pid @ 0x4d8"]
  end
  subgraph B["Run on kernel B"]
    SB["struct task_struct<br/>pid @ 0x4e0"]
  end
  PROG["BPF bytecode:<br/>load 4 bytes @ task+0x4d8"]
  SA --> PROG
  PROG -->|"same binary,<br/>NO relocation"| SB
  SB -->|"0x4d8 now points<br/>at WRONG field"| BAD["reads garbage"]

The portability problem in one picture. What it shows: the offset 0x4d8 is frozen into the bytecode when compiled against kernel A, but on kernel B the field has moved to 0x4e0, so the load reads the wrong memory. The insight to take: the bug is not a crash — it is silent wrong data. The program “works” (the verifier accepts a load of a valid pointer at a valid offset), it just reads the wrong field. This is why you cannot fix portability by “being careful”; you need the offset itself to be rewritten per-kernel.

The Pre-CO-RE Answer: BCC and Runtime Compilation

Before CO-RE, the dominant way to write portable BPF tools was BCC (the BPF Compiler Collection), and its strategy was to dodge the offset problem entirely by never freezing the offsets in the first place. A BCC tool ships its BPF program as C source text embedded in the application binary. At runtime, on the actual target host, BCC invokes an embedded copy of Clang/LLVM to compile that source against the local machine’s kernel headers — so the offsets it bakes in are always correct for the kernel it is about to run on (Nakryiko, BPF Portability and CO-RE).

This works, but the costs are severe, and they are exactly the costs CO-RE was built to eliminate:

  • Enormous binaries. Clang/LLVM is a large library. “Clang/LLVM combo is a big library, resulting in big fat binaries” — a trivial BCC tool that prints one counter can balloon to tens of megabytes because it is statically carrying a compiler.
  • Runtime resource cost. Compilation happens at tool startup, on the production host, every time the tool runs. As Nakryiko notes, this “will use a significant amount of resources, potentially tipping over a carefully balanced production workload” — a monitoring agent that spikes CPU and memory to recompile its BPF program on every restart is a genuine operational hazard.
  • Kernel-header dependency. BCC needs the running kernel’s headers (kernel-devel / linux-headers) installed on every target. These are routinely absent on production and container hosts, which strip them to save space. No headers, no compile, no tool.
  • Errors at runtime, not build time. Because compilation is deferred to the target, you discover “even most trivial compilation errors only in runtime,” on the customer’s host, instead of in CI.

BCC’s model is “ship the compiler, compile on every machine.” It is robust against layout changes precisely because it re-derives the layout each time — but it pays for that robustness with size, runtime cost, a header dependency, and lost compile-time checking. CO-RE keeps the robustness and discards every one of those costs.

The CO-RE Solution, in Shape

CO-RE inverts BCC’s bargain. Instead of shipping a compiler and compiling per-host, it compiles once on the developer’s machine and ships a small native binary, then makes the binary adaptable at load time by carrying enough metadata to fix up its own field offsets. Three ingredients make this possible.

First, BTF — the kernel’s self-description. BTF (BPF Type Format) is a compact encoding of C type information — every struct, every field, every field’s offset and size and signedness — roughly “DWARF debug info, but ~100x smaller” (kernel BTF doc; Nakryiko, BTF dedup). A kernel built with CONFIG_DEBUG_INFO_BTF=y embeds a complete BTF description of itself and exposes it to userspace at /sys/kernel/btf/vmlinux (Nakryiko, BTF dedup). This is the running kernel telling you, authoritatively, where every field of every struct actually lives on this machine right now. The full BTF story is in BTF (BPF Type Format).

Second, vmlinux.h — compiling against types without kernel headers. Rather than #include-ing fragile, version-specific kernel headers, a CO-RE program includes a single generated file, vmlinux.h, produced from BTF by dumping it back into C:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

This one header contains every type the kernel knows about, including internal ones never exported in normal headers, and it removes the kernel-devel dependency at compile time entirely (Nakryiko, BPF Portability and CO-RE). Crucially, which kernel’s vmlinux.h you compile against barely matters — you are compiling against type shapes, and CO-RE will fix up the offsets later. The vmlinux.h workflow has its own note: BTF and vmlinux.h.

Third, relocations — the metadata that makes the binary adaptable. When Clang compiles a CO-RE program (with clang --target=bpf -O2 -g -c, where the -g flag is what makes Clang emit BTF and CO-RE relocation metadata — confirmed in the kernel’s own LLVM reloc doc, which shows exactly this invocation producing .BTF and .BTF.ext ELF sections), it does not simply freeze offsets. For each field access marked for relocation, it records a small note in a special ELF section called .BTF.ext that says, in effect, “the instruction at this offset reads field pid (field index 0:N) of struct task_struct.” The actual byte offset is left as a placeholder. These notes are CO-RE relocations, and the full mechanism — how they are recorded, the twelve relocation kinds, and how the loader rewrites the bytecode — is the subject of CO-RE Relocations and Field Access.

flowchart TB
  subgraph DEV["Developer machine (once)"]
    SRC["BPF C source<br/>#include vmlinux.h"]
    CLANG["clang -target bpf -g -O2"]
    OBJ[".o ELF object<br/>bytecode + .BTF + .BTF.ext (relocs)"]
    SRC --> CLANG --> OBJ
  end
  subgraph TGT["Every target host (at load)"]
    LIBBPF["libbpf"]
    TBTF["/sys/kernel/btf/vmlinux<br/>(target kernel's BTF)"]
    PATCH["match relocs vs target BTF<br/>rewrite offsets in bytecode"]
    LOAD["BPF_PROG_LOAD<br/>(already-patched bytecode)"]
    LIBBPF --> PATCH
    TBTF --> PATCH
    PATCH --> LOAD
  end
  OBJ -->|"ship one small binary"| LIBBPF

The CO-RE pipeline. What it shows: compilation happens once on the developer’s machine and produces an ELF object whose .BTF.ext section carries relocation records; on each target host, libbpf reads that host’s /sys/kernel/btf/vmlinux, matches each relocation against the target’s real layout, rewrites the offsets in the bytecode, and only then submits the program to the kernel. The insight to take: the heavy work (parsing C, type-checking) happened once at build time; the per-host work is a cheap metadata-driven offset rewrite, not a compile. The target kernel never sees the relocations — it receives ordinary, already-correct bytecode (more in CO-RE Relocations and Field Access).

Why the Loader Does the Work — and Where

The single most important architectural fact about CO-RE is where the relocation happens. In the standard libbpf workflow — the one used by virtually every CO-RE tool — libbpf resolves the relocations in userspace, before the program is ever loaded. Reading the v6.12 libbpf source, the function bpf_object__relocate_core (tools/lib/bpf/libbpf.c, v6.12) parses the target BTF (from /sys/kernel/btf/vmlinux, or an override path), iterates every relocation record in .BTF.ext, computes the correct value against the target layout, and patches the in-memory bytecode — all before the BPF_PROG_LOAD syscall fires. The kernel receives bytecode that already has the right offsets; it never sees the CO-RE relocation records at all in this path.

This matters for the mental model: CO-RE is overwhelmingly a userspace, load-time transformation. The verifier sees normal, fully-resolved bytecode. (There is one exception — the “light skeleton” / gen_loader path, where relocation work is deferred and a kernel-resident copy of the relocation engine in kernel/bpf/relo_core.c does the patching — but that is a specialized path, not the default. The same relo_core.c file is compiled into both libbpf and the kernel precisely so the two paths share identical logic.) The full division of labor is detailed in CO-RE Relocations and Field Access and libbpf and the BPF Loader.

The Moving Parts, Named

CO-RE is a collaboration between a compiler, a metadata format, a loader, and the kernel’s self-description. The pieces, and where each lives:

  • Clang/LLVM (-target bpf -g) — the compiler. The -g flag makes it emit the .BTF section (the program’s own types) and the .BTF.ext section (function info, line info, and CO-RE relocation records). It is also Clang that provides the builtins — __builtin_preserve_access_index, __builtin_preserve_field_info, __builtin_btf_type_id, __builtin_preserve_enum_value — that generate the relocation records. (Confirmed: the v6.12 LLVM reloc doc shows clang --target=bpf -O2 -g -c test.c producing .BTF/.BTF.ext.)
  • .BTF and .BTF.ext — two ELF sections in the compiled object. .BTF holds the program’s type information; .BTF.ext holds per-instruction metadata including the CO-RE relocation records. Both are described in the kernel BTF doc.
  • vmlinux.h — the generated header that lets you compile against kernel types without kernel headers (see BTF and vmlinux.h).
  • libbpf — the userspace loader. It reads the ELF, reads target BTF, resolves relocations, patches bytecode, creates maps, and loads programs. The CO-RE-specific entry point is bpf_object__relocate_core. See libbpf and the BPF Loader.
  • Target BTF — the running kernel’s own BTF at /sys/kernel/btf/vmlinux (requires CONFIG_DEBUG_INFO_BTF=y). This is the ground truth libbpf matches against.
  • The relocation records and builtins — the BPF_CORE_READ family, bpf_core_field_exists(), and friends, plus the underlying relocation kinds. All of this is CO-RE Relocations and Field Access.

What CO-RE Buys, Concretely

The payoff is best seen by contrasting deployments. A BCC tool is a multi-tens-of-megabytes binary that, on startup, needs kernel-devel present and burns CPU recompiling its BPF program against local headers — and fails outright on a header-less container host. A CO-RE tool built with libbpf is a single small static binary (often well under a megabyte) that ships nothing but its own already-compiled bytecode plus a few kilobytes of relocation metadata, needs no compiler and no headers at runtime, starts instantly, and loads correctly on any kernel new enough to expose BTF (Nakryiko, BPF Portability and CO-RE). This is exactly why the modern eBPF ecosystem — bpftool, Cilium’s agent, the libbpf-tools rewrites of the classic BCC tools, observability agents like Parca and Pixie — standardized on CO-RE + libbpf. The portable static binary is the deployment story that made eBPF shippable as a product.

There is one hard prerequisite worth stating plainly: CO-RE needs the target kernel to expose BTF. If /sys/kernel/btf/vmlinux does not exist (an old kernel, or one built without CONFIG_DEBUG_INFO_BTF=y), there is no ground truth to relocate against. For such kernels the ecosystem bridges the gap with BTFHub, a community archive of pre-generated BTF blobs for thousands of distro kernels that never shipped their own, which libbpf can be pointed at as an external BTF source.

Uncertain

Verify: that CO-RE was first introduced in the Linux 5.2–5.4 era (2019). Reason: the introductory blog posts are dated to that period and __builtin_preserve_access_index landed in Clang around then, but I did not pin a single primary commit/release establishing the exact first kernel/libbpf/Clang versions in which the complete CO-RE flow worked end to end. To resolve: cross-check the libbpf CHANGELOG and the LLVM release notes for the first release shipping __builtin_preserve_access_index against the kernel BTF-exposure commit. The 6.12-pinned facts (the relocation kinds, relo_core.c behavior, the vmlinux.h workflow) are verified against source and are not affected by this date. #uncertain

Failure Modes and Common Misunderstandings

“CO-RE makes BPF programs version-independent automatically.” No. CO-RE relocates field offsets, sizes, signedness, and the existence of types/fields/enum values — the things recorded as relocations. It does not invent a field that does not exist on the target, nor reconcile a genuine semantic change (a field that was renamed and repurposed). If a program reads a field that simply is not present on the target kernel and you did not guard the read, the relocation fails and the instruction is poisoned — turned into a deliberately-invalid instruction so the verifier rejects the load if that path is reachable. Writing programs that degrade gracefully across kernels is a real skill, built on bpf_core_field_exists() and “struct flavors,” both covered in CO-RE Relocations and Field Access.

“CO-RE compiles on the target host.” This is the BCC model, and it is exactly what CO-RE eliminates. CO-RE compiles once, on the developer’s machine. The target-side work is offset patching, not compilation.

“The kernel does the relocation.” In the default libbpf path, no — libbpf does it in userspace and submits already-patched bytecode. The kernel-resident relocation engine exists only for the light-skeleton/gen_loader path.

“CO-RE needs DWARF debug info on the target.” No — it needs BTF, which is ~100x smaller than DWARF and is what kernels actually ship via CONFIG_DEBUG_INFO_BTF=y. DWARF is a build-time input from which BTF is derived; it is never needed at runtime.

Alternatives and When to Choose Them

  • BCC (runtime compilation). Still the right choice when you genuinely need to generate BPF C dynamically at runtime (e.g. an interactive tool that builds a probe from user input), or for quick one-off scripting where binary size and startup cost do not matter. For shippable, deployable tools, CO-RE has superseded it — most classic BCC tools have libbpf-CO-RE rewrites.
  • bpftrace. A higher-level tracing language; it compiles BPF at runtime (BCC-style under the hood historically) but trades portability concerns for expressiveness and interactivity. Right for ad-hoc investigation, not for shipping an agent. Lives in Linux Tracing and Observability MOC.
  • Kernel modules. The pre-eBPF way to read arbitrary kernel state. No verifier safety net, must be recompiled per kernel ABI, and can crash the box — exactly the fragility eBPF + CO-RE exists to avoid.
  • Plain libbpf without CO-RE. You can use libbpf and not touch a single kernel struct (e.g. an XDP program that only parses packet bytes). Then there is nothing to relocate and CO-RE is irrelevant — you get the small-binary benefits for free.

Production Notes

The libbpf-tools subdirectory of the BCC repository is the canonical real-world demonstration: it is a re-implementation of dozens of the classic BCC tracing tools as CO-RE + libbpf programs, each a small static binary with no runtime compiler. Cilium’s datapath and observability stack, Parca’s profiler, and Pixie all build on CO-RE for the same reason — they must ship one artifact that runs across a fleet of heterogeneous kernels. The standard operational gotcha is the missing-BTF kernel: older or minimally-built kernels lack /sys/kernel/btf/vmlinux, and the community answer is BTFHub plus libbpf’s external-BTF support, so the loader can be handed a matching BTF blob even when the kernel did not ship one. The other recurring lesson is that vmlinux.h should be treated as compile-input, not a kernel binding — you can compile against any reasonably recent kernel’s vmlinux.h and still run everywhere, because CO-RE fixes the offsets; pinning vmlinux.h to “the production kernel” is a misunderstanding of what CO-RE does.

See Also

Graph View

Backlinks