BPF Skeletons and bpftool
A BPF skeleton is a code-generated C header — by convention
prog.skel.h— that turns a compiled BPF object file into a typed, ergonomic interface for the userspace program that drives it. You produce it withbpftool gen skeleton prog.bpf.o > prog.skel.h; the header embeds a copy of the object’s bytecode and emits astructwith named fields for every map, program, and link, plus four lifecycle functions —<obj>__open,<obj>__load,<obj>__attach,<obj>__destroy— that wrap the raw libbpf object API (kernel.org libbpf overview; bpftool-gen(8), v6.12). The skeleton’s most powerful trick is mapping a BPF program’s global variables into memory-mapped.data/.rodata/.bssstructs, so userspace can set configuration before load — and because read-only.rodatais frozen before verification, the verifier treats those constants as known and can dead-code-eliminate whole branches.bpftoolis the wider Swiss-army knife: a single binary that inspects and manipulates programs, maps, links, BTF, and network/cgroup attachments. This note covers both — the generated skeleton and the tool that generates it.
This note is pinned to Linux 6.12 LTS (released 2024-11-17) for bpftool’s subcommand surface and to the in-tree bpftool and libbpf shipped with it. The skeleton ABI and the bpftool gen interface are stable and long-predate 6.12 (skeletons landed in libbpf in early 2020), but version-sensitive details — which subcommands exist, which attach types net/cgroup can show — are taken from the 6.12 bpftool documentation tree.
Mental Model
Think of the skeleton as the glue header generated from a .bpf.o, sitting between two worlds that otherwise speak through string lookups and raw file descriptors. Without it, a userspace loader opens the ELF object, walks it to find a map named "events" by string, calls bpf_object__find_map_by_name(obj, "events"), gets back an opaque struct bpf_map *, and repeats that dance for every map, program, and global. The skeleton replaces all of that with compile-time-checked struct field access: skel->maps.events, skel->progs.handle_exec, skel->bss->counter. A typo becomes a compiler error instead of a runtime NULL.
flowchart TB subgraph BUILD["Build time"] C["handler.bpf.c<br/>(BPF C source)"] OBJ["handler.bpf.o<br/>(BPF ELF: bytecode + BTF + maps + .rodata/.bss)"] SKEL["handler.skel.h<br/>(generated header:<br/>embedded object + typed accessors)"] C -->|"clang -target bpf"| OBJ OBJ -->|"bpftool gen skeleton"| SKEL end subgraph RUN["Userspace program (links handler.skel.h)"] OPEN["handler__open()<br/>-> in-memory bpf_object"] SET["skel->rodata->cfg = ...<br/>(set BEFORE load)"] LOAD["handler__load()<br/>-> BPF_MAP_CREATE + BPF_PROG_LOAD + verify"] ATT["handler__attach()<br/>-> BPF_LINK_CREATE per SEC()"] DES["handler__destroy()<br/>-> detach + close fds + munmap"] OPEN --> SET --> LOAD --> ATT --> DES end SKEL -.->|"#include"| OPEN
The skeleton’s place in the pipeline. What it shows: the BPF C source compiles to a .bpf.o ELF object that carries bytecode, BTF, map definitions, and the global-variable data sections; bpftool gen skeleton reads that object and emits a header embedding it together with typed accessors; the userspace program #includes the header and walks the open → (configure) → load → attach → destroy lifecycle. The insight to take: the configure step sits between open and load on purpose — that is the only window in which .rodata constants can be set before the verifier sees them, which is what unlocks dead-code elimination.
What bpftool gen skeleton Generates
Run bpftool gen skeleton handler.bpf.o > handler.skel.h (per bpftool-gen(8) v6.12). The generated header is named after the object’s base name (handler here) and contains four things.
1. An embedded copy of the object file. The documentation is explicit: “contents of source BPF object FILE is embedded within generated code and is thus not necessary to keep around” (bpftool-gen(8)). The ELF bytes are baked into the header as a byte array, so the resulting userspace binary is self-contained — there is no separate .o to ship or locate at runtime. This is the property that makes CO-RE-based tools deploy as a single static binary.
2. The main struct. A type named after the object — struct handler — that owns everything. Its fields are sub-structs giving named access to each component. From the libbpf-bootstrap minimal example, the shape is (Nakryiko, libbpf-bootstrap):
struct handler {
struct bpf_object_skeleton *skeleton; /* drives generic open/load/destroy */
struct bpf_object *obj; /* the underlying libbpf object */
struct { /* one pointer per map */
struct bpf_map *events;
struct bpf_map *rodata;
struct bpf_map *bss;
} maps;
struct { /* one pointer per program */
struct bpf_program *handle_exec;
} progs;
struct { /* one bpf_link* per program */
struct bpf_link *handle_exec;
} links;
struct handler__bss { __u64 counter; } *bss; /* mmap'd writable globals */
struct handler__rodata { __u32 target_pid; } *rodata; /* mmap'd read-only globals */
};The maps, progs, and links sub-structs are the typed replacements for bpf_object__find_map_by_name() / bpf_object__find_program_by_name() string lookups (kernel.org libbpf overview). The bss/rodata/data/kconfig pointers (only those that exist in the object appear) point at memory-mapped views of the global-variable sections — covered in detail below.
3. The four lifecycle functions, each prefixed with the object name:
handler__open()— “creates and opens BPF application,” instantiating the in-memorybpf_objectfrom the embedded bytes without touching the kernel (kernel.org libbpf overview). There is alsohandler__open_opts()to passbpf_object_open_opts, andhandler__open_and_load()to fuse the next step in.handler__load()— “instantiates, loads, and verifies BPF application parts”: it issuesBPF_MAP_CREATEfor every map andBPF_PROG_LOADfor every program, triggering the bpf() syscall and the verifier.handler__attach()— “attaches all auto-attachable BPF programs (it’s optional),” driving one [[BPF Links and Attachment Lifecycle|BPF_LINK_CREATE]] per program whoseSEC()annotation libbpf knows how to auto-attach, and storing the returnedstruct bpf_link *intoskel->links.<prog>.handler__destroy()— “detaches all BPF programs and frees up all used resources” — destroys links, closes program/map fds, andmunmaps the global sections.
4. Optional subskeletons. bpftool gen subskeleton produces a lighter header whose accessors “do not own the corresponding maps, programs, or global variables”; it offers only <obj>__open(bpf_object *) (to attach to an already-opened object, e.g. one loaded by a different process) and <obj>__destroy() (bpftool-gen(8)). This is how a sidecar inspects a BPF object it did not load.
Global Variables: .data, .rodata, .bss, and Dead-Code Elimination
This is the mechanism that makes skeletons more than convenience. A “global variable” in a BPF C program is not a kernel global in any ordinary sense — BPF programs have no .data segment they can address directly. Instead, libbpf gives each global-variable ELF section its own backing map: a single-element BPF_MAP_TYPE_ARRAY whose one value is the entire section’s bytes. The compiler turns each global access into a load from that array map at a fixed offset, and libbpf mmap()s the map into the userspace process so both sides see the same memory with no syscall per access (Nakryiko, libbpf-bootstrap; kernel.org libbpf overview). The three standard sections map to C semantics exactly:
.data— initialized, writable globals (int x = 5;). Skeleton fieldskel->data..bss— zero-initialized, writable globals (int y;). Skeleton fieldskel->bss..rodata—constglobals (const volatile int z = 0;). Skeleton fieldskel->rodata.
Writable globals are read and written through the mmap’d struct at any time: skel->bss->counter reflects the BPF side’s running counter, and you can poke configuration into .data/.bss even after load. The libbpf overview frames this as the headline benefit: the skeleton “memory maps global variables as a struct into user space,” enabling initialization before loading and inspection afterward (kernel.org libbpf overview).
The .rodata story is the interesting one. Read-only globals must be set between __open() and __load() — that is the only window before the kernel sees the program:
struct bootstrap_bpf *skel = bootstrap_bpf__open(); /* in-memory only */
skel->rodata->min_duration_ns = env.min_duration_ms * 1000000ULL;
int err = bootstrap_bpf__load(skel); /* now the kernel sees it */When __load() runs, libbpf populates the .rodata backing map with the bytes you set, then calls BPF_MAP_FREEZE on it. Freezing makes the map immutable from userspace — “no future syscall invocations may alter the map state” (bpf UAPI header, v6.12). Because the map is now provably constant, the verifier is allowed to treat each .rodata load as a known constant at verification time rather than an unknown scalar. Nakryiko spells out the consequence: setting min_duration_ns before load “makes the specific value of min_duration_ns variable known to the BPF verifier during the BPF program verification time,” which “allows BPF verifier to prune the dead code” (Nakryiko, libbpf-bootstrap). A branch like if (min_duration_ns && duration < min_duration_ns) return 0; collapses entirely when min_duration_ns is a known 0 — the verifier removes the unreachable arm, which both speeds the program and, crucially, can shrink it under the verifier’s complexity ceiling.
Two subtleties bite people. First, the const volatile idiom on .rodata globals is mandatory: volatile “is necessary to make sure Clang doesn’t optimize away the variable altogether,” since a plain const initialized to 0 would be folded away at compile time before libbpf ever gets to override it (Nakryiko, libbpf-bootstrap). Second, the dead-code-elimination benefit only applies to .rodata — writable .data/.bss globals stay unknown scalars to the verifier because userspace can change them at any moment.
A Worked Lifecycle
A canonical userspace driver using a skeleton reads almost identically to pseudocode (Nakryiko, libbpf-bootstrap; kernel.org libbpf overview):
#include "bootstrap.skel.h"
int main(int argc, char **argv)
{
struct bootstrap_bpf *skel;
int err;
skel = bootstrap_bpf__open(); /* 1. in-memory object, no kernel */
if (!skel) return 1;
skel->rodata->min_duration_ns = 1000000; /* 2. configure read-only globals */
err = bootstrap_bpf__load(skel); /* 3. create maps + load/verify progs */
if (err) goto cleanup;
err = bootstrap_bpf__attach(skel); /* 4. attach all auto-attach progs */
if (err) goto cleanup;
/* ... poll a ring buffer, read skel->bss->counter, etc. ... */
cleanup:
bootstrap_bpf__destroy(skel); /* 5. detach + free everything */
return err < 0 ? 1 : 0;
}Step 1 (__open) is purely userspace; the verifier is not involved. Step 2 writes into the mmap’d .rodata view. Step 3 (__load) is where the kernel work happens — BPF_MAP_CREATE per map, BPF_MAP_FREEZE on .rodata, BPF_PROG_LOAD (with verification) per program; a verifier rejection surfaces here. Step 4 (__attach) creates a [[BPF Links and Attachment Lifecycle|bpf_link]] per program from its SEC() and stores it in skel->links. Step 5 (__destroy) tears it all down. If you don’t need the open/configure split, bootstrap_bpf__open_and_load() fuses steps 1 and 3.
How does __attach() know where to attach? From the program’s SEC() annotation. libbpf’s auto-attach rules are documented per program type: a SEC("type/extras") string both selects the program type and carries attach details — SEC("tp/sched/sched_process_exec") attaches a tracepoint to sched:sched_process_exec, SEC("xdp") an XDP program, SEC("fentry/do_unlinkat") an fentry trampoline (kernel.org program types). Programs whose SEC() has no auto-attach rule are skipped by __attach() and must be attached by hand (e.g. an XDP program needs a target interface, which auto-attach cannot guess).
A Tour of bpftool
bpftool is the in-tree “tool for inspection and simple manipulation of eBPF programs and maps” (bpftool(8) v6.12). Its grammar is bpftool [OPTIONS] OBJECT { COMMAND }. The 6.12 object set is map, prog, link, cgroup, perf, net, feature, btf, gen, struct_ops, and iter (bpftool(8)). The seven most-used objects:
bpftool prog — programs. prog show lists every loaded program with its id, type, name, and the maps it uses. The two indispensable inspection commands are prog dump xlated, which prints the verified, post-rewrite eBPF instructions the kernel actually holds (after CO-RE relocation and verifier rewrites, optionally annotated with source lines), and prog dump jited, which prints the native machine code the JIT produced (bpftool-prog(8) v6.12). prog load/loadall load programs from an object and pin them; prog attach/detach wire a loaded program to a hook; prog run test-runs a program on a supplied input via BPF_PROG_TEST_RUN; prog profile and prog tracelog aid debugging.
bpftool map — maps. map show lists loaded maps with id, type, key/value sizes, and entry count. map dump prints “all entries in a given MAP” — every key/value pair (bpftool-map(8) v6.12). map lookup/update/delete/getnext manipulate individual entries; map create makes a map and pins it; map pin pins an existing map into bpffs.
bpftool link — the attachment objects. link show lists active links with id, type, the program id they hold, type-specific attributes, and the PIDs holding fds to each link. link pin pins a link into bpffs so the attachment outlives its creating process; link detach force-detaches a link from its hook while leaving the program loaded (bpftool-link(8) v6.12).
bpftool btf — type information. btf show lists loaded BTF objects; btf dump renders a BTF source in either raw indexed form or C syntax. bpftool btf dump file /sys/kernel/btf/vmlinux format c is the canonical way to generate vmlinux.h — the header of every kernel type that CO-RE programs #include (bpftool-btf(8) v6.12).
bpftool gen — code generation: gen skeleton (above), gen subskeleton, gen object (statically links several BPF objects into one, deduplicating BTF), and gen min_core_btf (produces a minimal BTF containing only the types a given program’s CO-RE relocations need) (bpftool-gen(8) v6.12).
bpftool net — networking attachments. net show lists XDP attachments (native/driver, generic, and offloaded), tc/TCX classifier and action attachments, flow-dissector programs, and netfilter/netkit hooks. net attach/net detach manage XDP and (since clsact/TCX) tc attachments (bpftool-net(8) v6.12). The page notes some attach types (sk_filter, lwt, seg6) are not shown and need iproute2 to inspect.
bpftool cgroup — cgroup attachments. cgroup show lists programs attached to one cgroup; cgroup tree walks the whole hierarchy. cgroup attach/detach attach a program to a cgroup with an attach type and optional flags — multi (a child’s program runs in addition to the parent’s, FIFO order) or override (a child’s program yields the parent’s) (bpftool-cgroup(8) v6.12).
Failure Modes and Gotchas
Setting .rodata after load silently does nothing useful. Once __load() has frozen the .rodata map, writes through skel->rodata->... either fault (it is mmap’d read-only after freeze) or are simply not what the program sees. Configuration that must reach the verifier must be set between __open() and __load(). A frequent bug is calling __open_and_load() (which fuses the two) and then trying to set .rodata.
Forgetting volatile on .rodata constants. A const int flag = 0; without volatile is constant-folded by Clang at compile time; libbpf’s later override never takes effect, and the program behaves as if the value were permanently its initializer. Always const volatile.
Skeleton out of sync with a stale header. Because the object is embedded in the skeleton, regenerating prog.skel.h is part of the build, not an afterthought. If the .bpf.c changes but the .skel.h is not regenerated, the userspace binary loads the old embedded bytecode. Build systems wire bpftool gen skeleton as a generated-header dependency precisely to avoid this.
prog dump xlated is not your source. The xlated dump shows the program after verifier rewrites, CO-RE relocations, and dead-code elimination — branches you wrote may be gone, helper calls may be inlined. This is a feature for debugging what the kernel runs, but a surprise if you expect a 1:1 mirror of your C.
Auto-attach skips programs it cannot place. __attach() only attaches programs whose SEC() carries enough information. XDP and tc programs typically need an interface/qdisc you must supply, so they are commonly attached by an explicit bpf_program__attach_xdp(skel->progs.x, ifindex) rather than via __attach().
Alternatives and When to Choose Them
Before skeletons (and still available), you drive libbpf through the generic object API directly: bpf_object__open_file(), then bpf_object__find_map_by_name() / bpf_object__find_program_by_name() by string, then bpf_object__load() and bpf_program__attach(). This is more verbose and loses compile-time checking, but it is the right tool when the set of maps/programs is not known at compile time (e.g. a generic loader that takes any .o on the command line). The libbpf note covers this layer.
The other historical alternative is BCC (BPF Compiler Collection), which embeds Clang/LLVM and compiles BPF C on the target host at runtime against the host’s kernel headers. BCC needs no skeleton because there is no ahead-of-time object — but it pays a heavy price: a multi-hundred-MB toolchain dependency on every machine and a compile on every run. The skeleton + CO-RE model is precisely the answer to BCC’s deployment weight: compile once into a small static binary, ship it everywhere.
For other languages, the skeleton model has analogues: the Go cilium/ebpf library generates Go bindings via bpf2go (a moral skeleton), and libbpf-rs generates Rust skeletons via a build script. They all rest on the same kernel mechanism — an embedded object, typed accessors, mmap’d globals, the bpf() syscall.
Production Notes
The skeleton + CO-RE + small-static-binary pattern is the de-facto standard for shipping eBPF tooling. The kernel’s own tools/bpf/runqslower and the upstream libbpf-bootstrap examples are built exactly this way; Nakryiko’s libbpf-bootstrap writeup is effectively the reference build (Nakryiko, libbpf-bootstrap). Observability vendors (the bpftrace/BCC successors, Pixie, Parca, and Cilium’s Hubble) lean on the same toolchain so a single agent binary runs across a fleet of differing kernels without per-host compilation.
bpftool itself is indispensable operationally: bpftool prog show and bpftool map dump are the first commands in any “what BPF is running on this box and what is in its maps” investigation, and bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h is step zero of nearly every CO-RE build. When a CO-RE program misbehaves on a particular kernel, bpftool prog dump xlated reveals what the relocations resolved to.
Uncertain
Verify: the exact field names and nesting of the generated skeleton
struct(e.g. whetherskeletonandobjboth appear, ordering ofmaps/progs/links) as emitted by the 6.12 in-treebpftool gen skeleton. Reason: the struct shape shown here is reconstructed from the libbpf-bootstrap writeup and the bpftool-gen(8) description rather than from a skeleton generated against 6.12 headers; minor cosmetic differences (field order, presence ofdata/kconfigpointers) are object-dependent. To resolve: runbpftool gen skeletonagainst a real.bpf.obuilt with 6.12bpftool/libbpfand diff the emitted struct. uncertain
See Also
- libbpf and the BPF Loader — the userspace library the skeleton’s functions wrap; the generic
bpf_object__*API - BPF Links and Attachment Lifecycle — what
<obj>__attach()creates and__destroy()tears down - BPF Object Pinning and Lifetime — how
bpftool {prog,map,link} pinkeeps objects alive past their creator - Map Pinning and bpffs — map-specific pinning, the sibling to object pinning
- The bpf() Syscall — the commands (
BPF_MAP_CREATE,BPF_PROG_LOAD,BPF_MAP_FREEZE) that__load()issues - BTF (BPF Type Format) — what
bpftool btf dumprenders; the basis ofvmlinux.h - CO-RE (Compile Once Run Everywhere) — why an embedded-object static binary runs across kernels
- eBPF Verifier — the dead-code elimination that frozen
.rodataenables - BPF Program Types — the
SEC()taxonomy that drives auto-attach - Linux eBPF MOC — parent map of content