vault

eBPF Tracing Program Types

15 min read

eBPF Tracing Program Types

A BPF program type (the enum bpf_prog_type value passed at BPF_PROG_LOAD time) tells the kernel verifier what context the program receives and which helper functions it may call. Of the ~30 program types in include/uapi/linux/bpf.h only a handful are about tracing — observing what the kernel and userspace are doing — and this note is the consumer’s map of which type attaches to which instrumentation source. The tracing types are BPF_PROG_TYPE_KPROBE (dynamic kernel/user instruction probes, plus the bulk kprobe_multi path), BPF_PROG_TYPE_TRACEPOINT and BPF_PROG_TYPE_RAW_TRACEPOINT (static, named kernel instrumentation points — cooked-arguments vs raw-arguments respectively), BPF_PROG_TYPE_PERF_EVENT (Performance Monitoring Unit counters and timer-driven sampling), and BPF_PROG_TYPE_TRACING (the BTF-typed function-boundary programs fentry/fexit/fmod_ret, attached through BPF trampolines). Every one of them runs the same loop: the kernel hits a hook, hands the verified program a context pointer describing what just happened, the program reads a few fields and — crucially — writes the result into a BPF map rather than copying the event out. This note frames the choice (“which type for which source”); the deep mechanics of each live in the sibling notes it links to.

This note is the tracing summary for the Linux Tracing and Observability MOC §7. It deliberately does not re-derive the BPF virtual machine, verifier, or trampoline internals — those belong to Linux eBPF MOC and the per-type notes. Its job is the map: source → program type → context → what you can do with it.

Mental Model — One Verifier, Many Doors Into the Kernel

Think of the BPF program type as the doorway a program is wired into. The BPF instruction set and verifier are the same regardless of type — the verifier always proves the program terminates, never reads out of bounds, and only calls helpers it is allowed to call. What the type decides is two things: (1) the context — the single pointer argument the program is handed, which is a different struct for each door (a struct pt_regs * of saved CPU registers for a kprobe, a typed argument array for an fentry program, a struct bpf_perf_event_data * for a perf event); and (2) the helper set — which bpf_*() functions the verifier permits, looked up per type in kernel/trace/bpf_trace.c via the per-type *_prog_func_proto callbacks.

The unifying insight is that the source of the event and the program type are nearly one-to-one. If you want to fire on an arbitrary kernel function the authors never marked, you reach for a kprobe program. If a stable tracepoint already exists at the spot you care about, you use a tracepoint program (cooked args) or a raw-tracepoint program (raw args, lower overhead). If you want to sample on a hardware counter overflow or a timer, that is a perf-event program. If you want typed access to a function’s arguments and return value with near-zero attach overhead, that is a tracing (fentry/fexit) program. Pick the source, and the type follows.

flowchart TB
  subgraph SRC["Instrumentation source"]
    K["Arbitrary kernel/user<br/>instruction (no static hook)"]
    T["Static tracepoint<br/>(TRACE_EVENT macro)"]
    P["PMU counter overflow<br/>or timer tick"]
    F["Function entry/exit<br/>(ftrace-patchable fn)"]
  end
  subgraph TYPE["BPF program type + context"]
    KP["BPF_PROG_TYPE_KPROBE<br/>ctx = struct pt_regs *"]
    TP["BPF_PROG_TYPE_TRACEPOINT<br/>ctx = cooked arg struct"]
    RTP["BPF_PROG_TYPE_RAW_TRACEPOINT<br/>ctx = raw u64 args[]"]
    PE["BPF_PROG_TYPE_PERF_EVENT<br/>ctx = bpf_perf_event_data *"]
    TR["BPF_PROG_TYPE_TRACING<br/>ctx = BTF-typed args[]"]
  end
  OUT["BPF map<br/>(aggregate) or ring buffer<br/>(rare events)"]
  K --> KP
  T --> TP
  T --> RTP
  P --> PE
  F --> TR
  KP --> OUT
  TP --> OUT
  RTP --> OUT
  PE --> OUT
  TR --> OUT

The tracing-program-type decision map. What it shows: each instrumentation source on the left maps to a specific bpf_prog_type and its context struct in the middle; whatever the type, the program’s output goes the same place on the right — a BPF map (the common case) or a ring buffer (rare must-export events). The insight to take: choosing a tracing tool is really choosing a source, and the program type follows almost mechanically; learning the five contexts is most of what separates the types.

The Five Tracing Program Types

BPF_PROG_TYPE_KPROBE — dynamic, register-level, universal

BPF_PROG_TYPE_KPROBE is the value at offset 2 in the bpf_prog_type enum (bpf.h v6.12), and it is the most general tracing door: it can fire on almost any kernel instruction (most usefully a function entry) and, with a different attach target, on any userspace instruction (a uprobe). The deep mechanism — the int3 breakpoint patch, the trap-and-single-step, the kretprobe return-address hijack — is covered in kprobe and uprobe BPF Programs; here the salient facts are the context and the attach families.

The context is a struct pt_regs * — the raw CPU register file captured at the probe point. In bpf_trace.c, kprobe_prog_is_valid_access enforces that the program may only read fields of struct pt_regs, read-only and aligned. Because the kernel hands you registers rather than a typed argument list, you must know the calling convention to extract arguments (on x86-64, first integer argument in rdi, second in rsi, …). This is what makes kprobe programs powerful but brittle: nothing checks that you read the right register, and a refactor that changes a function’s signature silently changes what your probe sees.

The helper set, from kprobe_prog_func_proto in bpf_trace.c, includes bpf_perf_event_output, bpf_get_stackid, bpf_get_stack, bpf_get_func_ip, bpf_get_attach_cookie, the optional bpf_override_return (only with CONFIG_BPF_KPROBE_OVERRIDE), plus everything from the base bpf_tracing_func_proto — which is where bpf_map_lookup_elem, bpf_map_update_elem, and bpf_map_delete_elem come from. That base set is the same across all the tracing types, which is why map-based aggregation (see In-Kernel Aggregation with BPF Maps) works identically no matter which door you came in through.

There are two attach families for one program type. The legacy path opens a perf_event of the kprobe PMU with perf_event_open(2) and then attaches the program with the PERF_EVENT_IOC_SET_BPF ioctl — one probe per file descriptor. The modern path is kprobe_multi (attach type BPF_TRACE_KPROBE_MULTI, present in the bpf_attach_type enum), which attaches one program to thousands of functions at once. kprobe_multi is built on fprobe, which itself sits on top of ftrace’s function-entry hook rather than int3 breakpoints. The payoff is attach speed: per the fprobe LWN coverage, the fprobe API “allows to attach probe on multiple functions at once very fast, because it works on top of ftrace,” and a measurement attaching 3342 functions completed in roughly 1.5 seconds — versus the per-function breakpoint-insertion cost of inserting thousands of individual int3 kprobes. The trade-off is that fprobe “limits the probe point to the function entry or return” — you cannot probe an arbitrary mid-function instruction with kprobe_multi, only entry and exit. The kernel data structure is struct bpf_kprobe_multi_link, which embeds a struct fprobe fp and arrays of addrs and cookies, with handlers kprobe_multi_link_handler (entry) and kprobe_multi_link_exit_handler (return). v6.12 also exposes BPF_TRACE_KPROBE_SESSION in the attach-type enum, pairing entry and return into a single session program.

Uncertain

Verify: that BPF_TRACE_KPROBE_SESSION is fully available and not gated behind a config/feature flag in the exact v6.12.0 release (it landed in the 6.10 cycle per upstream history, so 6.12 should carry it). Reason: confirmed present in the v6.12 bpf_attach_type enum but I did not read the session-link implementation in v6.12 to confirm its runtime semantics. To resolve: read kernel/trace/bpf_trace.c’s bpf_kprobe_multi_link_attach session handling on a v6.12 checkout. uncertain

BPF_PROG_TYPE_TRACEPOINT and BPF_PROG_TYPE_RAW_TRACEPOINT — static, named, stable-ish

Where kprobes probe anything, tracepoints probe the spots kernel developers deliberately marked with the TRACE_EVENT macro (a scheduler context switch, a block-I/O completion, a syscall entry). Both program types attach to those static points; they differ in how the arguments reach your program — and that difference is the whole point of having two types.

BPF_PROG_TYPE_TRACEPOINT (enum offset 5) receives the cooked trace record: the same fixed-layout structure that ftrace writes into its ring buffer and describes in the tracepoint’s format file under tracefs. The fields are stable across kernel versions in the sense that the format file is an ABI-ish contract; libbpf can generate a matching C struct so your program reads ctx->prev_pid by name. The cost is that the kernel has already formatted the record — copied and laid out the fields — before your program runs.

BPF_PROG_TYPE_RAW_TRACEPOINT (enum offset 17, with a writable variant BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE at offset 24) skips that formatting. It is handed the raw arguments — the actual u64 values passed to the tracepoint at the call site, as a struct bpf_raw_tracepoint_args whose args[] array holds the untyped arguments. There is no per-event formatting cost, which is why raw tracepoints are the lower-overhead choice for high-frequency tracepoints — but you must know the tracepoint’s argument list, and the values are untyped pointers/integers you cast yourself. The modern, BTF-typed evolution of this is the tp_btf/ attach (an ELF-section prefix that actually loads as BPF_PROG_TYPE_TRACING with attach type BPF_TRACE_RAW_TP), which gives you the raw tracepoint’s low overhead plus compiler-checked typed access to the arguments via BTF. The full breakdown lives in Tracepoint BPF Programs.

Helper-wise, both tracepoint flavors get bpf_perf_event_output, bpf_get_stackid, bpf_get_stack, bpf_get_attach_cookie, and the base map helpers (tp_prog_func_proto and the raw-tracepoint path raw_tp_prog_func_proto/tracing_prog_func_proto in bpf_trace.c). The practical guidance: prefer a tracepoint over a kprobe whenever one exists at the spot you care about, because it is a stable, documented contract that will not silently shift under you; prefer the raw/tp_btf variant when the tracepoint fires often enough that per-event formatting overhead matters.

BPF_PROG_TYPE_PERF_EVENT — counters and sampling

BPF_PROG_TYPE_PERF_EVENT (enum offset 7) attaches to a perf_event and fires on its overflow — making it the door for sampling and PMU-counter-driven tracing rather than deterministic per-call tracing. The backing event can be a hardware PMU counter (PERF_TYPE_HARDWARE — PERF_COUNT_HW_CPU_CYCLES, PERF_COUNT_HW_INSTRUCTIONS, PERF_COUNT_HW_CACHE_MISSES, per the perf_event_open(2) man page), a software event (PERF_TYPE_SOFTWARE — PERF_COUNT_SW_CPU_CLOCK, PERF_COUNT_SW_TASK_CLOCK, PERF_COUNT_SW_PAGE_FAULTS), a raw CPU-specific event (PERF_TYPE_RAW), or even a tracepoint (PERF_TYPE_TRACEPOINT). A sampling event “generates an overflow notification every N events, where N is given by sample_period,” or you set sample_freq and the kernel adjusts the period to hit a target rate. Each overflow runs your BPF program — so a perf-event program attached to PERF_COUNT_SW_CPU_CLOCK at 99 Hz is the canonical CPU profiler: every ~10 ms, on every CPU, grab the stack with bpf_get_stackid and bump a count in a map. That is exactly how on-CPU flame-graph collection is built without copying every sample to userspace.

The context is struct bpf_perf_event_data *, which in v6.12 exposes (via pe_prog_is_valid_access/pe_prog_convert_ctx_access) a regs field (the register state, like a kprobe’s pt_regs), a sample_period, and an addr. The helper set (pe_prog_func_proto) adds the perf-specific bpf_perf_prog_read_value and bpf_read_branch_records on top of the common stack/output/map helpers. The mechanics — PMU multiplexing, precise vs imprecise events, frequency vs period — are in perf_event BPF Programs.

BPF_PROG_TYPE_TRACING — BTF-typed function boundaries via trampolines

BPF_PROG_TYPE_TRACING (enum offset 26) is the modern function-boundary door. Its attach types — BPF_TRACE_FENTRY (run at function entry), BPF_TRACE_FEXIT (run at function return, with access to both arguments and the return value), and BPF_MODIFY_RETURN/fmod_ret (run at entry and able to change the return value, used for error injection and security policy) — all install via a BPF trampoline: a small generated piece of code that the kernel patches in at the function’s ftrace hook to marshal arguments and call your program. The defining advantage over a kprobe is that the arguments arrive BTF-typed: because the kernel ships BTF (BPF Type Format) describing every function’s signature, the verifier knows that fentry/tcp_v4_connect’s first argument is a struct sock *, so you read sk->__sk_common.skc_state by name with the compiler checking the access — no register-guessing, no manual casts. fexit is strictly more capable than a kretprobe because it sees the original arguments alongside the return value in one program; a kretprobe only sees the return.

The trade-off is reach: fentry/fexit only work on functions that ftrace can patch (not inlined, not notrace, attachable per BTF), so the universe of attachable functions is a subset of what a raw kprobe can reach — but for the functions in that subset, this is the lowest-overhead, most robust choice. The trampoline construction, the BTF argument marshalling, and fmod_ret’s return-rewriting are detailed in fentry fexit and BPF Trampolines. Note the ELF-section convention that confuses newcomers: fentry/, fexit/, fmod_ret/, and tp_btf/ sections all load as BPF_PROG_TYPE_TRACING with different attach types, not as different program types (libbpf program_types, v6.12).

How the Program Reads Context and Writes Maps

Whatever the door, the body is the same shape: read a few fields from the typed-by-program-type context, compute a key, and update a map. The verifier checks every context read against the per-type *_is_valid_access callback and every helper call against the per-type *_func_proto. Here is a kprobe program (libbpf C) that counts how often each process enters do_unlinkat, keyed by PID:

struct {
    __uint(type, BPF_MAP_TYPE_HASH);   // line 1: ordinary hash map...
    __type(key, __u32);                // line 2: ...keyed by PID (u32)...
    __type(value, __u64);              // line 3: ...counting u64 hits.
    __uint(max_entries, 10240);        // line 4: cap distinct keys.
} unlink_count SEC(".maps");
 
SEC("kprobe/do_unlinkat")              // line 5: program type KPROBE, attached to do_unlinkat entry
int BPF_KPROBE(count_unlink)           // line 6: BPF_KPROBE macro unpacks pt_regs into typed args
{
    __u32 pid = bpf_get_current_pid_tgid() >> 32;   // line 7: top 32 bits = TGID (the PID)
    __u64 *val = bpf_map_lookup_elem(&unlink_count, &pid);  // line 8: find this PID's counter
    if (val)
        __sync_fetch_and_add(val, 1);  // line 9: atomic bump if present
    else {
        __u64 one = 1;
        bpf_map_update_elem(&unlink_count, &pid, &one, BPF_NOEXIST);  // line 10: insert first hit
    }
    return 0;                          // line 11: must return; verifier requires it
}

Line 5’s SEC("kprobe/do_unlinkat") is what libbpf parses to pick BPF_PROG_TYPE_KPROBE and the attach target (libbpf program_types). Line 6’s BPF_KPROBE macro hides the struct pt_regs *ctx and would let you name function arguments (BPF_KPROBE(count_unlink, struct path *path)), reading them out of the right registers for you. Lines 8–10 are the map dance — this is where the event stays in the kernel: instead of shipping “PID X unlinked” to userspace, we incremented a counter. The same body, with SEC("tp/syscalls/sys_enter_unlinkat"), would be a tracepoint program reading cooked args; with SEC("fentry/do_unlinkat"), a tracing program reading path BTF-typed. The map-writing half is identical across all of them — which is exactly why the companion note In-Kernel Aggregation with BPF Maps is type-agnostic.

Common Misunderstandings

“fentry is just a faster kprobe.” Not quite. fentry/fexit give you typed arguments via BTF and fexit uniquely sees args-plus-return in one program, but they only attach to ftrace-patchable functions — a strictly smaller set than what a raw int3 kprobe can reach. Use fentry when the function is attachable (most non-inlined kernel functions are); fall back to a kprobe for the ones it cannot reach. See fentry fexit and BPF Trampolines.

“kprobe_multi and a plain kprobe are different program types.” They are the same BPF_PROG_TYPE_KPROBE; what differs is the attach type (BPF_TRACE_KPROBE_MULTI vs the legacy perf-event attach) and the backing mechanism (fprobe/ftrace vs int3). The program body and context are identical. The same one/many split exists for uprobes (BPF_TRACE_UPROBE_MULTI).

“Tracepoint programs are always cheaper than kprobes.” A cooked tracepoint program pays for the kernel formatting the record before your program runs. A raw tracepoint (or tp_btf) program skips that and is cheaper — but a plain BPF_PROG_TYPE_TRACEPOINT on a very hot tracepoint can cost more than you expect. Reach for the raw/tp_btf variant on high-frequency points.

“A perf-event program fires once per function call.” No — it fires on counter overflow (every sample_period events or at sample_freq Hz). It is a statistical sampler, not a deterministic per-event probe. If you need to see every single occurrence, you want a kprobe/tracepoint/fentry program, not a perf-event one.

Choosing a Type — The Decision

The honest one-liner for each: stable static point exists? use a tracepoint (raw/tp_btf if hot). Need typed args / args+return on an ordinary kernel function? use fentry/fexit. Need to probe something with no static hook, or a userspace function? use a kprobe/uprobe (kprobe_multi if attaching to many functions). Sampling on time or a hardware counter? use a perf-event program. The fallback ladder runs typed-and-stable → raw-and-stable → dynamic-and-universal: prefer fentry/tracepoint for robustness, drop to kprobe only when those cannot reach your target. The engine that makes all of them safe — verifier, JIT, maps, BTF/CO-RE — is owned by Linux eBPF MOC; this note only owns the which-door decision.

Production Notes

In practice the BCC and bpftrace toolkits hide the program-type choice behind probe syntax: a bpftrace kprobe:do_unlinkat compiles to a BPF_PROG_TYPE_KPROBE, tracepoint:syscalls:sys_enter_open to a tracepoint program, profile:hz:99 to a perf-event program, and fentry:vfs_read (bpftrace calls it fentry/kfunc) to a BPF_PROG_TYPE_TRACING program — see bpftrace. The shift from individual kprobes to kprobe_multi was a real production win for tools that attach to large function sets (e.g. tracing every function in a subsystem): attach time dropped from seconds-to-minutes of breakpoint insertion to well under a second on top of ftrace (LWN). The general production lesson is to prefer the most stable door that reaches your target — tracepoints and fentry survive kernel upgrades far better than register-reading kprobes, which is what you want for instrumentation that ships and runs for months.

See Also

Graph View