vault

kprobe and uprobe BPF Programs

17 min read

kprobe and uprobe BPF Programs

A kprobe BPF program (BPF_PROG_TYPE_KPROBE) is an eBPF program that fires when execution reaches an arbitrary instruction inside the kernel — most usefully the entry of a kernel function — and receives the CPU register state at that moment as a struct pt_regs *. The same program type, with a different attach target, becomes a uprobe program that fires at an instruction inside a userspace binary or shared library. The classic backing mechanism is the kernel’s kprobe/uprobe dynamic instrumentation: the kernel overwrites the target instruction’s first byte(s) with a breakpoint (int3 on x86), traps into a handler when the CPU hits it, and runs the attached BPF program before single-stepping the original instruction (kprobes.rst, v6.12). Because the probe reads raw registers rather than a typed kernel structure, kprobe programs are the unstable but universal tracing tool: they can attach to almost any function the compiler did not inline, but they break when a function’s signature, name, or arguments change. This note covers the program type, its pt_regs context, the entry-vs-return (kretprobe/uretprobe) split, the int3 trap mechanism, the two attach families — the legacy perf_event_open path and the modern kprobe_multi/uprobe_multi bpf_link — and the overhead model.

Mental Model

Think of a kprobe BPF program as a conditional breakpoint that the kernel itself plants and services, with your BPF code as the debugger script that runs at the breakpoint. Where a debugger would stop the process and hand control to a human, the kernel runs your verified BPF program in a few hundred nanoseconds and resumes. The program does not see the function’s C arguments by name or type; it sees the raw CPU registers, and it is your job to know the platform’s calling convention (on x86-64, first integer arg in rdi, second in rsi, and so on) to pull arguments out of pt_regs.

flowchart TB
  subgraph ATTACH["Two attach families (one program type)"]
    LEGACY["Legacy: perf_event_open(kprobe PMU)<br/>then PERF_EVENT_IOC_SET_BPF<br/>(one probe per fd)"]
    LINK["Modern: bpf_link<br/>kprobe_multi / uprobe_multi<br/>(thousands of probes, one link)"]
  end
  subgraph KERN["Kernel mechanism"]
    INT3["int3 breakpoint at target<br/>(single kprobe) OR<br/>ftrace __fentry__ hook (kprobe_multi)"]
    TRAP["trap / hook: save registers<br/>into struct pt_regs"]
    PROG["run BPF program<br/>ctx = struct pt_regs *"]
    RESUME["single-step original insn,<br/>resume execution"]
    INT3 --> TRAP --> PROG --> RESUME
  end
  LEGACY --> INT3
  LINK --> INT3
  PROG -. reads .-> ARGS["args via pt_regs<br/>(rdi/rsi/... = calling convention)"]
  PROG -. bpf_get_func_ip .-> IP["probed function address"]

The kprobe/uprobe BPF flow. What it shows: a single program type (BPF_PROG_TYPE_KPROBE) reached through two different attach machineries — the older one-probe-per-perf-event-fd path and the modern multi-attach bpf_link — both ending at the same in-kernel sequence: trap, capture registers into pt_regs, run the program, single-step, resume. The insight to take: the program you write is identical regardless of attach path; what differs is how many functions you reach with one syscall and how cheaply the probe is serviced (int3 trap vs ftrace hook). The context is always raw registers, never typed arguments — that is the defining limitation and the reason fexit exists.

The struct pt_regs Context

Every kprobe BPF program receives exactly one argument: a pointer to struct pt_regs, the architecture-specific snapshot of the CPU’s general-purpose registers at the probe site. The kernel enforces this at verification time. In kernel/trace/bpf_trace.c the access checker is blunt (bpf_trace.c, v6.12):

/* bpf+kprobe programs can access fields of 'struct pt_regs' */
static bool kprobe_prog_is_valid_access(int off, int size, enum bpf_access_type type,
                                        const struct bpf_prog *prog,
                                        struct bpf_insn_access_aux *info)
{
        if (off < 0 || off >= sizeof(struct pt_regs))
                return false;
        if (type != BPF_READ)
                return false;
        ...
}

The verifier permits read-only access to any byte offset within sizeof(struct pt_regs) and nothing beyond it. There is no typing: from the verifier’s view the context is a flat array of register-sized words. This is why kprobe programs must read function arguments through register macros (PT_REGS_PARM1(ctx), PT_REGS_PARM2(ctx), … in libbpf’s bpf_tracing.h, which expand to the right pt_regs field per architecture). On x86-64 the System V ABI puts the first six integer/pointer arguments in rdi, rsi, rdx, rcx, r8, r9; on arm64 they are in x0–x7. A kprobe program written against x86-64 register names will read garbage on arm64 unless it uses the architecture-neutral PT_REGS_PARMn macros — a frequent portability bug.

Crucially, reading an argument value (a register) is direct, but reading what a pointer argument points to is not. If rdi holds a struct task_struct *, the program cannot simply dereference it; that memory is not part of pt_regs and the verifier has no type for it. The program must call a probe-read helper — bpf_probe_read_kernel(&dst, sizeof(dst), src) for kernel addresses, or bpf_probe_read_user() for userspace addresses — which copies the bytes safely, returning an error instead of faulting if the address is bad (Nakryiko, BPF CO-RE reference). This is the central ergonomic and performance gap between kprobes and the BTF-typed fentry / tp_btf programs, where the verifier knows the argument types and lets you dereference kernel pointers directly.

Entry vs Return: kprobe/kretprobe and uprobe/uretprobe

A plain kprobe fires before the probed instruction executes — at function entry, this is the natural place to read arguments. A kretprobe fires after the function returns, which is where the return value lives (in rax on x86-64, read via PT_REGS_RC(ctx)). The two are different attach modes of the same program type; libbpf distinguishes them by ELF section name (SEC("kprobe/...") vs SEC("kretprobe/...")).

The return-probe mechanism is more elaborate than the entry probe, because a function has one entry but potentially many return paths. The kernel solves this by hijacking the return address. When a kretprobe is armed, the kernel first plants an ordinary kprobe at the function’s entry; when that entry probe fires, the kernel saves the real return address and overwrites it on the stack with the address of a trampoline. A second kprobe is registered at the trampoline. When the function eventually returns (via any path), control lands on the trampoline, that kprobe fires — running your return handler — and the kernel then restores the saved real return address and continues (kprobes.rst, v6.12). To track concurrent in-flight invocations (recursion, multiple threads in the same function), the kernel pre-allocates a pool of kretprobe_instance objects sized by the maxactive parameter; if the pool is exhausted, a return is missed and counted in nmissed. This is a real, observable data-loss mode on hot recursive functions.

The uprobe/uretprobe pair is the userspace analogue. A uprobe attaches to an offset inside an executable or shared library file (identified by path plus offset; the libbpf section form is SEC("uprobe/<path>:<function>"), so an absolute path produces SEC("uprobe//usr/bin/bash:readline") with the doubled slash); the kernel plants the breakpoint in the file-backed page so it fires whenever any process running that binary hits the instruction. Uprobe BPF programs are sleepable in their .s variants (SEC("uprobe.s/...")), because reading userspace memory can fault and block — reflected in libbpf’s section table, where uprobe.s+ and uretprobe.s+ are listed as sleepable (program_types.rst).

The int3 / Breakpoint Mechanism Behind Single kprobes

The mechanism that makes an arbitrary instruction into a probe site is instruction patching. Per the kernel documentation: “When a kprobe is registered, Kprobes makes a copy of the probed instruction and replaces the first byte(s) of the probed instruction with a breakpoint instruction (e.g., int3 on i386 and x86_64)” (kprobes.rst, v6.12). When any CPU executes that byte, it traps: “a trap occurs, the CPU’s registers are saved, and control passes to Kprobes via the notifier_call_chain mechanism.” Those saved registers are exactly the struct pt_regs your BPF program receives.

Kprobes then single-steps its private copy of the displaced original instruction — not the in-place instruction, because the in-place bytes are still the breakpoint and another CPU could be about to hit it. Single-stepping the out-of-line copy preserves correctness on SMP systems. After the copy executes, control returns to the instruction following the probe site.

This per-hit trap is not free: an int3 is a synchronous exception that flushes the pipeline and takes the slow exception entry path. To reduce the cost, the kernel supports optimized kprobes when CONFIG_OPTPROBES=y: instead of a breakpoint, the kernel replaces the target region with a jmp to a detour buffer containing code to push registers, call the handler trampoline, restore registers, run the displaced instructions, and jump back (kprobes.rst, v6.12). Replacing live kernel text with a jump is done carefully — synchronize_rcu() then stop_machine() — so no CPU is mid-execution in the patched region. An optimized kprobe avoids the exception entirely and is markedly cheaper than the int3 path, but it can only be applied where the surrounding instructions allow a jump to be inserted safely.

When a kprobe BPF program runs, its integer return value is interpreted by the kprobe handler. The source comment in trace_call_bpf spells out the contract: “0 - return from kprobe (event is filtered out); 1 - store kprobe event into ring buffer; Other values are reserved and currently alias to 1” (bpf_trace.c, v6.12). For pure BPF tracing (where the program does its own output via maps or the ring buffer) the return value is usually irrelevant, but it matters when the probe is also feeding the legacy ftrace event ring buffer.

bpf_get_func_ip: Which Function Fired?

When one BPF program is attached to many functions at once (the kprobe_multi case below), the program needs to know which function it is currently running in. The helper bpf_get_func_ip() returns the instruction pointer of the probed function. Its implementation differs by attach flavor. For a single kprobe it recovers the address from the live struct kprobe; for kprobe_multi it reads the entry IP that fprobe stashed in the current BPF context (bpf_trace.c, v6.12):

BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs)
{
        ...
        return get_entry_ip((uintptr_t)kp->addr);
}
 
BPF_CALL_1(bpf_get_func_ip_kprobe_multi, struct pt_regs *, regs)
{
        return bpf_kprobe_multi_entry_ip(current->bpf_ctx);
}

The loader picks the right one: kprobe_prog_func_proto switches on is_kprobe_multi(prog) and returns either bpf_get_func_ip_proto_kprobe_multi or the single-kprobe proto. From the program’s perspective you just call bpf_get_func_ip(ctx) and the kernel resolves the correct address. Returned addresses are typically resolved to symbol names in userspace by looking them up in /proc/kallsyms.

Attach Family 1 — Legacy perf_event_open + PERF_EVENT_IOC_SET_BPF

The original way to attach a BPF program to a kprobe goes through the perf subsystem. Since Linux 4.17 the kernel exposes a kprobe dynamic PMU (Performance Monitoring Unit) whose numeric type is read from sysfs: “these two dynamic PMUs create a kprobe/uprobe and attach it to the file descriptor generated by perf_event_open. The kprobe/uprobe will be destroyed on the destruction of the file descriptor” (perf_event_open(2)). The flow is:

  1. Read the PMU type id from /sys/bus/event_source/devices/kprobe/type (or .../uprobe/type).
  2. Call perf_event_open() with that type, setting config bit 0 (retprobe) for the return-probe variant, and pointing kprobe_func+probe_offset (kernel) or uprobe_path+probe_offset (userspace) at the target. This returns a perf event file descriptor that is a kprobe.
  3. ioctl(perf_fd, PERF_EVENT_IOC_SET_BPF, prog_fd) — attach the loaded BPF program to that perf event (PERF_EVENT_IOC_SET_BPF is _IOW('$', 8, __u32) in perf_event.h, v6.12).
  4. ioctl(perf_fd, PERF_EVENT_IOC_ENABLE, 0) — arm it. PERF_EVENT_IOC_DISABLE stops it; close(perf_fd) tears down the kprobe.

This path is “one probe per file descriptor”: attaching to N functions means N perf_event_open calls and N descriptors. It predates the bpf_link abstraction. A halfway-modern variant wraps the same perf fd in a BPF_PERF_EVENT bpf_link (see BPF Links and Attachment Lifecycle), giving the auto-detach-on-close lifetime guarantees of links while still being a single-probe attachment under the hood. In libbpf this whole dance is hidden behind bpf_program__attach_kprobe() / attach_uprobe().

Attach Family 2 — kprobe_multi and uprobe_multi bpf_link

The scaling problem with the legacy path is acute: tools like a syscall tracer want to attach to hundreds or thousands of functions, and N separate perf_event_open + int3 patches is slow to set up and expensive to run. The modern answer is the multi-attach link, which attaches one program to a whole set of functions in a single syscall.

kprobe_multi was merged in Linux 5.18 (2022): “Adds new link type BPF_TRACE_KPROBE_MULTI that attaches kprobe program through fprobe API. The fprobe API allows to attach probe on multiple functions at once very fast, because it works on top of ftrace” (Linux 5.18, kernelnewbies; LWN 885811). The key architectural shift is that kprobe_multi does not use int3 breakpoints — it rides fprobe, which sits on top of ftrace’s __fentry__ call sites (the NOP-patched function-entry hooks compiled in via -pg/-fpatchable-function-entry). Because ftrace already maintains a single dispatch hook per function and can flip thousands of them in batch, attaching to 3000+ functions takes well under two seconds versus the per-function overhead of individual kprobes (LWN 885811). The trade-off is positional: fprobe can only attach at function entry or return, not at an arbitrary offset inside a function the way a raw kprobe can.

The UAPI surface is a sub-struct of bpf_attr.link_create (bpf.h, v6.12):

struct {
        __u32           flags;
        __u32           cnt;
        __aligned_u64   syms;       /* array of symbol-name strings, OR */
        __aligned_u64   addrs;      /* array of resolved addresses */
        __aligned_u64   cookies;    /* per-probe bpf_get_attach_cookie value */
} kprobe_multi;

with the attach type BPF_TRACE_KPROBE_MULTI and a single flag, BPF_F_KPROBE_MULTI_RETURN = (1U << 0), selecting the return-probe (multi-kretprobe) variant. The link type is BPF_LINK_TYPE_KPROBE_MULTI = 8. In the kernel the link is struct bpf_kprobe_multi_link, which embeds struct fprobe fp plus sorted addrs, cookies, and the resolved module list — confirming the fprobe backend (bpf_trace.c, v6.12). libbpf exposes pattern matching: SEC("kprobe.multi/tcp_*") attaches to every kernel symbol matching the glob, with * and ? wildcards.

uprobe_multi is the userspace counterpart, merged in Linux 6.6 (2023): the new link “creates raw uprobes and attaches the BPF program to them without perf event involvement, which is faster and saves file descriptors” (LWN 940313). Its UAPI struct carries path, offsets, ref_ctr_offsets, cookies, cnt, flags, and pid (for filtering to one process), with attach type BPF_TRACE_UPROBE_MULTI, flag BPF_F_UPROBE_MULTI_RETURN, and link type BPF_LINK_TYPE_UPROBE_MULTI = 12 (bpf.h, v6.12). It also brought bpf_get_func_ip support to uprobes.

A further refinement, kprobe session (BPF_TRACE_KPROBE_SESSION, present in the 6.12 enum), was merged in Linux 6.10: it lets a single program instance run at both entry and return of the same function with shared per-session state and a return value (0/1) at entry deciding whether the return half runs (Linux 6.10, kernelnewbies).

// Modern multi-kprobe: one program over a whole subsystem's functions.
SEC("kprobe.multi/tcp_*")          // attach to every symbol matching tcp_*
int BPF_KPROBE(trace_tcp, struct sock *sk)  // BPF_KPROBE macro unpacks pt_regs args
{
        __u64 ip = bpf_get_func_ip(ctx);   // which tcp_* function fired
        // sk is the 1st arg pulled from pt_regs by the BPF_KPROBE macro;
        // to read INTO sk we still need a probe-read or CO-RE read:
        __u16 family = BPF_CORE_READ(sk, __sk_common.skc_family);
        bpf_printk("tcp fn %llx family %d\n", ip, family);
        return 0;
}

Line-by-line: the SEC glob tells libbpf to enumerate matching kallsyms and build one kprobe_multi link; BPF_KPROBE(name, args...) is a libbpf macro that casts the pt_regs context and extracts arguments using PT_REGS_PARMn so you can name them; bpf_get_func_ip(ctx) disambiguates the firing function; and because sk is a kernel pointer (not embedded data), reading its fields still goes through BPF_CORE_READ, which wraps bpf_probe_read_kernel with CO-RE relocations so the field offsets are fixed up for the running kernel.

Failure Modes

The function was inlined or is notrace. The single most common failure is “it’s not traceable (either non-existing, inlined, or marked as ‘notrace’).” Static and inlined functions have no out-of-line call site to probe; functions in the trace/kprobe machinery itself are marked notrace to prevent recursion. The probe attach simply fails. The fix is to find a non-inlined caller or use a tracepoint where one exists.

Reading a pointer’s target faults silently. Forgetting that pt_regs only gives you register values — and dereferencing a pointer argument directly in C — produces either a verifier rejection or, if it slips through, a bpf_probe_read returning -EFAULT. Pointer chasing must go through bpf_probe_read_kernel/_user (or BPF_CORE_READ), and the destination address space must match the pointer’s.

Wrong calling-convention assumptions. Hard-coding ctx->rdi works on x86-64 and breaks everywhere else; using PT_REGS_PARM1(ctx) is portable. Worse, kretprobes only have a meaningful return value in PT_REGS_RC — reading “arguments” in a kretprobe gives clobbered registers.

kretprobe missed returns. On hot recursive functions, maxactive exhaustion silently drops return events (nmissed climbs). Raising maxactive costs memory; the loss is invisible unless you check the counter.

Re-entrancy is blocked. trace_call_bpf increments a per-CPU bpf_prog_active; if a BPF program is already running on this CPU (e.g. a kprobe inside a function your own program called), the nested invocation is skipped and counted as a miss, not run — preventing infinite recursion but silently dropping events (bpf_trace.c, v6.12).

Overhead Model — int3 vs optimized vs fprobe vs fentry

The cost of a kprobe BPF program is attach machinery cost + program cost. The program cost (JIT-compiled BPF) is identical across attach paths; the machinery cost is not, and spans roughly an order of magnitude:

  • int3 single kprobe — slowest. A synchronous breakpoint exception per hit: pipeline flush, exception entry, register save, single-step of the displaced instruction. Order of a few hundred nanoseconds of pure overhead before your program even runs.
  • Optimized kprobe (CONFIG_OPTPROBES) — a jmp to a detour buffer instead of a trap, eliminating the exception. Substantially cheaper than int3 where it can be applied.
  • kprobe_multi over fprobe/ftrace — rides the ftrace __fentry__ hook, which is a direct call from a pre-patched NOP slot, not a trap. Far cheaper to attach (batch ftrace flip) and cheaper per-hit than int3, which is why mass-attach tools use it. It still goes through ftrace’s generic handler dispatch.
  • fexit — cheapest. A dedicated BPF trampoline at the function’s __fentry__ site jumps straight into the program with BTF-typed arguments and no pt_regs materialization, approaching the cost of a normal function call. This is why fentry is preferred over kprobe whenever the target is a traceable function on a BTF-enabled kernel.

The practical rule: reach for fexit first; fall back to kprobe_multi for mass attach; use a single int3 kprobe only when you must probe a specific offset inside a function (an instruction, not just entry/return) that fprobe and fentry cannot reach.

Alternatives and When to Choose Them

  • Tracepoints — when a stable, maintainer-blessed instrumentation point exists for what you want, prefer it: tracepoints have a (looser) ABI and survive refactors that rename internal functions. kprobes are the fallback for the vast surface that has no tracepoint.
  • fexit — same conceptual target (a kernel function’s entry/exit) but with BTF-typed arguments, direct kernel-pointer dereference, and lower overhead. Choose fentry over kprobe whenever the kernel has BTF (/sys/kernel/btf/vmlinux) and you only need entry/exit. kprobe wins only for arbitrary in-function offsets, very old kernels without BTF, or architectures lacking fentry/trampoline support.
  • uprobe vs USDT — for userspace, a raw uprobe attaches to any function offset; USDT (usdt+ section) attaches to statically-defined tracepoints a binary ships with, which are more stable. Use USDT where the binary provides it.

Production Notes

Mass-attach uprobes have a real, documented performance footprint: every process that maps the probed file takes the breakpoint, and uprobe trap handling is heavier than kprobe handling because it crosses the user/kernel boundary. The move to uprobe_multi (6.6) was explicitly motivated by avoiding per-probe perf-event file descriptors and the associated overhead at scale (LWN 940313). On the kernel side, the fprobe-backed kprobe_multi (5.18) is what made whole-subsystem tracing — “attach to every tcp_* function” — practical in tools like Cilium’s monitoring and the BCC/bpftrace kprobe:* wildcards (LWN 885811). For the front-end tooling (bpftrace, BCC) that turns these primitives into one-liners, see Linux Tracing and Observability MOC; the int3/ftrace mechanism itself is owned by that MOC, and this note covers it only as the substrate of the BPF program type.

See Also

Graph View

Backlinks