vault

fentry fexit and BPF Trampolines

16 min read

fentry fexit and BPF Trampolines

fentry and fexit are eBPF programs that run at the entry and exit of an arbitrary kernel (or BPF) function with very nearly zero overhead. They are the modern, BTF-typed answer to kretprobes: instead of trapping into the kernel through a software breakpoint, the kernel patches the function’s __fentry__ call-site to jump into a small, purpose-built, JIT-generated machine-code stub — the BPF trampoline — which saves the traced function’s arguments, calls your BPF program(s), and (for fexit) calls the original function and then runs the exit programs. All three live under one program type, BPF_PROG_TYPE_TRACING, distinguished by their attach type: BPF_TRACE_FENTRY, BPF_TRACE_FEXIT, and BPF_MODIFY_RETURN (the last lets you change the return value of whitelisted functions) (enum bpf_attach_type, v6.12 bpf.h). The mechanism was introduced by Alexei Starovoitov in Linux 5.5 (November 2019) as “a bridge between kernel functions, BPF programs and other BPF programs,” with the explicit goal that “there is practically zero overhead to call a set of BPF programs before or after [a] kernel function” (LWN, Introduce BPF trampoline). Because the trampoline knows the function’s signature from BTF, an fentry program reads args[0], args[1], … as typed values directly off the register-saved stack — no bpf_probe_read(), no manual pt_regs offset arithmetic.

This is the §5 leaf of the Linux eBPF MOC covering the fentry/fexit/fmod_ret family and the trampoline machinery they share with struct_ops and BPF-LSM.

Mental Model — Patch the Call-Site, Generate a Stub

Every function compiled with function tracing support (CONFIG_FUNCTION_TRACER, which the toolchain implements with -pg / -mfentry) begins with a call to a special symbol, __fentry__ (or the older mcount). At boot, the ftrace subsystem rewrites every one of those call instructions into a 5-byte NOP so the cost is just a no-op when nothing is attached. The BPF trampoline reuses that same patch-site: when you attach an fentry program, the kernel asks ftrace to convert the NOP into a direct call to a freshly generated trampoline image, using ftrace’s direct-call feature (CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS). The trampoline is not interpreted bytecode — it is architecture-specific machine code emitted on the fly by arch_prepare_bpf_trampoline() that knows exactly how many argument registers the traced function uses (from its BTF function model) and emits only the save/restore instructions actually needed.

flowchart TD
  CALLER["caller"] -->|"call foo()"| ENTRY
  subgraph FOO["foo() — entry patched by ftrace"]
    ENTRY["__fentry__ site<br/>(5-byte NOP, now a direct call)"]
    BODY["foo body"]
  end
  ENTRY -->|"jmp/call (patched)"| TRAMP
  subgraph TRAMP["BPF trampoline image (generated machine code)"]
    SAVE["save args R(di,si,dx,...) to stack<br/>(only as many as foo takes)"]
    FENTRY["call each fentry BPF prog<br/>(ctx = saved args)"]
    ORIG["call foo body (orig_call)<br/>only if fexit/fmod_ret present"]
    RET["capture return value"]
    FEXIT["call each fexit/fmod_ret BPF prog<br/>(ctx = args + ret)"]
    RESTORE["restore regs, return"]
    SAVE --> FENTRY --> ORIG --> RET --> FEXIT --> RESTORE
  end
  RESTORE --> BODY

The fentry/fexit path. What it shows: the traced function’s entry NOP is patched to jump into a generated trampoline that saves arguments, runs your fentry programs, optionally calls the original body, captures the return, runs your fexit programs, and restores state before letting foo() continue. The insight: there is no exception, no int3 breakpoint, no global trap handler dispatch — it is an ordinary, predictable call into tailored machine code, which is exactly why the overhead is a fraction of a kprobe’s. For a pure fentry attachment (no fexit), the trampoline does not call the original body at all (flag BPF_TRAMP_F_RESTORE_REGS); it restores registers and returns straight into foo()’s body, which then executes normally.

The contrast with kprobes is the whole point. A kprobe places a breakpoint (int3 on x86) at the target address; hitting it raises an exception, the CPU vectors into the kprobe handler, the handler single-steps or emulates the displaced instruction, and only then runs your BPF program. That trap-and-dispatch path costs hundreds of cycles. The trampoline replaces the trap with a direct call and the dispatch with hand-rolled, function-specific code — Brendan Gregg and others have measured fentry/fexit at roughly an order of magnitude cheaper than kprobe/kretprobe for the same instrumentation, though the exact ratio depends on the workload.

Uncertain

Verify: the specific “order of magnitude cheaper than kprobes” quantitative claim. Reason: not pinned to a primary-source benchmark fetched during this task; widely repeated in talks/blogs but figures vary by CPU and workload. To resolve: run bench from the kernel tools/testing/selftests/bpf trampoline benchmark, or cite a dated kernel-mailing-list benchmark. The direction (fentry/fexit cheaper, no trap) is firmly established by the design (LWN 804937); only the multiplier is unverified. uncertain

Mechanical Walk-through — From bpf() to a Live Trampoline

One trampoline per traced function, shared by all its programs

The kernel keeps a global hash table of trampolines, trampoline_table, keyed by a 64-bit key derived from the target’s BTF id (bpf_trampoline_compute_key). The table is sized 1 << 10 = 1024 buckets, with the comment that btf_vmlinux has ~22k attachable functions. 1k htab is enough. (v6.12 trampoline.c). The crucial design point: there is exactly one struct bpf_trampoline per traced function, and all fentry, fexit, and fmod_ret programs targeting that function share it. Each struct bpf_trampoline holds three program lists — one per kind — and a per-kind count:

enum bpf_tramp_prog_type {
	BPF_TRAMP_FENTRY,
	BPF_TRAMP_FEXIT,
	BPF_TRAMP_MODIFY_RETURN,
	BPF_TRAMP_MAX,
	BPF_TRAMP_REPLACE, /* more than MAX */
};
 
struct bpf_trampoline {
	struct hlist_node hlist;          /* link in trampoline_table */
	struct ftrace_ops *fops;          /* ftrace direct-call registration */
	struct mutex mutex;
	refcount_t refcnt;
	u32 flags;                        /* BPF_TRAMP_F_* */
	u64 key;
	struct {
		struct btf_func_model model;  /* nr_args, arg sizes — from BTF */
		void *addr;                   /* address of the traced function */
		bool ftrace_managed;
	} func;
	struct bpf_prog *extension_prog;
	struct hlist_head progs_hlist[BPF_TRAMP_MAX];
	int progs_cnt[BPF_TRAMP_MAX];     /* counter per kind */
	struct bpf_tramp_image *cur_image;/* the live machine-code image */
};

The btf_func_model is what makes typed argument access possible: btf_distill_func_proto() (called from the verifier during attach) walks the BTF prototype of the target and records how many arguments it has and the size of each. The trampoline generator then emits exactly enough mov [rsp+N], reg instructions to spill those argument registers onto the trampoline stack, where the BPF program reads them as a flat array u64 args[].

When userspace loads an fentry program with BPF_PROG_LOAD, it supplies attach_btf_id (the BTF id of the target function) and expected_attach_type = BPF_TRACE_FENTRY. The verifier’s check_attach_btf_id() path resolves the target. For BPF_TRACE_FENTRY/BPF_TRACE_FEXIT it requires the BTF type to be a function (btf_type_is_func), distills the prototype into the function model, and resolves the runtime address via kallsyms_lookup_name() (or, for module functions, find_kallsyms_symbol_value()) (v6.12 verifier.c).

Then, when the program is attached (via a BPF link, BPF_LINK_CREATE), bpf_trampoline_link_prog() runs. It looks up or creates the trampoline for the key, adds the program to the appropriate progs_hlist[], bumps progs_cnt[], and calls bpf_trampoline_update(). That update function is the engine room:

  1. It collects all currently-linked programs into a bpf_tramp_links array, one slot per kind.
  2. If the total drops to zero, it unregisters the patch and frees the image.
  3. Otherwise it computes the trampoline flags based on which kinds are present. If there are any fexit or fmod_ret programs, it sets BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME — meaning the trampoline must call the original function body and then continue to the exit programs. If it is fentry-only, it sets BPF_TRAMP_F_RESTORE_REGS instead — restore and return straight into the body, never calling it from the trampoline.
  4. It asks the arch backend for the required image size (arch_bpf_trampoline_size), refuses anything over PAGE_SIZE (-E2BIG), allocates a fresh image, and calls arch_prepare_bpf_trampoline() to emit the machine code.
  5. It marks the image executable (arch_protect_bpf_trampoline) and then installs it.

Installation: ftrace direct calls, or text-poke

Installation depends on whether the target sits at an ftrace patch-site. register_fentry() checks ftrace_location(); if the address is ftrace-managed, it uses register_ftrace_direct(tr->fops, ...) to point the function’s __fentry__ site at the trampoline. If the function is not ftrace-managed (e.g., it is another BPF program being extended), it falls back to bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr) — a raw, atomic instruction-rewrite using the architecture’s text-poke primitive (v6.12 trampoline.c):

static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
{
	void *ip = tr->func.addr;
	unsigned long faddr = ftrace_location((unsigned long)ip);
 
	if (faddr) {                       /* function is ftrace-managed */
		if (!tr->fops)
			return -ENOTSUPP;
		tr->func.ftrace_managed = true;
	}
	if (tr->func.ftrace_managed) {
		ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1);
		return register_ftrace_direct(tr->fops, (long)new_addr);
	}
	return bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); /* extend a BPF prog */
}

Subsequent program adds/removes do not re-patch the call-site — they generate a new image and atomically swap the call target with modify_fentry() (modify_ftrace_direct for ftrace-managed functions). This is why adding a second fentry program to an already-traced function is cheap and live: the old image keeps running until the new one is in place.

Why ftrace direct calls were needed

Before ftrace gained “direct calls,” every ftrace user funnelled through a shared dispatcher that walked a list of ftrace_ops callbacks — fine for function tracer, too slow and too indirect for BPF’s per-function tailored stubs. The direct-call mechanism lets a patch-site jump straight to one trampoline, bypassing the ftrace dispatcher entirely. On arm64 the direct-call plumbing landed later than x86 (LWN, bpf trampoline for arm64), which is why fentry/fexit availability historically varied by architecture.

fexit Reads Both Arguments and the Return Value

The headline advantage of fexit over kretprobe is that an fexit program sees the function’s input arguments and its return value simultaneously. A kretprobe fires only on return and, by itself, has lost the entry arguments — to correlate them you must place a kprobe on entry, stash the args in a map keyed by something like the thread id, and look them up again in the kretprobe. With fexit the trampoline has already saved the arguments on its own stack on the way in, so they are still there on the way out. The trampoline’s context layout places the arguments first, followed by the captured return value, and the helpers bpf_get_func_arg(), bpf_get_func_ret(), and bpf_get_func_arg_cnt() expose them. The verifier explicitly restricts bpf_get_func_ret():

-EOPNOTSUPP for tracing programs other than BPF_TRACE_FEXIT or BPF_MODIFY_RETURN. (v6.12 bpf.h helper docs)

That single line encodes the rule: only at exit (fexit) or at modify-return time does a return value exist to read.

fmod_ret / BPF_MODIFY_RETURN — Changing the Return Value

BPF_MODIFY_RETURN (introduced by KP Singh in Linux 5.6, March 2020, LWN, Introduce BPF_MODIFY_RET tracing progs) is the most powerful and most restricted of the three. An fmod_ret program runs like an fentry program but its own return value can override the traced function’s return value: if it returns non-zero, the trampoline can skip the original function entirely and propagate the program’s value as the function’s result (used for error injection and for LSM deny verdicts). Because letting arbitrary code rewrite any kernel function’s return is obviously dangerous, the verifier confines fmod_ret to a narrow allow-list. check_attach_modify_return() permits a target only if it is on the error-injection list or its name begins with the security_ prefix (v6.12 verifier.c):

static int check_attach_modify_return(unsigned long addr, const char *func_name)
{
	if (within_error_injection_list(addr) ||
	    !strncmp(SECURITY_PREFIX, func_name, sizeof(SECURITY_PREFIX) - 1))
		return 0;
	return -EINVAL;        /* "%s() is not modifiable" */
}

A function joins the error-injection list by being annotated ALLOW_ERROR_INJECTION(fn, ...) in kernel source (gated by CONFIG_FUNCTION_ERROR_INJECTION). So you can fmod_ret-inject failures into, say, should_failslab or a block-layer entry point that the kernel authors deliberately marked injectable — but you cannot fmod_ret an arbitrary function like tcp_sendmsg. The bpf_override_return() helper (a related, kprobe-era mechanism) is similarly fenced behind the error-injection list.

Trampoline Image Lifetime — percpu_ref + RCU Tasks

Swapping a live trampoline image is genuinely hard: while you are tearing down the old image, some CPU may be executing inside it right now. The kernel solves this with a layered reference-counting and grace-period scheme on struct bpf_tramp_image:

struct bpf_tramp_image {
	void *image;
	int size;
	struct bpf_ksym ksym;        /* so the image shows up in stack traces */
	struct percpu_ref pcref;     /* counts in-flight executions */
	void *ip_after_call;
	void *ip_epilogue;
	union { struct rcu_head rcu; struct work_struct work; };
};

The trampoline body itself does __bpf_tramp_enter() (a percpu_ref_get) on the way in and __bpf_tramp_exit() (a percpu_ref_put) on the way out for the fexit case, so pcref counts how many CPUs are mid-flight. When an image is retired, bpf_tramp_image_put() cannot free it immediately. Instead, it text-pokes the image’s ip_after_call to jump straight to the epilogue (so any future callers skip the fexit programs), then percpu_ref_kill()s the reference; once the count hits zero (all in-flight executions finished), a chain of call_rcu_tasks() / call_rcu_tasks_trace() callbacks waits out a “tasks” RCU grace period — which guarantees no task is sleeping on the trampoline’s assembly — before the image memory is finally freed (v6.12 trampoline.c). The comment in the source spells out why both mechanisms are needed: percpu_ref to protect trampoline itself and rcu tasks to protect trampoline asm not covered by percpu_ref. Tasks-RCU exists precisely because plain RCU’s grace period does not cover preemptible code that does not pass through a normal RCU read-side critical section — and a trampoline can be preempted mid-execution.

The enclosing struct bpf_trampoline is refcounted separately with a plain refcount_t: bpf_trampoline_get() looks it up (creating it on first use, refcount_set(&tr->refcnt, 1)) and bumps the count; bpf_trampoline_put() does refcount_dec_and_test() and, only when the last user drops, removes it from the hash table and frees its ftrace_ops.

Configuration / Code — A Worked fentry/fexit Pair

With libbpf and a CO-RE skeleton (see CO-RE (Compile Once Run Everywhere)), fentry/fexit programs are strikingly clean because of typed arguments. A program tracing tcp_connect(struct sock *sk):

#include "vmlinux.h"          /* generated from kernel BTF — gives struct sock */
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
 
/* SEC("fentry/<fn>") selects BPF_PROG_TYPE_TRACING + BPF_TRACE_FENTRY */
SEC("fentry/tcp_connect")
int BPF_PROG(on_connect_enter, struct sock *sk)   /* sk is TYPED, not pt_regs */
{
	__u32 dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
	bpf_printk("tcp_connect dport=%d", bpf_ntohs(dport));
	return 0;     /* fentry/fexit return value is ignored (not fmod_ret) */
}
 
/* fexit sees the SAME args PLUS the return value as a trailing parameter */
SEC("fexit/tcp_connect")
int BPF_PROG(on_connect_exit, struct sock *sk, int ret)
{
	bpf_printk("tcp_connect returned %d", ret);
	return 0;
}

Line-by-line: SEC("fentry/tcp_connect") is parsed by libbpf into prog_type = BPF_PROG_TYPE_TRACING, expected_attach_type = BPF_TRACE_FENTRY, and attach_btf_id resolved from tcp_connect’s BTF. The BPF_PROG() macro (from bpf_tracing.h) unpacks the trampoline’s saved-argument array into the named, typed parameters struct sock *sk — there is no pt_regs decoding because the trampoline already laid the args out by signature. BPF_CORE_READ(sk, __sk_common.skc_dport) is a CO-RE field read that survives kernel-layout changes. In the fexit variant, the trailing int ret parameter is the captured return value — the entire reason fexit exists.

For fmod_ret, the section is SEC("fmod_ret/<fn>"), the target must be error-injection-listed or security_*, and a non-zero return overrides the function result.

Failure Modes

  • Cannot attach: ... not found in kernel BTF — the target function name does not appear in /sys/kernel/btf/vmlinux. Static, inlined, or non-traceable functions have no BTF function id and cannot be fentry’d. Use a kprobe or a tracepoint instead.
  • %s() is not modifiable — you tried fmod_ret on a function that is neither error-injection-listed nor security_-prefixed. This is check_attach_modify_return() rejecting you; it is by design, not a bug.
  • Notrace / no __fentry__ site — functions marked notrace, and functions called before ftrace is initialized, have no patchable entry NOP and cannot be attached.
  • -E2BIG on attach — the generated trampoline image exceeded PAGE_SIZE. This happens for functions with many arguments combined with many attached programs; BPF_MAX_TRAMP_LINKS caps the number of programs per kind (38 on most arches, 27 on s390x in v6.12).
  • Architecture gaps — fentry/fexit require CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS. On architectures or kernels lacking it, attachment fails with -ENOTSUPP. arm64 gained support later than x86 (LWN 900453).
  • Recursion / self-tracing — attaching fentry to functions the trampoline itself calls would recurse; such functions are on a denylist and rejected.

Alternatives and When to Choose Them

NeedUseWhy
Trace a non-static, BTF-visible kernel function, low overhead, typed argsfentry/fexitDirect call, no trap; reads args by type
See entry args and return togetherfexitTrampoline saved args on the way in
Change a return value (error injection / LSM)fmod_retOnly allowed mechanism, but allow-listed targets only
Function has no BTF / is inlined / arbitrary instruction offsetkprobeBreakpoint works anywhere, no BTF needed
Stable, documented instrumentation pointtracepointABI-stable across kernel versions
Periodic sampling / PMU countersperf_event BPF ProgramsDriven by the perf subsystem, not function entry

The honest trade-off: fentry/fexit are faster and ergonomically far nicer than kprobes, but they only work on functions the compiler kept as real symbols with a BTF prototype. Kprobes are slower but can probe any address, including arbitrary instruction offsets inside a function. Tracepoints are slower to add (they must be coded into the kernel) but are an explicit, stable contract. Use fentry/fexit as the default for kernel-function tracing on a modern kernel, and fall back to kprobes for the long tail of functions BTF cannot see.

Production Notes

fentry/fexit are the backbone of modern observability tools. The libbpf-tools rewrites of classic BCC tools (opensnoop, tcpconnect, biolatency, …) prefer fentry/fexit where available because the lower overhead matters at high event rates. Cilium and other production eBPF systems use BPF-to-BPF fentry (the “attach a tracing program to another BPF program” capability from the original series, LWN 804937) to observe their own datapath programs without modifying them. The same trampoline machinery underpins struct_ops (and therefore sched_ext) and BPF-LSM — struct_ops uses trampoline flag BPF_TRAMP_F_INDIRECT/BPF_TRAMP_F_RET_FENTRY_RET so a BPF program can stand in for a kernel ops-table callback. This is why the trampoline is one of eBPF’s most reused mechanisms: learning it once explains tracing, security, and pluggable kernel structures at once.

See Also

Graph View

Backlinks