fentry fexit and BPF Trampolines

fentry and fexit are eBPF programs that run at the entry and exit of an arbitrary kernel (or BPF) function with very nearly zero overhead. They are the modern, BTF-typed answer to kretprobes: instead of trapping into the kernel through a software breakpoint, the kernel patches the function’s __fentry__ call-site to jump into a small, purpose-built, JIT-generated machine-code stub — the BPF trampoline — which saves the traced function’s arguments, calls your BPF program(s), and (for fexit) calls the original function and then runs the exit programs. All three live under one program type, BPF_PROG_TYPE_TRACING, distinguished by their attach type: BPF_TRACE_FENTRY, BPF_TRACE_FEXIT, and BPF_MODIFY_RETURN (the last lets you change the return value of whitelisted functions) (enum bpf_attach_type, v6.12 bpf.h). The mechanism was introduced by Alexei Starovoitov in Linux 5.5 (November 2019) as “a bridge between kernel functions, BPF programs and other BPF programs,” with the explicit goal that “there is practically zero overhead to call a set of BPF programs before or after [a] kernel function” (LWN, Introduce BPF trampoline). Because the trampoline knows the function’s signature from BTF, an fentry program reads args[0], args[1], … as typed values directly off the register-saved stack — no bpf_probe_read(), no manual pt_regs offset arithmetic.

This is the §5 leaf of the Linux eBPF MOC covering the fentry/fexit/fmod_ret family and the trampoline machinery they share with struct_ops and BPF-LSM.

Mental Model — Patch the Call-Site, Generate a Stub

Every function compiled with function tracing support (CONFIG_FUNCTION_TRACER, which the toolchain implements with -pg / -mfentry) begins with a call to a special symbol, __fentry__ (or the older mcount). At boot, the ftrace subsystem rewrites every one of those call instructions into a 5-byte NOP so the cost is just a no-op when nothing is attached. The BPF trampoline reuses that same patch-site: when you attach an fentry program, the kernel asks ftrace to convert the NOP into a direct call to a freshly generated trampoline image, using ftrace’s direct-call feature (CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS). The trampoline is not interpreted bytecode — it is architecture-specific machine code emitted on the fly by arch_prepare_bpf_trampoline() that knows exactly how many argument registers the traced function uses (from its BTF function model) and emits only the save/restore instructions actually needed.

flowchart TD
  CALLER["caller"] -->|"call foo()"| ENTRY
  subgraph FOO["foo() — entry patched by ftrace"]
    ENTRY["__fentry__ site<br/>(5-byte NOP, now a direct call)"]
    BODY["foo body"]
  end
  ENTRY -->|"jmp/call (patched)"| TRAMP
  subgraph TRAMP["BPF trampoline image (generated machine code)"]
    SAVE["save args R(di,si,dx,...) to stack<br/>(only as many as foo takes)"]
    FENTRY["call each fentry BPF prog<br/>(ctx = saved args)"]
    ORIG["call foo body (orig_call)<br/>only if fexit/fmod_ret present"]
    RET["capture return value"]
    FEXIT["call each fexit/fmod_ret BPF prog<br/>(ctx = args + ret)"]
    RESTORE["restore regs, return"]
    SAVE --> FENTRY --> ORIG --> RET --> FEXIT --> RESTORE
  end
  RESTORE --> BODY

The fentry/fexit path. What it shows: the traced function’s entry NOP is patched to jump into a generated trampoline that saves arguments, runs your fentry programs, optionally calls the original body, captures the return, runs your fexit programs, and restores state before letting foo() continue. The insight: there is no exception, no int3 breakpoint, no global trap handler dispatch — it is an ordinary, predictable call into tailored machine code, which is exactly why the overhead is a fraction of a kprobe’s. For a pure fentry attachment (no fexit), the trampoline does not call the original body at all (flag BPF_TRAMP_F_RESTORE_REGS); it restores registers and returns straight into foo()’s body, which then executes normally.

The five bytes everything hinges on

On x86-64 the patch site is exactly five bytes wide, and that number is not a coincidence — it is sizeof(call rel32). The architecture header states it outright: #define MCOUNT_INSN_SIZE 5 /* sizeof mcount call */, with MCOUNT_ADDR defined as ((unsigned long)(__fentry__)) (v6.12 arch/x86/include/asm/ftrace.h). The BPF JIT independently defines the same constant for its own poking: #define X86_PATCH_SIZE 5 (v6.12 bpf_jit_comp.c). ftrace’s two replacement generators are three lines each — ftrace_nop_replace() returns x86_nops[5], and ftrace_call_replace() returns text_gen_insn(CALL_INSN_OPCODE, ip, addr) (v6.12 arch/x86/kernel/ftrace.c). So the site is only ever one of two five-byte encodings:

packet-beta
0-7: "0xE8 — CALL rel32 opcode"
8-39: "rel32 — signed 32-bit displacement to the trampoline, computed as (target - (ip + 5))"

The armed state of a __fentry__ patch site: five bytes, one call rel32. What it shows: a near-call with a 32-bit signed displacement relative to the next instruction. The disarmed state occupies the same five bytes as a single multi-byte NOP (x86_nops[5]). The insight to take: because both states are exactly five bytes and the site is naturally aligned by the compiler, arming and disarming is a single text_poke of a fixed-width field — no instruction-boundary analysis, no relocation of surrounding code, and no need to displace and emulate anything. Contrast a kprobe, which must replace a variable-length instruction with a one-byte int3 and then arrange for the displaced instruction to execute somewhere. The whole performance story of fentry starts here, with the compiler having reserved a fixed-size, fixed-purpose hole at the top of every function.

The rel32 field also imposes the mechanism’s one hard geometric limit: the trampoline must land within ±2 GiB of the patch site, which is why trampoline images are allocated from the module/JIT memory region rather than arbitrary vmalloc space.

Two details complicate the picture slightly on modern builds. With Indirect Branch Tracking enabled, the compiler emits a four-byte endbr64 before the __fentry__ call, and both ftrace and the BPF JIT account for it — arch/x86/include/asm/ftrace.h defines FTRACE_MCOUNT_MAX_OFFSET as ENDBR_INSN_SIZE, and the trampoline generator, when it needs the address of the function body, skips it explicitly: if (is_endbr(*(u32 *)orig_call)) orig_call += ENDBR_INSN_SIZE; orig_call += X86_PATCH_SIZE;. That two-step skip is how BPF_TRAMP_F_SKIP_FRAME computes “the real start of foo(), past the ENDBR and past the patched call.” Second, -mfentry places the call at the top of the function, before the prologue — which is why the argument registers are still untouched when the trampoline runs, and is precisely what makes typed argument capture possible at all. The older -pg/mcount placement, after the prologue, would not have worked.

The contrast with kprobes is the whole point. A kprobe places a breakpoint (int3 on x86) at the target address; hitting it raises an exception, the CPU vectors into the kprobe handler, the handler single-steps or emulates the displaced instruction, and only then runs your BPF program. The trampoline replaces the trap with a direct call and the dispatch with hand-rolled, function-specific code. Three specific costs disappear: the #BP exception entry and its iret-shaped return, the walk of a shared handler list to find whose probe this was, and the construction of a full struct pt_regs — the trampoline spills only the nr_args registers the BTF function model says are live.

Quantifying the difference honestly requires care, because published multipliers are workload- and CPU-specific and are routinely quoted without their conditions. The kernel ships the apparatus to measure it yourself: tools/testing/selftests/bpf/benchs/bench_trigger.c defines directly comparable benchmarks over the same target function — BENCH_TRIG_KERNEL(kprobe, "kprobe"), kretprobe, kprobe-multi, kretprobe-multi, fentry, fexit and fmodret, all driven through the same batched trigger loop (v6.12 bench_trigger.c). Run bench trig-kprobe against bench trig-fentry on the machine you care about and the comparison is apples-to-apples by construction. Measured numbers for the kprobe side, with their dates and conditions attached, live in kprobes (§Overhead, With Real Numbers); the provider-level comparison from a tool user’s seat is in bpftrace (§kprobe: versus fentry:). What this note can assert from the design alone, and what LWN 804937 asserts, is the direction and its cause: no exception, no shared dispatch, no pt_regs.

Uncertain

Verify: any specific multiplier for fentry/fexit versus kprobe/kretprobe overhead. Reason: no primary-source benchmark run with stated hardware, kernel config and workload was obtained during this task; widely circulated figures (“~10x”) are undated and unconditioned, and the ratio genuinely moves with CPU generation (exception-entry cost), with CONFIG_MITIGATION_* settings, and with whether the kprobe is int3-backed, optimised, or ftrace-backed — the last of which shares the very same patch site as fentry and so is much closer in cost than the folklore suggests. To resolve: run bench trig-kprobe, bench trig-kretprobe, bench trig-fentry and bench trig-fexit from tools/testing/selftests/bpf on a named CPU and kernel build, and record the config alongside the numbers. uncertain

Mechanical Walk-through — From bpf() to a Live Trampoline

One trampoline per traced function, shared by all its programs

The kernel keeps a global hash table of trampolines, trampoline_table, keyed by a 64-bit key derived from the target’s BTF id (bpf_trampoline_compute_key). The table is sized 1 << 10 = 1024 buckets, with the comment that btf_vmlinux has ~22k attachable functions. 1k htab is enough. (v6.12 trampoline.c). The crucial design point: there is exactly one struct bpf_trampoline per traced function, and all fentry, fexit, and fmod_ret programs targeting that function share it. Each struct bpf_trampoline holds three program lists — one per kind — and a per-kind count:

enum bpf_tramp_prog_type {
	BPF_TRAMP_FENTRY,
	BPF_TRAMP_FEXIT,
	BPF_TRAMP_MODIFY_RETURN,
	BPF_TRAMP_MAX,
	BPF_TRAMP_REPLACE, /* more than MAX */
};
 
struct bpf_trampoline {
	struct hlist_node hlist;          /* link in trampoline_table */
	struct ftrace_ops *fops;          /* ftrace direct-call registration */
	struct mutex mutex;
	refcount_t refcnt;
	u32 flags;                        /* BPF_TRAMP_F_* */
	u64 key;
	struct {
		struct btf_func_model model;  /* nr_args, arg sizes — from BTF */
		void *addr;                   /* address of the traced function */
		bool ftrace_managed;
	} func;
	struct bpf_prog *extension_prog;
	struct hlist_head progs_hlist[BPF_TRAMP_MAX];
	int progs_cnt[BPF_TRAMP_MAX];     /* counter per kind */
	struct bpf_tramp_image *cur_image;/* the live machine-code image */
};

The btf_func_model is what makes typed argument access possible: btf_distill_func_proto() (called from the verifier during attach) walks the BTF prototype of the target and records how many arguments it has and the size of each. The trampoline generator then emits exactly enough mov [rsp+N], reg instructions to spill those argument registers onto the trampoline stack, where the BPF program reads them as a flat array u64 args[].

flowchart LR
  subgraph TBL["trampoline_table — 1024 hlist buckets, guarded by trampoline_mutex"]
    B0["bucket 0"]
    B1["bucket hash_64(key, 10)"]
    BN["bucket 1023"]
  end
  B1 --> TR["struct bpf_trampoline<br/>key = f(target BTF id)<br/>refcnt<br/>flags<br/>func.model (from BTF)<br/>func.addr (from kallsyms)<br/>func.ftrace_managed<br/>fops (ftrace registration)"]
  TR --> L0["progs_hlist[BPF_TRAMP_FENTRY]<br/>progs_cnt[0]"]
  TR --> L1["progs_hlist[BPF_TRAMP_FEXIT]<br/>progs_cnt[1]"]
  TR --> L2["progs_hlist[BPF_TRAMP_MODIFY_RETURN]<br/>progs_cnt[2]"]
  TR --> EXT["extension_prog<br/>(BPF_TRAMP_REPLACE — mutually<br/>exclusive with the three lists)"]
  TR --> IMG["cur_image -> struct bpf_tramp_image<br/>the ONE live machine-code blob"]
  L0 -.-> IMG
  L1 -.-> IMG
  L2 -.-> IMG
  style IMG fill:#e8f4e8,stroke:#4a4

One trampoline per function, three program lists, one image. What it shows: the global hash table maps a key derived from the target’s BTF id to a single bpf_trampoline, which holds three separate program lists — one per attach kind — and a pointer to the single currently-installed machine-code image. The insight to take: the three lists are inputs and the one image is the output, recomputed whenever any list changes. There is no dispatch table walked at runtime and no indirection between the lists and the image; the programs are compiled into the image as a straight-line sequence of call instructions. That is why per-hit cost scales linearly and predictably with the number of attached programs, why the count is capped, and why the image must be fully regenerated for every attach and detach rather than patched incrementally. It is also why extension_prog sits outside the three lists: an extension replaces the function rather than wrapping it, so it is incompatible with having a wrapping image at all, and __bpf_trampoline_link_prog() returns -EBUSY if you mix them.

Attaching: the verifier validates the target, then the trampoline links the program

When userspace loads an fentry program with BPF_PROG_LOAD, it supplies attach_btf_id (the BTF id of the target function) and expected_attach_type = BPF_TRACE_FENTRY. The verifier’s check_attach_btf_id() path resolves the target. For BPF_TRACE_FENTRY/BPF_TRACE_FEXIT it requires the BTF type to be a function (btf_type_is_func), distills the prototype into the function model, and resolves the runtime address via kallsyms_lookup_name() (or, for module functions, find_kallsyms_symbol_value()) (v6.12 verifier.c).

Then, when the program is attached (via a BPF link, BPF_LINK_CREATE), bpf_trampoline_link_prog() runs. It looks up or creates the trampoline for the key, adds the program to the appropriate progs_hlist[], bumps progs_cnt[], and calls bpf_trampoline_update(). That update function is the engine room:

1. It collects all currently-linked programs into a bpf_tramp_links array, one slot per kind.
2. If the total drops to zero, it unregisters the patch and frees the image.
3. Otherwise it computes the trampoline flags based on which kinds are present. If there are any fexit or fmod_ret programs, it sets BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME — meaning the trampoline must call the original function body and then continue to the exit programs. If it is fentry-only, it sets BPF_TRAMP_F_RESTORE_REGS instead — restore and return straight into the body, never calling it from the trampoline.
4. It asks the arch backend for the required image size (arch_bpf_trampoline_size), refuses anything over PAGE_SIZE (-E2BIG), allocates a fresh image, and calls arch_prepare_bpf_trampoline() to emit the machine code.
5. It marks the image executable (arch_protect_bpf_trampoline) and then installs it.

Installation: ftrace direct calls, or text-poke

Installation depends on whether the target sits at an ftrace patch-site. register_fentry() checks ftrace_location(); if the address is ftrace-managed, it uses register_ftrace_direct(tr->fops, ...) to point the function’s __fentry__ site at the trampoline. If the function is not ftrace-managed (e.g., it is another BPF program being extended), it falls back to bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr) — a raw, atomic instruction-rewrite using the architecture’s text-poke primitive (v6.12 trampoline.c):

static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
{
	void *ip = tr->func.addr;
	unsigned long faddr = ftrace_location((unsigned long)ip);
 
	if (faddr) {                       /* function is ftrace-managed */
		if (!tr->fops)
			return -ENOTSUPP;
		tr->func.ftrace_managed = true;
	}
	if (tr->func.ftrace_managed) {
		ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1);
		return register_ftrace_direct(tr->fops, (long)new_addr);
	}
	return bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); /* extend a BPF prog */
}

Subsequent program adds/removes do not re-patch the call-site — they generate a new image and atomically swap the call target with modify_fentry() (modify_ftrace_direct for ftrace-managed functions). This is why adding a second fentry program to an already-traced function is cheap and live: the old image keeps running until the new one is in place.

What an ftrace “direct call” actually is

This is the piece most descriptions skip, and it is the difference between a BPF trampoline and every other ftrace consumer. Ordinary ftrace users — the function tracer, kprobes running on its KPROBE_FLAG_FTRACE backend, function-graph — register a struct ftrace_ops with a callback. The patched call site does not jump to their code; it jumps to a shared ftrace dispatcher, which then walks the list of ftrace_ops registered for that address, and for each one whose filter matches, builds the register state that op asked for and calls its callback. That indirection is exactly what you want when several unrelated subsystems may be tracing the same function. It is also three things BPF cannot afford: a shared list walk, an indirect call, and a generic register-save policy (an op that sets FTRACE_OPS_FL_SAVE_REGS, as the kprobe backend must, gets a whole struct pt_regs built whether it needs one or not).

A direct call removes the dispatcher from the path. register_ftrace_direct(ops, addr) tells ftrace: for the address in this ops’ filter, do not route through the dispatcher at all — poke the site to call addr itself. The site becomes a plain call rel32 to the BPF trampoline image, and the trampoline is free to save exactly the registers it wants, because it was generated for this one function’s signature.

flowchart TD
  SITE["__fentry__ patch site in foo()"]
  SITE -->|"ordinary ftrace_ops"| DISP["ftrace dispatcher"]
  DISP --> WALK["walk ftrace_ops list<br/>for this ip"]
  WALK --> FILT["check each op's filter"]
  FILT --> REGS["build register state<br/>(full pt_regs if SAVE_REGS)"]
  REGS --> CB1["op A callback"]
  REGS --> CB2["op B callback"]
  SITE -->|"register_ftrace_direct()"| TRAMP["BPF trampoline image<br/>generated for foo's signature"]
  TRAMP --> SPILL["spill exactly nr_args registers"]
  SPILL --> PROG["call the JITed BPF program(s)"]
  style TRAMP fill:#e8f4e8,stroke:#4a4
  style DISP fill:#f6f0e0,stroke:#a94

Two ways out of the same five bytes. What it shows: the shared-dispatcher path is general — many independent ops, filtered, with a one-size-fits-all register-save policy — while the direct-call path is a single hard-wired jump into code generated for exactly one function. The insight to take: “direct” is about dispatch, not about patching. Both paths patch the same five bytes; the difference is what those five bytes point at. And there is an immediate consequence: a call site can host at most one direct call, because the five bytes hold one target. That is why there is exactly one struct bpf_trampoline per traced function, why all fentry/fexit/fmod_ret programs on that function must share it, and why the kernel must do real work when a direct call and an IPMODIFY user (a livepatch) both want the same site — the subject of a later section.

register_fentry() chooses between this path and a raw text-poke, and the whole decision is one call to ftrace_location() (v6.12 trampoline.c):

flowchart TD
  RF["register_fentry(tr, new_addr)"] --> LOC["faddr = ftrace_location(ip)"]
  LOC --> Q{"faddr != 0 ?<br/>i.e. is there an ftrace<br/>patch site at this address?"}
  Q -->|yes| FOPS{"tr->fops allocated?<br/>(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS)"}
  FOPS -->|no| ENOTSUPP["-ENOTSUPP<br/>attach fails"]
  FOPS -->|yes| MANAGED["tr->func.ftrace_managed = true<br/>ftrace_set_filter_ip(tr->fops, ip, 0, 1)<br/>register_ftrace_direct(tr->fops, new_addr)"]
  Q -->|no| POKE["bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr)<br/>raw atomic instruction rewrite"]
  POKE --> WHO["this is the BPF-to-BPF case:<br/>the target is another BPF program,<br/>which ftrace knows nothing about"]

How the kernel decides whether ftrace owns the site. What it shows: ftrace_location() is the sole discriminator. A kernel function compiled with function tracing has an ftrace record at its entry, so ftrace must be told; a JITed BPF program has a five-byte hole its own JIT reserved, which ftrace has never heard of, so the kernel pokes it directly. The insight to take: the fallback arm is not a legacy path — it is what makes fentry on a BPF program work, the second use case in Starovoitov’s original series. Attaching a tracing program to an XDP program uses the same trampoline machinery, but ftrace is not involved because there is no ftrace record to consult. Note also the -ENOTSUPP: on a kernel or architecture without CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS, tr->fops is never allocated (the field is inside an #ifdef), so attaching to a kernel function fails while attaching to a BPF program still works.

On arm64 the direct-call plumbing landed later than x86, and the arm64 series was explicitly reshaped around that gap: its v6 changelog notes that “since Mark is refactoring arm64 ftrace to support long jump and reduce the ftrace trampoline overhead, it’s not clear how we’ll attach bpf trampoline to regular kernel functions, so remove ftrace related patches for now”, shipping BPF-program attachment first and kernel-function attachment later (LWN, bpf trampoline for arm64). This is why fentry availability historically varied by architecture in a way that looks arbitrary from userspace: the trampoline generator was portable long before the installation mechanism was.

The Generated Stub Itself

Everything above describes how a trampoline gets installed. This section is about what is actually inside it, because the trampoline is not a fixed piece of assembly with a table — it is machine code emitted per function, and reading its shape explains every capability and every limit of fentry/fexit.

The stack frame it builds

__arch_prepare_bpf_trampoline() documents its own frame in a comment, and it is worth reading literally (v6.12 arch/x86/net/bpf_jit_comp.c):

        higher addresses
        +---------------------------+
 RBP+8  | return address            |  (pushed by the patched `call`)
 RBP+0  | saved RBP                 |
        +---------------------------+
 RBP-8  | return value              |  only if BPF_TRAMP_F_CALL_ORIG
        |                           |  or BPF_TRAMP_F_RET_FENTRY_RET
        +---------------------------+
        | reg_argN                  |  always
        | ...                       |
 -regs_off | reg_arg1               |  <-- THE BPF PROGRAM'S ctx POINTER
        +---------------------------+
 -nregs_off| regs count (nr_regs)   |  always -- backs bpf_get_func_arg_cnt()
        +---------------------------+
 -ip_off   | traced function's IP   |  only if BPF_TRAMP_F_IP_ARG
        +---------------------------+
 -rbx_off  | saved rbx              |  always (rbx carries the start timestamp)
        +---------------------------+
 -run_ctx_off | struct bpf_tramp_run_ctx |
        +---------------------------+
        | stack_argN                |  only if BPF_TRAMP_F_CALL_ORIG
        | ...                       |  (args 7+ must be re-pushed for the real call)
 -arg_stack_off | stack_arg1        |
        +---------------------------+
 RSP    | tail_call_cnt_ptr         |  only if BPF_TRAMP_F_TAIL_CALL_CTX
        +---------------------------+
        lower addresses

The generated trampoline’s stack frame, transcribed from the generator’s own comment with the conditions on each slot. An ASCII box diagram is used here rather than packet-beta because the offsets are not fixed constants — regs_off, nregs_off, ip_off, rbx_off, run_ctx_off and arg_stack_off are computed at generation time from the function’s argument count and the active flags, so there is no single bit-accurate layout to draw. What it shows: a frame assembled bottom-up by accumulating a running stack_size, where four of the eight regions are conditional. The insight to take: the single most important line is RBP - regs_off. That address, and nothing else, is what gets loaded into RDI as the BPF program’s context pointer — lea rdi, [rbp - regs_off]. From the program’s side the “context” of a tracing program is simply the spilled argument registers, laid out contiguously as u64 slots, which is why BPF_PROG() can unpack them into named typed parameters with no decoding at all, and why bpf_get_func_arg(n) is a bounds-checked load at ctx + 8n. The nr_regs slot immediately below is what bpf_get_func_arg_cnt() reads. And the return value living at a fixed RBP-8, above the arguments, is why an fexit program sees it as a trailing parameter after all the real ones.

Two subtleties in that frame reward a second look. First, a struct passed by value in registers consumes more than one slot: the generator pre-scans the model and, for any argument flagged BTF_FMODEL_STRUCT_ARG, adds (arg_size + 7) / 8 - 1 extra register slots, so nr_regs can exceed nr_args. If the total exceeds MAX_BPF_FUNC_ARGS (12) the generator returns -ENOTSUPP. Second, the stack_arg region exists only when the trampoline must call the original function and the function takes more than six arguments: x86-64 passes the first six in registers and the rest on the stack, so before calling the body the generator must copy those stack arguments into a fresh region — restore_regs() then save_args(..., for_call_origin=true). It even re-aligns: “make sure the stack pointer is 16-byte aligned if we need pass arguments on stack.”

The flags that decide the shape

Nine BPF_TRAMP_F_* bits parameterize the generator (v6.12 include/linux/bpf.h):

Flag	Bit	Effect on generated code
BPF_TRAMP_F_RESTORE_REGS	0	restore the argument registers and return into the function body; do not call it
BPF_TRAMP_F_CALL_ORIG	1	call the original function from inside the trampoline, between fentry and fexit programs
BPF_TRAMP_F_SKIP_FRAME	2	skip the trampoline’s own return address and return to the caller (add rsp, 8 before ret)
BPF_TRAMP_F_IP_ARG	3	store the traced function’s address in the frame, for bpf_get_func_ip()
BPF_TRAMP_F_RET_FENTRY_RET	4	propagate the fentry program’s return value — struct_ops only
BPF_TRAMP_F_ORIG_STACK	5	take the original function’s address from the stack instead of a baked-in constant
BPF_TRAMP_F_SHARE_IPMODIFY	6	an IPMODIFY user (livepatch) shares this site; set and cleared by ftrace callbacks
BPF_TRAMP_F_TAIL_CALL_CTX	7	cache and restore tail_call_cnt to avoid an infinite tail-call loop
BPF_TRAMP_F_INDIRECT	8	emit a CFI preamble so the image is safe to call indirectly — struct_ops

bpf_trampoline_update() derives most of them from what is attached, and the derivation is short enough to quote (v6.12 trampoline.c):

flowchart TD
  START["bpf_trampoline_update(tr)"] --> COLLECT["bpf_trampoline_get_progs():<br/>total = sum of progs_cnt[] over all 3 kinds"]
  COLLECT --> ZERO{"total == 0 ?"}
  ZERO -->|yes| UNREG["unregister_fentry()<br/>bpf_tramp_image_put(cur_image)<br/>site returns to a NOP"]
  ZERO -->|no| CLEAR["tr->flags &= (SHARE_IPMODIFY | TAIL_CALL_CTX)<br/>every other bit is recomputed from scratch"]
  CLEAR --> KIND{"any FEXIT or<br/>MODIFY_RETURN links?"}
  KIND -->|yes| ORIG["flags |= CALL_ORIG | SKIP_FRAME"]
  KIND -->|no| REST["flags |= RESTORE_REGS"]
  ORIG --> IPQ
  REST --> IPQ{"any prog with<br/>call_get_func_ip?"}
  IPQ -->|yes| IPARG["flags |= IP_ARG"]
  IPQ -->|no| LP
  IPARG --> LP{"SHARE_IPMODIFY and CALL_ORIG<br/>both set?"}
  LP -->|yes| OS["flags |= ORIG_STACK"]
  LP -->|no| SIZE
  OS --> SIZE["arch_bpf_trampoline_size()"]
  SIZE --> BIG{"size > PAGE_SIZE ?"}
  BIG -->|yes| E2BIG["-E2BIG"]
  BIG -->|no| GEN["allocate image, arch_prepare_bpf_trampoline(),<br/>arch_protect_bpf_trampoline()"]
  GEN --> INSTALL{"tr->cur_image exists?"}
  INSTALL -->|yes| MOD["modify_fentry(): swap the call target,<br/>old image keeps running"]
  INSTALL -->|no| REG["register_fentry(): first patch"]

Flag derivation and image installation, once per attach or detach. What it shows: the flags are recomputed from the current program population every time it changes — only SHARE_IPMODIFY and TAIL_CALL_CTX survive, because they describe the environment rather than the attachment. The insight to take: RESTORE_REGS and CALL_ORIG|SKIP_FRAME are mutually exclusive by construction (the source carries a NOTE: saying they “should not be set together”), and which one you get is decided solely by whether any fexit or fmod_ret program is attached. Adding a single fexit program to a function that already has ten fentry programs does not merely add a callback — it regenerates the image into a structurally different shape, one that now calls the function body itself and returns to the caller rather than into the body. And note the install branch: after the first register_fentry(), every subsequent change is a modify_fentry() that atomically swaps the call target while the old image is still executing. That is what makes attaching a second program to a hot function safe and live.

The emitted instructions, annotated

The generator’s comment carries a complete worked example for __be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev) — a two-argument function, so nr_args = 2. With only an fentry program attached (BPF_TRAMP_F_RESTORE_REGS), the emitted trampoline is:

push   rbp
mov    rbp, rsp
sub    rsp, 16                     ; space for skb and dev
push   rbx                         ; temp reg to carry the start timestamp
mov    qword ptr [rbp - 16], rdi   ; spill skb  (arg 1)
mov    qword ptr [rbp - 8],  rsi   ; spill dev  (arg 2)
call   __bpf_prog_enter            ; rcu_read_lock() and migrate_disable()
mov    rbx, rax                    ; remember start time if BPF stats are enabled
lea    rdi, [rbp - 16]             ; R1 == ctx of the BPF prog  <-- the spilled args
call   addr_of_jited_FENTRY_prog
movabsq rdi, 64bit_addr_of_struct_bpf_prog
mov    rsi, rbx                    ; prog start time
call   __bpf_prog_exit             ; rcu_read_unlock(), migrate_enable(), stats math
mov    rdi, qword ptr [rbp - 16]   ; restore skb -- the body must see them untouched
mov    rsi, qword ptr [rbp - 8]    ; restore dev
pop    rbx
leave
ret                                ; returns INTO eth_type_trans, just past the patch

Add an fexit program and the flags flip to CALL_ORIG | SKIP_FRAME; the same comment gives that variant, and the diff is the whole story:

sub    rsp, 24                     ; +8: space for skb, dev, AND the return value
...                                ; spill args, run fentry progs -- identical
mov    rdi, qword ptr [rbp - 24]   ; restore skb
mov    rsi, qword ptr [rbp - 16]   ; restore dev
call   eth_type_trans+5            ; <-- CALL THE BODY, skipping the patched 5 bytes
mov    qword ptr [rbp - 8], rax    ; capture the return value into the frame
call   __bpf_prog_enter
mov    rbx, rax
lea    rdi, [rbp - 24]             ; same ctx pointer -- args AND ret are now in it
call   addr_of_jited_FEXIT_prog    ; prog can access skb, dev, and the return value
movabsq rdi, 64bit_addr_of_struct_bpf_prog
mov    rsi, rbx
call   __bpf_prog_exit
mov    rax, qword ptr [rbp - 8]    ; restore the real return value into RAX
pop    rbx
leave
add    rsp, 8                      ; SKIP_FRAME: discard eth_type_trans's return addr
ret                                ; return to eth_type_trans's CALLER, not its body

The last three instructions are the ones to stare at. In the fentry-only case the trampoline rets back into the function body, which then runs and returns to its caller normally — the trampoline is a detour, and the function’s own frame is untouched. In the fexit case the trampoline has already called the body itself, so returning into it would run it twice; instead add rsp, 8 discards the return address that would have taken it there and the ret goes straight to the original caller. BPF_TRAMP_F_SKIP_FRAME is that add rsp, 8. From the caller’s point of view nothing has changed: it called eth_type_trans, and something returned with the right value in RAX.

sequenceDiagram
    autonumber
    participant C as caller
    participant T as trampoline image
    participant E as __bpf_prog_enter
    participant P as JITed BPF programs
    participant F as foo body
    C->>T: call foo -> patched 5 bytes land here
    T->>T: push rbp, sub rsp, spill nr_args regs
    T->>E: __bpf_prog_enter
    E->>E: rcu_read_lock, migrate_disable, set run_ctx
    E-->>T: start time, or 0 meaning skip
    T->>P: fentry prog(s), R1 = &frame[-regs_off]
    Note over P: sees arguments only.<br/>No return value exists yet.
    P-->>T: return value ignored
    T->>T: __bpf_prog_exit, restore arg regs
    alt fexit or fmod_ret attached (CALL_ORIG)
        T->>F: call foo+5, skipping the patch site
        F-->>T: RAX
        T->>T: store RAX at rbp-8
        T->>P: fexit prog(s), same ctx pointer
        Note over P: sees arguments AND return value
        T->>T: reload RAX from rbp-8, add rsp 8
        T-->>C: ret -- straight back to the caller
    else fentry only (RESTORE_REGS)
        T->>T: restore arg regs, leave
        T-->>F: ret into foo's body
        F-->>C: foo returns normally
    end

One trampoline, two control flows. What it shows: the fentry-only path is a detour that hands control back to the function body; the fexit path calls the body itself and returns past it. The insight to take: this is the mechanical reason fexit can see the return value and fentry cannot — not a policy, a stack layout. At the moment an fentry program runs, the function has not executed, so there is no return value anywhere in the machine; at the moment an fexit program runs, the trampoline has already captured RAX into its own frame at a known offset. It also explains why fexit costs strictly more than fentry: the fexit shape adds a real call to the body, a store, a second __bpf_prog_enter/__bpf_prog_exit pair, and a frame fix-up. Note step 5-6: __bpf_prog_enter can return 0, meaning “skip this program” — that is the per-program recursion guard (prog->active), and a skipped run is counted as a recursion_miss visible in bpftool prog show. bpftrace compares that per-program guard against the kprobe path’s global bpf_prog_active counter, which is a materially different failure mode.

The __bpf_prog_enter/__bpf_prog_exit pair is worth naming precisely, because it is where the trampoline’s non-generated cost lives. There are six enter variants and six matching exit variants — plain, _recur (with the prog->active guard), _lsm_cgroup, and sleepable versions of each — selected by bpf_trampoline_{enter,exit}() from the program’s type and sleepable flag. The plain one is four lines: rcu_read_lock(); migrate_disable(); run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx); return bpf_prog_start_time();. The sleepable ones take rcu_read_lock_trace() instead. The timing call is behind a static key — bpf_prog_start_time() reads sched_clock() only if (static_branch_unlikely(&bpf_stats_enabled_key)), so with BPF statistics off it costs one not-taken branch. Every one of these functions is marked notrace, for reasons the Attach Constraints section makes concrete.

Finally, the size ceiling. bpf_trampoline_update() refuses any image larger than PAGE_SIZE with -E2BIG, and the comment above BPF_MAX_TRAMP_LINKS explains where that budget goes: “Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 bytes on x86.” Roughly fifty bytes per attached program into four thousand is where the limit of 38 comes from.

fexit Reads Both Arguments and the Return Value

The headline advantage of fexit over kretprobe is that an fexit program sees the function’s input arguments and its return value simultaneously. A kretprobe fires only on return and, by itself, has lost the entry arguments — to correlate them you must place a kprobe on entry, stash the args in a map keyed by something like the thread id, and look them up again in the kretprobe. With fexit the trampoline has already saved the arguments on its own stack on the way in, so they are still there on the way out. The trampoline’s context layout places the arguments first, followed by the captured return value, and the helpers bpf_get_func_arg(), bpf_get_func_ret(), and bpf_get_func_arg_cnt() expose them. The verifier explicitly restricts bpf_get_func_ret():

-EOPNOTSUPP for tracing programs other than BPF_TRACE_FEXIT or BPF_MODIFY_RETURN. (v6.12 bpf.h helper docs)

That single line encodes the rule: only at exit (fexit) or at modify-return time does a return value exist to read. The three helpers all read the same frame, and their contracts line up exactly with the stack layout (v6.12 include/uapi/linux/bpf.h):

Helper	Reads	Fails with
bpf_get_func_arg(ctx, n, value)	the n-th argument register slot, zero-based	-EINVAL if n >= the traced function’s argument register count
bpf_get_func_ret(ctx, value)	the return-value slot at rbp-8	-EOPNOTSUPP for tracing programs other than BPF_TRACE_FEXIT or BPF_MODIFY_RETURN
bpf_get_func_arg_cnt(ctx)	the nr_regs slot	cannot fail; returns the count
bpf_get_func_ip(ctx)	the IP slot — present only if BPF_TRAMP_F_IP_ARG was set	requires a program that requested it, which is what sets the flag

Note the wording of the first: “Get n-th argument register”. It counts register slots, not C parameters. A struct passed by value occupies as many slots as it needs (size + 7) / 8, so for such a function the helper’s index and the source-level parameter index diverge — and bpf_get_func_arg_cnt() returns the register count, which is what nr_regs holds and what the diagram above labels “regs count”. The BPF_PROG() macro hides all of this when the signature is simple, which is why it is the right default; the helpers are the escape hatch for generic tracers that do not know the signature at compile time.

flowchart LR
  subgraph CTX["the ctx pointer both kinds receive — RBP minus regs_off"]
    direction TB
    A0["ctx[0] = arg 1"]
    A1["ctx[1] = arg 2"]
    AN["ctx[n-1] = arg n"]
    RV["ctx[n] = return value<br/>(this slot is rbp-8)"]
  end
  FEN["fentry program"] -->|"BPF_PROG unpacks<br/>ctx[0..n-1]"| A0
  FEX["fexit program"] -->|"BPF_PROG unpacks<br/>ctx[0..n-1] PLUS ctx[n]"| A0
  FEX -.->|"the trailing 'int ret' parameter"| RV
  FEN -.->|"slot exists but holds<br/>nothing meaningful yet"| RV
  style RV fill:#e8f4e8,stroke:#4a4

Why fexit takes one more parameter than the function it traces. What it shows: both program kinds get the same pointer to the same frame region; fexit simply reads one slot further. The insight to take: SEC("fexit/tcp_connect") int BPF_PROG(f, struct sock *sk, int ret) is not special syntax — the BPF_PROG macro is expanding a variadic argument list over a u64 array, and the trailing ret is slot n. This also explains a real footgun: declare the wrong number of arguments in an fexit program and you silently read the wrong slot, because nothing at the C level ties your parameter list to the traced function’s actual arity. The verifier catches the mismatch when the program’s own BTF is compared against the target’s FUNC_PROTO, which is a good reason to keep -g on and let BTF do the checking.

fmod_ret / BPF_MODIFY_RETURN — Changing the Return Value

BPF_MODIFY_RETURN (introduced by KP Singh in Linux 5.6, March 2020, LWN, Introduce BPF_MODIFY_RET tracing progs) is the most powerful and most restricted of the three. An fmod_ret program runs like an fentry program but its own return value can override the traced function’s return value: if it returns non-zero, the trampoline can skip the original function entirely and propagate the program’s value as the function’s result (used for error injection and for LSM deny verdicts). Because letting arbitrary code rewrite any kernel function’s return is obviously dangerous, the verifier confines fmod_ret to a narrow allow-list. check_attach_modify_return() permits a target only if it is on the error-injection list or its name begins with the security_ prefix (v6.12 verifier.c):

static int check_attach_modify_return(unsigned long addr, const char *func_name)
{
	if (within_error_injection_list(addr) ||
	    !strncmp(SECURITY_PREFIX, func_name, sizeof(SECURITY_PREFIX) - 1))
		return 0;
	return -EINVAL;        /* "%s() is not modifiable" */
}

A function joins the error-injection list by being annotated ALLOW_ERROR_INJECTION(fn, ...) in kernel source (gated by CONFIG_FUNCTION_ERROR_INJECTION). So you can fmod_ret-inject failures into, say, should_failslab or a block-layer entry point that the kernel authors deliberately marked injectable — but you cannot fmod_ret an arbitrary function like tcp_sendmsg. The bpf_override_return() helper (a related, kprobe-era mechanism) is similarly fenced behind the error-injection list. For the security_* arm there is a second gate at attach time: the original series states that “the modify_return programs are allowed for security hooks (with an extra CAP_MAC_ADMIN check) and functions whitelisted for error injection (ALLOW_ERROR_INJECTION)” (LWN 813724).

KP Singh’s cover letter sketches the intended control flow in pseudo-C, and it maps one-to-one onto what the generator emits:

int func_to_be_attached(int a, int b)
{   /* <--- do_fentry */
do_fmod_ret:
    <update ret by calling fmod_ret>
    if (ret != 0)
        goto do_fexit;
original_function:
    <side_effects_happen_here>
}   /* <--- do_fexit */

In the generated image that if is a real conditional branch, and the way it is emitted is a small piece of two-pass assembly worth seeing. invoke_bpf_mod_ret() runs each fmod_ret program, then compares its return value against zero and leaves a placeholder (v6.12 bpf_jit_comp.c):

/* cmp QWORD PTR [rbp - 0x8], 0x0 */
EMIT4(0x48, 0x83, 0x7d, 0xf8); EMIT1(0x00);
 
/* Save the location of the branch and Generate 6 nops
 * (4 bytes for an offset and 2 bytes for the jump) These nops
 * are replaced with a conditional jump once do_fexit (i.e. the
 * start of the fexit invocation) is finalized.
 */
branches[i] = prog;
emit_nops(&prog, 4 + 2);

The address of do_fexit is not known yet — it depends on how much code the original call and any remaining fmod_ret programs take — so the generator reserves six bytes per fmod_ret program and back-patches them once the layout is settled, aligning the target to 16 bytes first because the Intel optimization manual’s “Coding Rule 11: All branch targets should be 16-byte aligned.” Note the comparison target: [rbp - 0x8] is the return-value slot from the frame diagram above. An fmod_ret program writes its verdict into the same slot the original function’s return value would occupy, which is exactly why a non-zero verdict can substitute for it.

flowchart TD
  ENTRY["trampoline entry: spill args"] --> FEN["run fentry programs<br/>(return values ignored)"]
  FEN --> MOD1["run fmod_ret prog #1<br/>store its return at rbp-8"]
  MOD1 --> CMP1{"cmp [rbp-8], 0<br/>non-zero?"}
  CMP1 -->|"yes (jne, back-patched)"| FEXIT
  CMP1 -->|no| MOD2["run fmod_ret prog #2 ..."]
  MOD2 --> CMPN{"cmp [rbp-8], 0"}
  CMPN -->|yes| FEXIT
  CMPN -->|no| ORIG["call the original function body"]
  ORIG --> STORE["store its real return at rbp-8"]
  STORE --> FEXIT["do_fexit: 16-byte aligned<br/>run fexit programs"]
  FEXIT --> EPI["reload rax from rbp-8<br/>restore, add rsp 8, ret to caller"]
  style ORIG fill:#e8f4e8,stroke:#4a4
  style CMP1 fill:#f8e8e8,stroke:#a44
  style CMPN fill:#f8e8e8,stroke:#a44

How fmod_ret short-circuits a kernel function. What it shows: each fmod_ret program’s return value is tested immediately; the first non-zero one jumps past the original call straight to do_fexit, carrying its own value in the return slot. A chain of fmod_ret programs is therefore a short-circuiting sequence, and program order decides who wins. The insight to take: the green box is the side effect that never happens. This is not “observe and then alter the answer” — the function body is genuinely not executed, which is what makes fmod_ret useful for error injection (should_failslab returning -ENOMEM without ever allocating) and for LSM deny verdicts (security_* returning -EPERM without performing the operation). It is also exactly why the allow-list exists and is narrow: silently skipping the body of an arbitrary kernel function is a memory-safety and correctness hazard, not merely a policy question, so a function must have been deliberately marked — with ALLOW_ERROR_INJECTION() in its own source file, or by carrying the security_ prefix — before the verifier will permit it. Note the fexit programs still run on the short-circuit path; they observe the substituted value, not the real one.

There is one further route onto the allow-list that the check_attach_modify_return() snippet above does not show, because the verifier checks it first: btf_kfunc_is_modify_return(btf, btf_id, prog). A kfunc may be registered into a module’s fmod_ret ID set, which lets a module opt its own kfuncs into being modifiable without touching the error-injection machinery. The same set carries the KF_SLEEPABLE flag that decides whether a sleepable fmod_ret program may attach.

Trampoline Image Lifetime — percpu_ref + RCU Tasks

Swapping a live trampoline image is genuinely hard: while you are tearing down the old image, some CPU may be executing inside it right now. The kernel solves this with a layered reference-counting and grace-period scheme on struct bpf_tramp_image:

struct bpf_tramp_image {
	void *image;
	int size;
	struct bpf_ksym ksym;        /* so the image shows up in stack traces */
	struct percpu_ref pcref;     /* counts in-flight executions */
	void *ip_after_call;
	void *ip_epilogue;
	union { struct rcu_head rcu; struct work_struct work; };
};

The trampoline body itself does __bpf_tramp_enter() (a percpu_ref_get) on the way in and __bpf_tramp_exit() (a percpu_ref_put) on the way out for the fexit case, so pcref counts how many CPUs are mid-flight. When an image is retired, bpf_tramp_image_put() cannot free it immediately — and it takes two different teardown routes depending on the image’s shape, which the source distinguishes by whether im->ip_after_call is set (it is set only when the image contains a call to the original function, i.e. for fexit/fmod_ret images).

For an image that calls the original function, the first act is a text-poke: bpf_arch_text_poke(im->ip_after_call, BPF_MOD_JUMP, NULL, im->ip_epilogue) rewrites the instruction immediately after the original call into a jump straight to the epilogue. Any execution currently inside the original function will, on return, skip the fexit programs entirely — so “the progs will be freed even if the original function is still executing or sleeping.” Only then does teardown wait: call_rcu_tasks() to let the handful of assembly instructions before __bpf_tramp_enter finish, then percpu_ref_kill() to wait for every in-flight execution counted by pcref, then call_rcu_tasks() again for the few instructions after __bpf_tramp_exit, then finally a workqueue item that frees the memory.

For a fentry-only image there is no percpu_ref at all — the source is explicit that “the trampoline without fexit and fmod_ret progs doesn’t call original function and doesn’t use percpu_ref” — so teardown is call_rcu_tasks_trace() (to let sleepable programs finish) followed by call_rcu_tasks() (for everything else), then the free.

stateDiagram-v2
    [*] --> Allocated
    Allocated --> Live: arch_prepare_bpf_trampoline emits code<br/>arch_protect makes it executable<br/>register_fentry or modify_fentry installs it
    note right of Allocated
        ksym registered as bpf_trampoline_KEY
        so the image shows up in stack traces
    end note

    Live --> Retiring: a prog is added or removed<br/>a NEW image is generated and installed<br/>bpf_tramp_image_put on the old one
    Live --> Retiring: total prog count hits zero<br/>unregister_fentry restores the NOP

    state Retiring {
        [*] --> Choose
        Choose --> PokeEpilogue: image has ip_after_call<br/>(fexit or fmod_ret)
        Choose --> TraceGrace: fentry-only image
        PokeEpilogue --> TasksGrace1: text-poke ip_after_call to jump<br/>to ip_epilogue, so returns skip fexit
        TasksGrace1 --> KillRef: call_rcu_tasks waits out the<br/>first few asm instructions
        KillRef --> TasksGrace2: percpu_ref_kill waits for every<br/>in-flight execution to drop pcref
        TasksGrace2 --> Deferred: call_rcu_tasks waits out the<br/>epilogue asm instructions
        TraceGrace --> TasksGrace3: call_rcu_tasks_trace waits for<br/>sleepable programs
        TasksGrace3 --> Deferred: call_rcu_tasks waits for the rest
    }

    Retiring --> Freed: schedule_work then bpf_tramp_image_free<br/>ksym removed, JIT memory uncharged
    Freed --> [*]

The lifetime of one struct bpf_tramp_image, and why freeing it is not simply kfree. What it shows: an image is never mutated in place; every attach or detach builds a new image, installs it atomically, and retires the old one through a multi-stage grace period. The two branches inside Retiring are the fexit-shaped and fentry-shaped teardowns. The insight to take: two independent hazards are being covered, and neither mechanism alone suffices — the source comment names them as “percpu_ref to protect trampoline itself” and “rcu tasks to protect trampoline asm not covered by percpu_ref (which are few asm insns before __bpf_tramp_enter and after __bpf_tramp_exit).” The refcount cannot cover the instructions that run before the refcount is taken; Tasks RCU can, because its grace period is defined as “every task has voluntarily scheduled or is in userspace”, which is precisely the guarantee that no CPU is still parked inside a stretch of straight-line kernel assembly. Plain RCU would not do — a preemptible trampoline can be preempted mid-execution without ever passing a classic quiescent state. This is one of the clearest real examples of why Tasks RCU exists at all.

The !PREEMPT case gets a shortcut the code documents: “In !PREEMPT case the task that got interrupted in the first asm insns won’t go through an RCU quiescent state which the percpu_ref_kill will be waiting for. Hence the first call_rcu_tasks() is not necessary” — so on a non-preemptible kernel the fexit path drops straight to percpu_ref_kill(). The comment in the source spells out why both mechanisms are needed: percpu_ref to protect trampoline itself and rcu tasks to protect trampoline asm not covered by percpu_ref. Tasks-RCU exists precisely because plain RCU’s grace period does not cover preemptible code that does not pass through a normal RCU read-side critical section — and a trampoline can be preempted mid-execution.

The enclosing struct bpf_trampoline is refcounted separately with a plain refcount_t: bpf_trampoline_get() looks it up (creating it on first use, refcount_set(&tr->refcnt, 1)) and bumps the count; bpf_trampoline_put() does refcount_dec_and_test() and, only when the last user drops, removes it from the hash table and frees its ftrace_ops.

Sharing a Call Site with ftrace and Livepatching

A __fentry__ site can host only one direct call, but it can host a direct call and an ordinary ftrace_ops at the same time — the dispatcher and the direct call are arranged so both run. That works fine until the other user sets FTRACE_OPS_FL_IPMODIFY, which is a declaration that the callback intends to change regs->ip, i.e. to redirect execution somewhere else entirely. Kernel livepatching is the canonical IPMODIFY user: that is exactly how a patched function is replaced by its new version. The kprobe backend has an IPMODIFY ops too (kprobe_ipmodify_ops, discussed in kprobes).

The conflict is specific and subtle. A BPF trampoline with BPF_TRAMP_F_CALL_ORIG calls the original function by a baked-in constant address — orig_call is computed once at generation time as func_addr + ENDBR + 5, and emit_rsb_call(&prog, orig_call, ...) burns it into the image. If a livepatch has since redirected the function, that constant is stale: the trampoline would call the old, unpatched body, silently defeating the livepatch. Reading the return address from the stack instead would give the patched target, because the IPMODIFY user rewrote it.

BPF_TRAMP_F_ORIG_STACK is that alternative, and it changes two instructions in the generated code (v6.12 bpf_jit_comp.c):

if (flags & BPF_TRAMP_F_ORIG_STACK) {
        emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);  /* mov rbx, [rbp + 8] */
        EMIT2(0xff, 0xd3);                                  /* call *rbx          */
} else {
        emit_rsb_call(&prog, orig_call, ...);               /* call <constant>    */
}

[rbp + 8] is the return address slot from the frame diagram — whatever the caller (or the IPMODIFY user) put there. So the trampoline goes from a direct call to a constant, to an indirect call through a value read at runtime. Slower, and it gives up a correctly-predicted direct branch, which is exactly why it is not the default.

Negotiating the switch is a small protocol between ftrace and BPF, mediated by tr->fops->ops_func = bpf_tramp_ftrace_ops_func:

sequenceDiagram
    autonumber
    participant LP as livepatch (IPMODIFY ops)
    participant FT as ftrace core
    participant BT as bpf_tramp_ftrace_ops_func
    participant UP as bpf_trampoline_update
    LP->>FT: register an IPMODIFY ops on foo
    FT->>BT: FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER
    BT->>BT: tr->flags |= BPF_TRAMP_F_SHARE_IPMODIFY
    alt trampoline calls the original by constant
        BT->>UP: bpf_trampoline_update(tr, lock_direct_mutex = false)
        UP->>UP: SHARE_IPMODIFY and CALL_ORIG both set, so add ORIG_STACK
        UP->>UP: regenerate the image with call *rbx
        UP->>FT: modify_ftrace_direct_nolock to the new image
    else fentry-only trampoline
        BT-->>FT: nothing to do, no orig call to redirect
    end
    Note over LP,UP: the reverse command DISABLE_SHARE_IPMODIFY_PEER<br/>clears the flag and regenerates back to the fast form

How a livepatch and a BPF trampoline agree to share one patch site. What it shows: ftrace does not silently allow or refuse the overlap; it calls back into the BPF trampoline layer with an ftrace_ops_cmd, and the trampoline regenerates itself into a form that is compatible with someone else rewriting the return address. The insight to take: this is a genuine two-way protocol, not a lock. It even carries a retry: when the BPF attach is the one arriving second, bpf_tramp_ftrace_ops_func() cannot regenerate in place because it is already inside register_ftrace_direct(), so it sets the flag and returns -EAGAIN; bpf_trampoline_update() catches that, frees the image it just built, resets tr->fops->func and tr->fops->trampoline, and jumps back to its again: label to regenerate with ORIG_STACK set and retry the registration. The whole reason for this dance is the one-target-per-site constraint from the direct-call section — with only five bytes to point somewhere, coexistence has to be negotiated in the generated code rather than in the patch site.

The locking here is delicate enough that the source documents the order explicitly — “the normal locking order is tr->mutex ⇒ direct_mutex (ftrace.c) ⇒ ftrace_lock (ftrace.c)” — and because the ftrace-initiated commands arrive with direct_mutex already held, they take tr->mutex with mutex_trylock() and return -EAGAIN after a 1 ms sleep rather than risk an ABBA deadlock (v6.12 trampoline.c).

Attach Constraints — Which Functions Are Attachable, and Why

“Use fentry when you can” is only useful advice if you know when you cannot. The constraints come from four independent places, and conflating them makes the failures look arbitrary.

flowchart TD
  WANT["I want fentry/fexit on foo()"] --> NOTRACE{"is foo marked notrace,<br/>or does it run before ftrace init,<br/>or is it in a notrace translation unit?"}
  NOTRACE -->|yes| NO1["no __fentry__ call site exists.<br/>Nothing to patch. Use a kprobe."]
  NOTRACE -->|no| INLINE{"did the compiler inline foo,<br/>or make it a local static<br/>with no out-of-line copy?"}
  INLINE -->|yes| NO2["no symbol, no BTF FUNC record.<br/>'not found in kernel BTF'"]
  INLINE -->|no| BTFQ{"is CONFIG_DEBUG_INFO_BTF=y,<br/>and does foo appear as a<br/>BTF_KIND_FUNC in vmlinux BTF<br/>or in a module's split BTF?"}
  BTFQ -->|no| NO3["'attach_btf_id %u is not a function'"]
  BTFQ -->|yes| SIG{"does btf_distill_func_proto<br/>accept the signature?"}
  SIG -->|no| NO4["too many args / variadic /<br/>struct return / struct arg over 16 bytes /<br/>void argument"]
  SIG -->|yes| ADDR{"kallsyms_lookup_name(foo)<br/>returns an address?"}
  ADDR -->|no| NO5["'The address of function %s cannot be found'"]
  ADDR -->|yes| DENY{"is foo in the btf_id_deny set?"}
  DENY -->|yes| NO6["-EINVAL, would recurse"]
  DENY -->|no| DIRECT{"CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS?"}
  DIRECT -->|no| NO7["-ENOTSUPP"]
  DIRECT -->|yes| OK["attach succeeds"]
  style OK fill:#e8f4e8,stroke:#4a4

Seven gates between “I want to trace this” and a live trampoline. What it shows: the checks are ordered from “the compiler never emitted a hook” through “the type system says no” to “the runtime says no”, and each produces a different diagnostic. The insight to take: the first two gates are compiler facts and no kernel configuration can undo them — a notrace function has no five-byte hole and an inlined function has no callable body, so no amount of BTF helps. The middle gates are BTF facts, resolvable by building the kernel with CONFIG_DEBUG_INFO_BTF=y. Only the last is a kernel-config fact about the installation mechanism. When someone says “fentry doesn’t work on this function”, the useful next question is which of these seven it failed, and the kernel names each one differently on purpose.

Why notrace and inlining matter, concretely. notrace expands to an attribute that suppresses the -mfentry call for that function, so the five bytes simply are not there — this is applied to anything ftrace itself calls, which is why the whole __bpf_prog_enter/__bpf_prog_exit family in trampoline.c is declared static u64 notrace __bpf_prog_enter(...). Inlining is a different failure with the same symptom: an inlined function has no out-of-line body, so there is nothing to patch and typically no BTF_KIND_FUNC record either. This is why static helpers that look attachable in the source are frequently not attachable in the binary, and why the same program can attach on one kernel build and fail on another with different optimization settings. A function marked noinline is not merely a hint here — it is the difference between an attachable symbol and none.

The recursion denylist is a real, short list. kernel/bpf/verifier.c builds it with the BTF_ID macros described in BTF (BPF Type Format) and checks it with one call, btf_id_set_contains(&btf_id_deny, btf_id) (v6.12 verifier.c):

BTF_SET_START(btf_id_deny)
BTF_ID_UNUSED
#ifdef CONFIG_SMP
BTF_ID(func, migrate_disable)
BTF_ID(func, migrate_enable)
#endif
#if !defined CONFIG_PREEMPT_RCU && !defined CONFIG_TINY_RCU
BTF_ID(func, rcu_read_unlock_strict)
#endif
#if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_TRACE_PREEMPT_TOGGLE)
BTF_ID(func, preempt_count_add)
BTF_ID(func, preempt_count_sub)
#endif
#ifdef CONFIG_PREEMPT_RCU
BTF_ID(func, __rcu_read_lock)
BTF_ID(func, __rcu_read_unlock)
#endif
BTF_SET_END(btf_id_deny)

Read it against __bpf_prog_enter, which does rcu_read_lock(); migrate_disable();. Every name on the list is something the trampoline’s own prologue calls. Attaching to migrate_disable would mean: trampoline entered → __bpf_prog_enter → migrate_disable → its trampoline → __bpf_prog_enter → migrate_disable → unbounded recursion, ending in a stack overflow rather than a clean error. The #ifdefs are there because the list only needs to name functions that are actually out-of-line calls in this configuration — with CONFIG_PREEMPT_RCU=n, __rcu_read_lock is not a function at all. Note that the denylist is a narrow backstop, not the general defence: the general defence is notrace on the helpers themselves, and the per-program prog->active guard in __bpf_prog_enter_recur().

Attaching to other BPF programs has its own rules. Because fentry can target a BPF program, the verifier must prevent call chains: “Cannot nest tracing program attach more than once”, and for program extensions, “to avoid potential call chain cycles, prevent attaching of a program extension to another extension”. The most interesting one is asymmetric and the comment explains why at length — you may attach fentry/fexit to a BPF_PROG_TYPE_EXT program, but you may not extend an fentry/fexit program, because “if extending of fentry/fexit was allowed it would be possible to create long call chain fentry→extension→fentry→extension beyond reasonable stack size” (v6.12 verifier.c). Extensions and fentry/fexit are also mutually exclusive on the same trampoline: __bpf_trampoline_link_prog() returns -EBUSY if an extension program is already attached, and refuses an extension if any tracing program is.

Configuration / Code — A Worked fentry/fexit Pair

With libbpf and a CO-RE skeleton (see CO-RE (Compile Once Run Everywhere)), fentry/fexit programs are strikingly clean because of typed arguments. A program tracing tcp_connect(struct sock *sk):

#include "vmlinux.h"          /* generated from kernel BTF — gives struct sock */
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
 
/* SEC("fentry/<fn>") selects BPF_PROG_TYPE_TRACING + BPF_TRACE_FENTRY */
SEC("fentry/tcp_connect")
int BPF_PROG(on_connect_enter, struct sock *sk)   /* sk is TYPED, not pt_regs */
{
	__u32 dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
	bpf_printk("tcp_connect dport=%d", bpf_ntohs(dport));
	return 0;     /* fentry/fexit return value is ignored (not fmod_ret) */
}
 
/* fexit sees the SAME args PLUS the return value as a trailing parameter */
SEC("fexit/tcp_connect")
int BPF_PROG(on_connect_exit, struct sock *sk, int ret)
{
	bpf_printk("tcp_connect returned %d", ret);
	return 0;
}

Line-by-line: SEC("fentry/tcp_connect") is parsed by libbpf into prog_type = BPF_PROG_TYPE_TRACING, expected_attach_type = BPF_TRACE_FENTRY, and attach_btf_id resolved from tcp_connect’s BTF. The BPF_PROG() macro (from bpf_tracing.h) unpacks the trampoline’s saved-argument array into the named, typed parameters struct sock *sk — there is no pt_regs decoding because the trampoline already laid the args out by signature. BPF_CORE_READ(sk, __sk_common.skc_dport) is a CO-RE field read that survives kernel-layout changes. In the fexit variant, the trailing int ret parameter is the captured return value — the entire reason fexit exists.

For fmod_ret, the section is SEC("fmod_ret/<fn>"), the target must be error-injection-listed or security_*, and a non-zero return overrides the function result.

Failure Modes

• Cannot attach: ... not found in kernel BTF — the target function name does not appear in /sys/kernel/btf/vmlinux. Static, inlined, or non-traceable functions have no BTF function id and cannot be fentry’d. Use a kprobe or a tracepoint instead.

• %s() is not modifiable — you tried fmod_ret on a function that is neither error-injection-listed nor security_-prefixed. This is check_attach_modify_return() rejecting you; it is by design, not a bug.

• Notrace / no __fentry__ site — functions marked notrace, and functions called before ftrace is initialized, have no patchable entry NOP and cannot be attached.

• -E2BIG on attach — two different checks return this, and they mean different things.

  • __bpf_trampoline_link_prog() returns -E2BIG when the trampoline already holds BPF_MAX_TRAMP_LINKS programs — 38 on most architectures, 27 on s390x in v6.12. Note this is a total across all three kinds, not a per-kind cap: the check sums the counters first, for (i = 0; i < BPF_TRAMP_MAX; i++) cnt += tr->progs_cnt[i]; and then tests if (cnt >= BPF_MAX_TRAMP_LINKS). Thirty fentry programs plus nine fexit programs on one function is thirty-nine, and the thirty-ninth attach fails.
  • bpf_trampoline_update() returns -E2BIG when the generated image would exceed PAGE_SIZE. This is the constraint the link cap is a proxy for — roughly fifty bytes per program, plus the per-function frame setup — and a function with many arguments reaches it sooner.

• Architecture gaps — fentry/fexit require CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS. On architectures or kernels lacking it, attachment fails with -ENOTSUPP. arm64 gained support later than x86 (LWN 900453).

• Recursion / self-tracing — attaching fentry to functions the trampoline itself calls would recurse; those are named in the btf_id_deny BTF ID set and rejected with -EINVAL. See Attach Constraints above for the actual list and why each entry is on it. This is distinct from runtime recursion of a program into itself, which is caught by the per-program prog->active counter in __bpf_prog_enter_recur() and reported as a recursion_miss rather than an error.

Alternatives and When to Choose Them

Need	Use	Why
Trace a non-static, BTF-visible kernel function, low overhead, typed args	fentry/fexit	Direct call, no trap; reads args by type
See entry args and return together	fexit	Trampoline saved args on the way in
Change a return value (error injection / LSM)	fmod_ret	Only allowed mechanism, but allow-listed targets only
Function has no BTF / is inlined / arbitrary instruction offset	kprobe	Breakpoint works anywhere, no BTF needed
Stable, documented instrumentation point	tracepoint	ABI-stable across kernel versions
Periodic sampling / PMU counters	perf_event BPF Programs	Driven by the perf subsystem, not function entry

The honest trade-off: fentry/fexit are faster and ergonomically far nicer than kprobes, but they only work on functions the compiler kept as real symbols with a BTF prototype. Kprobes are slower but can probe any address, including arbitrary instruction offsets inside a function. Tracepoints are slower to add (they must be coded into the kernel) but are an explicit, stable contract. Use fentry/fexit as the default for kernel-function tracing on a modern kernel, and fall back to kprobes for the long tail of functions BTF cannot see.

The table above is deliberately short, because two sibling notes already carry the full decision from the other two directions and repeating them here would only let three copies drift apart. kprobes has the mechanism-by-mechanism comparison across kprobe, tracepoint, fentry/fexit, fprobe/kprobe_multi and uprobe, with attach cost, per-hit cost and “available since” columns, plus measured overhead numbers. bpftrace has the same choice as a tool user meets it — kprobe: versus fentry: in a one-liner, including the argN footguns and the provider-availability check. What this note owns is the layer underneath both: what the trampoline is, and therefore why the trade-offs come out the way they do.

Stability — These Are Not an ABI

fentry reads so much like a supported interface that it is worth being blunt about what it is not. The kernel’s design FAQ answers the question directly (v6.12 bpf_design_QA.rst):

Q: Attaching to arbitrary kernel functions is an ABI? A: NO. The kernel function prototypes will change, and BPF programs attaching to them will need to change. The BPF compile-once-run-everywhere (CO-RE) should be used in order to make it easier to adapt your BPF programs to different versions of the kernel.

And, closing the obvious follow-up:

Q: Marking a function with BTF_ID makes that function an ABI? A: NO. The BTF_ID macro does not cause a function to become part of the ABI any more than does the EXPORT_SYMBOL_GPL macro.

The same document says the same of tracepoints and of the places kprobes may attach. So attaching to tcp_connect is a use of an internal implementation detail, exactly as a kprobe on the same function is, and the fact that BTF gives you a typed struct sock *sk does not change that. Three practical consequences:

1. A signature change is a load-time failure, and that is the good outcome. If tcp_connect gains an argument, btf_distill_func_proto() produces a different btf_func_model, the trampoline lays out a different number of slots, and the program’s own BTF no longer matches — the attach is rejected. Compare a kprobe reading PT_REGS_PARM2, which will happily read whatever is in RSI now and report a plausible-looking wrong number. Loud beats silent.
2. Renaming or inlining a function removes it entirely, with no deprecation. Functions become static and get inlined between releases as a matter of ordinary refactoring, and nothing in the kernel’s process treats that as a break — because it is not one.
3. Production tools must degrade, not assume. This is why libbpf-tools and bpftrace scripts probe for the fentry provider and carry a kprobe fallback rather than requiring fentry, and why bpf_core_field_exists()-style feature testing belongs in any program meant to run on more than one kernel.

The security angle is the mirror image of the same fact. Loading a BPF_PROG_TYPE_TRACING program requires both CAP_BPF and CAP_PERFMON, and this is readable directly in bpf_prog_load() at the v6.12 tag (kernel/bpf/syscall.c):

bpf_cap = bpf_token_capable(token, CAP_BPF);
...
/* every program type except these two needs CAP_BPF */
if (type != BPF_PROG_TYPE_SOCKET_FILTER &&
    type != BPF_PROG_TYPE_CGROUP_SKB &&
    !bpf_cap)
        goto put_token;
 
if (is_net_admin_prog_type(type) && !bpf_token_capable(token, CAP_NET_ADMIN))
        goto put_token;
if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))
        goto put_token;

is_perfmon_prog_type() lists BPF_PROG_TYPE_TRACING explicitly, alongside KPROBE, TRACEPOINT, PERF_EVENT, RAW_TRACEPOINT, LSM, STRUCT_OPS (annotated “has access to struct sock”) and EXT (“extends any prog”). The wrapper matters as much as the capability: since the BPF token work, these are bpf_token_capable(token, cap) rather than bare capable(), and that function does bpf_ns_capable(token ? token->userns : &init_user_ns, cap) followed by an LSM hook — so with a delegated token the capability is checked against the token’s user namespace, not the initial one. Attaching to kernel functions is therefore delegatable to an unprivileged container in a way it historically was not, which is a meaningful change in the threat model and worth knowing before assuming “only root can fentry”.

The privilege is warranted: reading a traced function’s arguments is effectively read access to arbitrary kernel memory as soon as you follow a pointer. fmod_ret goes further and can alter kernel control flow, which is why it carries the narrow allow-list described above and, for security_* targets, an additional CAP_MAC_ADMIN check (LWN 813724). The denylist, the allow-list, the PAGE_SIZE image cap and the link cap are all there because the trampoline runs generated machine code at a kernel call site, which is about as privileged a position as exists.

Production Notes

fentry/fexit are the backbone of modern observability tools. The libbpf-tools rewrites of classic BCC tools (opensnoop, tcpconnect, biolatency, …) prefer fentry/fexit where available because the lower overhead matters at high event rates. Cilium and other production eBPF systems use BPF-to-BPF fentry (the “attach a tracing program to another BPF program” capability from the original series, LWN 804937) to observe their own datapath programs without modifying them. The same trampoline machinery underpins struct_ops (and therefore sched_ext) and BPF-LSM — struct_ops uses trampoline flag BPF_TRAMP_F_INDIRECT/BPF_TRAMP_F_RET_FENTRY_RET so a BPF program can stand in for a kernel ops-table callback. This is why the trampoline is one of eBPF’s most reused mechanisms: learning it once explains tracing, security, and pluggable kernel structures at once. bpf_prog_has_trampoline() is the kernel’s own predicate for “does this program go through one”: true for BPF_PROG_TYPE_TRACING with BPF_TRACE_FENTRY, BPF_TRACE_FEXIT or BPF_MODIFY_RETURN, and for BPF_PROG_TYPE_LSM with BPF_LSM_MAC.

Three things are worth knowing operationally, none of which are obvious from the API.

Trampolines are visible as kernel symbols. bpf_tramp_image_alloc() registers each image as a ksym named bpf_trampoline_<key> — snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu", key) — and bpf_image_ksym_add() additionally emits a PERF_RECORD_KSYMBOL event. So a bpf_trampoline_12345678 frame appearing in a stack trace or a perf profile is not corruption; it is the generated stub, and the number is the trampoline key derived from the target’s BTF id. Without this registration, any profile taken while a trampoline was on the stack would show an unresolved address, which is exactly the situation JIT-generated code normally creates.

Every attach and detach regenerates and reinstalls. This is cheap and live — the old image keeps executing until modify_fentry() swaps the target — but it is not free, and it is serialized on tr->mutex per function. A tool that attaches fifty programs to the same function does fifty image generations, each one larger than the last, and the fiftieth fails with -E2BIG. The retirement path also defers work through call_rcu_tasks chains and a workqueue, so images from a churning attach/detach loop accumulate briefly before being freed. Image memory is charged with bpf_jit_charge_modmem() against the JIT memory limit, so a runaway attacher hits that accounting rather than exhausting memory silently.

The recursion counter is the metric to watch. When __bpf_prog_enter_recur() finds prog->active already non-zero on this CPU it calls bpf_prog_inc_misses_counter() and returns 0, and the program simply does not run for that event. That is a silent data loss from the tool’s point of view — no error, just a missing sample — and it is surfaced only as the recursion_misses field in bpftool prog show. Any tracing tool attached to a function that its own helpers can reach should be checking it. bpftrace compares this per-program guard against the kprobe path’s global bpf_prog_active, which suppresses every BPF tracing program on the CPU rather than one, and is therefore the more damaging of the two.

See Also

• BPF Program Types — the §5 dispatcher; BPF_PROG_TYPE_TRACING is one row in it
• kprobe and uprobe BPF Programs — the trap-based predecessor fentry/fexit largely replace
• Tracepoint BPF Programs — the stable-ABI tracing alternative
• perf_event BPF Programs — sampling-driven tracing, a different attach model
• BTF (BPF Type Format) — supplies the function model that makes typed args possible
• BPF Kernel Functions (kfuncs) — kfuncs can also be registered as fmod_ret targets via KF_SLEEPABLE
• struct_ops and sched_ext — reuses the same trampoline machinery for ops tables
• BPF-LSM (Security Hooks) — security_* hooks are the other legal fmod_ret target
• kprobes — the breakpoint and optimized-kprobe mechanism itself, its ftrace backend, its blacklist, and measured overhead numbers; the fallback for every function BTF cannot see
• bpftrace — kprobe: versus fentry: as a tool user meets it, the provider availability check, and the recursion-guard comparison
• eBPF Verifier — check_attach_btf_id() lives there; it resolves the target before any of this machinery runs
• Dynamic ftrace and the mcount fentry Hook — the __fentry__ call sites and the NOP patching this note builds on
• Linux eBPF MOC — parent map
• Linux Tracing and Observability MOC — where the ftrace framework and higher-level tracing tools live