vault

fprobe and the Modern Function-Probe Path

14 min read

fprobe and the Modern Function-Probe Path

fprobe (short for “function probe”) is a thin in-kernel wrapper that lets you attach an entry handler and, optionally, an exit handler to many kernel functions at once, built directly on the ftrace function-hook machinery rather than on the breakpoint mechanism that ordinary kprobes use. Because it rides the compiler’s -pg/__fentry__ call site (see Dynamic ftrace and the mcount fentry Hook), a single fprobe can cover a whole filter list of functions with one ftrace_ops — no per-function breakpoint, no per-function trap — which is why it is dramatically cheaper than registering thousands of individual kprobes and is the foundation of the BPF kprobe.multi (“kprobe-multi”) attach type. At Linux 6.12 LTS, fprobe’s return path is implemented with rethook, the shared return-hook (shadow-stack) infrastructure — not the function-graph tracer; the “fprobe on fgraph” rewrite is a later development (see the version note below). This note describes the 6.12 implementation: register_fprobe/register_fprobe_ips/register_fprobe_syms, the entry and exit handlers, and how the ftrace-vs-breakpoint distinction makes fprobe fast-but-coarse where kprobes are slow-but-anywhere.

Uncertain

Verify: the exact kernel release in which “fprobe on fgraph” replaced the rethook return path (the brief states 6.14). Reason: the rewrite is a long Masami Hiramatsu patch series (v17 cover letter; LWN summary) whose stated goal is “to remove dependency of the rethook from fprobe so that the return hook code and shadow stack can be reduced,” but the primary sources fetched here did not pin the merge-window release. What is verified is the v6.12 state: kernel/trace/fprobe.c at tag v6.12 #include <linux/rethook.h> and uses rethook for exits. To resolve: check the git log --merges / changelog for the merge of commit-series tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph against v6.14. uncertain

Mental Model — One Hook, Many Functions

The single most important idea is that fprobe trades generality for throughput. A kprobe can be placed on almost any instruction in the kernel because it works by overwriting that instruction with a breakpoint (int3 on x86) and trapping into the kernel exception path. That universality is expensive: each probe is its own patched site, each hit is a trap, and attaching to 50,000 functions means 50,000 patched sites and 50,000 trap handlers. fprobe instead attaches only to function entries — specifically the __fentry__/mcount call that the compiler already emitted at the top of every traceable function when the kernel was built with -pg (see Dynamic ftrace and the mcount fentry Hook). ftrace already owns those call sites and already supports pointing a single ftrace_ops at an arbitrary set of them. fprobe simply registers one such ftrace_ops, hands ftrace the whole list of target addresses, and lets ftrace’s existing dispatch fan a call out to the fprobe handler for every function in the set.

flowchart TB
  subgraph BUILD["Compile time (-pg)"]
    F1["func_a:<br/>call __fentry__<br/>...body..."]
    F2["func_b:<br/>call __fentry__<br/>...body..."]
    F3["func_c:<br/>call __fentry__<br/>...body..."]
  end
  OPS["one struct ftrace_ops<br/>(fp-&gt;ops, filter = {a,b,c})"]
  ENTRY["fprobe_handler()<br/>recursion lock + entry_handler"]
  RH["rethook: hijack return addr<br/>(shadow stack node)"]
  EXIT["fprobe_exit_handler()<br/>on function return"]
  F1 -- "ftrace dispatch" --> OPS
  F2 -- "ftrace dispatch" --> OPS
  F3 -- "ftrace dispatch" --> OPS
  OPS --> ENTRY
  ENTRY -- "if exit_handler set" --> RH
  RH -- "function returns" --> EXIT

How a single fprobe covers many functions. What it shows: at build time every function carries a __fentry__ call; fprobe registers exactly one ftrace_ops whose filter names all the target functions, so ftrace’s existing dispatch routes every entry into one fprobe_handler. If an exit handler is wanted, the entry handler asks rethook to hijack the return address so the exit handler fires when the function returns. The insight to take: the per-function cost is just “being in the filter list,” not a separate patched breakpoint — that is why fprobe scales to tens of thousands of functions where individual kprobes would not.

Think of fprobe as the plural of the function-entry probe: where the classic kretprobe is “trap on entry, hijack the return for one function,” fprobe is “ftrace-hook a whole set of functions, optionally hijack each return.” The return-hijack half is delegated to a separate, reusable component — rethook — precisely so the same shadow-stack logic can be shared between kretprobes and fprobe.

Mechanical Walk-through — From register_fprobe_ips to the Exit Handler

The cleanest entry point to read is register_fprobe_ips(), because it takes the target functions as raw addresses and does the minimum of parsing. In kernel/trace/fprobe.c at v6.12 its core is three calls:

ret = ftrace_set_filter_ips(&fp->ops, addrs, num, 0, 0);
ret = fprobe_init_rethook(fp, num);
ret = register_ftrace_function(&fp->ops);

The first line, ftrace_set_filter_ips, programs the fprobe’s own struct ftrace_ops (fp->ops) so that its filter matches exactly the num addresses in addrs. This is the heart of the “many functions, one hook” property: fp->ops is a single object, and ftrace’s filter machinery (covered in ftrace Filtering and Triggers) lets that one object select an arbitrary subset of all traceable functions. The second line, fprobe_init_rethook, allocates the return-hook pool only if an exit handler was requested (more below). The third line, register_ftrace_function, activates the ftrace_ops — from this point ftrace will call into fprobe’s dispatch on every entry of every function in the filter.

register_fprobe() and register_fprobe_syms() are conveniences layered on top. register_fprobe(fp, filter, notfilter) takes glob-style strings — the documentation’s example is register_fprobe(&fp, "func*", "func2"), meaning “every function whose name starts with func, except func2” — resolves them to addresses, and funnels into the same ftrace_set_filter_ips path. register_fprobe_syms(fp, syms, num) takes an array of symbol names, resolves each via kallsyms, and does likewise. All three end up registering one ftrace_ops.

When a covered function is entered, ftrace dispatches to fprobe_handler. The first thing it does is take ftrace’s recursion lock:

bit = ftrace_test_recursion_trylock(ip, parent_ip);
if (bit < 0) {
    fp->nmissed++;
    return;
}
__fprobe_handler(ip, parent_ip, ops, fregs);
ftrace_test_recursion_unlock(bit);

ftrace_test_recursion_trylock guards against the handler re-entering itself (for example, if the handler calls a function that is also in the filter). If the lock cannot be taken, the event is dropped and fp->nmissed is incremented — fprobe deliberately misses rather than risk a recursion storm. The fprobe.rst documentation is explicit that this is a different model from kprobes: “Kprobes has per-cpu ‘current_kprobe’ variable which protects the kprobe handler from recursion in all cases,” whereas fprobe “uses only ftrace_test_recursion_trylock(),” which is lighter but permits the handler to run in more contexts (including interrupt context).

Inside __fprobe_handler, if the user set an exit_handler, fprobe arranges for the return to be caught. This is where rethook enters:

if (fp->exit_handler) {
    rh = rethook_try_get(fp->rethook);
    if (!rh) {
        fp->nmissed++;
        return;
    }
    fpr = container_of(rh, struct fprobe_rethook_node, node);
    fpr->entry_ip = ip;
    fpr->entry_parent_ip = parent_ip;
    if (fp->entry_data_size)
        entry_data = fpr->data;
}

rethook_try_get pulls a free node from the fprobe’s pre-allocated rethook pool. If the pool is exhausted — too many in-flight calls into the probed functions at once — rethook_try_get returns NULL, the exit is not hooked, and nmissed is incremented again. This is the second documented nmissed cause: fprobe “fails to setup the function exit because of the shortage of rethook (the shadow stack for hooking the function return).” When a node is obtained, the entry IP and parent IP are stashed in it, and a slice of per-call scratch storage (fpr->data, sized by entry_data_size) is exposed to the handlers so the entry handler can pass data forward to the exit handler. The entry callback then runs; if it returns non-zero, “the corresponding exit callback will be cancelled” — a way to say “I’m not interested in this call’s return.”

Finally, when an exit handler is in play, __fprobe_handler calls rethook_hook(rh, ftrace_get_regs(fregs), true). rethook works by replacing the function’s return address on the stack with the address of a trampoline and pushing a node describing the original return address onto a per-task shadow stack (rethook.c’s rethook_hook). When the probed function executes its ret, control lands in the rethook trampoline instead of the caller; rethook_trampoline_handler then runs every registered exit handler — for fprobe that is fprobe_exit_handler — restores the real return address, recycles the node back to the pool, and lets the function return to its true caller. The exit handler’s signature mirrors the entry handler’s:

typedef void (*fprobe_exit_cb)(struct fprobe *fp, unsigned long entry_ip,
                               unsigned long ret_ip, struct pt_regs *regs,
                               void *entry_data);

so the exit handler sees the same entry_ip, the ret_ip it is about to return to, the register state, and the entry_data blob the entry handler filled in.

The rethook dependency is the single most version-sensitive fact about 6.12 fprobe. kernel/trace/fprobe.c at tag v6.12 literally begins:

// SPDX-License-Identifier: GPL-2.0
/*
 * fprobe - Simple ftrace probe wrapper for function entry.
 */
#include <linux/err.h>
#include <linux/fprobe.h>
#include <linux/kallsyms.h>
#include <linux/kprobes.h>
#include <linux/rethook.h>
#include <linux/slab.h>
#include <linux/sort.h>
#include "trace.h"

The presence of #include <linux/rethook.h> and the absence of any function-graph header confirm: at 6.12, fprobe returns via rethook, not fgraph. The “fprobe on fgraph” work that removes this dependency lands later (see the uncertainty note about the exact release).

The struct fprobe and Its Fields

The user fills in a struct fprobe (from include/linux/fprobe.h at v6.12) and registers it. The meaningful fields are:

struct fprobe {
#ifdef CONFIG_FUNCTION_TRACER
    struct ftrace_ops ops;     /* the single ftrace hook covering all targets */
#endif
    unsigned long nmissed;     /* count of dropped entries/exits */
    unsigned int  flags;       /* FPROBE_FL_* */
    struct rethook *rethook;    /* return-hook pool (only if exit_handler set) */
    size_t entry_data_size;     /* per-call scratch shared entry->exit */
    int    nr_maxactive;        /* max simultaneous in-flight exits to track */
    fprobe_entry_cb entry_handler;
    fprobe_exit_cb  exit_handler;
};

ops is the one ftrace hook. nmissed is the diagnostic counter discussed above — a non-zero value means the probe is dropping events under pressure. entry_data_size reserves a scratch buffer per outstanding call so the entry handler can hand state to the exit handler (a timestamp, an argument value) without a side table. nr_maxactive bounds how many concurrent return-hooks fprobe will track — analogous to a kretprobe’s maxactive — and feeds the size of the rethook pool. The two callback typedefs:

typedef int  (*fprobe_entry_cb)(struct fprobe *fp, unsigned long entry_ip,
                                unsigned long ret_ip, struct pt_regs *regs,
                                void *entry_data);
typedef void (*fprobe_exit_cb)(struct fprobe *fp, unsigned long entry_ip,
                               unsigned long ret_ip, struct pt_regs *regs,
                               void *entry_data);

Note the entry callback returns int (non-zero cancels the exit hook) while the exit callback returns void. There are two flags in v6.12: FPROBE_FL_DISABLED (value 1, soft-disable without unregistering) and FPROBE_FL_KPROBE_SHARED (value 2, “the handler is shared with kprobes” — set before registration so fprobe and kprobe handlers do not nest dangerously on the same code path).

Configuration — Userspace Sees fprobe as Fprobe-Events

Most kernel developers never call register_fprobe directly; they use one of the consumers. Two matter most. First, the fprobe-events tracefs interface (documented in Documentation/trace/fprobetrace.rst at v6.12), which exposes fprobe through the same dynamic_events file used by uprobe events and eprobes. The synopsis is:

f[:[GRP/][EVENT]] SYM [FETCHARGS]            : probe on function entry
f[MAXACTIVE][:[GRP/][EVENT]] SYM%return [FETCHARGS] : probe on function exit

The leading f selects fprobe; SYM names the function; %return requests the exit variant; FETCHARGS like $arg1, $arg2, $retval pull function arguments and return values (with BTF enabled you can name parameters by name and even dereference struct fields with -> and .). The documentation frames it plainly: “Fprobe event is similar to the kprobe event, but limited to probe on the function entry and exit only.” A worked example writing into tracefs:

# Trace entry of every function whose name starts with "vfs_", in one event
cd /sys/kernel/tracing
echo 'f:myg/vfs_entry vfs_* $arg1' >> dynamic_events
echo 1 > events/myg/vfs_entry/enable
cat trace_pipe

Here a single fprobe event covers the whole vfs_* family — exactly the multi-function power fprobe provides, surfaced to userspace without any C.

Second, and more consequentially, BPF’s kprobe.multi link is implemented on fprobe. In kernel/trace/bpf_trace.c at v6.12, the link structure literally embeds an fprobe:

struct bpf_kprobe_multi_link {
    struct bpf_link link;
    struct fprobe   fp;
    unsigned long  *addrs;
    u64            *cookies;
    u32             cnt;
    ...
};

and attachment is:

if (!(flags & BPF_F_KPROBE_MULTI_RETURN))
    link->fp.entry_handler = kprobe_multi_link_handler;
if ((flags & BPF_F_KPROBE_MULTI_RETURN) || is_kprobe_session(prog))
    link->fp.exit_handler = kprobe_multi_link_exit_handler;
...
err = register_fprobe_ips(&link->fp, addrs, cnt);

This is why a bpftrace one-liner that attaches to thousands of kprobe:vfs_* functions in milliseconds is feasible: under the hood it is one register_fprobe_ips call against one ftrace_ops, not thousands of individual kprobe registrations. See fentry fexit and BPF Trampolines for the related-but-distinct BPF trampoline path (which is for single functions and runs without a trap at all), and bpftrace for the front-end.

Failure Modes and How to Diagnose Them

The dominant failure mode is dropped events, visible as a rising nmissed. There are exactly two causes in 6.12, both documented and both seen in the code above: (1) ftrace_test_recursion_trylock failed — the handler would have re-entered itself, so the whole entry+exit pair is skipped; (2) rethook_try_get returned NULL — the return-hook pool was exhausted, so only the exit is skipped (the entry handler already ran). The fix for the second is a larger nr_maxactive (more rethook nodes), exactly as you would raise a kretprobe’s maxactive. A workload that calls the probed functions very deeply or very concurrently is the classic trigger.

A second class of confusion is which functions are even probeable. fprobe rides ftrace, so it can only attach to functions ftrace can hook — those that received a __fentry__/mcount call site and are not on ftrace’s notrace blocklist (notably much of ftrace’s own machinery, and functions marked notrace). A kprobe, by contrast, can be placed mid-function on an arbitrary instruction. If a target “won’t probe” with fprobe but works with a kprobe, the function is almost certainly not ftrace-attachable. Check /sys/kernel/tracing/available_filter_functions — if the symbol is absent there, fprobe cannot reach it.

A third subtlety is the entry_ip vs the instruction pointer in regs. The fprobe.rst documentation warns that @entry_ip is “the ftrace address of the traced function” and may differ from the actual function entry, and that the IP in @regs at entry may not equal @entry_ip. Handlers that compute offsets from the IP must use the documented field, not assume they are identical.

Alternatives and When to Choose Them

Against kprobes: choose fprobe when you want to instrument function boundaries of many functions cheaply. Choose a kprobe when you need to probe a specific instruction inside a function, or a function ftrace cannot hook. The trade is generality (kprobe) versus throughput-at-scale (fprobe). Against kretprobes: a kretprobe is effectively “fprobe for one function’s return,” and indeed both now share the rethook shadow-stack infrastructure at 6.12 — fprobe is the multi-function generalization. Against fexit BPF trampolines: a BPF trampoline attaches to a single function and is faster still (it patches the __fentry__ call to jump straight to a generated trampoline, no ftrace dispatch loop, no recursion lock) but does not offer the “one hook, thousands of functions” property — that is exactly the niche kprobe.multi/fprobe fills. Against plain ftrace function tracer: the function tracer just records “function X was entered”; fprobe lets you run an arbitrary handler (and an exit handler) with access to arguments and return values.

Production Notes

The headline real-world consumer is BPF kprobe-multi, introduced precisely to make “attach to every function matching a glob” affordable for tools like bpftrace and retsnoop. The original fprobe series (Hiramatsu, lkml v8 cover letter) was motivated by giving BPF a fast multi-attach path that did not pay per-function kprobe costs. The later “fprobe on fgraph” rewrite (LWN.net, Multi-function graph and fprobe on fgraph) continues that arc: by re-hosting fprobe on the function-graph return mechanism, it removes the separate rethook pool and its shadow stack, shrinking memory and unifying the return-hook path — but, again, that is after 6.12 (see the uncertainty flag). For 6.12 production, the operational knobs are nr_maxactive (raise if nmissed climbs from rethook shortage) and awareness that fprobe handlers can run in interrupt context, so they must be non-blocking and re-entrancy-safe.

See Also

Graph View

Backlinks