kprobes

A kprobe (kernel probe) is Linux’s dynamic instruction-level instrumentation primitive: it lets you “dynamically break into any kernel routine and collect debugging and performance information non-disruptively” by trapping at “almost any kernel code address” (kprobes.rst, v6.12). The textbook mechanism is a software breakpoint: registration saves the original byte at the target address and overwrites it with a breakpoint instruction — int3 (0xCC) on x86 (core.c, v6.12). When any CPU executes that byte it traps, the CPU registers are captured into a struct pt_regs, your pre_handler runs, the kernel executes the displaced original instruction from a private out-of-line copy, your post_handler runs, and execution resumes after the probe site. That is the picture every write-up gives, and on a modern kernel it is the least likely of three paths to actually be taken: a probe that lands on a function’s ftrace call site is armed through ftrace instead, and a probe that passes the optimizer’s safety analysis is promoted to a jump. Unlike a tracepoint — a hook the kernel authors placed in advance — a kprobe can attach to a function, or an instruction, the authors never anticipated you would care about. That power is the whole point and also the whole risk: kprobes are bound to internal symbol names that change between kernels, and a blacklist of functions (the kprobe machinery itself, entry code, noinstr text) cannot be probed at all. kprobes are the engine behind tracefs kprobe-events and the eBPF BPF_PROG_TYPE_KPROBE program type; the BPF-consumer view is covered in kprobe and uprobe BPF Programs.

Version pin

Every source-code claim in this note was read from the Linux v6.12 tree via raw.githubusercontent.com, and file/line references mean that tree. v6.12 is a maintained long-term-support (LTS) release, not mainline — mainline has moved well past it — so this is a deliberate pin to a stable, still-supported vintage rather than a stale one. Where a mechanism changed materially before v6.12 the change is dated by its release; where something is known to have moved after v6.12 it is called out explicitly rather than silently folded in.

Mental Model

Think of a kprobe as the kernel planting a conditional breakpoint on itself and servicing it in software, where your handler is the debugger script that runs at the breakpoint. A traditional debugger stops the process and hands control to a human; the kernel instead runs your registered handler in-line, in a few hundred nanoseconds or less, and resumes — millions of times a second if the probed code is hot. The defining difficulty is that the kernel is live: other CPUs may be executing the very function you are patching, and there is no “stop the world” you can afford on a production box. Two tricks fall out of that constraint. First, arming overwrites only the first byte with the breakpoint, so the trap fires cleanly no matter where the other CPUs are and no matter how long the original instruction was. Second, the kernel never single-steps the in-place instruction (whose first byte is now a breakpoint); it runs a private copy stored elsewhere, so a second CPU racing through the same address still trips the breakpoint and is handled correctly. The kprobes documentation states the reasoning outright: “It would be simpler to single-step the actual instruction in place, but then Kprobes would have to temporarily remove the breakpoint instruction. This would open a small time window when another CPU could sail right past the probepoint” (kprobes.rst, v6.12).

flowchart TB
  subgraph REG["register_kprobe()"]
    R1["validate addr<br/>text? blacklisted? ftrace site?"]
    R2["save original byte -&gt; p-&gt;opcode<br/>copy original insn -&gt; p-&gt;ainsn.insn"]
    R3["arch_arm_kprobe:<br/>text_poke first byte = int3 (0xCC)"]
    R1 --> R2 --> R3
  end
  subgraph HIT["CPU hits the int3"]
    H1["trap -&gt; save registers into pt_regs"]
    H2["get_kprobe(addr) finds the kprobe"]
    H3["run pre_handler(p, regs)"]
    H4["execute the out-of-line copy<br/>(p-&gt;ainsn.insn)"]
    H5["run post_handler(p, regs, flags)"]
    H6["resume at next instruction"]
    H1 --> H2 --> H3 --> H4 --> H5 --> H6
  end
  R3 -.->|"breakpoint now live"| H1
  H3 -->|"returns != 0:<br/>handler changed regs-&gt;ip"| H6

The classic (breakpoint-based) kprobe lifecycle. What it shows: registration patches the target’s first byte to int3 and stashes both the original byte and a relocatable copy of the whole instruction; a hit traps, captures registers, runs your pre_handler, executes the copy (never the in-place breakpoint), runs your post_handler, and resumes. The insight to take: the breakpoint lives at the real address so that every CPU traps, but the original instruction is executed from a separate copy so that recovery is safe under SMP with no window where the probe is missing. A pre_handler that returns non-zero — meaning it rewrote regs->ip — skips the displaced-instruction step entirely and jumps wherever it pointed.

Three Arming Backends, One API

The single most useful thing to internalise about v6.12 kprobes is that register_kprobe() is a façade over three different patching mechanisms, chosen automatically, and that the int3 breakpoint is the fallback rather than the norm. The choice is made in two places in kernel/kprobes.c.

The first branch is in check_ftrace_location(), called at the very top of check_kprobe_address_safe() (kprobes.c, v6.12):

	unsigned long addr = (unsigned long)p->addr;
 
	if (ftrace_location(addr) == addr) {
#ifdef CONFIG_KPROBES_ON_FTRACE
		p->flags |= KPROBE_FLAG_FTRACE;
#else	/* !CONFIG_KPROBES_ON_FTRACE */
		return -EINVAL;
#endif
	}
	return 0;

If the requested address is a dynamic-ftrace call site — which, on a kernel built with CONFIG_DYNAMIC_FTRACE, is true of the entry of essentially every traceable function, because the compiler emitted an __fentry__ call there that ftrace turned into a five-byte nop — the kprobe is flagged KPROBE_FLAG_FTRACE and will be armed by registering an ftrace ops on that address, not by writing an int3. Read the #else branch too: on a kernel without CONFIG_KPROBES_ON_FTRACE, that same address is outright rejected with -EINVAL. This is why “you can’t kprobe the entry of that function” reports vary wildly across distributions — the failure mode is a build-configuration property, not a property of the function.

The second branch is the last statement of register_kprobe(): try_to_optimize_kprobe(p), which asks whether the breakpoint can be promoted to a five-byte jump into a detour buffer. That is the CONFIG_OPTPROBES path, and it is the subject of Optimized kprobes and the Breakpoint to Jump Path.

flowchart TD
  A["register_kprobe(p)"] --> B["_kprobe_addr():<br/>resolve symbol_name+offset<br/>via kallsyms"]
  B --> C{"ftrace_location(addr)<br/>== addr ?"}
  C -->|"yes, and<br/>CONFIG_KPROBES_ON_FTRACE=y"| D["KPROBE_FLAG_FTRACE<br/>arch_prepare_kprobe_ftrace():<br/>no out-of-line copy at all"]
  C -->|"yes, but<br/>CONFIG_KPROBES_ON_FTRACE=n"| E["-EINVAL<br/>(cannot probe here)"]
  C -->|"no"| F["arch_prepare_kprobe():<br/>can_probe()? get_insn_slot()<br/>copy + relocate the insn"]
  D --> G["arm_kprobe_ftrace():<br/>ftrace_set_filter_ip()<br/>+ register_ftrace_function()"]
  F --> H["arch_arm_kprobe():<br/>text_poke 1 byte = 0xCC<br/>text_poke_sync() IPI"]
  H --> I{"try_to_optimize_kprobe():<br/>OPTPROBES + no post_handler<br/>+ can_optimize()?"}
  I -->|"yes"| J["queue on optimizing_list;<br/>workqueue swaps 5 bytes<br/>for jmp to detour buffer<br/>[OPTIMIZED]"]
  I -->|"no"| K["stays a breakpoint probe"]
  G --> L["listed as [FTRACE]"]

How v6.12 decides which machinery arms a kprobe, traced from register_kprobe() in kernel/kprobes.c. What it shows: three mutually exclusive backends behind one API — ftrace call-site registration, a raw int3 breakpoint, and a jump-optimised detour — plus the configuration-dependent hard failure when a function-entry address hits ftrace on a kernel that cannot use it. The insight to take: on a mainstream distribution kernel (CONFIG_DYNAMIC_FTRACE=y, CONFIG_KPROBES_ON_FTRACE=y, CONFIG_OPTPROBES=y), a plain kprobe:some_function at offset 0 almost never plants an int3 — it becomes an ftrace registration, and only probes at a non-zero offset inside a function take the breakpoint path. The tags in /sys/kernel/debug/kprobes/list ([FTRACE], [OPTIMIZED]) tell you which one you actually got.

The user-visible consequences are worth stating plainly, because they explain a lot of otherwise-baffling behaviour:

	ftrace-backed ([FTRACE])	breakpoint (int3)	optimised ([OPTIMIZED])
Where it can attach	function entry (the __fentry__ site) only	any instruction boundary	any site that passes can_optimize()
What is patched	ftrace’s own nop/call site, via ftrace’s filter	one byte → 0xCC	five bytes → jmp rel32
Out-of-line copy	none (arch_prepare_kprobe_ftrace() sets p->ainsn.insn = NULL)	yes, one insn_slot	yes, a full detour buffer
Cost of a hit	one ftrace trampoline call	synchronous #BP exception	one taken jump
pre_handler may redirect regs->ip	yes	yes	no — silently ignored
post_handler supported	yes (forces FTRACE_OPS_FL_IPMODIFY ops)	yes	no — a post_handler blocks optimisation

Behavioural differences between the three arming backends, assembled from kernel/kprobes.c, arch/x86/kernel/kprobes/ftrace.c and Documentation/trace/kprobes.rst (all v6.12). The insight to take: the row that bites people is the last-but-one. The documentation’s “NOTE for geeks” says it directly — “when the probe is optimized, that [regs->ip] modification is ignored” — so a fault-injection or live-patch style handler that works on one machine can silently stop working on another purely because the probe got optimised there. The documented workarounds are to attach an empty post_handler (which disqualifies optimisation) or to set debug.kprobes_optimization=0.

struct kprobe and the Handler Contract

You drive a kprobe through a struct kprobe, declared in include/linux/kprobes.h. The v6.12 definition (kprobes.h, v6.12):

struct kprobe {
	struct hlist_node hlist;
	struct list_head list;		/* list of kprobes for multi-handler support */
	unsigned long nmissed;		/* number of times temporarily disarmed */
	kprobe_opcode_t *addr;		/* location of the probe point */
	const char *symbol_name;	/* user may indicate symbol name */
	unsigned int offset;		/* offset into the symbol */
	kprobe_pre_handler_t pre_handler;   /* called before addr is executed */
	kprobe_post_handler_t post_handler; /* called after addr is executed */
	kprobe_opcode_t opcode;		/* saved opcode replaced by breakpoint */
	struct arch_specific_insn ainsn;    /* copy of the original instruction */
	u32 flags;
};

The struct is unusual in that the caller and the kernel each own a disjoint half of it, and getting that division wrong is the single commonest programming error against this API.

flowchart LR
  subgraph YOU["You fill in (before register_kprobe)"]
    Y1["addr<br/><i>raw kernel address</i>"]
    Y2["symbol_name + offset<br/><i>resolved via kallsyms</i>"]
    Y3["pre_handler"]
    Y4["post_handler"]
    Y5["flags<br/><i>only KPROBE_FLAG_DISABLED accepted</i>"]
  end
  subgraph KERNEL["Kernel fills in (during/after registration)"]
    K1["opcode<br/><i>the displaced first byte</i>"]
    K2["ainsn<br/><i>arch copy of the whole insn,<br/>emulate_op, boostable, size</i>"]
    K3["hlist<br/><i>slot in kprobe_table[]</i>"]
    K4["list<br/><i>chain under an aggr kprobe</i>"]
    K5["nmissed<br/><i>hits that could not be serviced</i>"]
    K6["flags |= FTRACE / ON_FUNC_ENTRY / OPTIMIZED"]
  end
  Y1 -.->|"mutually exclusive:<br/>both set = -EINVAL"| Y2
  YOU --> RK["register_kprobe(p)"] --> KERNEL

Ownership of struct kprobe’s fields, per include/linux/kprobes.h and register_kprobe() in kernel/kprobes.c (v6.12). What it shows: which fields are inputs and which are kernel-managed outputs, and the mutual exclusivity of addr versus symbol_name+offset. The insight to take: p->flags is masked on entry — register_kprobe() executes p->flags &= KPROBE_FLAG_DISABLED; before doing anything else, so any other flag you set is silently discarded. nmissed is a diagnostic you read afterwards, not something you initialise; it is the counter that tells you your probe is silently losing events.

The fields you fill in to place a probe are addr (a raw kernel address) or symbol_name + offset (let the kernel resolve the address from kallsyms); the documentation is explicit that “if both are specified, kprobe registration will fail with -EINVAL”. The fields the kernel fills in are opcode (the original first byte it displaced) and ainsn (the architecture-specific copy of the original instruction plus the metadata described below). nmissed counts probe hits that could not be serviced — a re-entrant hit, or on the ftrace path a recursion the ftrace recursion guard rejected.

Two handler callbacks form the user-visible contract (kprobes.h, v6.12):

typedef int (*kprobe_pre_handler_t) (struct kprobe *, struct pt_regs *);
typedef void (*kprobe_post_handler_t) (struct kprobe *, struct pt_regs *,
				       unsigned long flags);

• pre_handler(p, regs) runs at the probe point, before the probed instruction executes. This is the natural place to read function arguments, which at function entry sit in the architecture’s argument registers inside pt_regs. Its return value is meaningful: “If you change the instruction pointer … in pre_handler, you must return !0 so that kprobes stops single stepping and just returns to the given address” (kprobes.rst, v6.12). A return of 0 means “carry on with normal processing”. The same document tells ordinary users to “return 0 here unless you’re a Kprobes geek.”
• post_handler(p, regs, flags) runs after the probed instruction has been executed, unless the pre_handler diverted execution. The documentation’s own note on the third argument is refreshingly candid: “flags always seems to be zero.” Attaching a post_handler is not free of side effects: it disqualifies the probe from jump optimisation, and on the ftrace path it forces the probe onto the kprobe_ipmodify_ops ftrace ops (FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY) rather than the plain kprobe_ftrace_ops.
• fault_handler is not a field of struct kprobe in v6.12 — it was removed from the generic struct. Fault handling during probe processing is done by the architecture’s kprobe_fault_handler(), which the fault paths reach on their own: gp_try_fixup_and_notify() in arch/x86/kernel/traps.c calls it directly for a #GP, and the page-fault handler in arch/x86/mm/fault.c reaches it through the generic kprobe_page_fault() wrapper. If the displaced instruction faults, kprobe_fault_handler() resets the current kprobe, points regs->ip back at the probe address, and lets the normal page-fault handler proceed (core.c, v6.12).

Multiple kprobes may share one address. When register_kprobe() finds an existing probe at p->addr via get_kprobe(), it calls register_aggr_kprobe(old_p, p) instead of arming a second breakpoint: an aggregate kprobe owns the patch site and chains the real probes on its list, calling each in turn. This is why struct kprobe carries both an hlist (its slot in the global address-keyed hash table) and a list (its position in an aggregate’s chain) — they are different memberships, not redundancy.

Registration, Step by Step

A kprobe is installed with register_kprobe(struct kprobe *kp) and removed with unregister_kprobe(). Registration is where the patching actually happens, and the ordering is load-bearing at every step (kprobes.c, v6.12):

int register_kprobe(struct kprobe *p)
{
	/* Adjust probe address from symbol */
	addr = _kprobe_addr(p->addr, p->symbol_name, p->offset, &on_func_entry);
	if (IS_ERR(addr))
		return PTR_ERR(addr);
	p->addr = addr;
 
	ret = warn_kprobe_rereg(p);            /* already registered? */
	if (ret)
		return ret;
 
	p->flags &= KPROBE_FLAG_DISABLED;      /* discard caller's other flags */
	p->nmissed = 0;
	INIT_LIST_HEAD(&p->list);
 
	ret = check_kprobe_address_safe(p, &probed_mod);   /* text? blacklist? */
	if (ret)
		return ret;
 
	mutex_lock(&kprobe_mutex);
	if (on_func_entry)
		p->flags |= KPROBE_FLAG_ON_FUNC_ENTRY;
 
	old_p = get_kprobe(p->addr);
	if (old_p) {                           /* someone is already here */
		ret = register_aggr_kprobe(old_p, p);
		goto out;
	}
 
	cpus_read_lock();
	mutex_lock(&text_mutex);               /* prevent concurrent text modification */
	ret = prepare_kprobe(p);               /* copy + relocate original insn */
	mutex_unlock(&text_mutex);
	cpus_read_unlock();
	if (ret)
		goto out;
 
	INIT_HLIST_NODE(&p->hlist);
	hlist_add_head_rcu(&p->hlist,
		       &kprobe_table[hash_ptr(p->addr, KPROBE_HASH_BITS)]);
 
	if (!kprobes_all_disarmed && !kprobe_disabled(p)) {
		ret = arm_kprobe(p);           /* plant the int3 / register ftrace ops */
		if (ret) {
			hlist_del_rcu(&p->hlist);
			synchronize_rcu();
			goto out;
		}
	}
 
	try_to_optimize_kprobe(p);             /* maybe promote to jmp */
out:
	mutex_unlock(&kprobe_mutex);
	if (probed_mod)
		module_put(probed_mod);
	return ret;
}

The two orderings that matter most are easy to miss in linear code. First, the kprobe is inserted into kprobe_table[] before it is armed: the hash-table insert is what makes get_kprobe(addr) able to find it, and if arming happened first there would be a window in which a CPU could take the trap and find no owning kprobe. Second, arm_kprobe() is the last thing that can fail, and its failure path unlinks the hash entry and calls synchronize_rcu() before returning — because the table is walked under RCU, the unlink alone is not enough to guarantee no CPU still holds a pointer.

sequenceDiagram
  autonumber
  participant M as your module
  participant K as kernel/kprobes.c
  participant A as arch/x86/.../core.c
  participant T as text_poke machinery
  participant C as other CPUs

  M->>K: register_kprobe(&kp)
  K->>K: _kprobe_addr(): kallsyms lookup<br/>symbol_name + offset -> addr
  K->>K: arch_adjust_kprobe_addr():<br/>skip ENDBR/BTI CFI landing pad at +0
  K->>K: check_ftrace_location(): ftrace site?
  K->>K: check_kprobe_address_safe():<br/>core_kernel_text / module text?<br/>blacklist? jump-label? static-call?<br/>find_bug()? __cfi_/__pfx_ symbol?
  Note over K: any failure here -> -EINVAL, nothing patched
  K->>K: mutex_lock(kprobe_mutex)
  K->>K: get_kprobe(addr) -> existing probe?
  alt address already probed
    K->>K: register_aggr_kprobe(): chain onto aggregate
  else first probe here
    K->>K: cpus_read_lock(); mutex_lock(text_mutex)
    K->>A: prepare_kprobe() -> arch_prepare_kprobe()
    A->>A: can_probe(): decode the function<br/>from its start to find insn boundary
    A->>A: get_insn_slot(): allocate ROX page slot
    A->>A: __copy_instruction(): copy + fix rel32 displacement
    A->>A: prepare_emulation(): pick emulate_op for<br/>jmp/call/jcc/loop/ret/pushf/popf
    A->>A: prepare_singlestep(): append reljump (boost)<br/>or a trailing int3
    A-->>K: p->ainsn populated
    K->>K: hlist_add_head_rcu() into kprobe_table[]
    K->>A: arm_kprobe() -> arch_arm_kprobe()
    A->>T: text_poke(p->addr, &int3, 1)
    T->>C: text_poke_sync(): IPI -> do_sync_core()
    C-->>T: serialising insn executed, prefetch discarded
    A->>A: perf_event_text_poke(): tell perf the text changed
  end
  K->>K: try_to_optimize_kprobe()
  K-->>M: 0

The full register_kprobe() path on x86-64, v6.12, from the caller down to the cross-CPU synchronisation. What it shows: address resolution and safety validation happen entirely before any text is touched; instruction copying happens under text_mutex with CPU hotplug held off; and the actual byte write is followed by an inter-processor interrupt to every other CPU. The insight to take: step 20 (text_poke_sync) is the step people forget exists. Writing the byte is not enough — another CPU may already have prefetched the old first byte into its pipeline and would sail straight past the probe. text_poke_sync() runs on_each_cpu(do_sync_core, ...), forcing every CPU to execute a serialising instruction and discard stale prefetch. That IPI is also why arming a few thousand probes one at a time is slow, and why the batch register_kprobes() and kprobe_multi interfaces exist.

Two details in that sequence deserve a sentence of their own because they are recent and undocumented in the prose docs. arch_adjust_kprobe_addr() exists to cope with control-flow-integrity landing pads: on a kernel built with Indirect Branch Tracking (x86 IBT) or Branch Target Identification (arm64 BTI), the first instruction of a function is an ENDBR/BTI C and probing it would be both useless and unsafe, so any offset inside the landing pad is remapped to “the first real instruction of the symbol”. And check_kprobe_address_safe() rejects __cfi_- and __pfx_-prefixed symbols outright via is_cfi_preamble_symbol() — those are the CFI type-hash preamble and the padding block that -fpatchable-function-entry puts before a function, not code you can meaningfully probe.

What Actually Gets Written: the Byte-Level View

arch_arm_kprobe() is four lines, and all of the subtlety is in what surrounds them (core.c, v6.12):

void arch_arm_kprobe(struct kprobe *p)
{
	u8 int3 = INT3_INSN_OPCODE;
 
	text_poke(p->addr, &int3, 1);
	text_poke_sync();
	perf_event_text_poke(p->addr, &p->opcode, 1, &int3, 1);
}
 
void arch_disarm_kprobe(struct kprobe *p)
{
	u8 int3 = INT3_INSN_OPCODE;
 
	perf_event_text_poke(p->addr, &int3, 1, &p->opcode, 1);
	text_poke(p->addr, &p->opcode, 1);
	text_poke_sync();
}

text_poke(p->addr, &int3, 1) overwrites exactly one byte with 0xCC. A single-byte write is what makes this safe without stopping the machine: x86 guarantees an aligned single-byte store is atomic with respect to instruction fetch, so no CPU can ever observe a half-written instruction. text_poke_sync() then issues the IPI described above. Note the asymmetry in the perf_event_text_poke() calls: on arm it is called after the poke, on disarm before it. The perf text-poke event tells a perf session recording the kernel’s instruction stream that these bytes changed, and it must bracket the window in which the old bytes are still executable — an ordering bug there would make a perf decode of the same window ambiguous.

Concretely, for a five-byte instruction at vfs_read+0x14:

BEFORE arming — the real text at p->addr
  addr:   +0    +1    +2    +3    +4    +5    +6
        +-----+-----+-----+-----+-----+-----+-----+
        | 48  | 89  | e5  | 41  | 57  | ..  | ..  |     mov %rsp,%rbp ; push %r15
        +-----+-----+-----+-----+-----+-----+-----+
          ^-- p->opcode will hold this byte (0x48)

AFTER arming — one byte changed, nothing else moved
  addr:   +0    +1    +2    +3    +4    +5    +6
        +-----+-----+-----+-----+-----+-----+-----+
        | CC  | 89  | e5  | 41  | 57  | ..  | ..  |     int3 ; (garbage tail, never executed)
        +-----+-----+-----+-----+-----+-----+-----+
          ^-- 0xCC = INT3_INSN_OPCODE

THE OUT-OF-LINE SLOT — p->ainsn.insn, on a separate ROX page
        +-----+-----+-----+ - - - - - - +-----+
        | 48  | 89  | e5  |             | CC  |   copy + trailing int3   (non-boosted)
        +-----+-----+-----+ - - - - - - +-----+
   or   +-----+-----+-----+-----+-----+-----+-----+-----+
        | 48  | 89  | e5  | e9  | rel32 (4 bytes)       |   copy + jmp back  (BOOSTED)
        +-----+-----+-----+-----+-----+-----+-----+-----+
                            ^-- synthesize_reljump() back to p->addr + insn->length

Before/after byte layout of a breakpoint-armed kprobe and the two possible shapes of its out-of-line slot, from arch_arm_kprobe() and prepare_singlestep() (v6.12 x86). Medium note: mermaid’s packet-beta is the vault default for byte layouts, but this is a before/after diff of two views of the same bytes plus a third buffer, which packet-beta cannot express in one figure, so an ASCII box diagram is used. What it shows: the in-place patch is a single byte at offset 0; the remaining bytes of the original instruction are left untouched and become unreachable garbage; the real instruction lives in a separate executable slot. The insight to take: the bytes at p->addr+1 … are stale but harmless — nothing executes them, because the only way into that region is the int3 at offset 0, and the handler resumes at p->addr + insn->length. It also explains why can_probe() must decode the function from its very start: to land on an instruction boundary at all, the decoder has to walk every preceding instruction.

The Hit Path, and the Myth of Single-Stepping

When a CPU executes the planted 0xCC it raises #BP, and do_int3() in arch/x86/kernel/traps.c — itself NOKPROBE_SYMBOL-marked — dispatches to kprobe_int3_handler() before falling back to the notify_die(DIE_INT3, ...) chain (core.c, v6.12):

int kprobe_int3_handler(struct pt_regs *regs)
{
	if (user_mode(regs))
		return 0;
 
	addr = (kprobe_opcode_t *)(regs->ip - sizeof(kprobe_opcode_t));
	kcb = get_kprobe_ctlblk();
	p = get_kprobe(addr);
 
	if (p) {
		if (kprobe_running()) {
			if (reenter_kprobe(p, regs, kcb))
				return 1;
		} else {
			set_current_kprobe(p, regs, kcb);
			kcb->kprobe_status = KPROBE_HIT_ACTIVE;
			if (!p->pre_handler || !p->pre_handler(p, regs))
				setup_singlestep(p, regs, kcb, 0);
			else
				reset_current_kprobe();
			return 1;
		}
	}
	...
	return 0;
}

Walking it: addr = regs->ip - 1 recovers the probe address, because the trap leaves regs->ip pointing past the one-byte breakpoint. get_kprobe(addr) looks the probe up in the global kprobe_table[] — the exact inverse of the hlist_add_head_rcu() done at registration. The compound condition if (!p->pre_handler || !p->pre_handler(p, regs)) is the return-value contract in executable form: no pre-handler, or one returning 0, falls through to setup_singlestep(); a non-zero return means the handler rewrote regs->ip and the displaced instruction must be skipped entirely.

Here is where most write-ups — and the kernel’s own prose documentation — are out of date. v6.12 x86 kprobes never single-step in the hardware sense. There is no TF flag being set, no #DB exception, and no kprobe_debug_handler(): grepping arch/x86/kernel/kprobes/core.c for X86_EFLAGS_TF or debug_handler at v6.12 returns nothing, and arch/x86/include/asm/kprobes.h declares kprobe_debug_handler() only in its #else branch — the one compiled when CONFIG_KPROBES is off, where it is a dead stub returning 0. (A comment in exc_debug_kernel() in arch/x86/kernel/traps.c still lists “Kprobes, consumed through kprobe_debug_handler()” as a user of TF single-stepping; that comment is stale with respect to the code beside it.) The source comment above resume_singlestep() states the actual design: “We also doesn’t use trap, but ‘int3’ again right after the copied instruction. Different from the trap single-step, ‘int3’ single-step can not handle the instruction which changes the ip register, e.g. jmp, call, conditional jmp, and the instructions which changes the IF flags because interrupt must be disabled around the single-stepping. Such instructions are software emulated, but others are single-stepped using ‘int3’.”

So the displaced instruction is executed by one of three mechanisms, selected at registration time by prepare_emulation() and prepare_singlestep():

flowchart TD
  S["pre_handler returned 0<br/>-&gt; setup_singlestep()"] --> D{"setup_detour_execution():<br/>is this probe OPTIMIZED?"}
  D -->|"yes"| OPT["jump into the detour buffer<br/>(see Optimized kprobes)"]
  D -->|"no"| E{"p-&gt;ainsn.emulate_op set?<br/>(prepare_emulation matched<br/>jmp / call / jcc / loop / ret /<br/>pushf / popf / indirect jmp+call)"}
  E -->|"yes"| EM["EMULATE: run int3_emulate_jmp/call/ret...<br/>directly on pt_regs.<br/>No second trap. kprobe_post_process()<br/>runs the post_handler immediately."]
  E -->|"no"| B{"p-&gt;ainsn.boostable?<br/>!CONFIG_PREEMPTION &amp;&amp; no post_handler<br/>&amp;&amp; can_boost(insn) &amp;&amp; room for a jmp32"}
  B -->|"yes"| BO["BOOST: reset_current_kprobe(),<br/>set regs-&gt;ip = p-&gt;ainsn.insn.<br/>The copy runs, then the appended<br/>jmp rel32 returns to p-&gt;addr+len.<br/>No second trap at all."]
  B -->|"no"| SS["int3 SINGLE-STEP: clear IF,<br/>set regs-&gt;ip = p-&gt;ainsn.insn.<br/>The copy runs, then the appended int3 traps.<br/>resume_singlestep() fixes up regs-&gt;ip,<br/>kprobe_post_process() runs post_handler."]

The three ways v6.12 x86 executes the instruction a breakpoint kprobe displaced, from setup_singlestep(), prepare_emulation() and prepare_singlestep() in arch/x86/kernel/kprobes/core.c. What it shows: control-flow instructions are emulated outright against pt_regs; simple instructions with a post_handler-free, non-preemptible probe get a jump appended to the copy so the second trap disappears; everything else pays a second int3. The insight to take: “kprobes single-steps the copy” is a two-trap cost model, and two of these three paths do not pay it. The middle path — the booster — is why the kernel’s own overhead table has a b column, and it is the difference between roughly 0.99 µs and 0.43 µs per hit on the reference hardware. can_boost() is a hand-maintained opcode bitmap (twobyte_is_boostable[] plus a switch over one-byte opcodes) in the same file; instructions that change IF, take exceptions, or are group-encoded are excluded.

Note the guard #if !defined(CONFIG_PREEMPTION) around the boost path and the identical condition inside prepare_singlestep(). Boosting means dropping the current-kprobe state before the copy runs; if the kernel could preempt in the middle of the copy, the bookkeeping needed to recover would not exist. The same constraint, for the same reason, gates jump optimisation. On a CONFIG_PREEMPT=y kernel — which most desktop and container-host distributions ship — you get neither, and the per-hit cost of a breakpoint kprobe is the full two-trap figure.

The ftrace Path: No Copy, No Trap, No Single-Step

When the probe is [FTRACE]-backed, none of the above happens. arch_prepare_kprobe_ftrace() is three lines and allocates nothing:

int arch_prepare_kprobe_ftrace(struct kprobe *p)
{
	p->ainsn.insn = NULL;
	p->ainsn.boostable = false;
	return 0;
}

Arming is ftrace_set_filter_ip(ops, addr, 0, 0) followed, for the first such probe, by register_ftrace_function(ops) — ordinary ftrace client registration (kprobes.c, v6.12). There are two ops, chosen per probe by whether it has a post_handler:

static struct ftrace_ops kprobe_ftrace_ops __read_mostly = {
	.func = kprobe_ftrace_handler,
	.flags = FTRACE_OPS_FL_SAVE_REGS,
};
 
static struct ftrace_ops kprobe_ipmodify_ops __read_mostly = {
	.func = kprobe_ftrace_handler,
	.flags = FTRACE_OPS_FL_SAVE_REGS | FTRACE_OPS_FL_IPMODIFY,
};

FTRACE_OPS_FL_SAVE_REGS is what makes this work at all: it tells ftrace to build a full struct pt_regs before calling the handler, so a kprobe pre_handler written against pt_regs sees exactly what it would have seen from a trap. FTRACE_OPS_FL_IPMODIFY declares that this ops may change regs->ip, which ftrace uses to refuse conflicting registrations (notably against live-patching). The handler then pretends a trap happened (ftrace.c, v6.12):

	unsigned long orig_ip = regs->ip;
	/* Kprobe handler expects regs->ip = ip + 1 as breakpoint hit */
	regs->ip = ip + sizeof(kprobe_opcode_t);
 
	__this_cpu_write(current_kprobe, p);
	kcb->kprobe_status = KPROBE_HIT_ACTIVE;
	if (!p->pre_handler || !p->pre_handler(p, regs)) {
		/*
		 * Emulate singlestep (and also recover regs->ip)
		 * as if there is a 5byte nop
		 */
		regs->ip = (unsigned long)p->addr + MCOUNT_INSN_SIZE;
		if (unlikely(p->post_handler)) {
			kcb->kprobe_status = KPROBE_HIT_SSDONE;
			p->post_handler(p, regs, 0);
		}
		regs->ip = orig_ip;
	}

Read that comment carefully — it is the whole trick. There is no instruction to displace, because the thing being “probed” is the five-byte ftrace call site itself. The handler fakes regs->ip = ip + 1 on the way in so that a pre-handler written for the trap ABI computes the right probe address, then fakes regs->ip = p->addr + MCOUNT_INSN_SIZE (the address after a five-byte nop) so that a post-handler sees what it would have seen after a real single-step, then restores the real orig_ip and lets ftrace return normally. Recursion is handled not by kprobes’ own reenter_kprobe() but by ftrace’s ftrace_test_recursion_trylock(); a rejected re-entry increments nmissed via kprobes_inc_nmissed_count(p) exactly as the trap path would.

The performance consequence is large and rarely stated: an ftrace-backed kprobe costs an ftrace trampoline call plus a pt_regs fill, not a synchronous exception. It is the reason a kprobe:vfs_read one-liner on a modern distro kernel is far cheaper than the 2005-era 0.5–1.0 µs figures in the documentation would suggest — those figures describe a code path your probe probably is not taking.

The Blacklist: What Cannot Be Probed, and Why

A kprobe traps into kernel code, so probing the wrong function causes infinite recursion or a double fault. The kernel therefore maintains a blacklist and validates every registration against it. check_kprobe_address_safe() is the gate (kprobes.c, v6.12):

	/* Ensure the address is in a text area, and find a module if exists. */
	*probed_mod = NULL;
	if (!core_kernel_text((unsigned long) p->addr)) {
		*probed_mod = __module_text_address((unsigned long) p->addr);
		if (!(*probed_mod)) {
			ret = -EINVAL;
			goto out;
		}
	}
	/* Ensure it is not in reserved area. */
	if (in_gate_area_no_mm((unsigned long) p->addr) ||
	    within_kprobe_blacklist((unsigned long) p->addr) ||
	    jump_label_text_reserved(p->addr, p->addr) ||
	    static_call_text_reserved(p->addr, p->addr) ||
	    find_bug((unsigned long)p->addr) ||
	    is_cfi_preamble_symbol((unsigned long)p->addr)) {
		ret = -EINVAL;
		goto out;
	}

That is six independent rejections, and lumping them all under “the blacklist” hides what is actually going on. They fall into three genuinely different categories.

flowchart TD
  A["candidate probe address"] --> B{"core_kernel_text()<br/>or module text?"}
  B -->|no| X1["-EINVAL<br/><i>not executable kernel text at all</i>"]
  B -->|yes| C{"in_gate_area_no_mm()?"}
  C -->|yes| X2["-EINVAL<br/><i>vsyscall gate page</i>"]
  C -->|no| D{"within_kprobe_blacklist()?"}
  D -->|yes| X3["-EINVAL<br/><i>recursion hazard</i>"]
  D -->|no| E{"jump_label_text_reserved()<br/>or static_call_text_reserved()?"}
  E -->|yes| X4["-EINVAL<br/><i>bytes are themselves runtime-patched;<br/>two patchers would collide</i>"]
  E -->|no| F{"find_bug()?"}
  F -->|yes| X5["-EINVAL<br/><i>a WARN/BUG ud2 site with<br/>bug-table metadata</i>"]
  F -->|no| G{"is_cfi_preamble_symbol()<br/>__cfi_* / __pfx_* ?"}
  G -->|yes| X6["-EINVAL<br/><i>CFI type hash / patchable padding,<br/>not real code</i>"]
  G -->|no| H{"module? __init text already freed?"}
  H -->|yes| X7["-ENOENT"]
  H -->|no| OK["accepted<br/>(then arch_prepare_kprobe -&gt; can_probe()<br/>may still return -EILSEQ)"]

  subgraph BL["within_kprobe_blacklist() consults..."]
    L1["arch_within_kprobe_blacklist():<br/>__kprobes_text_start..end"]
    L2["__noinstr_text_start..end<br/><i>(added by populate_kprobe_blacklist)</i>"]
    L3["arch_populate_kprobe_blacklist():<br/>__entry_text_start..end"]
    L4["_kprobe_blacklist section<br/><i>every NOKPROBE_SYMBOL()</i>"]
    L5["loaded modules' kprobe_blacklist[]<br/><i>+ their noinstr/kprobes text</i>"]
    L6["suffixed symbols: strip after '.'<br/>and re-check the base symbol"]
  end
  D -.-> BL

Every rejection path in check_kprobe_address_safe() and the five-plus sources within_kprobe_blacklist() actually consults (v6.12 kernel/kprobes.c, arch/x86/kernel/kprobes/core.c). What it shows: the “blacklist” is one branch out of six, and the blacklist itself is a union of a linker section, two address ranges, per-module lists, and a symbol-name normalisation step. The insight to take: all six return the same undifferentiated -EINVAL, which is why “kprobe registration failed” is such an unhelpful error — the kernel does not tell you which check rejected you. /sys/kernel/debug/kprobes/blacklist only shows the fourth branch’s contents, so an address rejected by, say, jump_label_text_reserved() will not appear there and the failure looks inexplicable.

The recursion-hazard blacklist is the interesting one, and the documentation gives the rationale directly: “Kprobes can probe most of the kernel except itself. This means that there are some functions where kprobes cannot probe. Probing (trapping) such functions can cause a recursive trap (e.g. double fault) or the nested probe handler may never be called” (kprobes.rst, v6.12). It is assembled from five sources:

1. __kprobes text. arch_within_kprobe_blacklist() is a __weak function whose generic body rejects [__kprobes_text_start, __kprobes_text_end) — the section that the __kprobes function annotation places code in.
2. noinstr text. populate_kprobe_blacklist() adds [__noinstr_text_start, __noinstr_text_end) at boot. This is not mentioned in kprobes.rst, and it is a big deal: noinstr is the section for code that runs before the kernel’s instrumentation infrastructure is safe to use — the low-level entry/exit and context-tracking code. If you have wondered why an entire cluster of __ functions around syscall entry silently refuse probes, this is why.
3. Architecture-added ranges. On x86, arch_populate_kprobe_blacklist() is exactly one call: kprobe_add_area_blacklist(__entry_text_start, __entry_text_end) — the interrupt and syscall entry stubs.
4. NOKPROBE_SYMBOL(). Any kernel developer can mark one function unprobeable; the macro emits an entry into the _kprobe_blacklist linker section, which populate_kprobe_blacklist() walks at boot. kernel/kprobes.c itself applies it 17 times and arch/x86/kernel/kprobes/core.c 16 times — including to the handlers you just read, kprobe_int3_handler, setup_singlestep, resume_singlestep, kprobe_post_process, and kprobe_ftrace_handler.
5. Modules. add_module_kprobe_blacklist() extends the list when a module carrying NOKPROBE_SYMBOL() annotations (or noinstr/__kprobes sections) is loaded, and remove_module_kprobe_blacklist() retracts it on unload.

within_kprobe_blacklist() adds one more subtlety on top: if the direct address check fails, it looks up the symbol name at that address, truncates it at the first ., and re-checks the base symbol. That handles compiler-generated suffixed clones — foo.cold, foo.constprop.0, foo.isra.0 — so that blacklisting foo blacklists its clones too. Without it, NOKPROBE_SYMBOL(foo) would be trivially defeated by GCC’s own optimiser.

You can read the resulting list at runtime:

# mount | grep debugfs || mount -t debugfs none /sys/kernel/debug
# wc -l /sys/kernel/debug/kprobes/blacklist
# head -3 /sys/kernel/debug/kprobes/blacklist
0xffffffff81c00000-0xffffffff81c01000	entry_SYSCALL_64
...
# cat /sys/kernel/debug/kprobes/list
ffffffff812a4e40  k  vfs_read+0x0    [FTRACE]
ffffffff812a51c0  r  tcp_v4_rcv+0x0
ffffffff8130a882  k  ext4_file_write_iter+0x22    [OPTIMIZED]

The three debugfs files are created at boot by debugfs_kprobe_init() (kprobes.c, v6.12): list (mode 0400), enabled (0600), and blacklist (0400). list’s columns are address, type (k for kprobe, r for kretprobe), symbol+offset, and status tags — [GONE] for a probe on a now-invalid address such as freed module init text, [DISABLED], [OPTIMIZED], and [FTRACE]. Reading those tags is the fastest way to find out which backend a probe actually landed on. enabled is a global kill switch: echoing 0 disarms every registered probe without changing any probe’s own disabled state, and echoing 1 re-arms them — note that a probe individually marked [DISABLED] stays disabled through both.

Beyond the explicit checks, three practical limits apply:

• Inlined functions. “If you install a probe in an inline-able function, Kprobes makes no attempt to chase down all inline instances of the function and install probes there. gcc may inline a function without being asked, so keep this in mind if you’re not seeing the probe hits you expect” (kprobes.rst, v6.12). The failure is silent: registration succeeds against whatever out-of-line copy survived (or fails with -EINVAL if none did), and you simply get fewer hits than the call graph implies.
• Mid-instruction addresses on CISC. “With CISC architectures (such as i386 and x86_64), the kprobes code does not validate if the kprobe.addr is at an instruction boundary. Use offset with caution.” In practice can_probe() does decode the function from its symbol start to find boundaries, and returns -EILSEQ from arch_prepare_kprobe() when the address is not one — but this is an arch courtesy, not a guarantee the generic API makes.
• __switch_to on x86-64. arch/x86/kernel/kprobes/core.c carries a kretprobe-specific blacklist with exactly one entry: {"__switch_to", }, commented “This function switches only current task, but doesn’t switch kernel stack.” register_kretprobe() checks it and returns -EINVAL. Note the scope: this table is consulted only by register_kretprobe(), and __switch_to() in arch/x86/kernel/process_64.c is marked __notrace_funcgraph but not NOKPROBE_SYMBOL, so the entry kprobe is not rejected by this path.

Uncertain

Verify: whether a plain register_kprobe() on x86-64 __switch_to succeeds in practice. Reason: Documentation/trace/kprobes.rst (v6.12) states “Kprobes doesn’t support return probes (or kprobes) on the x86_64 version of __switch_to(); the registration functions return -EINVAL”, but the only code enforcing it that I could find is kretprobe_blacklist[] in arch/x86/kernel/kprobes/core.c, which kernel/kprobes.c consults only inside register_kretprobe(). __switch_to in arch/x86/kernel/process_64.c at v6.12 carries __visible __notrace_funcgraph but no NOKPROBE_SYMBOL(). Either the documentation overstates the check, or the entry probe is rejected by some path I did not locate (e.g. it landing in a section that populate_kprobe_blacklist() covers). To resolve: on a v6.12 machine, echo 'p:sw __switch_to' > /sys/kernel/tracing/kprobe_events and see whether it takes; or grep a full v6.12 tree for other __switch_to blacklisting. uncertain

Probe Lifecycle

A registered kprobe is not simply on or off. It moves through a small state machine driven by four independent inputs — the caller’s enable/disable, the global kprobes/enabled switch, the optimiser workqueue, and module unload — and the tags in /sys/kernel/debug/kprobes/list are a direct readout of that state.

stateDiagram-v2
  [*] --> Unregistered
  Unregistered --> Prepared: register_kprobe()<br/>checks pass, insn copied,<br/>in kprobe_table[]
  Prepared --> Armed: arm_kprobe()<br/>int3 planted / ftrace ops registered
  Prepared --> Disabled: registered with<br/>KPROBE_FLAG_DISABLED
  Armed --> Optimizing: try_to_optimize_kprobe()<br/>queued on optimizing_list
  Optimizing --> Optimized: kprobe_optimizer workqueue:<br/>synchronize_rcu() then<br/>text_poke_bp() 5 bytes -> jmp
  Optimizing --> Armed: dequeued before the swap<br/>(disable / another probe lands<br/>inside the optimized region)
  Optimized --> Unoptimizing: disable, unregister,<br/>or a post_handler appears
  Unoptimizing --> Armed: jmp replaced by original<br/>bytes + int3 at byte 0
  Armed --> Disabled: disable_kprobe()
  Disabled --> Armed: enable_kprobe()
  Armed --> AllDisarmed: echo 0 &gt; kprobes/enabled
  AllDisarmed --> Armed: echo 1 &gt; kprobes/enabled
  Optimized --> Gone: probed module unloaded
  Armed --> Gone: probed module unloaded
  Gone --> [*]: unregister_kprobe()
  Disabled --> [*]: unregister_kprobe()
  Armed --> [*]: unregister_kprobe()

  note right of Optimized
    /sys/kernel/debug/kprobes/list tag [OPTIMIZED]
  end note
  note right of Disabled
    tag [DISABLED]; global switch does
    NOT clear a probe's own disabled state
  end note
  note right of Gone
    tag [GONE]; address no longer valid
  end note

The kprobe lifecycle as implemented across register_kprobe(), arm_kprobe(), __disable_kprobe(), the kprobe_optimizer workqueue and debugfs_kprobe_init() (v6.12 kernel/kprobes.c). What it shows: optimisation is an asynchronous, reversible promotion, not a registration-time decision, and the global disarm switch is a separate axis from per-probe disable. The insight to take: Optimizing → Armed is not an error path. A probe can bounce back out of the optimising queue because someone registered a second probe inside the five bytes it wanted to overwrite, or because a post_handler was added by an aggregate. That means the same script can produce [OPTIMIZED] on one run and nothing on the next, with a 10× difference in per-hit cost and a silent change in whether pre_handler may redirect regs->ip.

Overhead, With Real Numbers

The kernel documentation carries two overhead tables, and they are the best publicly available primary measurements of this mechanism. They are old, but they are measured, and the ratios between them are what matters.

The unoptimised figures date to 2005 hardware (kprobes.rst, v6.12): “On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0 microseconds to process. Specifically, a benchmark that hits the same probepoint repeatedly, firing a simple handler each time, reports 1-2 million hits per second… A return-probe hit typically takes 50-75% longer than a kprobe hit. When you have a return probe set on a function, adding a kprobe at the entry to that function adds essentially no overhead.”

Architecture / CPU	kprobe (k)	kretprobe (r)	k + r on same function
i386 — Pentium M, 1495 MHz	0.57 µs	0.92 µs	0.99 µs
x86_64 — Opteron 246, 1994 MHz	0.49 µs	0.80 µs	0.82 µs
ppc64 — POWER5 (gr), 1656 MHz	0.77 µs	1.26 µs	1.45 µs

The optimised table is measured on one machine, an Intel Xeon E5410 at 2.33 GHz, and is the more useful of the two because every column shares the hardware:

Path	i386	x86-64	Relative to plain kprobe (x86-64)
k — unoptimised kprobe (two traps)	0.80 µs	0.99 µs	1.00×
b — boosted kprobe (single-step skipped)	0.33 µs	0.43 µs	0.43×
o — jump-optimised kprobe	0.05 µs	0.06 µs	0.06×
r — unoptimised kretprobe	1.10 µs	1.24 µs	1.25×
rb — boosted kretprobe	0.61 µs	0.68 µs	0.69×
ro — optimised kretprobe	0.33 µs	0.30 µs	0.30×

Per-hit cost of each kprobe execution path, from the “Optimized Probe Overhead” table in Documentation/trace/kprobes.rst (v6.12), measured by Masami Hiramatsu on an Intel Xeon E5410 @ 2.33 GHz. The same numbers appear in the original v10 patch posting (LWN 375232, Feb 2010), which adds the summary “An optimized kprobe is about 5 times faster than a kprobe” — an understatement against its own table, which shows 16×. The insight to take: the ladder is not two rungs but three. Removing the second trap (boosting) buys 2.3×; removing the first trap as well (jump optimisation) buys another 7×. That is why the boosted path exists at all, and why disabling optimisation with sysctl debug.kprobes_optimization=0 costs you far less than the 16× headline suggests — you fall back to the boosted path, not the naive one, provided the kernel is CONFIG_PREEMPT=n.

Two caveats keep these numbers honest. First, the absolute values are from 2008-era silicon; only the ratios should be carried forward. Second — and this is the caveat that matters most in 2026 — none of these rows describes the path a modern function-entry kprobe takes, which is the ftrace backend (§The ftrace Path), and the kernel documentation has no measured table for it. The original patch posting also quantifies the memory cost of optimisation, which is the one number that has not aged: “it just uses ~200 bytes, so, even if you use ~10,000 probes, it just consumes a few MB” (LWN 375232).

Uncertain

Verify: the per-hit cost of an ftrace-backed ([FTRACE]) kprobe on current hardware, relative to the int3 and optimised paths. Reason: Documentation/trace/kprobes.rst at v6.12 gives measured tables only for the breakpoint, boosted and jump-optimised paths; the ftrace path was added in 2012 (LWN 499110) and no measured figure was published with it, nor added to the document since. Everything above about it being “much cheaper than a trap” is a mechanism argument (trampoline call plus pt_regs fill versus a synchronous exception), not a measurement. To resolve: benchmark on a v6.12 box with CONFIG_KPROBES_ON_FTRACE=y — e.g. attach at a symbol’s +0 (ftrace path) and at +N past the prologue (int3 path) on the same hot function and compare, or read a published microbenchmark. uncertain

Failure Modes and Gotchas

Almost everything that goes wrong with kprobes goes wrong quietly. There is no exception, no dmesg line, and no non-zero exit status for most of it — you simply get fewer events than you expected, or a handler that has no effect. The list below is ordered by how often each one wastes an afternoon.

The probe registers and never fires, because the function was inlined. This is the single most common kprobes disappointment, and the documentation warns about it in one sentence: “gcc may inline a function without being asked, so keep this in mind if you’re not seeing the probe hits you expect” (kprobes.rst, v6.12). The subtlety is that the symbol can still exist in /proc/kallsyms — the compiler may emit an out-of-line copy for the one caller it could not inline, or the symbol may be a static inline in a header with no out-of-line body at all. A concrete v6.12 case: blk_account_io_start, the block-layer function that half the world’s I/O-latency scripts attach to, is declared static inline void blk_account_io_start(struct request *req) in block/blk-mq.c at v6.12 — it is not an independent symbol and cannot be probed. It was a global function in block/blk-core.c as recently as v5.16. The correct diagnostic is not grep /proc/kallsyms (which can be misleading for a function whose only surviving instance is a .constprop clone) but grep '^symbol$' /sys/kernel/tracing/available_filter_functions, which lists exactly the addresses ftrace — and therefore the [FTRACE] kprobe backend — can actually attach to.

Registration fails with -EINVAL and the kernel will not say which check rejected you. As §The Blacklist showed, check_kprobe_address_safe() has six independent rejection branches that all return the same errno, and /sys/kernel/debug/kprobes/blacklist only enumerates the contents of one of them. An address rejected because it sits inside a jump-label patch site, a static call, a WARN_ON bug table entry, or a __cfi_/__pfx_ preamble will not appear in that file, and the failure looks arbitrary. When you go through the tracefs interface instead of the C API you get a much better error: /sys/kernel/tracing/error_log records the parse or registration failure with a caret under the offending token, which is the fastest triage available.

Your handler rewrites regs->ip and nothing happens. Only on optimised probes, only on some machines, and never with a warning. The kprobes documentation is explicit — “when the probe is optimized, that modification is ignored” — and whether a given probe gets optimised depends on the workqueue, on the kernel’s CONFIG_PREEMPTION setting, and on whether some other probe happens to sit inside the five bytes the optimiser wanted. Fault injection, live patching, and syscall-argument rewriting all run into this. The two documented defences are to attach a do-nothing post_handler (which disqualifies the probe from optimisation, per kprobes.rst) or to turn the optimiser off globally with sysctl -w debug.kprobes_optimization=0.

$arg1 is garbage because the probe is not at offset 0. Documentation/trace/kprobetrace.rst states the restriction twice over: $argN is “only for the probe on function entry (offs == 0)”, and even there “this argument access is best effort, because depending on the argument type, it may be passed on the stack. But this only support the arguments via registers” (kprobetrace.rst, v6.12). A probe at func+0x20 is past the prologue: the argument registers have very likely been clobbered, and $arg1 will return whatever is in %rdi now, silently. Structures passed by value, and anything beyond the sixth integer argument on x86-64, live on the stack and are not reachable through $argN at all. Each probe is also capped at 128 fetch arguments.

nmissed climbs and you never look at it. A kprobe hit that cannot be serviced is dropped, not queued, and the only record is the per-probe nmissed counter. The documented rule is blunt: “Kprobes makes no attempt to prevent probe handlers from stepping on each other — e.g., probing printk() and then calling printk() from a probe handler. If a probe handler hits a probe, that second probe’s handlers won’t be run in that instance, and the kprobe.nmissed member of the second probe will be incremented.” On the [FTRACE] backend the same counter is bumped by kprobes_inc_nmissed_count() when ftrace’s own recursion guard, ftrace_test_recursion_trylock(), refuses re-entry. The practical consequence is that a probe on a function that your own tracing infrastructure calls — the allocator, the ring buffer, printk — produces a biased sample, not a complete one, and the bias is invisible unless you read nmissed.

kretprobe BUG! in dmesg, or return events quietly missing. A kretprobe needs a pre-allocated instance per in-flight call; maxactive sets how many. “It’s not a disaster if you set maxactive too low; you’ll just miss some probes,” says the documentation — and the miss count lands in the kretprobe’s own nmissed. The louder failure is the mismatched-call-and-return case: “If the number of times a function is called does not match the number of times it returns, registering a return probe on that function may produce undesirable results,” and the kernel prints kretprobe BUG!: Processing kretprobe ... @ .... do_exit() is handled specially; the documentation admits the general case is not solved. This is covered in depth in kretprobes.

Return probes corrupt stack traces of the probed function. Because a kretprobe replaces the return address with a trampoline address, “stack backtraces and calls to __builtin_return_address() will typically yield the trampoline’s address instead of the real return address for kretprobed functions.” If you are simultaneously profiling and return-probing the same function, the profile is wrong in a way that looks like a symbolisation bug.

Your handler sleeps and the box dies. “Probe handlers are run with preemption disabled or interrupt disabled, which depends on the architecture and optimization state… In any case, your handler should not yield the CPU (e.g., by attempting to acquire a semaphore, or waiting I/O).” Note the “depends on the optimization state” clause: kretprobe handlers and optimised kprobe handlers run on x86 with interrupts enabled, plain breakpoint handlers with them disabled. Code that accidentally depends on interrupts being off will work until the probe gets optimised.

The probe silently becomes a no-op when its module unloads. /sys/kernel/debug/kprobes/list marks it [GONE]. The struct kprobe is still registered and still consuming a hash-table slot; it is simply pointed at text that no longer exists. Nothing fires and nothing warns.

The symbol you probed is not the symbol you probed last year. kprobes bind to internal names with no stability guarantee whatsoever, and the block layer’s I/O-accounting entry point is the canonical cautionary tale:

Kernel	Where blk_account_io_start lives	Probeable?
v5.4 – v5.16	global function in block/blk-core.c	yes — kprobe:blk_account_io_start works
v5.17	split: static inline blk_account_io_start() wrapper + static void __blk_account_io_start() in block/blk-mq.c	only the __-prefixed inner function, and only if not inlined
v6.12	static inline in block/blk-mq.c; no out-of-line symbol; body now calls the block_io_start tracepoint	no — use the tracepoint

Five years of drift in one symbol, verified by fetching block/blk-core.c and block/blk-mq.c at each tag from raw.githubusercontent.com. What it shows: a function that a widely-copied BCC tool (biolatency.py) attached to by name ceased to be a symbol at all, in three steps, without any deprecation notice — and the kernel replaced it with a stable tracepoint on the way. The insight to take: this is exactly why Brendan Gregg’s advice is “use static tracepoints (tracepoints/USDT) instead of dynamic tracing (kprobes/uprobes) wherever possible… Dynamic tracing is an unstable API, so your programs will break if the code it’s instrumenting changes from one release to another” (Gregg, Linux eBPF Tracing Tools). He says the same thing about his own tool in the same document: biolatency.py “was written before eBPF had tracepoint support… It should be rewritten to use tracepoints, as they are a stable API.”

Putting the diagnostics together, the triage order that actually resolves these fastest:

flowchart TD
  START["kprobe is not behaving"] --> Q1{"did register_kprobe /<br/>kprobe_events even succeed?"}
  Q1 -->|"no, -EINVAL"| E1["read /sys/kernel/tracing/error_log<br/>then check, in order:<br/>is it kernel text? blacklisted?<br/>jump-label / static-call / bug-table?<br/>__cfi_ / __pfx_ preamble?"]
  Q1 -->|"no, -EILSEQ"| E2["address is mid-instruction:<br/>can_probe() decoded the function<br/>and your offset is not a boundary"]
  Q1 -->|"no, -ENOENT"| E3["module __init text already freed,<br/>or symbol not in kallsyms"]
  Q1 -->|"yes"| Q2{"any hits at all?"}
  Q2 -->|"zero hits"| Z1{"is the symbol in<br/>available_filter_functions?"}
  Z1 -->|"no"| Z2["inlined or static inline:<br/>probe a caller, or use<br/>the tracepoint instead"]
  Z1 -->|"yes"| Z3["code path genuinely not taken;<br/>or probe is [GONE] (module unloaded);<br/>or kprobes/enabled == 0"]
  Q2 -->|"fewer than expected"| Y1["read nmissed in<br/>debug/kprobes/list accounting:<br/>re-entrancy or ftrace recursion guard<br/>is dropping hits"]
  Q2 -->|"hits, but wrong data"| W1{"probe at offset 0?"}
  W1 -->|"no"| W2["$argN is invalid past the prologue<br/>-- registers already clobbered"]
  W1 -->|"yes"| W3["arg passed on the stack, or<br/>&gt;6 integer args on x86-64:<br/>$argN cannot reach it"]
  Q2 -->|"handler runs but<br/>regs-&gt;ip edit ignored"| V1["check the tag in<br/>debug/kprobes/list.<br/>[OPTIMIZED] ignores ip changes.<br/>Add an empty post_handler, or<br/>sysctl debug.kprobes_optimization=0"]

Triage tree for a misbehaving kprobe, assembled from the rejection branches in check_kprobe_address_safe(), the nmissed accounting in kernel/kprobes.c, and the documented $argN and optimisation restrictions (v6.12). What it shows: the four distinct classes of failure — rejected, never hit, under-counted, and wrong data — each with a different first diagnostic. The insight to take: the branch people skip is the middle one. “Zero hits” and “fewer hits than expected” have completely different causes; the second is invisible unless you deliberately read nmissed, and a probe on anything the tracing infrastructure itself touches will always be in that state.

One last piece of advice from the kernel’s own documentation deserves quoting verbatim, because it is the correct attitude toward a mechanism that lets you write to pt_regs on a live kernel: “A probe handler can modify the environment of the probed function… So Kprobes can be used, for example, to install a bug fix or to inject faults for testing. Kprobes, of course, has no way to distinguish the deliberately injected faults from the accidental ones. Don’t drink and probe.”

Alternatives and When to Choose Them

kprobes is the oldest of the kernel’s dynamic-instrumentation mechanisms and by 2026 it is no longer the default choice for most work. Four newer mechanisms have taken over specific parts of its job, and the right question is not “kprobe or not” but “which of these five things am I actually doing”.

Tracepoints — the stable-API answer

A tracepoint is a hook the kernel authors declared with TRACE_EVENT, with a named, versioned argument struct exposed through a tracefs format file. Where one exists for what you want, it is strictly better than a kprobe: it survives kernel upgrades, its arguments are typed and documented, it is patched in and out with a static key so it costs a nop when disabled, and it cannot be inlined away. The block-layer example from the previous section is the argument in miniature: v6.12’s blk_account_io_start() is unprobeable, and its first statement is trace_block_io_start(req) — the maintainers replaced the accidental kprobe target with a deliberate one. The cost is coverage: there are on the order of two thousand tracepoints against roughly fifty thousand traceable functions, and if the thing you care about is not one of them, you cannot add one without patching the kernel.

fentry/fexit BPF trampolines — the fast path for eBPF

Introduced with the BPF trampoline in Linux v5.5 (verified by tag: kernel/bpf/trampoline.c returns HTTP 404 at v5.4 and HTTP 200 at v5.5), fentry/fexit are described by their author as “roughly equivalent to kprobe/kretprobe. Unlike k[ret]probe there is practically zero overhead to call a set of BPF programs before or after a kernel function” (Starovoitov, Introduce BPF trampoline, Nov 2019). The mechanism is genuinely different from an [FTRACE] kprobe, and the difference is where the speed comes from. register_fentry() in kernel/bpf/trampoline.c (v6.12) resolves ftrace_location(ip) and then calls register_ftrace_direct() — an ftrace direct call, which jumps straight to a JIT-generated trampoline. A kprobe on the same site registers a FTRACE_OPS_FL_SAVE_REGS ops, which makes ftrace materialise a full struct pt_regs on every hit. The fentry trampoline instead saves only the argument registers the BPF program declared it needs, in the target function’s own calling convention, so a fentry/vfs_read program reads args->file as a typed struct file * rather than decoding %rdi out of a pt_regs.

The price is a hard dependency on BTF (BPF Type Format): the attach path goes through bpf_check_attach_target() in kernel/bpf/verifier.c, which takes a btf_id, so the kernel must have been built with CONFIG_DEBUG_INFO_BTF=y. It is also entry/exit-only — there is no fentry equivalent of kprobe:func+0x40 — and a single trampoline holds at most BPF_MAX_TRAMP_LINKS programs, defined in include/linux/bpf.h as 38 on most architectures and 27 on s390x, with the comment “Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 bytes on x86.” See BTF (BPF Type Format) and kprobe and uprobe BPF Programs.

fprobe and kprobe_multi — the mass-attachment answer

Documentation/trace/fprobe.rst states the purpose in one line: fprobe “is a wrapper of ftrace (+ kretprobe-like return callback) to attach callbacks to multiple function entry and exit… Compared with kprobes and kretprobes, fprobe gives faster instrumentation for multiple functions with single handler” (fprobe.rst, v6.12). Both fprobe and its BPF-facing sibling kprobe_multi landed in v5.18 (verified by tag: kernel/trace/fprobe.c is 404 at v5.17, 200 at v5.18; bpf_kprobe_multi_link_attach appears in kernel/trace/bpf_trace.c on the same boundary).

The problem they solve is not per-hit cost but attach cost. Every plain arch_arm_kprobe() ends in a text_poke_sync() IPI to every CPU (§Registration); arming ten thousand probes one at a time means ten thousand cross-CPU synchronisations. ftrace_set_filter_ips() batches the whole set into one update. The measured effect, from the patch series cover letter:

# perf stat --null -r 5 ./src/bpftrace -e 'kprobe:x* { } i:ms:1 { exit(); } '
Attaching 2 probes...
Attaching 3342 functions
...
1.4960 +- 0.0285 seconds time elapsed  ( +- 1.91% )

— 3,342 kernel functions attached in 1.5 seconds (Olsa, bpf: Add kprobe multi link, Feb 2022); the earlier RFC describes the goal as making attachment “comparable to ftrace tracer attachment speed” (Olsa, RFC, Jan 2022). The ceiling is generous: MAX_KPROBE_MULTI_CNT in kernel/trace/bpf_trace.c (v6.12) is 1U << 20, i.e. 1,048,576 addresses in one attach. The restriction, stated in the same cover letter, is inherited from ftrace: “this limits the probe point to the function entry or return.” v6.10 added a further refinement — kprobe sessions (is_kprobe_session() in bpf_trace.c, absent at v6.9, present at v6.10), which give one program both hooks with a shared cookie and let the entry hook tell the kernel to skip the exit hook entirely (BPF tracing performance, LWN, June 2024). This whole family is covered in fprobe and the Modern Function-Probe Path.

Changed after the v6.12 pin

fprobe was reimplemented on top of the function-graph tracer’s shadow stack, replacing the rethook mechanism, in v6.14 — verified by fetching kernel/trace/fprobe.c at three tags: rethook appears 30 times and fgraph zero times at both v6.12 and v6.13, and the counts invert to zero and 18 at v6.14. Jiri Olsa laid out the plan and its rationale (consolidating “two parallel implementations”, at the cost of “one extra page per process”) at LSFMM+BPF 2024 (LWN 978335). Nothing in this note’s v6.12 description of fprobe’s interface changes; the return-hook implementation underneath it does.

ftrace directly, and eprobe

If all you need is “which functions ran, and in what order”, the function tracer and function-graph tracer do it with no probe registration at all and with the lowest per-function cost available, because they are the mechanism the ftrace-backed kprobe is borrowing. And if a tracepoint gives you a pointer but not the field inside it, an eprobe attaches a fetch expression to an existing trace event rather than planting a new probe — a stable-API way to get one more field.

The comparison, and the decision

	kprobe (int3/ftrace/optimised)	tracepoint	fentry/fexit	fprobe / kprobe_multi	uprobe
Where it can attach	any kernel instruction boundary	only where a TRACE_EVENT exists	function entry + exit	function entry + exit	any userspace instruction
API stability	none — internal symbol names	stable, versioned format file	none (symbol + BTF type)	none (symbol names)	none (binary offsets)
Argument access	pt_regs decoding; $argN only at offset 0	typed struct fields	typed, via BTF	pt_regs-style	pt_regs decoding
Needs BTF	no	no	yes (CONFIG_DEBUG_INFO_BTF)	no	no
Attach cost for N sites	N × text_poke_sync() IPI	N static-key flips	N ftrace-direct registrations	one batched ftrace update (3,342 fns ≈ 1.5 s)	N inode/offset patches
Per-hit cost	0.06–1.0 µs depending on backend (§Overhead)	~nop when off	“practically zero” (author’s claim, unmeasured here)	ftrace trampoline	trap into kernel from userspace
Available since	2.6.9 (2004)	2.6.24-era TRACE_EVENT	v5.5	v5.18	3.5
Survives inlining	no	yes	no	no	no

Honest comparison of the five kernel-instrumentation mechanisms a tracing script can target, with each version claim pinned by source-tag existence checks rather than memory. The insight to take: only one row genuinely separates kprobes from everything newer, and it is the first one. fentry, fprobe, and kprobe_multi are all ftrace-based and therefore all entry-and-exit only; the moment you need func+0x40 — a specific branch inside a function, a particular call site, a loop body — the breakpoint kprobe is the only mechanism in the kernel that can do it. Everything else about kprobes has been superseded.

flowchart TD
  Q0["I need to observe something<br/>in the kernel"] --> Q1{"is there a tracepoint<br/>for it?"}
  Q1 -->|yes| T["use the tracepoint.<br/>Stable API, survives inlining,<br/>nop when disabled."]
  Q1 -->|"a tracepoint exists but<br/>lacks the field I need"| EP["eprobe: attach a fetch<br/>expression to that event"]
  Q1 -->|no| Q2{"do I need a point<br/>inside a function,<br/>not its entry or exit?"}
  Q2 -->|yes| KP["breakpoint kprobe<br/>kprobe:func+0xNN<br/>-- the only mechanism<br/>that can do this"]
  Q2 -->|no| Q3{"attaching to many<br/>functions at once?"}
  Q3 -->|"yes, hundreds+"| KM["kprobe_multi / fprobe<br/>one batched ftrace update;<br/>3342 fns in ~1.5 s"]
  Q3 -->|"no, a handful"| Q4{"kernel built with<br/>CONFIG_DEBUG_INFO_BTF?<br/>and do I want typed args?"}
  Q4 -->|yes| FE["fentry / fexit<br/>ftrace-direct call,<br/>no pt_regs fill,<br/>typed arguments"]
  Q4 -->|no| KF["plain kprobe at offset 0<br/>-- lands on the [FTRACE] backend"]
  Q0 --> Q5{"is it in userspace?"}
  Q5 -->|yes| U["uprobe / USDT"]

Choosing a kernel instrumentation mechanism in the v6.12 era. What it shows: the decision is driven by three questions in order — does a stable hook already exist, do I need a mid-function address, and how many sites am I attaching to. The insight to take: the tree only reaches “breakpoint kprobe” through one branch, and that branch is exactly the capability nothing else replicates. If your answer to “entry or exit?” is “entry”, you are choosing between four ftrace-based mechanisms and the classic int3 machinery is not one of them.

Production Notes

Twenty years of quietly changing underneath the same API

kprobes’ public interface — register_kprobe(), a pre_handler, a post_handler — has barely moved since 2004, while the machinery under it has been replaced roughly every five years. That is the reason so much writing about kprobes is confidently wrong: the API it describes is still correct, and the mechanism it describes has been superseded twice.

timeline
  title Mechanism changes under a stable kprobes API
  2004 (2.6.9) : kprobes merged -- int3 breakpoint plus out-of-line single-step of a copied instruction
  2005 : djprobe posted, the jump-probe prototype that became optprobes (LWN 157751)
  2010 (v2.6.34) : CONFIG_OPTPROBES arrives -- a 5-byte jmp detour, 0.99us down to 0.06us per hit on x86-64
  2013 (v3.9) : arch/x86/kernel/kprobes/ftrace.c -- probes at function entry are armed through ftrace, not int3
  2019 (v5.5) : BPF trampoline -- fentry/fexit via register_ftrace_direct(), typed args, no pt_regs fill
  2022 (v5.18) : fprobe and kprobe_multi -- one batched ftrace update instead of N text_poke_sync IPIs
  2024 (v6.10) : kprobe sessions -- one BPF program for entry and exit with a shared cookie
  2025 (v6.14) : after this note's pin, fprobe reimplemented on fgraph's shadow stack and rethook removed

Evolution of the machinery behind register_kprobe(). What it shows: five distinct arming/execution mechanisms introduced over two decades, each dated by fetching a file or symbol at consecutive release tags rather than from memory (opt.c’s CONFIG_OPTPROBES: absent v2.6.33, present v2.6.34; kprobes/ftrace.c: 404 at v3.8, 200 at v3.9; bpf/trampoline.c: 404 at v5.4, 200 at v5.5; trace/fprobe.c and bpf_kprobe_multi_link_attach: absent v5.17, present v5.18; is_kprobe_session: absent v6.9, present v6.10). The insight to take: any document that describes kprobes as “planting an int3 and single-stepping” is describing the 2004 mechanism. On a v6.12 distribution kernel that path is taken only for probes at a non-zero offset, and even then the single-step is emulated or boosted away (§The Hit Path).

What to check before trusting a measurement

The mechanism a probe lands on is a build-configuration property of the machine you are on, and it changes the per-hit cost by more than an order of magnitude. Before comparing two numbers from two boxes, check that these agree:

Config / knob	Where to read it	Why it changes your results
CONFIG_KPROBES	zgrep CONFIG_KPROBES= /proc/config.gz or /boot/config-$(uname -r)	without it, nothing works at all
CONFIG_KPROBES_ON_FTRACE	same	off ⇒ probing a function entry returns -EINVAL outright (the #else in check_ftrace_location())
CONFIG_OPTPROBES	same	off ⇒ no [OPTIMIZED] probes; ~16× more per-hit cost on the paths that would have qualified
CONFIG_PREEMPTION	same	on ⇒ no boosting and no optimisation; you pay the full two-trap cost. Most desktop and container-host kernels ship CONFIG_PREEMPT=y
CONFIG_KALLSYMS / _ALL	same	kallsyms_lookup_name() backs symbol-name resolution; without it only raw addresses work
CONFIG_DEBUG_INFO_BTF	same	required for fentry/fexit and for CO-RE-portable BPF; irrelevant to plain kprobes
debug.kprobes_optimization	sysctl debug.kprobes_optimization	echoing 0 unoptimises existing probes, not just future ones
per-probe backend actually chosen	cat /sys/kernel/debug/kprobes/list	the [FTRACE] / [OPTIMIZED] / [DISABLED] / [GONE] tags are the ground truth

The eight things that decide which of the mechanisms in this note your probe actually got, per Documentation/trace/kprobes.rst Appendices A and B and the #ifdef guards in kernel/kprobes.c and arch/x86/kernel/kprobes/core.c (v6.12). The insight to take: the last row supersedes the other seven. Do not reason from the config about which backend you got — read the tag. A benchmark that does not report the tag is not reproducible.

How anyone actually uses kprobes in 2026

Almost nobody calls register_kprobe() from a kernel module any more. There are four practical front doors, in ascending order of how often they are used:

1. A kernel module calling register_kprobe(). Still the only route if you need a post_handler, need to rewrite regs->ip, or need to run arbitrary C. Note also that this is the standard workaround for a different problem: EXPORT_SYMBOL_GPL(kallsyms_lookup_name) was removed from kernel/kallsyms.c between v5.6 and v5.7 (verified: the export appears once at v5.6 and zero times at v5.7 and v6.12), so out-of-tree modules that need an unexported symbol’s address now commonly register a throwaway kprobe on kallsyms_lookup_name and read the resolved p->addr. That the kernel’s own probe mechanism is the accepted way around the kernel’s own export restriction is worth knowing before you are surprised by it in someone else’s driver.
2. tracefs kprobe_events. echo 'p:myprobe vfs_read file=$arg1' > /sys/kernel/tracing/kprobe_events, then enable it under events/kprobes/myprobe/enable. Documentation/trace/kprobetrace.rst notes that /sys/kernel/tracing/dynamic_events is the newer unified interface covering kprobes, uprobes, fprobes and eprobes together. Errors land in error_log, which is far more informative than the C API’s undifferentiated -EINVAL. See kprobe and uprobe Event Interface.
3. perf probe. The reason to reach for it over raw kprobe_events is local variables: “If you build your kernel with debug info (CONFIG_DEBUG_INFO=y), you can find which register/stack is assigned to which local variable or arguments by using perf-probe” (kprobes.rst, v6.12). It reads DWARF and emits the right fetch expression, which is not something you want to do by hand.
4. eBPF, via bpftrace or BCC. This is how the overwhelming majority of kprobe hits in production are generated. bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }' aggregates in-kernel through a BPF map and never ships a per-event record to userspace (Gregg, A thorough introduction to bpftrace). The consumer side is kprobe and uprobe BPF Programs; the aggregation side is In-Kernel Aggregation with BPF Maps.

One historical interface is gone: jprobes — a kprobe variant that let a handler mirror the probed function’s prototype — is documented as “now a deprecated feature. People who are depending on it should migrate to other tracing features or use older kernels,” with trace-events and perf probe named as the replacements. It was removed from the kernel entirely; if you find a tutorial using register_jprobe(), the tutorial predates 4.15.

Operating rules that hold up

Prefer a tracepoint; reach for a kprobe when there isn’t one. This is Brendan Gregg’s standing advice — “Use static tracepoints (tracepoints/USDT) instead of dynamic tracing (kprobes/uprobes) wherever possible. It’s often not possible, but do try” (Gregg, Linux eBPF Tracing Tools) — and the blk_account_io_start history in §Failure Modes is what happens when you do not follow it. Concretely: a kprobe-based script is a tool you will re-port at every kernel upgrade, and you should budget for that rather than be surprised by it.

Read nmissed before you believe a count. A kprobe on a hot, re-entrant, or allocator-adjacent function silently drops hits. /sys/kernel/debug/kprobes/list does not print nmissed; the C API exposes it on your own struct kprobe, and the tracefs kprobe events surface a lost-event count through the trace buffer’s own overrun/lost accounting. Either way, an event count from a probe you have not checked for misses is a lower bound, not a measurement.

Attach in bulk with kprobe_multi, not in a loop. The text_poke_sync() IPI per probe is the cost that dominates wide attaches. bpftrace uses the kprobe_multi link automatically for wildcard probes on kernels that have it (v5.18+); on older kernels the same one-liner will take tens of seconds where the modern path takes 1.5 (Olsa, Feb 2022).

Budget the per-hit cost against the call rate, not against a feeling. Using the v6.12 documentation’s measured optimised-probe table as ratios: an optimised probe at ~0.06 µs on a function called a million times a second costs about 6% of one CPU; the same probe on the unoptimised two-trap path at ~0.99 µs costs roughly 99% of one CPU — the difference between “unnoticeable” and “you just took a core offline”. Since whether you land on the fast path depends on CONFIG_PREEMPTION, on the optimiser workqueue, and on whether another probe is sitting in your five bytes, the safe assumption for a hot function is the slow number. This is also the strongest argument for moving hot probes to fentry or a tracepoint rather than tuning kprobes.

Do not probe what your probe depends on. The kernel will happily let you register a kprobe on printk, the slab allocator, or the scheduler, and the result is nmissed inflation at best. The blacklist covers only the code that would crash; the code that merely gives you a biased sample is your problem.

Resolved during this pass

bpftrace does select kprobe_multi automatically, and the selection is a runtime capability probe rather than a version check. BpfProgram::set_expected_attach_type() in src/bpfprogram.cpp at bpftrace v0.26.1 sets attach_type = BPF_TRACE_KPROBE_MULTI whenever the probe is a kprobe/kretprobe, its wildcard has expanded to a non-empty funcs list, and no module was specified — with the in-source comment “We want to avoid kprobe_multi when a module is specified because the BPF_TRACE_KPROBE_MULTI link type does not currently support the module:function syntax.” A session-expanded probe on a kernel where has_kprobe_session() is true gets BPF_TRACE_KPROBE_SESSION instead. BPFfeature::has_kprobe_multi() in src/bpffeature.cpp decides support by simply trying to create such a link once and caching the result. bpftrace --no-feature kprobe_multi forces the one-probe-at-a-time path. Read from the v0.26.1 release tarball (codeload). Note the consequence for §Failure Modes: kprobe:mymod:myfunc and kprobe:myfunc take different kernel attach paths, so a module-qualified wildcard is still the slow one-IPI-per-probe route.

See Also

• fentry fexit and BPF Trampolines — the BPF trampoline mechanism this note’s fentry section describes; the faster modern path for the same attach points

The other half of this mechanism

• kretprobes — the return-probe sibling: how maxactive, the return trampoline, and rethook work, and why a return probe costs 25–50% more per hit than an entry probe
• Optimized kprobes and the Breakpoint to Jump Path — the CONFIG_OPTPROBES machinery this note hands off to: can_optimize(), the detour buffer, the kprobe_optimizer workqueue, and the 16× measured speed-up
• uprobes — the same idea applied to userspace text: an inode+offset instead of a kernel symbol, with its own single-step and trampoline design

What replaced it, and what it sits on

• fprobe and the Modern Function-Probe Path — fprobe and kprobe_multi (v5.18), the batched-attach answer to the text_poke_sync() IPI problem
• Dynamic ftrace and the mcount fentry Hook — the __fentry__ call sites that the [FTRACE] kprobe backend registers against; read this to understand why a probe at offset 0 is not an int3 at all
• The Function Tracer and The Function Graph Tracer — when “which functions ran” is the whole question and no probe is needed
• Static Keys and Tracepoint Patching — the other runtime text-patching subsystem in the kernel, and one of the six things check_kprobe_address_safe() refuses to probe on top of
• Tracepoints and The Trace Event Subsystem — the stable-API alternative; prefer these where one exists
• eprobe Event-Based Probes — attaching a fetch expression to an existing trace event instead of planting a new probe

How you actually drive it

• kprobe and uprobe Event Interface — the tracefs kprobe_events / dynamic_events syntax, $argN, fetch types, and error_log
• kprobe and uprobe BPF Programs — BPF_PROG_TYPE_KPROBE, the consumer side of every probe in this note
• BTF (BPF Type Format) — what fentry/fexit needs and plain kprobes do not
• bpftrace and BCC BPF Compiler Collection — the front-ends that generate essentially all production kprobe hits
• bpftrace vs BCC vs ftrace — choosing between them
• In-Kernel Aggregation with BPF Maps — why a kprobe handler should summarise rather than emit
• perf Profiling Tool — perf probe, the DWARF-aware way to probe a local variable mid-function
• The tracefs Filesystem — where every file path in this note lives

Context

• Static vs Dynamic Tracing — the framing distinction this note is the canonical example of
• Observability Overhead and Safety — the general version of §Overhead and the nmissed problem
• Linux Tracing and Observability MOC — parent map (§4, Dynamic Instrumentation)
• Linux eBPF MOC — sibling map, owning the verifier/JIT/map machinery that the BPF consumers of kprobes compile to