kprobes

A kprobe (kernel probe) is Linux’s dynamic instruction-level instrumentation primitive: it lets you “dynamically break into any kernel routine and collect debugging and performance information non-disruptively” by trapping at “almost any kernel code address” (kprobes.rst, v6.12). The mechanism is a software breakpoint: when a kprobe is registered, the kernel saves the original instruction byte(s) at the target address and overwrites the first byte with a breakpoint instruction — int3 (0xCC) on x86 (core.c, v6.12). When any CPU executes that byte, it traps, the CPU registers are saved into a struct pt_regs, your pre_handler runs, the kernel then single-steps an out-of-line copy of the displaced original instruction, your post_handler runs, and execution resumes at the instruction after the probe. Unlike a tracepoint — a hook the kernel authors placed in advance — a kprobe can attach to a function or even an instruction the authors never anticipated you would care about. That power is the whole point and also the whole risk: kprobes are tied to internal symbol names that can change between kernels, and a small blacklist of functions (the kprobe machinery itself, some entry code) cannot be probed at all. kprobes are the engine behind tracefs kprobe-events and the eBPF BPF_PROG_TYPE_KPROBE program type; the BPF-consumer view is covered in kprobe and uprobe BPF Programs.

Mental Model

Think of a kprobe as the kernel planting a conditional breakpoint on itself and servicing it in software, where your handler is the debugger script that runs at the breakpoint. A traditional debugger would stop the process and hand control to a human; the kernel instead runs your registered handler in-line, in a few hundred nanoseconds, and resumes — millions of times a second if the probed code is hot. The defining difficulty is that the kernel is live: other CPUs may be executing the very function you are patching, so the kprobe machinery is built around two careful tricks. First, it overwrites only the first byte with the breakpoint, so the trap fires cleanly. Second, it never single-steps the in-place instruction (whose first byte is now a breakpoint); it single-steps a private copy stored elsewhere, so a second CPU racing through the same address still trips the breakpoint and is handled correctly.

flowchart TB
  subgraph REG["register_kprobe()"]
    R1["validate addr<br/>(text? blacklisted? ftrace site?)"]
    R2["save original byte -> p->opcode<br/>copy original insn -> p->ainsn.insn"]
    R3["arch_arm_kprobe:<br/>text_poke first byte = int3 (0xCC)"]
    R1 --> R2 --> R3
  end
  subgraph HIT["CPU hits the int3"]
    H1["trap -> save registers into pt_regs"]
    H2["get_kprobe(addr) finds the kprobe"]
    H3["run pre_handler(p, regs)"]
    H4["single-step the out-of-line copy<br/>(p->ainsn.insn)"]
    H5["run post_handler(p, regs, flags)"]
    H6["resume at next instruction"]
    H1 --> H2 --> H3 --> H4 --> H5 --> H6
  end
  R3 -.->|"breakpoint now live"| H1
  H3 -->|"returns !=0:<br/>handler changed regs->ip"| H6

The kprobe lifecycle. What it shows: registration patches the target’s first byte to int3 and stashes both the original byte and a relocatable copy of the whole instruction; a hit traps, captures registers, runs your pre_handler, single-steps the copy (not the in-place breakpoint), runs your post_handler, and resumes. The insight to take: the breakpoint lives at the real address (so every CPU traps), but the original instruction is executed from a separate copy (so single-stepping is safe under SMP). A pre_handler that returns non-zero — meaning it rewrote regs->ip — skips the single-step entirely and jumps wherever it pointed.

struct kprobe and the Handler Trio

You drive a kprobe through a struct kprobe, declared in include/linux/kprobes.h. The v6.12 definition (kprobes.h, v6.12):

struct kprobe {
	struct hlist_node hlist;
	struct list_head list;		/* list of kprobes for multi-handler support */
	unsigned long nmissed;		/* number of times temporarily disarmed */
	kprobe_opcode_t *addr;		/* location of the probe point */
	const char *symbol_name;	/* user may indicate symbol name */
	unsigned int offset;		/* offset into the symbol */
	kprobe_pre_handler_t pre_handler;   /* called before addr is executed */
	kprobe_post_handler_t post_handler; /* called after addr is executed */
	kprobe_opcode_t opcode;		/* saved opcode replaced by breakpoint */
	struct arch_specific_insn ainsn;    /* copy of the original instruction */
	u32 flags;
};

The fields you fill in to place a probe are addr (a raw kernel address) or symbol_name + offset (let the kernel resolve the address from kallsyms); specifying both fails with -EINVAL. The fields the kernel fills in are opcode (the original first byte it displaced) and ainsn (the architecture-specific copy of the whole original instruction, used for out-of-line single-stepping). nmissed counts probe hits that could not be serviced (e.g. a re-entrant hit). The three handler callbacks are the user-visible contract:

• pre_handler(struct kprobe *p, struct pt_regs *regs) — runs at the breakpoint, before the probed instruction executes. This is the natural place to read function arguments (which, at function entry, sit in the architecture’s argument registers inside pt_regs). Its return value is meaningful: “If you change the instruction pointer … in pre_handler, you must return !0 so that kprobes stops single stepping and just returns to the given address” (kprobes.rst, v6.12). A return of 0 means “carry on with normal single-step processing.”
• post_handler(struct kprobe *p, struct pt_regs *regs, unsigned long flags) — runs after the probed instruction has been single-stepped, unless the pre_handler diverted execution. Useful for inspecting state the instruction produced.
• fault_handler — not a field of struct kprobe in v6.12 (it was removed from the generic struct); fault handling during single-stepping is done by the architecture’s kprobe_fault_handler(). If the single-stepped instruction itself faults (e.g. a page fault), kprobe_fault_handler() resets the current kprobe, points regs->ip back at the probe address, and lets the normal page-fault handler proceed (core.c, v6.12).

The handler typedefs make the signatures concrete (kprobes.h, v6.12):

typedef int (*kprobe_pre_handler_t) (struct kprobe *, struct pt_regs *);
typedef void (*kprobe_post_handler_t) (struct kprobe *, struct pt_regs *,
				       unsigned long flags);

Registration: register_kprobe()

A kprobe is installed with register_kprobe(struct kprobe *kp) and removed with unregister_kprobe(). Registration is where the breakpoint is actually planted, and the v6.12 flow in kernel/kprobes.c shows the careful sequence (kprobes.c, v6.12):

int register_kprobe(struct kprobe *p)
{
	...
	/* Adjust probe address from symbol */
	addr = _kprobe_addr(p->addr, p->symbol_name, p->offset, &on_func_entry);
	...
	ret = check_kprobe_address_safe(p, &probed_mod);   /* text? blacklist? */
	if (ret)
		return ret;
 
	mutex_lock(&kprobe_mutex);
	...
	cpus_read_lock();
	mutex_lock(&text_mutex);                           /* prevent text modification */
	ret = prepare_kprobe(p);                           /* copy original insn */
	mutex_unlock(&text_mutex);
	cpus_read_unlock();
	...
	hlist_add_head_rcu(&p->hlist,
		&kprobe_table[hash_ptr(p->addr, KPROBE_HASH_BITS)]);
 
	if (!kprobes_all_disarmed && !kprobe_disabled(p)) {
		ret = arm_kprobe(p);                       /* plant the int3 */
		...
	}
	try_to_optimize_kprobe(p);                         /* maybe promote to jmp */
out:
	mutex_unlock(&kprobe_mutex);
	...
}

The steps in order: _kprobe_addr() resolves symbol_name+offset to an address if needed; check_kprobe_address_safe() rejects non-text addresses and blacklisted regions (covered below); prepare_kprobe() makes the relocatable copy of the original instruction into p->ainsn; the kprobe is inserted into a global hash table kprobe_table[] keyed by address (this is how the trap handler later finds the kprobe via get_kprobe(addr)); and finally arm_kprobe() plants the breakpoint. The try_to_optimize_kprobe() call at the end is the optional promotion to a jump-based optimized kprobe — deferred to Optimized kprobes and the Breakpoint to Jump Path.

The architecture-specific arming is where the int3 is written (core.c, v6.12):

void arch_arm_kprobe(struct kprobe *p)
{
	u8 int3 = INT3_INSN_OPCODE;
 
	text_poke(p->addr, &int3, 1);
	text_poke_sync();
	perf_event_text_poke(p->addr, &p->opcode, 1, &int3, 1);
}
 
void arch_disarm_kprobe(struct kprobe *p)
{
	u8 int3 = INT3_INSN_OPCODE;
 
	perf_event_text_poke(p->addr, &int3, 1, &p->opcode, 1);
	text_poke(p->addr, &p->opcode, 1);
	text_poke_sync();
}

text_poke(p->addr, &int3, 1) overwrites exactly one byte — the first byte of the target instruction — with 0xCC. text_poke_sync() then issues an inter-processor interrupt (via on_each_cpu(do_sync_core, ...)) so every other CPU executes a serializing instruction and discards any stale prefetched bytes — without this, a CPU that had already prefetched the old first byte could miss the new breakpoint. Disarming is the mirror: write the saved p->opcode back over the breakpoint and sync. (The text_poke byte-patching machinery itself, including why a single-byte poke is safe, is the same code-patching infrastructure described in Static Keys and Code Patching.)

The int3 Trap Handler: What Happens on a Hit

When a CPU executes the planted 0xCC, it raises the breakpoint exception, and the kernel’s exception path dispatches to kprobe_int3_handler(). The v6.12 core (core.c, v6.12):

int kprobe_int3_handler(struct pt_regs *regs)
{
	kprobe_opcode_t *addr;
	struct kprobe *p;
	struct kprobe_ctlblk *kcb;
 
	if (user_mode(regs))
		return 0;
 
	addr = (kprobe_opcode_t *)(regs->ip - sizeof(kprobe_opcode_t));
	kcb = get_kprobe_ctlblk();
	p = get_kprobe(addr);
 
	if (p) {
		if (kprobe_running()) {
			if (reenter_kprobe(p, regs, kcb))
				return 1;
		} else {
			set_current_kprobe(p, regs, kcb);
			kcb->kprobe_status = KPROBE_HIT_ACTIVE;
			/*
			 * If we have no pre-handler or it returned 0, we
			 * continue with normal processing. ...
			 */
			if (!p->pre_handler || !p->pre_handler(p, regs))
				setup_singlestep(p, regs, kcb, 0);
			else
				reset_current_kprobe();
			return 1;
		}
	}
	...
	return 0;
}

Walking it: addr = regs->ip - 1 recovers the probe address (the trap left regs->ip pointing past the one-byte breakpoint). get_kprobe(addr) looks up the struct kprobe in the global hash table — this is the inverse of the hlist_add_head_rcu() at registration. If found, and no kprobe is already running on this CPU, the handler records the current kprobe and calls p->pre_handler(p, regs). The conditional if (!p->pre_handler || !p->pre_handler(p, regs)) is the return-value contract in action: if there is no pre-handler, or it returns 0, the kernel proceeds to setup_singlestep(); if the pre-handler returns non-zero (it rewrote regs->ip to redirect execution), the single-step is skipped and the current kprobe is reset. setup_singlestep() arranges to execute the out-of-line copy p->ainsn.insn — the relocated original instruction — and a follow-up trap (a second int3 placed right after the copy, or the hardware single-step) brings control back to resume_singlestep() and kprobe_post_process(), which runs the post_handler and points regs->ip at the instruction following the original probe site. The int3 exception disables interrupts and the handler keeps preemption off for the whole probe, because, as the documentation notes, “int3 and debug trap disables irqs and we clear IF while singlestepping, it must be no preemptible.”

This per-hit trap is not free. The kprobe documentation’s benchmark figures put an unoptimized kprobe hit at roughly 0.5–1.0 microseconds on 2005-era hardware, and a return probe at 50–75% more (kprobes.rst, v6.12). The synchronous breakpoint exception flushes the pipeline and takes the slow exception-entry path; that overhead is the motivation for optimized kprobes, which replace the trap with a jump and bring the per-hit cost down to ~0.05–0.1 microseconds — an order of magnitude — where the surrounding instructions allow it (kprobes.rst, v6.12). The breakpoint-to-jump promotion is its own topic; see Optimized kprobes and the Breakpoint to Jump Path.

The Blacklist: What Cannot Be Probed

A kprobe traps into kernel code, so probing the wrong function can cause infinite recursion or a double fault. The kernel therefore maintains a blacklist and validates every registration against it. The check is in check_kprobe_address_safe() (kprobes.c, v6.12):

	if (in_gate_area_no_mm((unsigned long) p->addr) ||
	    within_kprobe_blacklist((unsigned long) p->addr) ||
	    jump_label_text_reserved(p->addr, p->addr) ||
	    static_call_text_reserved(p->addr, p->addr) ||
	    find_bug((unsigned long)p->addr) ||
	    is_cfi_preamble_symbol((unsigned long)p->addr)) {
		ret = -EINVAL;
		goto out;
	}

The blacklist has two parts. First, a default architectural blacklist: arch_within_kprobe_blacklist() rejects anything in the __kprobes text section, and arch_populate_kprobe_blacklist() adds the entry text (__entry_text_start … __entry_text_end) — the low-level interrupt/syscall entry stubs (core.c, v6.12). The documentation states the rationale plainly: “Kprobes can probe most of the kernel except itself … Probing (trapping) such functions can cause a recursive trap (e.g. double fault) or the nested probe handler may never be called” (kprobes.rst, v6.12). The kprobe core itself (kernel/kprobes.c, much of arch/x86/kernel/kprobes/core.c), the page-fault handler, and notifier_call_chain are excluded — registering there returns -EINVAL. Second, an extensible per-function blacklist: a kernel developer marks a function unprobeable with the NOKPROBE_SYMBOL(name) macro, which adds it to the _kprobe_blacklist section that within_kprobe_blacklist() consults. The check above also refuses addresses that overlap jump-label or static-call patch sites (jump_label_text_reserved, static_call_text_reserved) — because those bytes are themselves runtime-patched and a kprobe there would collide — and CFI preambles.

Beyond the explicit blacklist, two practical limits apply. Probing an inlined function fails silently in the sense that “Kprobes makes no attempt to chase down all inline instances” — if gcc inlined the function, there is no out-of-line copy to probe (kprobes.rst, v6.12). And a few functions are excluded for stack reasons: x86-64’s __switch_to() cannot be return-probed because the CPU runs on a non-current-task stack across it.

Return Probes and Re-Entrancy (Briefly)

A plain kprobe fires before an instruction; a kretprobe (return probe) fires when a function returns. Because a function has one entry but many possible return paths, a kretprobe works by planting an entry kprobe that saves the real return address and overwrites it on the stack with a trampoline address; a second kprobe on the trampoline fires on the way out and restores the real return address. The pool of in-flight return instances is sized by maxactive, and exhaustion silently increments nmissed. This is its own atom — see kretprobes. Re-entrancy is handled defensively in the trap handler shown above: if kprobe_running() is already true on this CPU (a probe hit inside your own handler), reenter_kprobe() is taken and the nested hit is counted, not deeply recursed — the documentation notes “if a probe handler hits a probe, that second probe’s handlers won’t be run in that instance, and the kprobe.nmissed member of the second probe will be incremented.”

Failure Modes and Diagnostics

-EINVAL on register. The address is non-text, blacklisted, an inlined/notrace function with no out-of-line site, mid-instruction on a CISC arch, or overlaps a patch site. The fix is to probe a different, real symbol — grep the target in /proc/kallsyms to confirm it exists and is not inlined; for a function with no probeable site, fall back to a tracepoint or a non-inlined caller.

Handler must not sleep. “Probe handlers are run with preemption disabled or interrupt disabled … your handler should not yield the CPU (e.g., by attempting to acquire a semaphore, or waiting I/O)” (kprobes.rst, v6.12). Blocking in a pre_handler/post_handler is a bug. Kprobes also do not allocate memory or take mutexes except during registration/unregistration.

Missed hits under re-entrancy. Probing a function that your own handler calls (the classic printk example) means the nested hit is dropped and nmissed climbs. Diagnose by reading /sys/kernel/debug/kprobes/list, which shows each probe’s hit and missed counts.

Wrong argument register / ABI assumptions. A pre_handler reading function arguments must use the architecture’s calling convention out of pt_regs (on x86-64, first integer arg in rdi, second in rsi, …). Hard-coding x86-64 register names breaks on arm64. In a post_handler or kretprobe, “argument” registers may already be clobbered; the return value lives in the ABI’s return register (rax on x86-64).

How Userspace Reaches kprobes

Few users call register_kprobe() directly — that is a kernel-module API. The common paths layer on top of it:

• tracefs kprobe-events. Writing to /sys/kernel/tracing/kprobe_events creates a kprobe-backed trace event: echo 'p:myprobe do_sys_open' > .../kprobe_events plants a kprobe at do_sys_open and exposes it under tracing/events/kprobes/myprobe/ with the usual enable, format, filter, and trigger files (kprobe-event tracing docs). This is the file-interface front-end, owned by the trace-event subsystem.
• eBPF BPF_PROG_TYPE_KPROBE. A BPF program attached to a kprobe runs as the probe’s handler, receiving the captured struct pt_regs * as its context. “eBPF builds upon kprobes by allowing users to attach safe, sandboxed … code to kernel functions” (BPF_PROG_TYPE_KPROBE, eBPF docs). This is the modern default for kprobe-based tracing and the substrate behind bpftrace’s kprobe:* and BCC tools — the BPF-program details (the pt_regs context, the legacy perf_event_open vs modern kprobe_multi attach families, the overhead comparison against fentry) are covered in kprobe and uprobe BPF Programs.

Alternatives and When to Choose Them

Reach for a kprobe when no stable instrumentation point exists for what you need to see. If the kernel ships a tracepoint for the event, prefer it: a tracepoint is zero-cost when off, has a looser ABI, and survives the internal-function renames that break kprobes. If you only need a function’s entry or exit on a BTF-enabled kernel, fentry/fexit BPF trampolines (fentry fexit and BPF Trampolines) reach the same target with BTF-typed arguments and near-function-call overhead — far cheaper than a raw int3 kprobe. The unique niche where a raw kprobe wins is probing a specific instruction at an arbitrary offset inside a function (not just entry/return), which neither tracepoints, fprobe-based kprobe_multi, nor fentry can do. For mass attach across thousands of functions, kprobe_multi (which rides ftrace’s __fentry__ hook, not int3) is the right tool; see kprobe and uprobe BPF Programs. The practical hierarchy: tracepoint if one exists → fentry/fexit for entry/exit on modern kernels → kprobe_multi for mass attach → a single int3 kprobe only for a precise in-function offset.

Production Notes

The single most-cited limitation in production is instability: a kprobe is bound to an internal symbol name, address, and (implicitly) calling convention, all of which can change across kernel versions, so a tool that works on one kernel may silently fail to attach (or, worse, read the wrong register) on another. This is why tracepoints and BTF-typed fentry are preferred where available, and why kallsyms-based address resolution and CO-RE-style relocations exist in the BPF tooling. The per-hit int3 overhead also matters on hot paths: tracing a function called millions of times a second with an unoptimized kprobe imposes a measurable tax, which is why CONFIG_OPTPROBES jump-promotion and the ftrace-based kprobe_multi path were developed. Even so, the kprobe remains the universal fallback — when you need to see what a function the kernel authors never instrumented is doing, on a running production box, with no recompilation, a kprobe (most often via bpftrace or a BCC tool) is the tool that reaches it.

See Also

• kretprobes — return probes: hijacking the return address to fire on function exit, maxactive/nmissed
• Optimized kprobes and the Breakpoint to Jump Path — promoting the int3 to a jmp into a detour buffer for ~10x lower per-hit cost
• kprobe and uprobe BPF Programs — the eBPF BPF_PROG_TYPE_KPROBE consumer view: pt_regs context, attach families, overhead model
• Static Keys and Tracepoint Patching — the static counterpart: pre-placed, zero-cost-when-off tracepoints vs dynamic kprobes
• Static Keys and Code Patching — the text_poke byte-patching machinery the arm/disarm path uses
• uprobes — the userspace analogue: patch a breakpoint into a binary’s text instead of kernel text
• fentry fexit and BPF Trampolines — the typed, lower-overhead alternative for function entry/exit on BTF kernels
• Linux Tracing and Observability MOC — parent map; kprobes are the dynamic-instrumentation engine in §4