Static Keys and Tracepoint Patching

A Linux tracepoint — a named, structured instrumentation hook the kernel developers place at a semantically meaningful spot (a scheduler context-switch, a block-I/O completion, a system-call entry) — is always compiled in but pays literally nothing when no one is listening. The mechanism that buys that guarantee is the static key: every tracepoint owns an embedded struct static_key key, and the inline trace_<name>() body gates its call to the probe machinery behind static_key_false(&__tracepoint_<name>.key) (tracepoint.h, v6.12). When the key is off, that branch is patched into a fall-through nop, so the hot path executes one already-decoded no-op and moves on; registering the first probe flips the key on by rewriting that instruction live, and from then on the branch jumps into the probe-iteration code. This is the single design decision that makes it affordable to ship a kernel with thousands of always-present tracepoints. This note is the tracepoint-specific application of static keys; the generic jump-label / asm goto / text_poke machinery beneath them — including the int3-breakpoint live-patching trick — is owned by Static Keys and Code Patching, which this note cross-links rather than re-derives.

Mental Model

Think of a tracepoint as a doorway in a hot corridor that is bricked over by default. Walking the corridor, you pass a flat wall where the doorway would be — no door to check, no handle to turn, nothing to read; the wall is just part of the corridor and you stride past it for free. The moment a single observer (a perf session, an ftrace event-enable, a BPF program) wants to watch that doorway, a crew comes out and knocks a real door into the wall: the corridor is briefly stopped, the bricks become a door (the nop becomes a jmp), and from then on everyone passing the doorway is routed through the room behind it where the observers stand. When the last observer leaves, the crew bricks the door back up. The passing is always free; only opening and closing the door is expensive, and that happens on a slow administrative path, almost never.

flowchart TB
  subgraph OFF["No probes registered (default)"]
    A["hot kernel path<br/>(scheduler / block / syscall)"] --> B["trace_foo():<br/>static_key_false(&tp.key)<br/>= patched NOP"]
    B --> C["fall through<br/>(zero cost)"]
  end
  subgraph ON["First probe registered"]
    D["hot kernel path"] --> E["trace_foo():<br/>branch = patched JMP"]
    E --> F["__traceiter_foo /<br/>static_call(tp_func_foo)"]
    F --> G["walk RCU probe array<br/>(struct tracepoint_func[])"]
    G --> H["call each probe(data, args)"]
  end
  OFF -->|"tracepoint_probe_register():<br/>0->1 funcs => static_key_enable(&tp.key)"| ON
  ON -->|"tracepoint_probe_unregister():<br/>1->0 funcs => static_key_disable(&tp.key)"| OFF

The two states of a tracepoint, gated by its static key. What it shows: with no probes attached, the trace_foo() inline compiles to a no-op branch and control falls straight through; attaching the first probe transitions the function-count 0→1, which calls static_key_enable(&tp->key) and patches the no-op into a jump that reaches the RCU-protected probe array. The insight to take: the “is anyone listening?” test is not a memory load on the hot path — it is encoded directly in the instruction stream, so a disabled tracepoint costs one nop. The expensive work (live code patching, IPIs to every CPU) happens once, on the registration path, under a mutex.

The struct tracepoint and Its Embedded Key

Every tracepoint is a single statically-allocated struct tracepoint placed in a dedicated ELF section (__tracepoints). Its v6.12 definition makes the gating field explicit (tracepoint-defs.h, v6.12):

struct tracepoint {
	const char *name;		/* Tracepoint name */
	struct static_key key;
	struct static_call_key *static_call_key;
	void *static_call_tramp;
	void *iterator;
	void *probestub;
	int (*regfunc)(void);
	void (*unregfunc)(void);
	struct tracepoint_func __rcu *funcs;
};

Walking the fields that matter for gating and dispatch:

• key is the struct static_key — the on/off switch for this tracepoint. Its job is exactly the job of any static key: the fast path reads the key’s value from the instruction stream, never from this struct, and only the slow registration path touches key directly to flip the patched code.
• funcs is an RCU-protected array of struct tracepoint_func — the list of currently-registered probe callbacks. It is __rcu-annotated because the fast path dereferences it under RCU read-side protection while the registration path replaces it wholesale.
• static_call_key / static_call_tramp wire the tracepoint into the kernel’s static-call machinery, a sibling of static keys that patches an indirect call target into a direct call. In v6.12 a tracepoint with exactly one probe calls that probe through a patched direct call (cheaper, and Spectre-retpoline-friendly) instead of looping over an array.
• iterator points at the generated __traceiter_<name>() function that loops over funcs when there is more than one probe.
• regfunc / unregfunc are optional callbacks fired on the 0→1 and 1→0 transitions (used, for example, by the syscall tracepoints to enable syscall trace gathering only while watched).

The companion struct tracepoint_func is the per-probe record (tracepoint-defs.h, v6.12):

struct tracepoint_func {
	void *func;
	void *data;
	int prio;
};

Each entry holds the probe function pointer (func), an opaque per-probe data cookie passed back to it, and a priority prio controlling insertion order. The array is NULL-terminated by a sentinel entry whose func is NULL, which is how the iterator knows where to stop.

What trace_<name>() Compiles To

The TRACE_EVENT/DECLARE_TRACE macros generate, for every tracepoint foo, an inline trace_foo() whose entire purpose is to be free when off. The v6.12 generator macro is __DECLARE_TRACE (tracepoint.h, v6.12):

#define __DECLARE_TRACE(name, proto, args, cond, data_proto)		\
	extern int __traceiter_##name(data_proto);			\
	DECLARE_STATIC_CALL(tp_func_##name, __traceiter_##name);	\
	extern struct tracepoint __tracepoint_##name;			\
	static inline void trace_##name(proto)				\
	{								\
		if (static_key_false(&__tracepoint_##name.key))		\
			__DO_TRACE(name,				\
				TP_ARGS(args),				\
				TP_CONDITION(cond), 0);			\
		...							\
	}

Line by line, the body the compiler actually emits is the single statement if (static_key_false(&__tracepoint_foo.key)) __DO_TRACE(...). The crucial part is static_key_false(): this is not an ordinary if testing a memory word. As Static Keys and Code Patching explains in full, static_key_false() expands (on asm goto-capable architectures) to a branch site that the compiler lays out as a 5-byte no-op on x86 in the key’s default-false state, with a record in the __jump_table ELF section that tells the kernel how to rewrite it. So in the disabled state — the overwhelmingly common state — trace_foo() is a single nop followed by the fall-through. The tracepoint’s key is never loaded from memory on this path; its value is baked into the code.

The tracepoint is initialized default-false. Look at the static initializer in DEFINE_TRACE_FN (tracepoint.h, v6.12):

	struct tracepoint __tracepoint_##_name	__used			\
	__section("__tracepoints") = {					\
		.name = __tpstrtab_##_name,				\
		.key = STATIC_KEY_INIT_FALSE,				\
		.static_call_key = &STATIC_CALL_KEY(tp_func_##_name),	\
		...							\
		.funcs = NULL };					\

.key = STATIC_KEY_INIT_FALSE is the whole “nop when off” guarantee in one line: the key starts false, which means the branch starts as a no-op, which means a freshly booted kernel with no tracing active pays nothing for every one of its tracepoints. .funcs = NULL says there are no probes yet. The header exposes the same test publicly as tracepoint_enabled(tp), which is literally static_key_false(&(__tracepoint_##tp).key) (tracepoint-defs.h, v6.12) — used by code that wants to gate an out-of-line trace call behind the same zero-cost check (a header may include tracepoint-defs.h just to reach key, without pulling in the heavier tracepoint.h).

The RCU-Protected Probe Array and Dispatch

When the branch is taken (the key is on), control reaches __DO_TRACE, which is where the registered probes actually run. The v6.12 body (tracepoint.h, v6.12):

#define __DO_TRACE(name, args, cond, rcuidle)				\
	do {								\
		...							\
		preempt_disable_notrace();				\
		...							\
		__DO_TRACE_CALL(name, TP_ARGS(args));			\
		...							\
		preempt_enable_notrace();				\
	} while (0)

The probe call is bracketed by preempt_disable_notrace() / preempt_enable_notrace(). This is the read-side of an RCU-sched critical section: by disabling preemption around the probe call, the tracepoint core guarantees that any in-flight probe will finish before an RCU grace period elapses, which is precisely what lets the registration path free the old probe array safely. The dispatch itself, with CONFIG_HAVE_STATIC_CALL, is __DO_TRACE_CALL:

#define __DO_TRACE_CALL(name, args)					\
	do {								\
		struct tracepoint_func *it_func_ptr;			\
		void *__data;						\
		it_func_ptr =						\
			rcu_dereference_raw((&__tracepoint_##name)->funcs); \
		if (it_func_ptr) {					\
			__data = (it_func_ptr)->data;			\
			static_call(tp_func_##name)(__data, args);	\
		}							\
	} while (0)

rcu_dereference_raw(...->funcs) is the RCU read of the probe array — it loads the current funcs pointer with the memory ordering that pairs with the rcu_assign_pointer() on the registration side, so a reader either sees the complete old array or the complete new one, never a torn intermediate. When there is exactly one probe, static_call(tp_func_foo)(...) dispatches through the patched static call directly to that single probe (or, when there is more than one, the static call points at the generated iterator __traceiter_foo, which walks the array). The iterator’s loop is the multi-probe path (tracepoint.h, v6.12):

	it_func_ptr =							\
		rcu_dereference_raw((&__tracepoint_##_name)->funcs);	\
	if (it_func_ptr) {						\
		do {							\
			it_func = READ_ONCE((it_func_ptr)->func);	\
			__data = (it_func_ptr)->data;			\
			((void(*)(void *, proto))(it_func))(__data, args); \
		} while ((++it_func_ptr)->func);			\
	}

The do { ... } while ((++it_func_ptr)->func) walks the array entry by entry, calling each probe with its own data, and stops when it reaches the NULL-func sentinel. The comment in the header underlines the invariant that makes this safe: “it_func[0] is never NULL because there is at least one element in the array when the array itself is non NULL” — i.e. the array pointer is either NULL (no probes) or points at a fully-populated, sentinel-terminated array, never an empty-but-non-NULL one.

The Registration Path: Where the Key Gets Flipped

The whole “free until watched” promise rests on one fact: the static key is toggled exactly on the 0↔1 probe-count transitions, inside kernel/tracepoint.c, under the tracepoints_mutex. A caller (ftrace, perf, BPF, or a kernel module) registers a probe via tracepoint_probe_register(tp, probe, data), which routes through tracepoint_add_func(). The pivotal code (tracepoint.c, v6.12):

	switch (nr_func_state(tp_funcs)) {
	case TP_FUNC_1:		/* 0->1 */
		...
		/* Set static call to first function */
		tracepoint_update_call(tp, tp_funcs);
		/* Both iterator and static call handle NULL tp->funcs */
		rcu_assign_pointer(tp->funcs, tp_funcs);
		static_key_enable(&tp->key);
		break;
	case TP_FUNC_2:		/* 1->2 */
		...
	case TP_FUNC_N:		/* N->N+1 (N>1) */
		rcu_assign_pointer(tp->funcs, tp_funcs);
		...
	}

The decisive line is static_key_enable(&tp->key), reached only on the TP_FUNC_1 (0→1) case — the registration of the first probe. That single call is what physically patches the nop at every trace_foo() site into a jmp, using the live code-patching machinery described in Static Keys and Code Patching (jump_label_update() → the architecture transform → text_poke_bp() with the int3 breakpoint trick on a running SMP system). Adding a second, third, or Nth probe does not re-patch the code — the key is already on — it only swaps in a larger funcs array via rcu_assign_pointer(). Note also the ordering: tracepoint_update_call() and rcu_assign_pointer(tp->funcs, ...) make the probe array reachable before static_key_enable() opens the branch, so a CPU that takes the freshly-patched jump is guaranteed to find a fully-populated array, not a NULL one. The rcu_assign_pointer() carries the smp_store_release() ordering the header comment relies on: “the new probe callbacks array is consistent before setting a pointer to it. This array is referenced by __DO_TRACE.”

The symmetric teardown is in tracepoint_remove_func() on the 1→0 transition (tracepoint.c, v6.12):

	case TP_FUNC_0:		/* 1->0 */
		/* Removed last function */
		if (tp->unregfunc && static_key_enabled(&tp->key))
			tp->unregfunc();
 
		static_key_disable(&tp->key);
		/* Set iterator static call */
		tracepoint_update_call(tp, tp_funcs);
		/* Both iterator and static call handle NULL tp->funcs */
		rcu_assign_pointer(tp->funcs, NULL);
		...

Removing the last probe calls static_key_disable(&tp->key), which patches the jmp back to a nop, returning the tracepoint to its free state. Crucially, the old probe array is not freed inline — release_probes(old) defers the free through chained call_rcu() + call_srcu() callbacks so that any reader currently inside the preempt-disabled __DO_TRACE section finishes first. This is the same RCU discipline that makes the array swaps on the 2→1, N→N-1 paths safe. Whenever a module unregisters its last probe it must additionally call tracepoint_synchronize_unregister() before the module’s text is freed — the header is blunt: it “must be called between the last tracepoint probe unregistration and the end of module exit to make sure there is no caller executing a probe when it is freed.”

A Worked Example: One Tracepoint, Three Listeners

Consider sched_switch, fired on every context switch — one of the hottest tracepoints in the kernel. In a freshly booted system with no tracing, the scheduler’s __schedule() path contains a trace_sched_switch() call that has compiled to a single nop; the scheduler pays nothing across billions of context switches. Now three things happen in sequence:

1. An operator runs echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable. ftrace registers its probe; tracepoint_add_func() sees the 0→1 transition, installs the probe array, and calls static_key_enable(&__tracepoint_sched_switch.key). The nop at the scheduler call site is live-patched to a jmp. From this instant, every context switch routes through __DO_TRACE and records an event into the ftrace ring buffer.
2. A perf record -e sched:sched_switch session starts. perf registers its own probe on the same tracepoint. This is the 1→2 transition: the key is already on, so no code is patched; the array simply grows from one entry to two, and the static call switches from the single-probe direct call to the iterator. Both ftrace and perf now receive every switch.
3. A bpftrace -e 'tracepoint:sched:sched_switch { @ = count(); }' one-liner attaches. That is the 2→3 transition; again no patching, the array grows to three. All three consumers get every event.

When each listener detaches, the array shrinks back down the same way, and only when the last one leaves does static_key_disable() re-brick the call site into a nop. This is the “one source, many consumers” property the Linux Tracing and Observability MOC highlights: the static key and probe array are shared substrate, so attaching the second and third tools is nearly free, and the disabled steady state is genuinely zero-cost.

Why Not Just Test a Flag in the Struct?

The naive alternative is if (tp->enabled) trace_foo(...), reading a boolean field of struct tracepoint. It is correct, and the branch predictor learns the branch is never taken — but the load still happens on every pass, occupying a cache line that is otherwise dead weight. Multiplied across thousands of tracepoints embedded in the hottest paths of the scheduler, networking stack, and block layer, those dead loads compete for load-store bandwidth and evict useful cache lines. The kernel’s static-key documentation frames the cost precisely: even a cheap memory test’s overhead “increases when the memory cache comes under pressure” (static-keys.rst, v6.12), and the original LWN write-up puts it the same way — “Each test must fetch a value from memory, adding to the pressure on the cache and hurting performance” (LWN, Jump labels, 2010). The static-key answer is to not read the flag at all on the fast path and instead bake its value into the code, re-baking only on the rare toggle. For state that is read billions of times and changed almost never — the exact profile of “is anyone tracing this?” — that is the right bargain, and it is why the whole static-tracing design rests on it.

Failure Modes and Subtleties

The cost is the toggle, not the steady state. Enabling a widely-instrumented tracepoint patches the call site and (because the key may gate many trace_foo() inlines, and because the patch issues IPIs to serialize every CPU’s instruction cache) produces a brief, observable latency blip. This is acceptable because toggles are administrative and rare, but real-time-sensitive systems prefer to enable the tracepoints they need up front rather than flip them under load. The detailed IPI/text_poke cost model lives in Static Keys and Code Patching.

A disabled tracepoint is free; an enabled but unfiltered one is not. The zero-cost guarantee covers only the off state. Once any consumer attaches, every hit runs __DO_TRACE, dereferences the array, and calls each probe — real per-event work. Filtering (ftrace filter files, perf filters, BPF early-return) reduces output volume but does not remove the per-hit dispatch cost; the cheapest tracepoint is still the one nobody is watching.

RCU must be watching at the call site. Because __DO_TRACE runs inside a preempt-disabled RCU read-side section, a tracepoint placed in code where RCU is not active (deep idle, certain entry paths) is a bug. v6.12 guards this with a CONFIG_LOCKDEP check that warns "RCU not watching for tracepoint" and, historically, with a separate trace_foo_rcuidle() variant using SRCU for the idle path. Placing tracepoints is therefore a kernel-developer responsibility, not something a user does.

Architectures without asm goto lose the guarantee. On a platform lacking jump-label support, static_key_false() degrades to an ordinary atomic_read() of the key — exactly the memory-tested flag the mechanism was invented to avoid. The tracepoint is still correct everywhere; the “free when off” property is specific to HAVE_JUMP_LABEL architectures (x86, arm64, and friends). Pin any “tracepoints are free” claim to a supported arch.

Alternatives and When to Choose Them

A static tracepoint is the right tool when a kernel maintainer has placed a stable, named hook for the event you care about: it is zero-cost when off, has a (loose) ABI via its format file, and survives refactors better than probing internal symbols. Where no tracepoint exists, the dynamic alternative is a kprobe — which can instrument almost any instruction by patching in a breakpoint at runtime, at the cost of being tied to internal symbol names and a heavier per-hit trap. The two are complementary: tracepoints for the curated, stable surface; kprobes for the vast surface no one anticipated tracing. For a flag that flips frequently or from atomic context, a static key (and hence a tracepoint’s gating) is the wrong mechanism entirely — the patch cost would dwarf any read savings; a READ_ONCE-tested flag is correct there. That trade-off and the full alternatives comparison belong to Static Keys and Code Patching.

Production Notes

The practical payoff is visible every time you attach a tool to a production box: you can perf record a hot scheduler tracepoint, or point a BPF program at a block-I/O completion tracepoint, and the system pays for instrumentation only from the moment you attach — because until then the tracepoint was a nop. Distribution kernels lean on this hard: a modern kernel ships with thousands of tracepoints compiled in (/sys/kernel/tracing/available_events enumerates them), and the static-key gating is the only reason that pervasive, always-present instrumentation does not tax the idle, untraced common case. The same struct tracepoint.funcs array and key are what the kprobe-based and BPF event machinery hook into when you create dynamic events (/sys/kernel/tracing/kprobe_events, per the kprobe-event tracing docs) — but those dynamic probes are a different substrate (kprobes); the static tracepoint path described here is the curated, compiled-in one.

See Also

• Static Keys and Code Patching — the generic jump-label / asm goto / text_poke / int3 machinery this note sits on top of; owns the live-patching mechanics
• Tracepoints — the tracepoint concept and the TRACE_EVENT macro that generates the trace_<name>() inlines gated here
• The TRACE_EVENT Macro — how a tracepoint declaration expands into the struct, iterator, and format file
• The Trace Event Subsystem — the ring buffers and format files that consume tracepoint output
• Syscall Tracepoints sys_enter and sys_exit — a concrete tracepoint family using regfunc/unregfunc on the 0↔1 transition
• kprobes — the dynamic counterpart: probe any instruction by patching a breakpoint, when no tracepoint exists
• Read-Copy-Update Fundamentals — the RCU discipline that protects the funcs probe array across registration
• Linux Tracing and Observability MOC — parent map; “static keys make tracing free when off” is one of its cross-cutting themes