Tracepoints

A tracepoint is a named, static instrumentation hook that a kernel developer places by hand at a semantically meaningful spot in the source — a scheduler context switch (sched_switch), a block-I/O completion (block_rq_complete), a slab allocation (kmalloc). Each tracepoint compiles down to a tiny inline trace_<name>() function whose body is guarded by a static key, so that when no probe is attached the call site is a literal nop and costs nothing but a few bytes of text and a record in a metadata section (Documentation/trace/tracepoints.rst, v6.12). When one or more probe callbacks are registered at runtime, the static key is patched live and each call to trace_<name>() invokes every registered probe in the caller’s own execution context, passing it the tracepoint’s typed arguments. Tracepoints are the shared substrate of Linux tracing: the same hook can be written to an ftrace ring buffer, sampled by perf, or handed raw to an eBPF program — learn the source once and every front-end unlocks. The infrastructure was contributed by Mathieu Desnoyers and has lived in the kernel since 2.6.28; the TRACE_EVENT() macro that made it usable at scale was added by Steven Rostedt in 2009 (Rostedt, LWN, March 2010).

This note is about the tracepoint mechanism: the hook, its static-key guard, the probe-registration state machine, what the TRACE_EVENT() macro actually generates, and the tracepoint’s contested ABI status. The tracefs control surface that exposes enabled tracepoints as files is The tracefs Filesystem; the ring-buffer machinery behind the records is The Trace Event Subsystem; the jump-label mechanism the guard rests on is Static Keys and Tracepoint Patching.

All code, line numbers, and struct layouts below were read from the Linux 6.12 LTS tree via raw.githubusercontent.com on 2026-09-04. v6.12 is a maintained long-term-support release; mainline had moved on to the 7.x series by that date, so anything stated about later kernels is dated explicitly.

Mental Model — A Patched-Out Call Site With a Probe List

Think of a tracepoint as three pieces that the compiler and linker assemble for you. (1) A call site: an if-guarded call buried in the function you care about. (2) A struct tracepoint living in a dedicated ELF section (__tracepoints) — it holds the tracepoint’s name, its static key, and a pointer to an RCU-protected array of registered probe functions. (3) A set of register/unregister functions (register_trace_<name>()) that consumers call to attach or detach a probe.

The crucial trick is the guard. The if (static_key_false(&tp.key)) in front of the call is not an ordinary branch reading a global from memory — it is a jump-label site. While the key is “false,” the kernel has patched the instruction stream so that the branch is a nop falling straight through to the rest of the function; the probe-dispatch block is jumped over and never touched. When the first probe registers, static_key_enable() rewrites the live .text to make the branch taken. This is why a kernel can ship thousands of always-compiled-in tracepoints with effectively zero steady-state cost — a disabled tracepoint is one nop, not a load-and-compare.

flowchart TB
  subgraph FN["kernel function, e.g. __schedule"]
    A["... real work ..."]
    G{"static_key_false<br/>&amp;__tracepoint_sched_switch.key"}
    A --> G
    G -- "key FALSE:<br/>patched to a 5-byte nop,<br/>falls through" --> Z["continue function"]
    G -- "key TRUE:<br/>patched to jmp" --> DT["__DO_TRACE<br/>cond check, then<br/>preempt_disable_notrace"]
    DT --> IT["__DO_TRACE_CALL:<br/>1 probe = static_call direct<br/>2+ probes = iterator loop"]
    IT --> Z
  end
  subgraph TP["struct tracepoint, in section __tracepoints"]
    K["key: struct static_key"]
    SC["static_call_key + tramp"]
    F["funcs: RCU array of<br/>tracepoint_func<br/>func, data, prio"]
  end
  IT -. "rcu_dereference_raw" .-> F
  IT -. "dispatches through" .-> SC
  G -. "guarded by" .-> K
  REG["register_trace_sched_switch probe, data<br/>-> tracepoint_probe_register<br/>-> tracepoint_add_func"] -. "0 to 1 probe:<br/>static_key_enable" .-> K
  REG -. "rcu_assign_pointer" .-> F

A tracepoint call site inside a kernel function, the struct tracepoint that backs it, and the registration path. What it shows: three separate objects — the guarded call site in .text, the metadata struct in __tracepoints, and the registration entry point — connected only by the static key and the RCU-protected funcs array. The insight to take: the disabled path performs no memory access at all, because the branch decision was baked into the instruction stream by code patching rather than read from a variable; and the probe array is read under RCU inside a preempt_disable window, so probe removal needs a grace period rather than a lock on the hot path.

What a Declaration Generates For the Tracepoint

Almost every tracepoint in the kernel is written with TRACE_EVENT(), and that macro is famously opaque: one declaration in include/trace/events/<subsys>.h silently produces a dozen distinct C objects. The full clause-by-clause and stage-by-stage expansion is owned by The TRACE_EVENT Macro; what matters here is the shape of the fan-out and, specifically, the artifacts that constitute the tracepoint itself.

The mechanism is multi-pass header inclusion. include/trace/define_trace.h is included at the bottom of every event header, deliberately outside that header’s own include guard, and it works by redefining TRACE_EVENT and its siblings and then #include-ing the very same header again. In v6.12 the header is re-read nine times in total: once for the tracepoint pass, seven times inside trace/trace_events.h, once inside trace/perf.h, and once inside trace/bpf_probe.h. (The count is nine re-reads producing four consumers; trace_events.h accounts for seven of them.) Every pass sees the same declaration text and expands it into a completely different C construct.

flowchart TB
  SRC["TRACE_EVENT sched_switch<br/>TP_PROTO / TP_ARGS /<br/>TP_STRUCT__entry / TP_fast_assign / TP_printk"]
  SRC --> DT["include trace/define_trace.h<br/>fires only in the one .c file that<br/>defines CREATE_TRACE_POINTS"]

  DT --> P0["pass 0<br/>TRACE_EVENT redefined to DEFINE_TRACE"]
  P0 --> O0["THE TRACEPOINT ITSELF:<br/>struct tracepoint __tracepoint_sched_switch<br/>__traceiter_sched_switch  the N-probe iterator<br/>__probestub_sched_switch  an empty stub<br/>DEFINE_STATIC_CALL tp_func_sched_switch<br/>name string + self-pointer in 3 ELF sections"]

  DT --> TE["include trace/trace_events.h<br/>7 stages = 7 more re-reads"]
  TE --> OT["THE FTRACE TRACE-EVENT:<br/>trace_event_raw_sched_switch  record layout<br/>trace_event_raw_event_sched_switch  the probe<br/>trace_event_fields_...  drives the format file<br/>struct trace_event_call in section _ftrace_events"]

  DT --> PF["include trace/perf.h<br/>only if CONFIG_PERF_EVENTS"]
  PF --> OP["THE PERF PROBE:<br/>perf_trace_sched_switch<br/>perf_trace_buf_alloc + perf_fetch_caller_regs<br/>+ perf_trace_run_bpf_submit"]

  DT --> BP["include trace/bpf_probe.h<br/>only if CONFIG_BPF_EVENTS"]
  BP --> OB["THE BPF RAW BINDING:<br/>__bpf_trace_sched_switch -> bpf_trace_run4<br/>struct bpf_raw_event_map in __bpf_raw_tp_map<br/>typedef btf_trace_sched_switch  read by BTF"]

  O0 -.-> HOOK["trace_sched_switch  called from __schedule"]
  OT -.-> HOOK
  OP -.-> HOOK
  OB -.-> HOOK

One TRACE_EVENT() declaration expanded through include/trace/define_trace.h (v6.12), grouped by consumer rather than by stage. What it shows: four independent consumers — the bare tracepoint, the ftrace trace-event, perf, and BPF — are each generated by re-reading the same header with a different set of macro definitions in force, and all four ultimately hang off one call site. The insight to take: the pass-0 column is the tracepoint; the other three columns are probes that register against it. That is the whole architecture of Linux tracing in one picture — a single static hook, and a set of independently compiled, independently attachable consumers. It also explains a build-time gotcha: CREATE_TRACE_POINTS must be defined in exactly one translation unit, because pass 0 is the only pass that defines rather than declares __tracepoint_sched_switch.

Pass 0, in detail — everything DEFINE_TRACE emits

Pass 0 is where the tracepoint is born. define_trace.h redefines TRACE_EVENT (and DECLARE_TRACE, and DEFINE_EVENT) to DEFINE_TRACE(name, proto, args), which is a thin wrapper over DEFINE_TRACE_FN(name, NULL, NULL, proto, args) in include/linux/tracepoint.h. That macro emits six things:

Artifact	Section / linkage	Purpose
__tpstrtab_<name>[]	__tracepoints_strings	The tracepoint’s name as a string, kept out of the struct so the struct stays small
struct tracepoint __tracepoint_<name>	__tracepoints	The whole tracepoint: name pointer, static_key initialised FALSE, static-call key and trampoline, iterator, probestub, optional regfunc/unregfunc, and funcs = NULL
tracepoint_ptr_t __tracepoint_ptr_<name>	__tracepoints_ptrs	A pointer to the struct. Iteration uses this array, not the struct array, because “we have no guarantee that gcc and the linker won’t up-align the tracepoint structures”
int __traceiter_<name>(void *__data, proto)	normal text	The multi-probe iterator: walks the funcs array and calls each probe in turn
void __probestub_<name>(void *__data, proto)	normal text	An empty function, used as a placeholder when a probe slot must be filled but not called (notably when func_remove() cannot allocate a smaller array)
DEFINE_STATIC_CALL(tp_func_<name>, __traceiter_<name>)	static-call machinery	The patchable indirect call the dispatch path goes through, initially pointing at the iterator

The other half of the tracepoint — the parts a caller and a consumer see — comes from __DECLARE_TRACE, which runs in every ordinary .c file that merely #includes the event header. It emits trace_<name>() (the call site), trace_<name>_enabled(), register_trace_<name>(), register_trace_prio_<name>(), unregister_trace_<name>(), and check_trace_callback_type_<name>() — the last being an empty function whose only purpose is to make the compiler type-check a candidate probe against the exact TP_PROTO signature. perf.h and bpf_probe.h both call it (check_trace_callback_type_##call(perf_trace_##template)), with a comment that says exactly why: “it is only here as a build time check to make sure that if the tracepoint handling changes, the perf probe will fail to compile unless it too is updated.”

DECLARE_EVENT_CLASS and DEFINE_EVENT — one class, many hooks

Stage 1 of trace_events.h reveals that TRACE_EVENT is not a primitive at all:

#define TRACE_EVENT(name, proto, args, tstruct, assign, print) \
	DECLARE_EVENT_CLASS(name, PARAMS(proto), PARAMS(args),  \
			    PARAMS(tstruct), PARAMS(assign), PARAMS(print)); \
	DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args));

It is a one-to-one class plus a single instance. From the tracepoint’s point of view the important consequence is that DEFINE_EVENT is what creates a distinct struct tracepoint — so several independent tracepoints can share one record layout and one probe function. The scheduler does exactly this: DECLARE_EVENT_CLASS(sched_wakeup_template, ...) is instanced three times, producing sched:sched_waking, sched:sched_wakeup, and sched:sched_wakeup_new as three separately enableable tracepoints on one class. Stage 7 shows the sharing explicitly — .class = &event_class_##template and .print_fmt = print_fmt_##template, but .tp = &__tracepoint_##call. The class-versus-instance split, and the seven stages that build the class, are walked in full in The TRACE_EVENT Macro.

A DECLARE_TRACE() used without the trace-event machinery produces only the pass-0 and __DECLARE_TRACE artifacts: a hook with no record layout, no format file, and no entry in tracefs. Those exist (a handful of subsystem-internal hooks, and the hooks BPF raw_tp programs bind to) and are discussed under Tracepoint vs Trace Event below.

The Static Key — Why a Disabled Tracepoint Costs Nothing

Everything about tracepoints economically follows from one property: a disabled tracepoint is free. That is what lets distributions ship a kernel with several thousand of them compiled in and enabled-on-demand, and it is the single strongest argument for choosing a tracepoint over a kprobe when both are available.

The mechanism is jump labels, and the kernel’s own documentation is unusually blunt that tracepoints are why they exist: “Currently, tracepoints are implemented using a conditional branch. The conditional check requires checking a global variable for each tracepoint… As we increase the number of tracepoints in the kernel this overhead may become more of an issue… Although tracepoints are the original motivation for this work, other kernel code paths should be able to make use of the static keys facility” (Documentation/staging/static-keys.rst, v6.12).

A jump label replaces the load-test-branch with a patchable instruction. On x86, arch/x86/include/asm/jump_label.h emits either a literal five-byte no-op or a jmp that objtool rewrites to one, and records the site in a __jump_table section entry:

arch_static_branch()  emits, at label 1:

   .byte BYTES_NOP5              <-- the patch site: 5 bytes, one instruction
   .pushsection __jump_table, "aw"
   .align
   .long  1b - .                 <-- rel32: offset to the patch site
   .long  %l[l_yes] - .          <-- rel32: offset to the "taken" target
   _ASM_PTR %c0 + %c1 - .        <-- rel64: offset to (struct static_key *) | branch bit
   .popsection

  key DISABLED                      key ENABLED
  +---------------------------+     +---------------------------+
  | 0f 1f 44 00 00   nop5     |     | e9 xx xx xx xx   jmp l_yes|
  +---------------------------+     +---------------------------+
  | ...function continues...  |     | ...function continues...  |
  +---------------------------+     +---------------------------+

The x86-64 jump-label patch site and its __jump_table entry (v6.12), drawn as ASCII because the thing being shown is an instruction-stream rewrite, not a bit-field layout — mermaid packet-beta would misrepresent it. What it shows: the entire “is this tracepoint on?” decision is five bytes of code, plus a 16-byte metadata record living outside .text that tells the patcher where those five bytes are, where to jump, and which key owns them. The insight to take: the low bit of the key pointer (%c1) encodes the branch polarity, which is how one table serves both static_branch_likely and static_branch_unlikely; and because the offsets are stored as relative .longs, the table is position-independent and survives module relocation.

The measured cost, from the same document (numbers taken on 3.3.0-rc2 and unchanged in the v6.12 text, so treat them as illustrative of the shape of the win rather than as current absolute figures):

Metric	Without jump labels	With jump labels	Delta
Disabled-branch code size	mov (6B) + test (2B) + jne (2B) = 10 bytes	one 5-byte nop	5 bytes saved per site
Function footprint (the 80-byte sys_getppid example, incl. padding)	80 bytes	64 bytes	20% smaller
perf bench sched pipe, branches	208,368,926	206,859,359	−0.7%
perf bench sched pipe, branch-misses	5,569,188 (2.67%)	4,884,119 (2.36%)	−12%
perf bench sched pipe, cycles	1,474,374,262	1,432,559,428	−2.8%
perf bench sched pipe, elapsed	1.6016 s	1.5794 s	−1.4%

Static-key cost, as measured and published in Documentation/staging/static-keys.rst. What it shows: the win is not primarily instruction count — it is 0.2% fewer instructions — but branch predictor pressure, down 12%, and memory traffic, since the removed mov was a load of a global whose cache line “may be shared with other memory accesses.” The insight to take: the numbers are small per-site precisely because they are meant to be multiplied by thousands of sites on hot paths. The trade the document states outright is: “changing branch direction is expensive but branch selection is basically ‘free’.” Patching costs a stop_machine-class synchronisation; checking costs nothing. That asymmetry is why tracepoints are cheap to have and comparatively slow to toggle — and why toggling thousands of events at once (echo 1 > events/enable) visibly takes time.

stateDiagram-v2
    [*] --> Disabled : boot / DEFINE_TRACE<br/>key = STATIC_KEY_INIT_FALSE<br/>funcs = NULL
    Disabled --> Patching : first register_trace_x()<br/>tracepoint_add_func 0 to 1<br/>calls static_key_enable
    Patching --> Enabled : arch_jump_label_transform<br/>text_poke_bp rewrites the<br/>5-byte nop into a jmp
    Enabled --> Unpatching : last unregister_trace_x()<br/>tracepoint_remove_func 1 to 0<br/>calls static_key_disable
    Unpatching --> Disabled : jmp rewritten back to nop
    Enabled --> Enabled : 2nd..Nth probe registers<br/>key already true,<br/>NO patching happens

    note right of Disabled
      Cost at the call site:
      one 5-byte nop.
      No load, no compare,
      no branch prediction slot.
    end note
    note right of Enabled
      Cost at the call site:
      taken jmp + preempt_disable
      + one static_call
      + the probe body.
    end note
    note right of Patching
      Not free. x86 uses
      HAVE_JUMP_LABEL_BATCH so a
      mass enable batches the
      text_poke IPIs.
    end note

The lifecycle of one tracepoint’s static key. What it shows: the key has two stable states and two transient patching states, and the transitions are driven purely by the count of registered probes crossing zero — not by which consumer registered. The insight to take: the second and subsequent registrations do not re-patch. This is why enabling sched_switch for both ftrace and a BPF program costs one patch, not two; and why the expensive part of echo 1 > events/enable is the thousands of first-registrations, which is exactly the case x86’s HAVE_JUMP_LABEL_BATCH exists to amortise.

The user-visible payoff of the key is trace_<name>_enabled(), generated for every tracepoint by __DECLARE_TRACE:

static inline bool trace_foo_bar_enabled(void) {
	return static_key_false(&__tracepoint_foo_bar.key);
}

This is the idiom for tracepoints whose arguments are expensive to compute:

if (trace_foo_bar_enabled()) {          /* itself a jump label: whole block is jumped over */
	int i, tot = 0;
	for (i = 0; i < count; i++)
		tot += calculate_nuggets();  /* only runs when someone is listening */
	trace_foo_bar(tot);
}

The documentation is precise about why the inner call must stay inside the block: “The trace_<tracepoint>() should always be within the block of the if (trace_<tracepoint>_enabled()) to prevent races between the tracepoint being enabled and the check being seen.” And it names the payoff: “the advantage of using the trace_<tracepoint>_enabled() is that it uses the static_key of the tracepoint to allow the if statement to be implemented with jump labels and avoid conditional branches.”

There is a header-safe sibling. trace_<name>_enabled() lives in tracepoint.h, which drags in the whole heavy machinery; including that from a widely used header bloats the kernel. tracepoint-defs.h therefore offers tracepoint_enabled(foo_bar) — the identical static_key_false() test — together with DECLARE_TRACEPOINT(foo_bar), and the documented pattern is to test in the header and call a one-line wrapper defined in a .c file.

Mechanical Walk-through — From Call Site to Probe

Consider trace_sched_switch(...), called from the scheduler’s __schedule(). The inline that __DECLARE_TRACE generates is, with line-continuations stripped:

static inline void trace_sched_switch(proto) {
    if (static_key_false(&__tracepoint_sched_switch.key))
        __DO_TRACE(sched_switch, TP_ARGS(args), TP_CONDITION(cond), 0);
    if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) {
        WARN_ONCE(!rcu_is_watching(), "RCU not watching for tracepoint");
    }
}

The guard. static_key_false() is the jump-label test described above. The trailing WARN_ONCE under CONFIG_LOCKDEP runs unconditionally — outside the key test — and that is deliberate: the comment in tracepoint.h explains that “tracepoints require RCU to be active, and it should always warn at the tracepoint site if it is not watching, as it will need to be active when the tracepoint is enabled.” Catching the bug only once someone enables the tracepoint would be far too late.

The condition. cond for a plain DECLARE_TRACE is cpu_online(raw_smp_processor_id()) — a sanity guard that suppresses the tracepoint on a CPU that is not fully online. DECLARE_TRACE_CONDITION and TRACE_EVENT_CONDITION && the author’s own predicate onto that, which is how events like sched_stat_runtime avoid firing in states where their arguments would be meaningless.

The dispatch (__DO_TRACE). When the key is enabled, control enters __DO_TRACE:

#define __DO_TRACE(name, args, cond, rcuidle) do {                  \
    int __maybe_unused __idx = 0;                                   \
    if (!(cond))                                                    \
        return;                                                     \
    if (WARN_ONCE(RCUIDLE_COND(rcuidle), "Bad RCU usage for tracepoint")) \
        return;                                                     \
    preempt_disable_notrace();                                      \
    if (rcuidle) { __idx = srcu_read_lock_notrace(&tracepoint_srcu); \
                   ct_irq_enter_irqson(); }                         \
    __DO_TRACE_CALL(name, TP_ARGS(args));                           \
    if (rcuidle) { ct_irq_exit_irqson();                            \
                   srcu_read_unlock_notrace(&tracepoint_srcu, __idx); } \
    preempt_enable_notrace();                                       \
} while (0)

Three things matter. First, preempt_disable_notrace() brackets the probe call — this, combined with an RCU grace period at unregistration, is what makes detaching a probe safe without any lock on the hot path: a reader either sees the probe and runs to completion within the non-preemptible window, or does not see it at all. Second, notrace on every helper keeps the function tracer from recursing into the tracepoint machinery and deadlocking. Third, RCUIDLE_COND(rcuidle) is architecture-dependent: on CONFIG_ARCH_WANTS_NO_INSTR architectures (x86-64 and arm64 among them) it is simply (rcuidle), meaning the rcuidle path always warns and returns, because as the comment says such architectures “are expected to have sanitized entry and idle code that disallow any/all tracing/instrumentation when RCU isn’t watching.” Elsewhere it is (rcuidle && in_nmi()), because SRCU cannot be used from an NMI.

__DO_TRACE_CALL and the static-call optimisation. This is the clever part. On architectures with CONFIG_HAVE_STATIC_CALL:

it_func_ptr = rcu_dereference_raw((&__tracepoint_sched_switch)->funcs);
if (it_func_ptr) {
    __data = it_func_ptr->data;
    static_call(tp_func_sched_switch)(__data, args);
}

A static call is to a function pointer what a static key is to a branch: a patched, direct call. When exactly one probe is attached, static_call() has been rewritten to jump straight to that probe — no indirect branch, and therefore no retpoline or IBT cost on hardware with Spectre-v2 mitigations. When two or more probes are attached, the static call is repointed at the tracepoint’s iterator, __traceiter_<name>(), which walks the array:

it_func_ptr = rcu_dereference_raw((&__tracepoint_sched_switch)->funcs);
if (it_func_ptr) {
    do {
        it_func = READ_ONCE((it_func_ptr)->func);
        __data  = (it_func_ptr)->data;
        ((void(*)(void *, proto))(it_func))(__data, args);
    } while ((++it_func_ptr)->func);
}

The array of struct tracepoint_func { void *func; void *data; int prio; } is NULL-terminated by an entry with a null func. On architectures without static calls, __DO_TRACE_CALL simply always calls the iterator — correct, just one indirect branch slower in the single-probe case.

sequenceDiagram
    autonumber
    participant K as __schedule<br/>(kernel code)
    participant SK as static key<br/>(patched .text)
    participant SC as static_call<br/>tp_func_sched_switch
    participant IT as __traceiter_<br/>sched_switch
    participant P1 as probe 1<br/>trace_event_raw_event_...<br/>(ftrace)
    participant P2 as probe 2<br/>__bpf_trace_...<br/>(BPF raw_tp)

    K->>SK: trace_sched_switch(preempt, prev, next, prev_state)
    Note over SK: 5-byte nop if no probes:<br/>returns immediately, zero cost
    SK-->>K: (disabled case ends here)

    rect rgb(238,242,248)
    Note over K,P2: enabled case, ONE probe attached
    K->>SK: trace_sched_switch(...)
    SK->>K: branch taken, enter __DO_TRACE
    K->>K: cond check, then preempt_disable_notrace()
    K->>SC: static_call, patched DIRECT to probe 1
    SC->>P1: probe1(funcs[0].data, args)
    P1-->>SC: return
    K->>K: preempt_enable_notrace()
    end

    rect rgb(245,240,235)
    Note over K,P2: enabled case, TWO probes attached
    K->>SC: static_call, now patched to the ITERATOR
    SC->>IT: __traceiter_sched_switch(NULL, args)
    IT->>IT: rcu_dereference_raw(tp->funcs)
    IT->>P1: funcs[0].func(funcs[0].data, args)
    P1-->>IT: return
    IT->>P2: funcs[1].func(funcs[1].data, args)
    P2-->>IT: return
    Note over IT: loop ends at the<br/>NULL func terminator
    IT-->>K: return
    end

One sched_switch firing, in all three regimes. What it shows: the disabled case never leaves the caller; the one-probe case reaches the probe through a direct patched call; the multi-probe case pays one extra hop into the iterator, which then walks an RCU-protected array. The insight to take: the probe runs in the caller’s own context — inside __schedule(), with preemption disabled, on the same stack. That is the defining property of a tracepoint and the source of most of its rules: a probe may not sleep, may not fault on user memory carelessly, must be notrace, and its cost is directly added to the traced path’s latency. It also shows what “two consumers” costs: one extra indirect call, not two full dispatches.

Registration — The Probe-Count State Machine

A consumer attaches a probe with register_trace_<name>(probe, data), a generated wrapper over tracepoint_probe_register(&__tracepoint_<name>, probe, data). That lands in tracepoint_add_func() in kernel/tracepoint.c. The function appends the new {func, data, prio} to a freshly allocated, priority-ordered copy of the array and publishes it with rcu_assign_pointer(). What happens besides the publish depends entirely on how many probes the array now holds, classified by nr_func_state() into four cases:

enum tp_func_state { TP_FUNC_0, TP_FUNC_1, TP_FUNC_2, TP_FUNC_N };
 
static enum tp_func_state nr_func_state(const struct tracepoint_func *tp_funcs) {
	if (!tp_funcs)            return TP_FUNC_0;
	if (!tp_funcs[1].func)    return TP_FUNC_1;
	if (!tp_funcs[2].func)    return TP_FUNC_2;
	return TP_FUNC_N;	/* 3 or more */
}

stateDiagram-v2
    direction LR
    [*] --> F0
    state "TP_FUNC_0<br/>funcs = NULL<br/>key OFF, call site = nop" as F0
    state "TP_FUNC_1<br/>one probe<br/>key ON, static_call = probe (DIRECT)" as F1
    state "TP_FUNC_2<br/>two probes<br/>key ON, static_call = iterator" as F2
    state "TP_FUNC_N<br/>three or more<br/>key ON, static_call = iterator" as FN

    F0 --> F1 : add. tp_rcu_cond_sync(1_0_1)<br/>update_call to funcs[0].func<br/>rcu_assign_pointer<br/>static_key_enable
    F1 --> F0 : remove. unregfunc()<br/>static_key_disable<br/>update_call to iterator<br/>rcu_assign_pointer NULL<br/>tp_rcu_get_state(1_0_1)
    F1 --> F2 : add. update_call to iterator FIRST,<br/>then rcu_assign_pointer
    F2 --> F1 : remove. rcu_assign_pointer,<br/>cond_sync(N_2_1),<br/>then update_call to funcs[0].func
    F2 --> FN : add. rcu_assign_pointer only
    FN --> FN : add or remove.<br/>rcu_assign_pointer only
    FN --> F2 : remove. rcu_assign_pointer only

The tracepoint_add_func() / tracepoint_remove_func() state machine, v6.12. What it shows: only two transitions touch the static key — 0 to 1 and 1 to 0 — and only two touch the static call — 0 to 1/2 to 1 (point it at the single probe) and 1 to 2/1 to 0 (point it at the iterator). Everything else is a bare pointer publish. The insight to take: look at the ordering asymmetry between 1 to 2 and 2 to 1. Adding a second probe repoints the static call to the iterator before publishing the larger array, so no reader can ever reach a two-element array through a direct single-probe call. Removing back to one publishes the shorter array first and only then repoints the static call. Both orders exist so that a reader mid-dispatch always sees a dispatcher that is safe for whatever array it loads — and the extra tp_rcu_cond_sync calls close the remaining window where a 1→0→1 or N→…→2→1 sequence could hand a stale data pointer to a freshly repointed static call.

tracepoint_update_call() is the one-line summary of that middle column:

static void tracepoint_update_call(struct tracepoint *tp, struct tracepoint_func *tp_funcs) {
	void *func = tp->iterator;
	/* Synthetic events do not have static call sites */
	if (!tp->static_call_key)
		return;
	if (nr_func_state(tp_funcs) == TP_FUNC_1)
		func = tp_funcs[0].func;
	__static_call_update(tp->static_call_key, tp->static_call_tramp, func);
}

The early return is worth noting: dynamically created synthetic events (built at runtime through tracefs, not compiled in) have a struct tracepoint with no static call site at all, since there is no compiled-in trace_<name>() inline to patch. They always go through the iterator.

Two more details from tracepoint_add_func(). First, if (tp->regfunc && !static_key_enabled(&tp->key)) ret = tp->regfunc(); — a tracepoint may carry a registration hook run on the 0→1 transition, which is how the syscall tracepoints arrange for the syscall entry/exit slow path to be enabled before their first probe can fire. Second, freeing. release_probes(old) does not free the old array immediately; it goes through call_rcu(rcu_free_old_probes) which in turn does call_srcu(&tracepoint_srcu, srcu_free_old_probes). Two chained grace periods — RCU then SRCU — because the dispatch path may have used either, depending on the rcuidle path.

That double grace period is exactly what tracepoint_synchronize_unregister() waits out, and it is why the function is defined as it is:

static inline void tracepoint_synchronize_unregister(void) {
	synchronize_srcu(&tracepoint_srcu);
	synchronize_rcu();
}

Calling it is mandatory before a module that registered a probe may exit. The .rst states the contract: it “must be called before the end of the module exit function to make sure there is no caller left using the probe. This, and the fact that preemption is disabled around the probe call, make sure that probe removal and module unload are safe.” Skip it and you have the classic use-after-free: one CPU frees the module text while another is executing inside the probe.

Probes are dispatched in priority order, with TRACEPOINT_DEFAULT_PRIO = 10; register_trace_prio_<name>() lets a consumer that must observe state before another consumer mutates it order itself first. And tracepoint_probe_register_prio_may_exist() exists for attach paths that want to be idempotent rather than error on a duplicate {probe, data} pair — BPF’s bpf_probe_register() uses exactly this.

Tracepoint vs Trace Event — The Distinction People Conflate

A tracepoint is only the hook: the struct tracepoint, the static-key guard, the trace_<name>() inline, the iterator, and the register/unregister functions. By itself it has no ring buffer, no format file, and no representation under /sys/kernel/tracing/events/. A bare tracepoint declared with DECLARE_TRACE() is invisible to ftrace and perf; it can only be consumed by in-kernel code that calls register_trace_<name>() directly.

A trace event is a tracepoint plus the trace-event machinery: a generated record layout, a probe that copies the arguments into a ring-buffer entry, a format file describing that layout, and a directory under events/<system>/<name>/ with enable, filter, trigger, and id files.

Corbet’s summary of Rostedt’s 2017 Kernel Summit talk puts it as crisply as anyone has: “People talk about ‘tracepoints’, but there are actually two mechanisms in the kernel. Internally, a tracepoint is a simple marker in the code, a hook to which a kernel function can be attached. What user space sees as a tracepoint is actually a ‘trace event’, which is a specific interface that is implemented using the internal tracepoints. Without trace events, there is no interface visible to user space” (LWN, 27 October 2017).

	Tracepoint (DECLARE_TRACE)	Trace event (TRACE_EVENT)
Declared with	DECLARE_TRACE / DECLARE_TRACE_CONDITION	TRACE_EVENT / DECLARE_EVENT_CLASS + DEFINE_EVENT
Generates	pass-0 artifacts + __DECLARE_TRACE inlines	all of that, plus the 7 stages, perf.h, bpf_probe.h
Record layout	none	struct trace_event_raw_<call>
format file	none	yes, from trace_event_fields_<call>[]
Visible in tracefs events/	no	yes
Enableable by echo 1 > .../enable	no	yes
Attachable by perf	no	yes, via the id as perf_event_attr.config
Attachable by BPF tracepoint:	no	yes
Attachable by BPF raw_tracepoint: / tp_btf:	yes	yes
In-kernel consumer via register_trace_<name>()	yes	yes

The two-tier structure of Linux static instrumentation. What it shows: the row that distinguishes the tiers is “visible in tracefs” — everything userspace-facing hangs off the trace-event layer, and the tracepoint layer is a kernel-internal hook. The insight to take: the raw_tracepoint row is the interesting one. A BPF raw tracepoint binds to the hook, not to the trace event, which is why a bare DECLARE_TRACE is still reachable from BPF and why raw tracepoints skip all record-building cost. Every trace event has a tracepoint; not every tracepoint is a trace event.

The record a trace event actually writes

When a trace event fires, trace_event_raw_event_<call>() reserves a slot in the per-CPU ring buffer and fills it. Every record begins with the same 8-byte common header, struct trace_entry, defined in include/linux/trace_events.h and registered as fields by trace_define_common_fields() in kernel/trace/trace_events.c:

packet-beta
0-15: "type (u16) — the event id, matches events/<sys>/<ev>/id"
16-23: "flags (u8) — irqs-off, need-resched, hardirq/softirq"
24-31: "preempt_count (u8) — holds preempt_count AND migrate_disable"
32-63: "pid (int) — the traced task's PID"

struct trace_entry, the 8-byte header on every ftrace record (v6.12). What it shows: exactly four fields, in this order, at these offsets — which is why every format file starts with common_type at offset 0, common_flags at 2, common_preempt_count at 3, and common_pid at 4 before any event-specific field. The insight to take: common_type is the event id, so the ring buffer is self-describing at the record level: a reader takes the u16, looks up the corresponding format, and knows how to parse the rest. The preempt_count byte is doing double duty — the v6.12 source comments it “Holds both preempt_count and migrate_disable” — so tools that decode it as a plain preemption depth are wrong on PREEMPT_RT.

After the header come the fixed fields, laid out exactly as written in TP_STRUCT__entry, followed by a flexible char __data[] tail for variable-length data. Variable-length fields do not sit inline; instead the fixed part holds a 32-bit locator:

packet-beta
0-15: "offset — byte offset from the START of the record to the data"
16-31: "length — the data's length in bytes"

The __data_loc locator word written by __dynamic_array / __string / __cpumask fields (v6.12). What it shows: a 32-bit word split into a 16-bit offset and a 16-bit length, exactly as decoded by stage 3’s accessors: __get_dynamic_array(f) is ((void *)__entry + (__entry->__data_loc_##f & 0xffff)) and __get_dynamic_array_len(f) is ((__entry->__data_loc_##f >> 16) & 0xffff). The insight to take: the 16-bit fields are a hard ceiling — a single dynamic field cannot exceed 65,535 bytes, and it cannot start more than 65,535 bytes into the record. The format file advertises these as type "__data_loc char[]" with size:4, which is the signal to a userspace parser that the four bytes are a locator rather than the data. The newer __rel_loc variant stores its offset relative to the end of the locator field itself rather than the start of the record, which makes a record relocatable — that is why BPF and trace_pipe_raw consumers must check which of the two an event uses.

Putting the pieces together, a sched_switch record on x86-64 looks like this:

 offset  size  field                       source
 ------  ----  --------------------------  -------------------------------
   0       2   common_type                 struct trace_entry.type
   2       1   common_flags                struct trace_entry.flags
   3       1   common_preempt_count        struct trace_entry.preempt_count
   4       4   common_pid                  struct trace_entry.pid
 ------  ----  --------------------------  -------------------------------
   8      16   prev_comm[TASK_COMM_LEN]    __array(char, prev_comm, 16)
  24       4   prev_pid                    __field(pid_t, prev_pid)
  28       4   prev_prio                   __field(int,   prev_prio)
  32       8   prev_state                  __field(long,  prev_state)
  40      16   next_comm[TASK_COMM_LEN]    __array(char, next_comm, 16)
  56       4   next_pid                    __field(pid_t, next_pid)
  60       4   next_prio                   __field(int,   next_prio)
 ------  ----  --------------------------  -------------------------------
  64       0   char __data[]               (empty: no dynamic fields here)

A sched_switch ring-buffer record, reconstructed from the v6.12 TP_STRUCT__entry and struct trace_entry. Drawn as an ASCII offset table rather than packet-beta because at 64 bytes a bit-accurate grid would be unreadable. What it shows: the whole record is fixed-size — TASK_COMM_LEN is a compile-time constant and there are no __string fields — so trace_event_get_offsets_sched_switch() returns zero and the reserve is a constant sizeof(*entry). The insight to take: these offsets are not guaranteed by anything; they are whatever the compiler chose for that struct on that architecture with that TASK_COMM_LEN. That is precisely why the format file exists and why a tool must read it rather than hard-code offsets. Note the natural alignment padding you would get if prev_state were int rather than long — the layout is architecture-dependent, and a 32-bit kernel produces a different table.

Uncertain

Verify: the exact offsets in the table above against a running v6.12 kernel’s events/sched/sched_switch/format. Reason: they were computed by hand from the v6.12 TP_STRUCT__entry assuming x86-64 natural alignment and TASK_COMM_LEN == 16; the kernel’s own Documentation/trace/events.rst example of a format file is demonstrably stale (it shows sched_wakeup with success and cpu fields and a common_tgid, none of which match the v6.12 sched_wakeup_template), so the docs cannot be used to cross-check. To resolve: cat /sys/kernel/tracing/events/sched/sched_switch/format on a 6.12 box. uncertain

The Stability Question — Two Tiers, One Honest Answer

Whether tracepoints are a stable userspace ABI is one of the more genuinely contested points in the kernel, and getting it right matters because tools depend on it. The honest answer has two tiers, and conflating them is the source of most of the confusion.

There is no formal guarantee. Tracepoints and trace-event format files are not enumerated in Documentation/ABI/stable. No document promises a tracepoint will exist next release, or that its fields will not change.

In practice, Linus Torvalds treats widely used tracepoints as covered by “don’t break userspace.” The settled-by-fiat position, per Corbet’s 2017 Kernel Summit report, is that “Torvalds wants to make a guarantee to user-space tools that works in 99% of the cases.” Rostedt — the ftrace maintainer, who wrote TRACE_EVENT() — disagrees with that policy: he “said he disagrees with that decision, but it doesn’t matter, since Torvalds has the final say.” The canonical breakage is powertop, which “broke some years ago when a variable was removed from a tracepoint”; Ted Ts’o’s takeaway from that episode was that “self-describing formats do not work as a solution to this problem,” because a tool can depend on information that is simply no longer present, or can ignore the format data entirely.

The 2017 discussion ended with a decision that shaped the next decade: rather than expand formal tracepoint guarantees, “support will be added to make it easy for an application to attach a BPF script to any function in the kernel, with access to that function’s arguments.” Torvalds explicitly said that if a popular script broke because a function was removed, “he would not see it as a regression that needs to be fixed” — but that it should be read as a signal that the kernel ought to expose that information properly. That is the origin of the modern fentry/BTF-based tracing that bpftrace and kprobes both sit on.

What actually changes, with a dated example

The two-tier answer is not hand-waving; it is visible in the source. Take the kernel’s single most-used tracepoint, sched:sched_switch:

Aspect	v5.13 – v5.17	v5.18 – v6.12
TP_PROTO	(bool preempt, struct task_struct *prev, struct task_struct *next) — 3 args	(bool preempt, struct task_struct *prev, struct task_struct *next, unsigned int prev_state) — 4 args
TP_ARGS	(preempt, prev, next)	(preempt, prev, next, prev_state)
TP_STRUCT__entry	prev_comm[], prev_pid, prev_prio, prev_state, next_comm[], next_pid, next_prio	byte-for-byte identical
format file / record layout	unchanged	unchanged
ftrace / perf / BPF tracepoint: consumers	keep working	keep working
BPF raw_tracepoint: / tp_btf: reading arg3	n/a (only 3 args existed)	new argument appears; a program compiled against the 3-arg signature is now reading a tracepoint whose bpf_raw_event_map.num_args is 4
In-tree register_trace_sched_switch() callers	3-arg probe	must be updated to a 4-arg probe or the build fails

Verified by curl against raw.githubusercontent.com/torvalds/linux/<tag>/include/trace/events/sched.h for v5.13, v5.15, v5.16, v5.17, v5.18, and v6.12 on 2026-09-04. What it shows: the in-kernel signature of sched_switch gained a fourth argument between v5.17 and v5.18, while its userspace-visible record layout did not change at all. The insight to take: this is the two-tier stability model made concrete. The trace event — the format file, the field names, the record — is what Torvalds’s guarantee actually protects, and it held. The tracepoint — the TP_PROTO signature that raw-tracepoint and in-kernel consumers bind to — is not protected in the same way and did change in a normal release. Choose your attach point accordingly: tracepoint:sched:sched_switch in bpftrace survived this; a hand-written raw_tp program with a hard-coded 3-argument context did not necessarily.

The mirror image of this argument — the case for tracepoints — is worked out in detail in bpftrace, which carries a verified example of the opposite failure: the out-of-line helpers __blk_account_io_start and __blk_account_io_done were folded into their static inline callers between v6.3 and v6.4, so both kprobe attach points simply ceased to exist in a point release, with no deprecation and no warning. The biolatency-kp.bt tool that used them silently attaches nothing on a modern kernel; the tracepoint-based biolatency.bt keeps working because block:block_bio_queue is part of the declared trace-event surface. Read the two examples together and the rule falls out: the tracepoint’s record layout is the most stable thing in Linux tracing, the tracepoint’s TP_PROTO is moderately stable, and a kernel function’s symbol is not stable at all.

Uncertain

Verify: that sched_switch’s TP_PROTO change in v5.18 broke no userspace consumer. Reason: the record layout is verified identical, and the trace-event path is therefore safe, but I did not enumerate the out-of-tree BPF tp_btf programs of the era to confirm none read a fourth argument’s absence. The change also predates the widespread use of BTF-relocated tracepoint arguments, which would mask it. To resolve: check libbpf’s and bcc’s git history around the v5.18 window for sched_switch compatibility shims. uncertain

Two practical rules follow. First, parse the format file, do not hard-code offsets — the whole point of stage 4 and the print fmt: line is that a tool can adapt to a layout change it did not anticipate. Second, treat obscure tracepoints as unstable. Well-established families (scheduler, block, syscalls, networking) are effectively frozen because too much depends on them; a tracepoint added last release in a driver has no such protection, and Rostedt’s own position is that it should not.

BPF Raw Tracepoints — Skipping the Marshalling

A BPF program can attach to a tracepoint in three different ways, and the difference between them is entirely about how much work happens before your program runs.

flowchart TB
    TP["trace_sched_switch(preempt, prev, next, prev_state)<br/>static key ON"]

    TP --> A["probe: trace_event_raw_event_sched_switch<br/>(the ftrace probe, stage 6)"]
    A --> A1["reserve ring-buffer slot<br/>run TP_fast_assign: memcpy comms, copy pids<br/>commit"]
    A1 --> A2["BPF program type<br/>BPF_PROG_TYPE_TRACEPOINT<br/>ctx = the FORMATTED record"]

    TP --> B["probe: perf_trace_sched_switch<br/>(trace/perf.h)"]
    B --> B1["perf_trace_buf_alloc<br/>perf_fetch_caller_regs<br/>run TP_fast_assign"]
    B1 --> B2["perf_trace_run_bpf_submit<br/>-> BPF via perf ring buffer"]

    TP --> C["probe: __bpf_trace_sched_switch<br/>(trace/bpf_probe.h)"]
    C --> C1["CAST_TO_U64 each arg<br/>bpf_trace_run4(link, a0,a1,a2,a3)"]
    C1 --> C2["BPF program type<br/>BPF_PROG_TYPE_RAW_TRACEPOINT<br/>ctx = the RAW ARGUMENTS as u64[]"]

    C2 --> D["tp_btf variant:<br/>same path, but BTF gives the<br/>args real types, so ctx->prev is a<br/>typed struct task_struct *"]

The three BPF attach paths onto one tracepoint (v6.12). What it shows: all three register an ordinary probe against the same struct tracepoint, but the ftrace and perf paths build a full formatted record before your program sees anything, while the raw path does nothing but widen each argument to u64. The insight to take: raw_tracepoint is strictly less work — no ring-buffer reserve, no memcpy of two 16-byte comm strings, no commit — which is why it is the default for aggregating tools. The cost is that you get untyped u64s, unless BTF is available, in which case tp_btf gives you the same raw path with the original pointer types restored. That is the modern default.

The raw binding is generated by include/trace/bpf_probe.h, which emits, per event class, a trampoline whose whole body is an argument widen-and-forward:

static notrace void
__bpf_trace_sched_switch(void *__data, bool preempt, struct task_struct *prev,
                         struct task_struct *next, unsigned int prev_state)
{
    bpf_trace_run4(__data, CAST_TO_U64(preempt), CAST_TO_U64(prev),
                           CAST_TO_U64(next),    CAST_TO_U64(prev_state));
}

and, per event, a discovery record placed in its own ELF section:

static union {
    struct bpf_raw_event_map event;
    btf_trace_sched_switch   handler;   /* typedef void (*)(void *, proto) — read by BTF */
} __bpf_trace_tp_map_sched_switch __used __section("__bpf_raw_tp_map") = {
    .event = {
        .tp        = &__tracepoint_sched_switch,
        .bpf_func  = __bpf_trace_sched_switch,
        .num_args  = 4,                 /* COUNT_ARGS(args) */
        .writable_size = 0,
    },
};

bpf_get_raw_tracepoint(name) in kernel/trace/bpf_trace.c searches that section (and the equivalent section in loaded modules) to resolve a name like sched_switch to this record. bpf_probe_register() then does two safety checks before attaching:

if (prog->aux->max_ctx_offset > btp->num_args * sizeof(u64))
    return -EINVAL;                     /* program reads past the last real argument */
if (prog->aux->max_tp_access > btp->writable_size)
    return -EINVAL;                     /* program writes to a non-writable tracepoint */
return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func, link);

The first check is the verifier’s answer to the TP_PROTO stability problem discussed above: a program that reads ctx[3] on a tracepoint whose num_args is 3 is rejected at load time with -EINVAL rather than reading garbage. The second exists for the small family of writable tracepoints (declared with DECLARE_TRACE_WRITABLE/DEFINE_EVENT_WRITABLE, used by LSM-style hooks) where a BPF program is permitted to modify a buffer the tracepoint passed it; __CHECK_WRITABLE_BUF_SIZE makes the compiler verify at build time that the declared size matches sizeof(*first_arg).

There is a hard structural limit worth knowing. bpf_probe.h defines __CAST1 through __CAST12 and comments: “tracepoints with more than 12 arguments will hit build error.” kernel/trace/bpf_trace.c correspondingly defines bpf_trace_run1 through bpf_trace_run12. A tracepoint with more than twelve arguments cannot be built on a CONFIG_BPF_EVENTS kernel. This is not a documented policy anywhere in Documentation/; it is a compile-time wall discoverable only by reading the header. The related program types are surveyed in Tracepoint BPF Programs and eBPF Tracing Program Types.

Defining a Tracepoint — Worked Example, With a Correction

Two files are involved. First, a header under include/trace/events/ declares the tracepoint:

/* include/trace/events/subsys.h */
#undef TRACE_SYSTEM
#define TRACE_SYSTEM subsys              /* groups events under events/subsys/ */
 
#if !defined(_TRACE_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
#define _TRACE_SUBSYS_H
#include <linux/tracepoint.h>
 
DECLARE_TRACE(subsys_eventname,                       /* unique, global name */
    TP_PROTO(int firstarg, struct task_struct *p),    /* probe signature */
    TP_ARGS(firstarg, p));                            /* argument names */
 
#endif /* _TRACE_SUBSYS_H */
 
/* This part must be OUTSIDE the header guard */
#include <trace/define_trace.h>

The TRACE_HEADER_MULTI_READ term in the guard condition is the mechanism that makes the multi-pass expansion possible: define_trace.h #defines it before each re-include, so the guard lets the body through again. The #include <trace/define_trace.h> must sit outside the guard for the same reason.

Second, in exactly one .c file, the tracepoint is instantiated and called:

/* subsys/file.c */
#define CREATE_TRACE_POINTS               /* flips the header into "generate" mode */
#include <trace/events/subsys.h>
 
void somefct(void) {
    ...
    trace_subsys_eventname(arg, task);    /* the call site — a nop when off */
    ...
}

Uncertain

This corrects a stale statement in the kernel’s own documentation, and in the previous revision of this note. Documentation/trace/tracepoints.rst instructs you to write an explicit DEFINE_TRACE(subsys_eventname); in the .c file, with one argument. That form has not compiled since v5.10: include/linux/tracepoint.h has defined #define DEFINE_TRACE(name, proto, args) — three arguments — from v5.10 onward (v5.9 and earlier had the one-argument form), because the static-call rework needs the prototype in order to emit __traceiter_<name> and the static-call trampoline. Verified by fetching include/linux/tracepoint.h and Documentation/trace/tracepoints.rst at v5.4, v5.5, v5.6, v5.7, v5.8, v5.9, v5.10, v5.15, v6.0, v6.6, v6.12 and v6.17 on 2026-09-04: the macro changed at v5.10, and the .rst still shows the one-argument form at v6.17, six years later. In practice you do not write DEFINE_TRACE by hand at all — define_trace.h emits it for you from the DECLARE_TRACE in the header, which is why nobody noticed. What remains true is that CREATE_TRACE_POINTS must appear in exactly one translation unit, or the linker sees multiple definitions of __tracepoint_subsys_eventname. Reason for flagging rather than silently fixing: the in-tree doc is authoritative-looking and wrong, so anyone checking this note against docs.kernel.org will see a contradiction. To resolve: read include/linux/tracepoint.h at any tag from v5.10 onward. uncertain

A consumer attaches a probe like so:

static void my_probe(void *data, int firstarg, struct task_struct *p) { /* ... */ }
/*              ^^^^ every probe takes a leading void *data, from data_proto */
 
register_trace_subsys_eventname(my_probe, NULL);    /* attach: 0->1 enables the key */
/* ... */
unregister_trace_subsys_eventname(my_probe, NULL);  /* detach: 1->0 disables the key */
tracepoint_synchronize_unregister();                /* wait out in-flight probes */

Note the leading void *data on the probe. __DECLARE_TRACE is invoked with data_proto = PARAMS(void *__data, proto), so every probe’s real signature is the tracepoint’s TP_PROTO with a private data pointer prepended. That pointer is the data you passed at registration, and it is per-registration: it is how the ftrace probe receives its struct trace_event_file * (which instance’s buffer to write to) while the BPF probe receives its struct bpf_raw_tp_link *, from the same probe array on the same tracepoint.

To export a tracepoint for use by modules, add EXPORT_TRACEPOINT_SYMBOL_GPL(name) — which, since v5.10, exports three symbols: __tracepoint_<name>, __traceiter_<name>, and the static-call key. net/core/net-traces.c is the canonical example, exporting kfree_skb, napi_poll, tcp_send_reset and a couple of dozen others this way.

Failure Modes and Subtleties

Calling a tracepoint where RCU isn’t watching. Probe callbacks run in an RCU read-side critical section; invoking trace_<name>() from the idle loop or from entry/exit code before RCU is re-enabled is a bug — the probe’s rcu_dereference would be unsafe. The historical workaround was a separate trace_<name>_rcuidle() variant using SRCU (still generated by __DECLARE_TRACE_RCU in v6.12, but only for built-in code — it is compiled out under #ifndef MODULE). On CONFIG_ARCH_WANTS_NO_INSTR architectures the rcuidle path now always warns and returns, because the entry code is sanitised; the path is on its way out. The CONFIG_LOCKDEP WARN_ONCE(!rcu_is_watching(), ...) at every call site is what catches violations, and it fires whether or not the tracepoint is enabled.

Tracepoints in header files bloat the kernel. trace_<name>() is a non-trivial inline; calling it from a widely included header inlines the whole dispatch machinery into every translation unit. tracepoint-defs.h spells out the fix: include tracepoint-defs.h (not tracepoint.h), declare with DECLARE_TRACEPOINT(foo_bar), test with tracepoint_enabled(foo_bar), and call a wrapper defined in a .c file. There is a second reason beyond size, given in the .rst: “tracepoints in header files can have side effects if a header is included from a file that has CREATE_TRACE_POINTS set” — the multi-pass machinery would try to define your tracepoint in someone else’s translation unit.

Forgetting tracepoint_synchronize_unregister() in a module. The classic use-after-free: a module unregisters its probe and immediately unloads while another CPU is mid-probe. Because release_probes() defers through both call_rcu and call_srcu, the window is real. This is the single most common tracepoint bug in out-of-tree code.

Assuming a probe may sleep. It may not. The probe runs inside preempt_disable_notrace(), in the caller’s context, possibly in interrupt or NMI context, possibly holding the caller’s locks. might_sleep() in a probe is a bug; so is unbounded work, since the probe’s cost lands directly in the traced path’s latency.

Double registration. Registering the same {probe, data} pair twice returns an error from func_add(). Use tracepoint_probe_register_prio_may_exist() when idempotent attach is genuinely wanted — but note the warn parameter exists precisely so that the ordinary path still WARN_ON_ONCEs, because a duplicate registration is usually a refcounting bug.

Expecting the argument list to be stable. See the stability section: TP_PROTO is a weaker contract than the format file. Bind to the trace event when you can.

Expecting a bare DECLARE_TRACE to show up in tracefs. It will not. If available_events does not list your tracepoint, check whether it was declared with TRACE_EVENT or with plain DECLARE_TRACE — the latter has no trace-event layer at all.

Alternatives and When to Choose Them

	Tracepoint	kprobe	fprobe / fentry	USDT
Placed by	kernel author, at compile time	you, at runtime	you, at runtime	application author, at compile time
Can instrument	only pre-declared sites	almost any kernel instruction, incl. mid-function	function entry/exit only	only pre-declared sites, in userspace
Disabled cost	one 5-byte nop	zero (nothing installed)	zero	one nop
Enabled cost	direct call, no trap	int3 trap, or optimised jmp	direct trampoline call	int3 trap into a uprobe
Argument access	typed, by name	raw registers, arch-specific	typed via BTF	typed by the probe’s declared args
Survives a kernel upgrade	usually — see the two-tier model	often not: symbols vanish or go static inline	same exposure as kprobe	tied to the application, not the kernel
Right for	production, always-on, known-in-advance	exploratory, one-off, “no hook exists”	fast BPF on a known function	userspace application internals

Choosing an instrumentation source. What it shows: the trade is placement-time against reach. A tracepoint can only be where someone already put one, but it is typed, cheap, and comparatively durable; a kprobe reaches anywhere but binds to an internal symbol. The insight to take: the “survives a kernel upgrade” row is the one that costs people real time. The bpftrace note carries the worked case — __blk_account_io_start going static inline between v6.3 and v6.4, silently removing a kprobe attach point that a widely used tool depended on. Gregg’s advice, quoted there, is the operating rule: “try to use the static types first, before the dynamic ones, so that your programs are more stable.”

The decision procedure in practice is short. Is there a tracepoint? Use it. Is there a tracepoint but the record-building cost is too high for the event rate? Use raw_tracepoint/tp_btf on the same hook. Is there no tracepoint but there is a stable exported function? Use fentry (with BTF) or a kprobe, and accept version fragility. Is the point of interest inside a function? A kprobe:function+offset is the only option, and it will break — kprobes covers why. Need the same thing in userspace? uprobes for arbitrary instructions, USDT for the static analogue of a tracepoint.

Production Notes

Tracepoints are the only instrumentation most organisations allow to run continuously. The static key is the reason: a fleet-wide kernel can carry every tracepoint compiled in, and enabling a handful costs a code-patch plus the per-event probe. The kprobes note’s overhead figures make the contrast concrete — a trapping kprobe is orders of magnitude more expensive per hit than a tracepoint’s direct call.

The _enabled() guard is not optional on hot paths. If a tracepoint’s arguments require walking a list, taking a lock, or formatting a string, that work happens before the call and therefore even when the tracepoint is off — unless it is wrapped in if (trace_x_enabled()). This is a routine source of “why did adding a disabled tracepoint cost 2%” regressions.

Enabling thousands of events at once is not free. echo 1 > events/enable performs a first-registration on every trace event, each of which is a 0→1 transition and therefore a live text patch. x86’s HAVE_JUMP_LABEL_BATCH batches the text_poke IPIs via arch_jump_label_transform_queue()/text_poke_finish() (see arch/x86/kernel/jump_label.c), which is why it completes in a reasonable time at all — but it is still a measurable stall.

Two consumers on one tracepoint is normal and cheap. ftrace writing to a tracefs buffer while a BPF program aggregates the same event costs exactly one extra hop into __traceiter_<name>. There is no second patch and no second dispatch. This is what makes it safe for a monitoring agent to hold a tracepoint open in its own tracefs instance while an engineer attaches bpftrace to the same hook interactively.

Prefer the trace event for anything you must maintain. Every field name in a format file is a promise the kernel community has, in practice, kept for a decade; a TP_PROTO is not. Tools that parse format at runtime — trace-cmd, perf, libtraceevent, bpftrace’s tracepoint: provider — inherit that durability. Tools that bake in offsets or argument counts do not.

See Also

• The TRACE_EVENT Macro — the seven-stage expansion, clause by clause; this note covers the tracepoint half, that one covers the trace-event half
• The Trace Event Subsystem — turns an enabled tracepoint into a ring-buffer record and a tracefs events/ entry
• Trace Event Format Files — the self-describing format file that makes a trace event parseable, and the reason not to hard-code offsets
• Static Keys and Tracepoint Patching — the jump-label mechanism that makes a disabled tracepoint a literal nop
• Static Keys and Code Patching — the general-purpose facility tracepoints motivated
• The tracefs Filesystem — where enabled tracepoints surface as files, and how instances/ lets consumers coexist
• Tracepoint BPF Programs — raw_tp/tp_btf programs that attach directly to a tracepoint’s raw arguments
• Syscall Tracepoints sys_enter and sys_exit — the generated tracepoint family on the syscall boundary, and a real user of regfunc
• kprobes — the dynamic counterpart; read its “Alternatives” section for the other side of the stability argument
• bpftrace — the tool most people reach tracepoints through, with the verified biolatency case study
• Static vs Dynamic Tracing — the framing this whole distinction sits inside
• Linux Tracing and Observability MOC — the parent map (§3, static instrumentation)