vault

Static vs Dynamic Tracing

12 min read

Static vs Dynamic Tracing

The deepest organizing distinction in Linux instrumentation is whether a probe point is static — placed by a developer at compile time and frozen into the binary — or dynamic — patched into already-running code at an arbitrary address. Static instrumentation (kernel tracepoints and userspace USDT, “Userspace Statically Defined Tracing”) is a stable, named interface that costs essentially nothing when disabled, because static-key patching turns it into a nop; its limitation is that you can only observe where a developer chose to place a hook. Dynamic instrumentation (kprobes/kretprobes in the kernel, uprobes in user space) can probe almost any instruction by overwriting it with a breakpoint or jump at runtime, giving total coverage — but it is tied to volatile internal symbols that can vanish or change meaning between kernel versions, and it costs more per hit. The two are not rivals: Brendan Gregg’s standing advice is to “use static tracepoints (tracepoints/USDT) instead of dynamic tracing (kprobes/uprobes) wherever possible. It’s often not possible, but do try” (Gregg, SCALE 2017). Crucially, both kinds feed the same downstream consumers — the trace-event ring buffers, perf, and eBPF — so the choice is about the probe’s properties, not the tooling.

Mental Model — Frozen-In versus Patched-In

Picture a compiled kernel function as a sequence of machine instructions. A static probe is a hook the author welded into that sequence before compilation: a TRACE_EVENT macro expands into a call site that, when the tracepoint is off, the compiler and runtime collapse to a single no-op instruction. The probe has a stable name (sched:sched_switch), a documented set of fields, and the kernel maintainers consider breaking it a regression. The cost: it exists only there, and adding a new one requires patching and rebuilding the kernel.

A dynamic probe is one you graft on at runtime. kprobes copies the target instruction aside and overwrites its first bytes with a breakpoint; when the CPU trips the breakpoint, the kernel traps into your handler (kprobes.rst, v6.12). You can do this to almost any address — functions the authors never imagined anyone would trace — without recompiling. The cost: you are attaching to an internal symbol with no API contract. If tcp_sendmsg gets renamed, inlined, or split in the next release, your probe silently stops matching or attaches to the wrong thing.

flowchart LR
  subgraph STATIC["Static — author-placed, compile time"]
    direction TB
    S1["Tracepoint<br/>(TRACE_EVENT macro)"]
    S2["USDT probe<br/>(nop + ELF note)"]
    SP["Property:<br/>stable name + ABI,<br/>nop when off,<br/>only where placed"]
  end
  subgraph DYN["Dynamic — you-placed, runtime"]
    direction TB
    D1["kprobe / kretprobe<br/>(breakpoint patch)"]
    D2["uprobe<br/>(binary text patch)"]
    DP["Property:<br/>any instruction,<br/>costlier per hit,<br/>tied to volatile symbols"]
  end
  CONS["Shared consumers:<br/>trace-event buffers · perf · eBPF"]
  S1 --> CONS
  S2 --> CONS
  D1 --> CONS
  D2 --> CONS

Static and dynamic instrumentation compared. What it shows: two families of sources with opposite trade-offs (left: stable/free/limited-coverage; right: arbitrary-coverage/costlier/unstable), both funnelling into the same set of consumers. The insight to take: the static-vs-dynamic choice is a property of the probe (stability, overhead, coverage), not of the tool — once a source fires, it is the identical trace-event/perf/eBPF machinery downstream.

How Static Instrumentation Works

A kernel tracepoint is created with the TRACE_EVENT macro (or the lower-level DECLARE_TRACE/DEFINE_TRACE pair). The tracepoints documentation describes the result as code that “provides a hook to call a function (probe) that you can provide at runtime,” toggled on or off (tracepoints.rst, v6.12). When the author writes trace_sched_switch(...) at the relevant spot, that call expands into something logically equivalent to “if this tracepoint is enabled, call every registered probe.” Probes attach with register_trace_<name>() and detach with unregister_trace_<name>(), with tracepoint_synchronize_unregister() required before a module exits so no in-flight probe is mid-execution (ibid.).

The magic that makes a disabled tracepoint free is the static key (jump label). The static-keys documentation explains the mechanism: in the disabled state the straight-line code path contains “a single atomic ‘no-op’ instruction (5 bytes on x86)”; flipping the key patches that no-op into a jump to the out-of-line probe-calling code (static-keys.rst, v6.12). The whole feature was born to make tracepoints cheap: “The original impetus came from reducing tracepoint overhead,” because tracepoints “are often dormant (disabled) and provide no direct kernel functionality” (ibid.). The tracepoints doc confirms the residual cost when off is merely “checking a condition for a branch” plus a little space — and with jump labels even that branch is elided (tracepoints.rst, v6.12). The full mechanism is covered in Static Keys and Tracepoint Patching.

TRACE_EVENT does more than place a hook — it also generates the trace-event machinery. Steven Rostedt, its author, designed it so one macro yields a callable tracepoint, a ring-buffer record layout, and a self-documenting format file, and deliberately decoupled from any single tracer: “TRACE_EVENT() is also used by perf, LTTng and SystemTap” (Rostedt, LWN 379903). This is why a static tracepoint is simultaneously readable by ftrace, perf, and eBPF. See The TRACE_EVENT Macro.

The userspace analogue is USDT (“Userspace Statically Defined Tracing”). A developer places a probe macro in application source; at compile time it becomes “a no-op assembly instruction at the probe site and writing an ELF note” recording the probe’s name and argument locations (LWN 753601). “The runtime overhead of an inactive probe is the cost of executing a no-op instruction” (ibid.) — exactly the same bargain as a kernel tracepoint, in user space. Big applications ship them: “MySQL, PostgreSQL, Java, and Node.js offer USDT probe support” (ibid.). See USDT Userspace Statically Defined Tracing.

How Dynamic Instrumentation Works

A kprobe instruments the kernel by surgery on live code. When you register one, the kernel “makes a copy of the probed instruction and replaces the first byte(s) of the probed instruction with a breakpoint instruction (e.g., int3 on i386 and x86_64)” (kprobes.rst, v6.12). When a CPU hits that breakpoint it traps, the kernel runs your pre_handler, then single-steps the saved copy of the original instruction out of line (so the original code stream is undisturbed), runs the post_handler, and returns. Single-stepping the displaced copy rather than the in-place instruction is what makes kprobes safe under concurrency.

A kretprobe (return probe) hooks a function’s exit. The kernel “saves a copy of the return address, and replaces the return address with the address of a ‘trampoline’”; when the function returns, “control passes to the trampoline and that probe is hit,” the return handler runs, and execution resumes at the saved address (kprobes.rst, v6.12). Because it must track concurrent invocations, you set maxactive; unset, it defaults to max(10, 2*NR_CPUS) (ibid.). See kretprobes.

To shrink overhead, kprobes can be optimized (CONFIG_OPTPROBES=y): instead of a trapping breakpoint, the kernel patches in “a jump instruction instead of a breakpoint instruction,” after safety checks confirm the jump target “lies entirely within one function,” nothing jumps into the optimized region, and each replaced instruction “can be executed out of line” (kprobes.rst, v6.12). Optimization is skipped if the probe has a post_handler. See Optimized kprobes and the Breakpoint to Jump Path.

The userspace counterpart is the uprobe, which patches the text of a target binary or library. The uprobe tracer doc states “Uprobe based trace events are similar to kprobe based trace events,” specified as p[:[GRP/][EVENT]] PATH:OFFSET [FETCHARGS] with a return-probe form using the r prefix (uprobetracer.rst, v6.12). Because you compute the byte offset, getting it wrong is dangerous: Brendan Gregg warns that “if you use the wrong address, say, mid-way through a multi-byte instruction, the target process will either crash or be in a corrupted state” (Gregg 2015). See uprobes.

The Three Trade-off Axes

Stability. This is the headline difference. A tracepoint is a maintained interface with a stable name and documented fields; a kprobe attaches to a raw internal symbol with no contract. Gregg states the consequence bluntly: “Dynamic tracing is an unstable API, so your programs will break if the code it’s instrumenting changes from one release to another” (Gregg, SCALE 2017). A bpftrace script built on tracepoint:syscalls:sys_enter_openat will keep working across kernels; the same script built on kprobe:do_sys_openat2 may break the moment that function is renamed, split, or inlined away.

Overhead. A disabled static probe is free (a nop). A dynamic probe is never quite free, because something was patched in. The kprobes doc gives concrete numbers (circa-2005 hardware, but the ratios still teach the lesson): an unoptimized kprobe hit is on the order of 0.5 microseconds, a return probe “typically take[s] 50-75% longer than a kprobe hit,” and a jump-optimized kprobe drops to “0.07 to 0.1 microseconds” (kprobes.rst, v6.12). When enabled, a tracepoint and an optimized kprobe are in the same ballpark; the static win is mainly that the disabled state is truly zero, so you can compile thousands in.

Coverage. Here dynamic wins decisively. A kprobe can “trap at almost any kernel code address” (kprobes.rst, v6.12) — every function, even ones with no tracepoint. The exceptions are the kprobe blacklist: code that implements kprobes itself, plus functions like do_page_fault and notifier_call_chain where probing “can cause a recursive trap (e.g. double fault),” and anything tagged NOKPROBE_SYMBOL() or __kprobes (ibid.). A tracepoint, by contrast, exists only where a developer placed one — comprehensive for well-instrumented subsystems (scheduler, block layer, networking), absent everywhere else.

Both Feed the Same Consumers

The unifying fact — and the reason this is one distinction rather than two separate toolchains — is that once a probe fires, the machinery downstream is identical. Dynamic probes are exposed as trace events through tracefs: you add a kprobe event by writing p:myprobe do_sys_open ... to /sys/kernel/tracing/kprobe_events, and “Unlike the tracepoint-based event, this can be added and removed dynamically, on the fly,” after which it behaves exactly like any static tracepoint event (kprobetrace.rst, v6.12). Uprobe events work the same way through uprobe_events (uprobetracer.rst, v6.12). From perf’s view both are just sources: perf_event_open(2)’s type field accepts PERF_TYPE_TRACEPOINT for the static ones, and dynamic probes register as dynamic PMUs reachable the same way (perf_event_open(2)). And eBPF attaches to all four (tracepoint, kprobe, uprobe, usdt) with the same program model. Julia Evans’ layered framing captures this: probes and tracepoints are both “data sources,” distinct from the “mechanisms for collecting data” that consume them (Evans 2017). The pipeline is detailed in The Trace Event Pipeline.

Worked Examples — The Same Question, Both Ways

# STATIC: the syscall has a maintained tracepoint. Stable across kernels.
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'
 
# DYNAMIC: probe the underlying kernel function directly. More coverage, less stable.
bpftrace -e 'kprobe:do_sys_openat2 { @[comm] = count(); }'
 
# DYNAMIC via raw tracefs (no eBPF): add a kprobe event, enable it, read it.
cd /sys/kernel/tracing
echo 'p:myopen do_sys_openat2' > kprobe_events     # add a dynamic probe as a trace event
echo 1 > events/kprobes/myopen/enable              # turn it on
cat trace_pipe                                      # read the stream
echo '-:myopen' > kprobe_events                     # remove it

Commentary: the first two commands answer the identical question (“which processes are opening files?”) from opposite sides of the static/dynamic line. The tracepoint: form rides a maintained interface and will survive kernel upgrades; the kprobe: form reaches a function with no tracepoint but is hostage to that symbol’s stability. The raw-tracefs block shows that a dynamic probe is registered dynamically (echo ... > kprobe_events) yet, once registered, is consumed through the exact same events/.../enable and trace_pipe files a static tracepoint uses — the consumer side does not know or care which family the source came from (kprobetrace.rst, v6.12).

Failure Modes

Dynamic probe attaches to the wrong thing — silently. A kprobe on a function that got renamed simply fails to attach (a loud error), but a probe on a function whose semantics changed while keeping its name attaches fine and reports subtly wrong data. There is no compiler to catch this; only knowledge of the kernel version. This is the core hazard Gregg’s “prefer static” rule exists to avoid.

Inlining erases dynamic targets. A small static helper the compiler inlines into its callers has no standalone symbol to probe, so a kprobe on it cannot attach even though the source function “exists.” Tracepoints survive inlining because the hook is anchored to a tracepoint object, not a function symbol.

Wrong uprobe offset corrupts the target. Because uprobes patch userspace text by raw offset, a miscalculated offset landing inside a multi-byte instruction crashes or corrupts the traced process (Gregg 2015). Symbol-resolving front-ends (bpftrace, perf probe) mitigate this by computing offsets for you.

Over-tracing a hot path. Even cheap probes hurt at high frequency: Gregg notes “you can trace too much, slowing the target process” (Gregg 2015). A kprobe on a function called millions of times per second multiplies the per-hit cost into real overhead — the argument for in-kernel aggregation via eBPF rather than streaming every event out.

When to Choose Which

Reach for static when the instrumentation exists and you need stability — production monitoring, long-lived scripts, dashboards, and anything that must survive kernel upgrades. Tracepoints in well-instrumented subsystems (scheduler, block I/O, networking, syscalls via Syscall Tracepoints sys_enter and sys_exit) and USDT in well-instrumented applications are the right default. Reach for dynamic when you are exploring — chasing a function with no tracepoint, narrowing down a bug, answering a one-off question — and you accept that the script is version-specific. In practice the workflow is often: explore with a kprobe/uprobe, and if the spot proves valuable, lobby for (or contribute) a real tracepoint so the next person gets a stable interface. The honest comparison of front-ends that consume both lives in bpftrace vs BCC vs ftrace.

See Also

Graph View

Backlinks