vault

ftrace Framework

14 min read

ftrace Framework

ftrace (“function tracer”) is the Linux kernel’s native, built-in tracing framework — the in-tree machinery that lets you observe what the kernel is doing without recompiling it, loading a module, or running any userspace agent. Despite the name, ftrace is far more than a function tracer: it is a collection of pluggable tracers (function call tracing, call-graph tracing, several latency tracers, hardware-noise tracers) sitting on top of a shared per-CPU ring buffer and the kernel’s trace-event subsystem, and it is the substrate that higher-level tools such as trace-cmd and the KernelShark GUI drive underneath. Its defining characteristic is that everything is controlled through files: you turn tracers on and off, set filters, and read results by writing to and reading from files under the tracefs mount at /sys/kernel/tracing (ftrace.rst, v6.12; trace.c header). The framework was written largely by Steven Rostedt and Ingo Molnar, growing out of the real-time preemption (PREEMPT_RT) latency tracers (trace.c copyright header).

This note describes ftrace as a framework — what it is, the pluggable-tracer model, the substrate it rests on, and how the pieces relate. The file-based control surface gets its own treatment in The tracefs Filesystem; the individual tracers (The Function Tracer, The Function Graph Tracer, ftrace Latency Tracers) and the low-level compiler hook (Dynamic ftrace and the mcount fentry Hook) are separate atoms. All facts here are pinned to Linux 6.12 LTS (the VERSION = 6 / PATCHLEVEL = 12 “Baby Opossum Posse” release, per the v6.12 Makefile).

Mental Model — One Buffer, Many Pluggable Tracers, All Driven Through Files

The cleanest way to think about ftrace is as a small operating system for tracing: there is one shared resource (a per-CPU ring buffer of trace records), a registry of tracers that each know how to fill that buffer with a particular kind of data, exactly one tracer active at a time (selected through the current_tracer file), and a file-based “system call” interface (tracefs) through which userspace drives everything. The trace-event subsystem — the static tracepoints compiled into the kernel — is a second, parallel producer that can write into the same buffers independently of which tracer is current.

flowchart TB
  subgraph PROD["Producers (fill the ring buffer)"]
    direction TB
    FN["function tracer<br/>(every fn entry)"]
    FG["function_graph tracer<br/>(entry + exit, call graph)"]
    LAT["latency tracers<br/>(irqsoff / wakeup / ...)"]
    OSN["osnoise / timerlat / hwlat<br/>(noise detection)"]
    TE["trace events<br/>(static tracepoints)"]
  end
  CT["current_tracer file<br/>(selects ONE active tracer)"]
  RB["per-CPU ring buffer<br/>(one buffer per CPU)"]
  subgraph CONS["Consumers (drain the buffer)"]
    TRACE["trace<br/>(snapshot, non-consuming)"]
    PIPE["trace_pipe<br/>(live, consuming stream)"]
    TOOLS["trace-cmd / KernelShark<br/>perf, bpftrace also read events"]
  end
  CT -->|activates one of| FN & FG & LAT & OSN
  FN & FG & LAT & OSN --> RB
  TE -->|independent of current_tracer| RB
  RB --> TRACE & PIPE
  TRACE & PIPE --> TOOLS

ftrace as a pluggable-tracer framework. What it shows: producers (the pluggable tracers, plus the always-available trace-events) write into a single per-CPU ring buffer; the current_tracer file picks exactly one tracer to be active; consumers drain the buffer either as a static snapshot (trace) or a live consuming stream (trace_pipe). The insight to take: the function-class tracers are mutually exclusive (one current_tracer at a time), but trace events are orthogonal and can be enabled alongside whatever tracer is current — which is why you can run, say, the function tracer and simultaneously record sched_switch events into the same buffer.

What ftrace Actually Is — A Collection of Tracers Plus the Event Machinery

It is a common misconception that “ftrace” means “the function tracer.” In the kernel’s own framing, ftrace is the framework, and the function tracer is just one tracer registered with it. The ftrace.rst documentation opens by calling ftrace “an internal tracer designed to help out developers and designers of systems to find what is going on inside the kernel,” and immediately notes that “the name ftrace was the name given by the function tracer, the original tracer,” but that ftrace “is now a framework of several assorted tracing utilities.”

Concretely, the framework comprises three intertwined pieces:

  1. A registry of pluggable tracers. Each tracer is a struct tracer registered into the kernel’s trace_types linked list via register_tracer(). When you write a name into current_tracer, the kernel walks trace_types looking for a matching name; an unknown name is rejected. (The duplicate-name guard in register_tracer() — for (t = trace_types; t; t = t->next) { if (strcmp(type->name, t->name) == 0) ... } — is visible in the v6.12 trace.c, and the same list is what tracing_set_tracer() searches when validating a current_tracer write.)

  2. A shared per-CPU ring buffer. Every tracer and every trace event writes records into a lockless, per-CPU ring buffer (Steven Rostedt’s ring_buffer implementation, distinct from the printk log buffer). Having one buffer per CPU is what keeps tracing scalable: a CPU writes to its own buffer with no cross-CPU locking. The tracer_tracing_on() / tracer_tracing_off() helpers gate writing by calling ring_buffer_record_on/off() on tr->array_buffer.buffer (v6.12 trace.c).

  3. The trace-event and dynamic-event machinery. Beyond the built-in tracers, ftrace hosts the trace-event subsystem (static tracepoints surfaced under events/) and the dynamic events — kprobe, uprobe, fprobe, and eprobe events you create at runtime by writing to control files. All of these deposit their records into the same ring buffers and are described by self-documenting format files.

The unifying idea — and the reason ftrace is pleasant to use over SSH on a machine with nothing installed — is that all of this is driven through files. There is no special binary, no library, no syscall you must call from C. echo, cat, and shell redirection are a complete ftrace client. This file-based control model is the subject of The tracefs Filesystem.

The current_tracer Pluggable-Tracer Model

The heart of the framework is the current_tracer file. Reading it shows the active tracer; writing a tracer’s name activates it. Per ftrace.rst: “This is used to set or display the current tracer that is configured. Changing the current tracer clears the ring buffer content as well as the ‘snapshot’ buffer.” That clear-on-switch behavior is important: switching tracers is destructive to whatever was already captured, so you read out a trace before changing tracers, not after.

Only one tracer of the function class is active at a time. The set of names you may write is exactly the set listed in available_tracers — itself just the contents of trace_types, filtered to those compiled into this kernel. The documentation describes available_tracers as “the different types of tracers that have been compiled into the kernel,” and notes that “the tracers listed here can be configured by echoing their name into current_tracer.”

The principal tracers, with their documented one-line descriptions (ftrace.rst, v6.12):

  • nop — “the ‘trace nothing’ tracer. To remove all tracers from tracing simply echo ‘nop’ into current_tracer.” This is the default. nop does not stop trace events from being recorded — it simply means no function-class tracer is plugged in — which is why you can enable events/ tracepoints with nop as the current tracer.
  • function — “Function call tracer to trace all kernel functions.” Records each kernel function’s entry. Covered in depth in The Function Tracer.
  • function_graph — “Similar to the function tracer except that the function tracer probes the functions on their entry whereas the function graph tracer traces on both entry and exit of the functions.” This produces the familiar indented, brace-matched call-graph output with per-function durations. Covered in The Function Graph Tracer.
  • The latency tracers — irqsoff (“Traces the areas that disable interrupts and saves the trace with the longest max latency”), preemptoff (“Similar to irqsoff but traces and records the amount of time for which preemption is disabled”), preemptirqsoff (the combination), and the wakeup family wakeup / wakeup_rt / wakeup_dl (max scheduling latency for, respectively, any task, real-time tasks, and SCHED_DEADLINE tasks). These are detailed in ftrace Latency Tracers.
  • The noise tracers — hwlat (“the Hardware Latency tracer … used to detect if the hardware produces any latency”), plus osnoise and timerlat, are covered in OSNoise and Timerlat Tracers.
  • Other built-ins — blk (“the block tracer … used by the blktrace user application”), mmiotrace, and branch (likely/unlikely tracing) round out the list, availability depending on CONFIG_* options.

A worked switch looks like this:

cd /sys/kernel/tracing
cat available_tracers        # -> function_graph function wakeup wakeup_rt ... nop
cat current_tracer           # -> nop   (default)
echo function > current_tracer   # plug in the function tracer
cat trace | head             # read the captured records (snapshot)
echo nop > current_tracer    # unplug; back to tracing nothing

Line by line: available_tracers enumerates what this kernel supports; current_tracer shows nop until you change it; writing function activates entry tracing of every kernel function (subject to set_ftrace_filter — see ftrace Filtering and Triggers); trace is read non-destructively; writing nop cleanly disables. Note that writing an unsupported name (e.g. a tracer not compiled in) fails with -EINVAL because tracing_set_tracer() cannot find it in trace_types.

How It Bottoms Out — The Compiler Hook and the Ring Buffer

The function-class tracers need a way to get control at the entry of every kernel function. They obtain it from a compiler-inserted hook. When the kernel is built with profiling enabled, the compiler emits a call to a thunk — historically mcount, on modern x86 __fentry__ — at the start of every traceable function (the -pg GCC option, refined to -pg -mfentry). At boot, dynamic ftrace rewrites every one of those call sites into a nop, so an un-traced kernel pays essentially nothing; enabling the function tracer patches the relevant nops back into calls to ftrace’s trampoline. The ftrace.rst documentation gestures at this: when dynamic ftrace is configured, “the code is dynamically modified … to disable calling of the function profiler (mcount). This lets tracing be configured in with practically no overhead in performance” (ftrace.rst, v6.12). The full mechanism — text patching, the trampoline, ftrace_ops, the safety dance with stop_machine / IPIs — is the subject of Dynamic ftrace and the mcount fentry Hook and is deliberately not re-explained here.

Once a tracer has control, it formats a record and commits it to the current CPU’s ring buffer. That buffer is sized by buffer_size_kb (per-CPU kilobytes), gated by tracing_on, and read out through trace (snapshot) or trace_pipe (consuming stream) — all of which live in The tracefs Filesystem.

Listing What Is Available — available_tracers and available_events

Two read-only files tell you what this particular kernel can do. available_tracers lists the function-class tracers compiled in (the names valid for current_tracer). available_events lists every static tracepoint compiled in, in subsystem:event form — per events.rst, “The events which are available for tracing can be found in the file /sys/kernel/tracing/available_events,” and the same events are also browsable as the directory tree under events/. The two are independent axes: available_tracers is the tracer dimension (what fills the buffer wholesale), available_events is the static instrumentation dimension (named hooks you toggle individually).

cat available_tracers
# function_graph function wakeup_dl wakeup_rt wakeup ... blk mmiotrace nop
 
wc -l available_events
# 1800+   (thousands of subsystem:event pairs, kernel-config dependent)
 
grep '^sched:' available_events
# sched:sched_switch
# sched:sched_wakeup
# ...

available_tracers is a short, space-separated list; available_events is one subsystem:event per line and is typically large (a stock distro kernel exposes well over a thousand). You enable a tracer by writing to current_tracer and an event by writing 1 to its events/<subsys>/<event>/enable or by appending subsys:event to set_event (see The tracefs Filesystem and Trace Event Format Files).

ftrace as a Substrate — What Drives It From Above

ftrace is rarely operated by hand in production; instead it is the engine under friendlier tools. trace-cmd is the canonical command-line front-end: it configures current_tracer, enables events, sets filters, starts/stops tracing, and extracts the binary ring-buffer contents into a .dat file, all by manipulating the very tracefs files described here. KernelShark is the GTK GUI that visualizes those .dat files as a timeline. Both are covered in trace-cmd and KernelShark. Because the trace-event ring buffers are a shared substrate, the same tracepoints ftrace exposes are also readable by perf (through perf_event_open) and by eBPF programs — one source, many consumers, the central theme of the Linux Tracing and Observability MOC.

Failure Modes and Common Misunderstandings

“I switched tracers and lost my data.” Writing to current_tracer clears the ring buffer (documented behavior). Always read trace before changing tracers.

“nop means tracing is off.” Not quite. nop means no function-class tracer is active, but trace events you enabled under events/ keep recording into the buffer. To actually stop all recording, write 0 to tracing_on (or disable the events). This trips people up constantly.

“My tracer name isn’t accepted.” If a name in your script isn’t in available_tracers, the kernel returns -EINVAL from tracing_set_tracer() because it isn’t in trace_types. The tracer may simply not be compiled into this kernel (e.g. function_graph requires CONFIG_FUNCTION_GRAPH_TRACER, the latency tracers require their respective CONFIG_*_TRACER options).

“available_tracers is suspiciously short.” It only lists compiled-in tracers. A minimal or hardened kernel may ship with nop and little else. The function tracer requires CONFIG_FUNCTION_TRACER; dynamic ftrace requires CONFIG_DYNAMIC_FTRACE.

“The buffer overran and I lost events.” The per-CPU ring buffer is fixed-size and, by default, overwriting (newest events evict oldest). High-rate function tracing on a busy kernel overruns small buffers in milliseconds. Increase buffer_size_kb, narrow the trace with set_ftrace_filter, or switch to a consuming trace_pipe read — see The tracefs Filesystem and ftrace Filtering and Triggers.

Alternatives and When to Choose Them

ftrace is the right tool when you want low-overhead, always-available, in-kernel tracing with zero setup — function call flow, call graphs, latency hot spots, or static tracepoint streams — on a machine where you cannot install anything. Its limits are that it ships raw event streams to userspace and does little in-kernel aggregation: if you want a histogram or a per-key counter without paying to copy every event out, bpftrace / BCC aggregating in BPF maps is the better fit (ftrace does have histogram triggers, covered in ftrace Filtering and Triggers, but they are less flexible than eBPF). For CPU profiling — “where are the cycles going?” — perf sampling and flame graphs are the standard, not ftrace. And for portable, structured, low-overhead production event tracing across kernel and userspace, LTTng is the dedicated framework. The honest summary: ftrace owns kernel function/latency tracing and the raw tracepoint stream; eBPF owns in-kernel aggregation and programmability; perf owns sampling/counting; they overlap at the shared trace-event substrate.

Production Notes

In practice ftrace is the first thing reached for during a live kernel investigation precisely because it needs nothing installed. A typical triage is: mount tracefs if needed, echo function_graph > current_tracer, narrow with echo <fn> > set_ftrace_filter, watch trace_pipe. For anything beyond ad-hoc poking, engineers use trace-cmd record/report so the capture is a portable .dat file analyzable later in KernelShark (trace-cmd and KernelShark).

Uncertain

Verify: the exact contents of available_tracers on a given running 6.12/6.18 box, and which tracers a particular distro kernel compiles in. Reason: available_tracers is entirely CONFIG_*-dependent and varies by build — the ftrace.rst list documents possible tracers, not a guaranteed set. To resolve: cat /sys/kernel/tracing/available_tracers on the target kernel and cross-check each name against its CONFIG_*_TRACER option in /boot/config-$(uname -r). uncertain

Uncertain

Verify: the precise list of trace_create_file() calls in init_tracer_tracefs() / tracer_init_tracefs() and the exact tracing_set_tracer() validation path in v6.12. Reason: kernel/trace/trace.c is ~10k lines and the relevant functions were beyond the fetched/parsable excerpt; the register_tracer() duplicate-name loop and the trace_types list were read directly, but the file-creation list and the current_tracer-write -EINVAL path are inferred from documented behavior plus the visible trace_types traversal rather than read line-by-line. To resolve: read kernel/trace/trace.c around tracing_set_tracer and init_tracer_tracefs in the v6.12 tree. uncertain

See Also

Graph View

Backlinks