ftrace (“function tracer”) is the Linux kernel’s native, built-in tracing framework — the in-tree machinery that lets you observe what the kernel is doing without recompiling it, loading a module, or running any userspace agent. Despite the name, ftrace is far more than a function tracer: it is a collection of pluggable tracers (function call tracing, call-graph tracing, several latency tracers, hardware-noise tracers) sitting on top of a shared per-CPU ring buffer and the kernel’s trace-event subsystem, and it is the substrate that higher-level tools such as trace-cmd and the KernelShark GUI drive underneath. Its defining characteristic is that everything is controlled through files: you turn tracers on and off, set filters, and read results by writing to and reading from files under the tracefs mount at /sys/kernel/tracing (ftrace.rst, v6.12; trace.c header). The framework was written largely by Steven Rostedt and Ingo Molnar, growing out of the real-time preemption (PREEMPT_RT) latency tracers (trace.c copyright header).
This note describes ftrace as a framework — what it is, the pluggable-tracer model, the __fentry__ mechanism it bottoms out on, and how the pieces relate. Scope, stated up front, because the neighbours are close: the filesystem — mounting tracefs, the instances/ mechanism, per_cpu/, and the file-creation machinery in fs/tracefs/ — belongs to The tracefs Filesystem; this note covers only the subset of files that select and steer a tracer. The individual tracers get their own atoms (The Function Tracer, The Function Graph Tracer, ftrace Latency Tracers, OSNoise and Timerlat Tracers), as does the ftrace_ops/trampoline layer (Dynamic ftrace and the mcount fentry Hook) and the BPF direct-call path (fentry fexit and BPF Trampolines). What is covered here that they do not repeat is the framework question: how a tracer is registered, how one becomes current, and why an untraced kernel function costs a single nop.
All source facts are pinned to Linux 6.12 — the VERSION = 6 / PATCHLEVEL = 12 “Baby Opossum Posse” release, per the v6.12 Makefile — because that is this vault’s pin, not because it is the only long-term kernel. As of 2026-09-12, kernel.org/releases.json lists mainline 7.3-rc2, latest stable 7.2.5, and five longterm branches: 6.18.51, 6.12.109, 6.6.156, 6.1.187, 5.15.220. The machine used for the demonstrations below runs 7.1.13 (a stable, not a longterm, release), so where its output is quoted it is labelled as such and the pinned-source claims are kept separate from it.
Mental Model — One Buffer, Many Pluggable Tracers, All Driven Through Files
The cleanest way to think about ftrace is as a small operating system for tracing: there is one shared resource (a per-CPU ring buffer of trace records), a registry of tracers that each know how to fill that buffer with a particular kind of data, exactly one tracer active at a time (selected through the current_tracer file), and a file-based “system call” interface (tracefs) through which userspace drives everything. The trace-event subsystem — the static tracepoints compiled into the kernel — is a second, parallel producer that can write into the same buffers independently of which tracer is current.
flowchart TB
subgraph PROD["Producers (fill the ring buffer)"]
direction TB
FN["function tracer<br/>(every fn entry)"]
FG["function_graph tracer<br/>(entry + exit, call graph)"]
LAT["latency tracers<br/>(irqsoff / wakeup / ...)"]
OSN["osnoise / timerlat / hwlat<br/>(noise detection)"]
TE["trace events<br/>(static tracepoints)"]
end
CT["current_tracer file<br/>(selects ONE active tracer)"]
RB["per-CPU ring buffer<br/>(one buffer per CPU)"]
subgraph CONS["Consumers (drain the buffer)"]
TRACE["trace<br/>(snapshot, non-consuming)"]
PIPE["trace_pipe<br/>(live, consuming stream)"]
TOOLS["trace-cmd / KernelShark<br/>perf, bpftrace also read events"]
end
CT -->|activates one of| FN & FG & LAT & OSN
FN & FG & LAT & OSN --> RB
TE -->|independent of current_tracer| RB
RB --> TRACE & PIPE
TRACE & PIPE --> TOOLS
ftrace as a pluggable-tracer framework. What it shows: producers (the pluggable tracers, plus the always-available trace-events) write into a single per-CPU ring buffer; the current_tracer file picks exactly one tracer to be active; consumers drain the buffer either as a static snapshot (trace) or a live consuming stream (trace_pipe). The insight to take: the function-class tracers are mutually exclusive (one current_tracer at a time), but trace events are orthogonal and can be enabled alongside whatever tracer is current — which is why you can run, say, the function tracer and simultaneously record sched_switch events into the same buffer.
What ftrace Actually Is — A Collection of Tracers Plus the Event Machinery
It is a common misconception that “ftrace” means “the function tracer.” In the kernel’s own framing, ftrace is the framework, and the function tracer is just one tracer registered with it. The ftrace.rst documentation opens by calling ftrace “an internal tracer designed to help out developers and designers of systems to find what is going on inside the kernel,” and immediately notes that “the name ftrace was the name given by the function tracer, the original tracer,” but that ftrace “is now a framework of several assorted tracing utilities.”
Concretely, the framework comprises three intertwined pieces:
A registry of pluggable tracers. Each tracer is a struct tracer registered into the kernel’s trace_types linked list via register_tracer(). When you write a name into current_tracer, the kernel walks trace_types looking for a matching name; an unknown name is rejected. (The duplicate-name guard in register_tracer() — for (t = trace_types; t; t = t->next) { if (strcmp(type->name, t->name) == 0) ... } — is visible in the v6.12 trace.c, and the same list is what tracing_set_tracer() searches when validating a current_tracer write.)
A shared per-CPU ring buffer. Every tracer and every trace event writes records into a lockless, per-CPU ring buffer (Steven Rostedt’s ring_buffer implementation, distinct from the printk log buffer). Having one buffer per CPU is what keeps tracing scalable: a CPU writes to its own buffer with no cross-CPU locking. The tracer_tracing_on() / tracer_tracing_off() helpers gate writing by calling ring_buffer_record_on/off() on tr->array_buffer.buffer (v6.12 trace.c).
The trace-event and dynamic-event machinery. Beyond the built-in tracers, ftrace hosts the trace-event subsystem (static tracepoints surfaced under events/) and the dynamic events — kprobe, uprobe, fprobe, and eprobe events you create at runtime by writing to control files. All of these deposit their records into the same ring buffers and are described by self-documenting format files.
The unifying idea — and the reason ftrace is pleasant to use over SSH on a machine with nothing installed — is that all of this is driven through files. There is no special binary, no library, no syscall you must call from C. echo, cat, and shell redirection are a complete ftrace client. This file-based control model is the subject of The tracefs Filesystem.
The current_tracer Pluggable-Tracer Model
The heart of the framework is the current_tracer file. Reading it shows the active tracer; writing a tracer’s name activates it. Per ftrace.rst: “This is used to set or display the current tracer that is configured. Changing the current tracer clears the ring buffer content as well as the ‘snapshot’ buffer.” That clear-on-switch behavior is important: switching tracers is destructive to whatever was already captured, so you read out a trace before changing tracers, not after.
Only one tracer of the function class is active at a time. The set of names you may write is exactly the set listed in available_tracers — itself just the contents of trace_types, filtered to those compiled into this kernel. The documentation describes available_tracers as “the different types of tracers that have been compiled into the kernel,” and notes that “the tracers listed here can be configured by echoing their name into current_tracer.”
The complete set of tracers is enumerable from the source rather than from documentation: every one is a struct tracer with a .name field, handed to register_tracer(). Grepping the v6.12 kernel/trace/ tree for those definitions yields fifteen tracers, and no more. The descriptions below pair each .name found in the source with the behaviour ftrace.rst documents for it.
Every memory-mapped I/O access a driver makes. “MMIO accesses are recorded via page faults. Just before __ioremap() returns, the mapped pages are marked as not present” (mmiotrace.rst). Built to reverse-engineer binary drivers — and brutal: it “takes all but one CPU off-line during run-time activation”
Block-layer request lifecycle events; “used by the blktrace user application.”
The fifteen ftrace tracers in Linux 6.12, enumerated by grepping kernel/trace/ for struct tracer definitions and their register_tracer() calls. What it shows: the tracers sort into four families — do-nothing (nop), function-flow (function, function_graph), max-latency (irqsoff…wakeup_dl), and noise/hardware (hwlat, osnoise, timerlat) — plus two special-purpose outliers (mmiotrace, branch) and one that exists to feed a userspace tool (blk). The insight to take: the latency tracers are a fundamentally different shape from the function tracers. They do not stream records; they keep only the single worst observation and the trace that produced it, which is why they interact with tracing_max_latency and tracing_thresh and why reading them is a different workflow.
Every gate in that column was read from kernel/trace/Kconfig at v6.12, and two of them are default n — IRQSOFF_TRACER (“Interrupts-off Latency Tracer”, depends on TRACE_IRQFLAGS_SUPPORT) and PREEMPT_TRACER (“Preemption-off Latency Tracer”, depends on PREEMPTION). BRANCH_TRACER is stricter still: it depends on TRACE_BRANCH_PROFILING, which a production kernel never enables because it instruments every likely() in the tree.
This has a concrete, checkable consequence. On the Fedora 44 kernel this note was researched on, /boot/config-7.1.13-200.fc44.x86_64 contains:
$ grep -E 'IRQSOFF_TRACER|PREEMPT_TRACER|PROFILE_ANNOTATED' /boot/config-$(uname -r)# CONFIG_IRQSOFF_TRACER is not set# CONFIG_PREEMPT_TRACER is not set# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
so irqsoff, preemptoff, preemptirqsoff and branchdo not exist on this machine, while nop, function, function_graph, wakeup, wakeup_rt, wakeup_dl, hwlat, osnoise, timerlat, mmiotrace and blk all do (CONFIG_NOP_TRACER, CONFIG_FUNCTION_TRACER, CONFIG_FUNCTION_GRAPH_TRACER, CONFIG_SCHED_TRACER, CONFIG_HWLAT_TRACER, CONFIG_OSNOISE_TRACER, CONFIG_TIMERLAT_TRACER, CONFIG_MMIOTRACE, CONFIG_BLK_DEV_IO_TRACE are all =y). Note also that preemptirqsoff is registered only under #if defined(CONFIG_IRQSOFF_TRACER) && defined(CONFIG_PREEMPT_TRACER) (trace_irqsoff.c:747) — it is not its own config symbol, which is why you cannot turn it on by itself. This is the general lesson about available_tracers: it is a property of the build, not of the kernel version.
Three names that look like tracers but are not: traceon, traceoff, stacktrace, dump and cpudump also appear as .name fields in trace_functions.c (lines 915–935), but they are struct ftrace_func_command entries — filter commands you attach to a function, not things you write into current_tracer. They are tabulated under Filtering below.
A worked switch looks like this:
cd /sys/kernel/tracingcat available_tracers # -> function_graph function wakeup wakeup_rt ... nopcat current_tracer # -> nop (default)echo function > current_tracer # plug in the function tracercat trace | head # read the captured records (snapshot)echo nop > current_tracer # unplug; back to tracing nothing
Line by line: available_tracers enumerates what this kernel supports; current_tracer shows nop until you change it; writing function activates entry tracing of every kernel function (subject to set_ftrace_filter — see ftrace Filtering and Triggers); trace is read non-destructively; writing nop cleanly disables. Note that writing an unsupported name (e.g. a tracer not compiled in) fails with -EINVAL because tracing_set_tracer() cannot find it in trace_types — the full set of ways that write can fail is tabulated in What a current_tracer Write Actually Does below. (The comment lines show the shape of each file’s contents, not output captured from this machine; see the permission note further down.)
How It Bottoms Out — The __fentry__ Call Site, Nopped at Boot
This is the mechanism that makes ftrace possible at all, and it is worth understanding precisely rather than by analogy, because “the kernel patches itself” undersells how carefully it is done.
Step 1 — The compiler plants a call at every function entry
Building the kernel with -pg makes GCC emit a call to a profiling thunk at the top of every function; adding -mfentry moves that call to the very first instruction, before the frame pointer is set up, and renames the thunk __fentry__. On x86-64 the emitted instruction is a 5-byte near call with a 32-bit relative displacement. MCOUNT_INSN_SIZE is defined as exactly 5 /* sizeof mcount call */ (asm/ftrace.h:12), and MCOUNT_ADDR is ((unsigned long)(__fentry__)).
You can see this shape without any kernel access at all, because the same GCC flags work on ordinary userspace code. Compiling the hot_work() function from uprobes with gcc -O2 -pg -mfentry on this Fedora 44 box (GCC 16.2.1) gives:
$ objdump -d demo_pg | sed -n '/<hot_work>:/,+2p'0000000000400550 <hot_work>: 400550: e8 6b fe ff ff call 4003c0 <__fentry__@plt> 400555: 48 85 ff test %rdi,%rdi
e8 6b fe ff ff — opcode e8 (call rel32) plus a four-byte displacement — is byte-for-byte the shape ftrace finds at the top of every traceable kernel function. The function’s real first instruction, test %rdi,%rdi, sits five bytes in.
Step 2 — The build records every call site, and boot nops them all out
The build records the address of each of those call sites into a dedicated section, which the linker gathers between the symbols __start_mcount_loc and __stop_mcount_loc. At boot, ftrace_init() (ftrace.c:7738) reads that array:
count = __stop_mcount_loc - __start_mcount_loc;if (!count) { pr_info("ftrace: No functions to be traced?\n"); goto failed;}pr_info("ftrace: allocating %ld entries in %ld pages\n", count, DIV_ROUND_UP(count, ENTRIES_PER_PAGE));ret = ftrace_process_locs(NULL, __start_mcount_loc, __stop_mcount_loc);
ftrace_process_locs() allocates one struct dyn_ftrace record per call site, and ftrace_update_code() then calls ftrace_nop_initialize() → ftrace_init_nop() on each, which overwrites the 5-byte call __fentry__ with a 5-byte nop (ftrace.c:3722, :2818). Steven Rostedt states the design in one sentence: “when this configuration is enabled all of the mcount caller locations are stored and at boot time are converted into NOPs. These locations are saved and used to enable tracing when the function tracer is activated” (LWN, January 2010).
The nop is not arbitrary. ftrace_nop_replace() returns x86_nops[5] (arch/x86/kernel/ftrace.c:66), and on 64-bit BYTES_NOP5 is 0x0f,0x1f,0x44,0x00,0x00 (asm/nops.h:60) — a single 5-byte nopl 0x0(%rax,%rax,1), chosen so the CPU retires one instruction rather than five one-byte 0x90s. Disassembling those exact bytes confirms what the site looks like before and after:
$ printf '\x0f\x1f\x44\x00\x00\xe8\x00\x00\x00\x00' > bytes.bin$ objdump -D -b binary -m i386:x86-64 -M att bytes.bin | tail -3 0: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) <- untraced: this is what runs 5: e8 00 00 00 00 call 0xa <- traced: patched back to a call
That is the whole performance story. An untraced kernel function pays exactly one 5-byte nop — not a branch, not a test of a global flag, not an indirect call. This is why CONFIG_FUNCTION_TRACER=y is safe to ship in a distribution kernel, and why ftrace.rst can claim dynamic ftrace “lets tracing be configured in with practically no overhead in performance.”
The count is observable on a live system even without tracefs access, because ftrace_init() prints it. On this box:
$ journalctl -k | grep ftraceSep 10 16:46:31 beast kernel: ftrace: allocating 64148 entries in 252 pagesSep 10 16:46:31 beast kernel: ftrace: allocated 252 pages with 6 groups
64,148 call sites were nopped out during this boot — one per traceable function in this kernel image — and the struct dyn_ftrace array tracking them cost 252 pages, a little under 1 MiB. That array is the permanent memory price of dynamic ftrace; the code patching itself is free at rest.
flowchart TB
subgraph BUILD["Build time"]
GCC["gcc -pg -mfentry<br/>emits 'call __fentry__'<br/>as insn #0 of every function"]
REC["call-site addresses collected<br/>into the __mcount_loc section"]
GCC --> REC
end
subgraph BOOT["Boot — ftrace_init()"]
CNT["count = __stop_mcount_loc - __start_mcount_loc<br/>(64,148 on this box)"]
ALLOC["ftrace_process_locs():<br/>one struct dyn_ftrace per site<br/>(252 pages)"]
NOP["ftrace_init_nop() on each:<br/>e8 xx xx xx xx -> 0f 1f 44 00 00"]
CNT --> ALLOC --> NOP
end
subgraph RUN["echo function > current_tracer"]
SEL["ftrace_replace_code():<br/>walk dyn_ftrace records,<br/>select those matching set_ftrace_filter"]
Q["text_poke_queue() each one<br/>0f 1f 44 00 00 -> e8 -> ftrace trampoline"]
FIN["text_poke_finish()"]
SEL --> Q --> FIN
end
REC --> CNT
NOP -.->|"kernel now runs at<br/>full speed, one nop per fn"| SEL
The three-stage life of an ftrace call site. What it shows: the compiler plants a call, the boot code turns all of them into nops while recording where they are, and enabling a tracer turns the selected ones back into calls. The insight to take: the expensive work — finding every function entry — happens once, at build and boot. At steady state, enabling tracing on a hundred functions means patching a hundred 5-byte sequences, which is why narrowing with set_ftrace_filter first means the enable step patches only the sites you asked for rather than all 64,148. (The kernel’s own text_poke_bp_batch() comment confirms the scale is real — it calls cond_resched() every 256 sites specifically because “function tracing can enable thousands of places that need to be updated” and the softlockup watchdog would otherwise fire.)
Step 3 — Patching live kernel text without stopping the machine
Rewriting instructions in a running SMP kernel is the dangerous part: another CPU may be executing, or about to fetch, the very bytes you are changing. There is a generic fallback that solves this by brute force — ftrace_run_stop_machine() (ftrace.c:2927), reached through the __weak default arch_ftrace_update_code():
x86 does not use it.arch/x86/kernel/ftrace.c provides its own arch_ftrace_update_code() (line 256) which calls ftrace_modify_all_code() directly, and x86’s ftrace_replace_code() batches the work through text_poke_queue() / text_poke_finish(). This is a correction worth making loudly, because a great many write-ups — including older ones that were accurate at the time — still say ftrace uses stop_machine() on x86. It does not, and the kernel says so in as many words. From the kerneldoc on text_poke_bp_batch() (alternative.c:2233):
Modify multi-byte instruction by using int3 breakpoint on SMP. We completely avoid stop_machine() here, and achieve the synchronization using int3 breakpoint.
The protocol it describes is a three-phase commit against concurrent instruction fetch:
Phase
Action
Why it is safe
1
Write a one-byte int3 (0xcc) over the first byte of every site to be patched, then text_poke_sync()
A one-byte store is atomic with respect to instruction fetch. Any CPU that arrives now traps into poke_int3_handler(), which emulates the intended new instruction (int3_emulate_call, int3_emulate_jmp, int3_emulate_ret, int3_emulate_jcc) instead of executing garbage
2
Write bytes 2..N of each new instruction, then sync
No CPU can be executing those bytes — they are all parked behind the int3 on byte 1
3
Replace the int3 with the real first byte, then sync
The instruction becomes whole and live in one atomic store
The text_poke_bp_batch() protocol. What it shows: how the kernel swaps a 5-byte instruction on a running SMP system without pausing any CPU. The insight to take: the int3 is used as a temporary trap door, not as instrumentation. A CPU that hits it mid-patch is handed the new instruction’s semantics by the exception handler and carries on. This is the same text_poke machinery behind static keys, jump labels and alternatives — ftrace is a client of it, not the owner.
Two details in that function are pure ftrace accommodations. text_poke_bp_batch() calls cond_resched() with an explicit comment naming the workload: “Function tracing can enable thousands of places that need to be updated. This can take quite some time, and with full kernel debugging enabled, this could cause the softlockup watchdog to trigger. This function gets called every 256 entries added to be patched.” And ftrace_replace_code() on x86 runs two passes over all the dyn_ftrace records — the first verifying that every site currently contains exactly the bytes ftrace believes are there (ftrace_verify_code(), which bails out via ftrace_bug() on mismatch), the second doing the queuing. Verify-everything-then-patch-everything means a corrupted or unexpectedly-modified call site aborts the whole update rather than leaving the kernel half-patched.
Once a tracer has control, it formats a record and commits it to the current CPU’s ring buffer. That buffer is sized by buffer_size_kb (per-CPU kilobytes), gated by tracing_on, and read out through trace (snapshot) or trace_pipe (consuming stream) — all of which live in The tracefs Filesystem.
Listing What Is Available — available_tracers and available_events
Two read-only files tell you what this particular kernel can do. available_tracers lists the function-class tracers compiled in (the names valid for current_tracer). available_events lists every static tracepoint compiled in, in subsystem:event form — per events.rst, “The events which are available for tracing can be found in the file /sys/kernel/tracing/available_events,” and the same events are also browsable as the directory tree under events/. The two are independent axes: available_tracers is the tracer dimension (what fills the buffer wholesale), available_events is the static instrumentation dimension (named hooks you toggle individually).
The illustrative shapes are:
cat available_tracers# a short, space-separated list, e.g.# function_graph function wakeup_dl wakeup_rt wakeup hwlat osnoise timerlat blk mmiotrace nopwc -l available_events# one 'subsystem:event' pair per line; a stock distro kernel exposes well over a thousandgrep '^sched:' available_events# sched:sched_switch# sched:sched_wakeup
Not executed on this host — illustrative, not captured
The three commands above show the format of each file, not output read from this machine. /sys/kernel/tracing is mode 0700 here, so an unprivileged process cannot read either file. The exact failing command:
This is a permission wall, not a missing feature: /proc/mounts shows tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0, and the kernel config has the tracers built in. The tracer list predicted for this kernel from its /boot/config-* file is given in the tracer table above; no ftrace session is reproduced anywhere in this note, and every console block that is real output is labelled as such. To resolve: run the commands as root and paste the result. uncertain
available_tracers is a short, space-separated list; available_events is one subsystem:event per line and is typically large. You enable a tracer by writing to current_tracer and an event by writing 1 to its events/<subsys>/<event>/enable or by appending subsys:event to set_event (see The tracefs Filesystem and Trace Event Format Files).
A third listing is the one this note’s mechanism section already touched: available_filter_functions is neither of these axes. It is the userspace view of the dyn_ftrace array built at boot — one line per nopped-out __fentry__ call site — which is why its length tracks the ftrace: allocating N entries line in the kernel log rather than any CONFIG list. On the box above that would be on the order of 64,148 entries, minus the ones ftrace declines to trace (its own internals, notrace-annotated functions, and anything in .init.text that has already been freed).
The function_graph Pairing — Why Two Hooks, Not One
The difference between function and function_graph is not cosmetic, and it explains a whole class of “why does my trace look like that” questions. function registers a single callback that fires at entry. function_graph registers a fgraph_ops with two callbacks — .entryfunc = trace_graph_entry and .retfunc = trace_graph_return (trace_functions_graph.c:289–290) — and gets the return hook the same way a kretprobe does: by hijacking the return address on the stack and substituting a kernel trampoline. On x86-64 that trampoline is return_to_handler (arch/x86/kernel/ftrace_64.S:348), which calls ftrace_return_to_handler() to pop the saved address. The real return address lives on a per-task shadow stack of struct ftrace_ret_stack frames — FGRAPH_FRAME_SIZE bytes each, indexed by task->curr_ret_stack (fgraph.c).
sequenceDiagram
autonumber
participant K as Kernel function do_sys_open()
participant FT as ftrace trampoline
participant FG as function_graph ops
participant SS as Per-task return-address stack
participant RB as Ring buffer
K->>FT: patched 'call' at the __fentry__ site
FT->>FG: entryfunc = trace_graph_entry()
FG->>SS: push the real return address, record depth
FG->>K: overwrite the on-stack return address with return_to_handler
FG->>RB: emit a graph ENTRY record (func, depth)
Note over K: the function body runs, calling deeper functions
K->>FT: function returns — lands on return_to_handler instead
FT->>FG: retfunc = trace_graph_return()
FG->>SS: pop the saved return address, compute elapsed time
FG->>RB: emit a graph RETURN record (func, depth, duration)
FG->>K: resume at the real return address
How function_graph gets both ends of a call. What it shows: the entry hook comes from the same patched __fentry__ call site the plain function tracer uses, but the exit hook is manufactured by rewriting the return address and keeping the original on a per-task stack. The insight to take: three consequences follow. Durations exist only because the two records are paired, so function_graph is the tracer to reach for when you want to know how long something took. The shadow stack has a finite depth, which is why max_graph_depth exists. And any code that reads its own return address — __builtin_return_address(), some unwinders — sees the trampoline while function_graph is active, which is the source of a small set of genuinely weird bugs.
The pairing is what produces the familiar brace-matched output; ftrace.rst shows it (v6.12, “Graph tracer” section):
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | sys_open() {
0) | do_sys_open() {
0) | getname() {
0) | kmem_cache_alloc() {
0) 1.382 us | __might_sleep();
0) 2.478 us | }
Read the duration column carefully: a line ending in { has no duration because the function has not returned yet; the matching } line carries the total. A function with no traced callees is collapsed to a single line ending in ; with its duration inline — that is print_graph_entry_leaf() (trace_functions_graph.c:705) noticing that the entry and return records are adjacent. So __might_sleep(); 1.382 us and kmem_cache_alloc() { ... } 2.478 us are the same information in two different layouts.
The Control Surface — Which Files Belong to the Tracer
Scope boundary
The tracefs Filesystem owns the filesystem: how tracefs is mounted, the instances/ mechanism that gives each consumer its own buffer set, per-CPU directories, and the file-creation machinery in fs/tracefs/. This note owns the tracer. What follows is only the subset of files that select, steer and drain a tracer, listed here so the mechanism above is usable — not a tracefs reference.
Selects the one active tracer. Writing it clears the ring buffer and the snapshot buffer — read your trace first. An unknown name is rejected with -EINVAL.
available_tracers
read-only
The names legal for current_tracer — i.e. trace_types, filtered to what was compiled in. Build-dependent, as shown above.
tracing_on
read/write
0 stops all recording into the buffer without changing current_tracer. The real off switch.
trace
read
Non-consuming snapshot, human-formatted. Reading it twice gives the same data.
trace_pipe
read
Consuming live stream — records are removed as you read, and the read blocks when the buffer is empty.
buffer_size_kb
read/write
Per-CPU buffer size in KiB. The total is this times the CPU count.
buffer_percent
read/write
How full a buffer must get before a blocked trace_pipe reader is woken.
trace_options / options/
read/write
Per-tracer and global output toggles (funcgraph-abstime, sym-offset, function-fork, …).
tracing_max_latency
read/write
Used only by the latency tracers: the current maximum, resettable by writing 0.
tracing_thresh
read/write
Record only latencies above this threshold (µs).
set_ftrace_filter
read/write
The allow-list of functions to trace; also accepts filter commands (below).
set_ftrace_notrace
read/write
The deny-list. Wins over set_ftrace_filter if a function is in both.
set_graph_function
read/write
For function_graph: trace these functions and everything they call.
set_graph_notrace
read/write
Suppress tracing from entry to exit of these functions — the way to hide a noisy subtree.
set_ftrace_pid / set_event_pid
read/write
Restrict to specific tasks. With the function-fork option set, children are added and exited PIDs removed automatically.
available_filter_functions
read-only
Every function ftrace can trace — i.e. the dyn_ftrace records built at boot.
available_filter_functions_addrs
read-only
Same plus the patch-site address, which “can differ from /proc/kallsyms address” (the __fentry__ site is five bytes into a function whose kallsyms entry is its start).
dyn_ftrace_total_info
read-only
“The number of functions that have been converted to nops and are available to be traced” — the userspace view of the boot-time count printed in dmesg.
enabled_functions
read-only
Every function that currently has a callback attached, with flag letters: R = callback wants saved registers, I = it may modify regs->ip, D = a non-ftrace (BPF) direct trampoline, O = ops located above the entry point, M = this function was ever ip-modified or direct-called (never cleared).
touched_functions
read-only
Same format, but every function ever traced. grep ' M ' touched_functions finds everything that was ever live-patched or BPF-attached.
max_graph_depth
read/write
Caps function_graph recursion depth.
function_profile_enabled
read/write
Turns on per-function call counts (and, with function_graph, time) readable under trace_stat/function<cpu>.
trace_clock
read/write
local (default), global, counter, x86-tsc, … — the timestamp source.
trace_marker
write
Userspace writes a string straight into the kernel’s buffer, interleaved with kernel events. The one place a userspace program can annotate a kernel trace.
The tracer-facing tracefs files. What it shows: the surface splits cleanly into four groups — select a tracer, steer what it records, size and gate the buffer, drain it. The insight to take:current_tracer and tracing_on are not the same switch, and conflating them is the single most common ftrace mistake. current_tracer=nop silences the function tracers but leaves every enabled trace event recording; tracing_on=0 silences everything at the buffer.
Filtering — What Actually Gets Traced
Filtering is where ftrace stops being a firehose. set_ftrace_filter accepts four syntactic forms, and it is worth knowing all of them because the naive one is painfully slow.
Form
Example
Notes
Exact name
echo sys_read > set_ftrace_filter
> replaces the list, >> appends, echo > clears it
glob(7) pattern
echo 'hrtimer_*' > set_ftrace_filter
<m>*, *<m>, *<m>*, <m1>*<m2> — always quote, or the shell expands it against your cwd
Index number
echo 5 > set_ftrace_filter
“A number (starting with ‘1’) written will instead select the same corresponding at the line position of the available_filter_functions file”
Negation
echo '!sys_read' >> set_ftrace_filter
Removes an entry (or a command)
The performance gap between the first two forms is not subtle. Rostedt measured it: grep sched available_filter_functions > set_ftrace_filter “took several seconds,” because “each function name written into the filter file will be processed individually” against all ~24,000 available functions — “over 300 * 24331 (7,299,300) comparisons.” A glob is matched once per available function instead (LWN, January 2010). The index form (ftrace.rst) exists for the same reason: “since processing strings can be quite expensive and requires a check of all functions registered to tracing, instead an index can be written into this file.”
On top of the function selector, set_ftrace_filter accepts commands in the form <function>:<command>:<parameter>. These are the struct ftrace_func_command entries mentioned earlier, and they turn the filter file into a tiny event-driven control language:
Command
Form
What it does
mod
write*:mod:ext3
Restrict the pattern to one module. Supports module globbing: !*:mod:* disables all module functions but keeps the kernel, *write*:mod:!* restricts to the kernel only, *write*:mod:*snd* matches a module glob
traceoff
__schedule_bug:traceoff:5
Stop recording when this function is hit, at most N times (omit N for no limit)
traceon
<fn>:traceon
The inverse — start recording on a hit
snapshot
native_flush_tlb_others:snapshot:1
Take a snapshot of the buffer when this function is hit
enable_event
try_to_wake_up:enable_event:sched:sched_switch:2
Turn a trace event on when this function runs. The tracepoint is activated immediately but kept in “soft” disabled mode until the trigger fires
disable_event
schedule:disable_event:sched:sched_switch
The inverse
dump
<fn>:dump
Dump the whole ring buffer to the console on a hit — for cases where the machine will not survive to be read normally
cpudump
<fn>:cpudump
Same, but only this CPU’s buffer
stacktrace
<fn>:stacktrace
Record a stack trace on each hit
The set_ftrace_filter command language. What it shows: the filter file is not just a list — it is where you attach conditional behaviour to function hits without writing any code. The insight to take:<fn>:traceoff is the single most useful entry in this table. It is how you capture the events leading up to a failure on a machine that is about to become unusable: arm the tracer, set traceoff on the function that signals the bug, and the buffer freezes with the last N microseconds before it. Commands are cumulative and removed by prefixing ! — and !<fn>:traceoff:0 (with the count) removes a counted command while !<fn>:traceoff removes an uncounted one, which is an easy thing to get wrong.
ftrace as a Substrate — What Drives It From Above
ftrace is rarely operated by hand in production; instead it is the engine under friendlier tools. trace-cmd is the canonical command-line front-end: it configures current_tracer, enables events, sets filters, starts/stops tracing, and extracts the binary ring-buffer contents into a .dat file, all by manipulating the very tracefs files described here. KernelShark is the GTK GUI that visualizes those .dat files as a timeline. Both are covered in trace-cmd and KernelShark. Because the trace-event ring buffers are a shared substrate, the same tracepoints ftrace exposes are also readable by perf (through perf_event_open) and by eBPF programs — one source, many consumers, the central theme of the Linux Tracing and Observability MOC.
flowchart TB
subgraph HOOK["Kernel-side hooks (the instrumentation)"]
FE["__fentry__ call sites<br/>patched nop <-> call"]
TP["static tracepoints<br/>(compiled-in)"]
KP["kprobes / uprobes / fprobes<br/>(dynamic)"]
end
subgraph FT["ftrace framework — THIS NOTE"]
TR["struct tracer registry<br/>(trace_types, current_tracer)"]
RBUF["per-CPU ring buffer"]
TR --> RBUF
end
TFS["tracefs — The tracefs Filesystem note<br/>files, instances/, per_cpu/"]
subgraph UP["Consumers"]
TC["trace-cmd / KernelShark"]
PERF["perf — via perf_event_open()"]
BPF["BPF programs — via bpf() + trampolines"]
end
FE --> TR
TP --> RBUF
KP --> RBUF
TP --> PERF
TP --> BPF
FE --> BPF
RBUF --> TFS
TFS --> TC
Where the ftrace framework sits. What it shows: the instrumentation hooks at the bottom are shared, not owned by ftrace. Tracepoints feed ftrace’s ring buffer, perf’s per-event buffers, and BPF programs equally; the __fentry__ call sites feed both ftrace’s tracers and BPF trampolines. The insight to take: three notes divide this picture and none of them owns the whole thing. This note owns the middle box — the tracer registry and how a tracer gets control. The tracefs Filesystem owns the file layer above it. Tracepoints, kprobes, uprobes and fentry fexit and BPF Trampolines own the hooks below it. “ftrace vs perf vs eBPF” is a question about which consumer you pick, not about which instrumentation exists.
Failure Modes and Common Misunderstandings
“I switched tracers and lost my data.” Writing to current_tracerclears the ring buffer (documented behavior). Always read trace before changing tracers.
“nop means tracing is off.” Not quite. nop means no function-class tracer is active, but trace events you enabled under events/ keep recording into the buffer. To actually stop all recording, write 0 to tracing_on (or disable the events). This trips people up constantly.
“My tracer name isn’t accepted.” If a name in your script isn’t in available_tracers, the kernel returns -EINVAL from tracing_set_tracer() because it isn’t in trace_types. The tracer may simply not be compiled into this kernel (e.g. function_graph requires CONFIG_FUNCTION_GRAPH_TRACER, the latency tracers require their respective CONFIG_*_TRACER options).
“available_tracers is suspiciously short.” It only lists compiled-in tracers. A minimal or hardened kernel may ship with nop and little else. The function tracer requires CONFIG_FUNCTION_TRACER; dynamic ftrace requires CONFIG_DYNAMIC_FTRACE.
“echo function_graph > current_tracer returns Device or resource busy.” You have a trace_pipe reader open somewhere. tracing_set_tracer() checks tr->trace_ref and returns -EBUSY with the source comment “If trace pipe files are being read, we can’t change the tracer” (trace.c:6244). The same -EBUSY also appears if a conditional snapshot is armed and you are switching to a tracer that uses the max-latency buffer.
“This tracer works at the top level but not in my instance.”trace_ok_for_array() gates some tracers to the top-level buffer only; a write inside instances/<name>/ returns -EINVAL for those. The instances/ mechanism itself belongs to The tracefs Filesystem.
“The first echo into current_tracer after boot was slow.” Expected. Step 2 of the write path expands the ring buffer from its minimal boot size to trace_buf_size across all CPUs the first time a tracer is selected.
“available_filter_functions doesn’t contain a function I can see in /proc/kallsyms.” Two different lists. kallsyms enumerates symbols; available_filter_functions enumerates __fentry__ call sites ftrace recorded at boot. Functions marked notrace, ftrace’s own internals, anything inlined away, and .init.text code freed after boot are in the first list and not the second. Relatedly, available_filter_functions_addrs prints the patch-site address, which the documentation warns “can differ from /proc/kallsyms address.”
“The buffer overran and I lost events.” The per-CPU ring buffer is fixed-size and, by default, overwriting (newest events evict oldest). High-rate function tracing on a busy kernel overruns small buffers in milliseconds. Increase buffer_size_kb, narrow the trace with set_ftrace_filter, or switch to a consuming trace_pipe read — see The tracefs Filesystem and ftrace Filtering and Triggers.
Alternatives and When to Choose Them
ftrace is the right tool when you want low-overhead, always-available, in-kernel tracing with zero setup — function call flow, call graphs, latency hot spots, or static tracepoint streams — on a machine where you cannot install anything. Its limits are that it ships raw event streams to userspace and does little in-kernel aggregation: if you want a histogram or a per-key counter without paying to copy every event out, bpftrace / BCC aggregating in BPF maps is the better fit (ftrace does have histogram triggers, covered in ftrace Filtering and Triggers, but they are less flexible than eBPF). For CPU profiling — “where are the cycles going?” — perf sampling and flame graphs are the standard, not ftrace. And for portable, structured, low-overhead production event tracing across kernel and userspace, LTTng is the dedicated framework. The honest summary: ftrace owns kernel function/latency tracing and the raw tracepoint stream; eBPF owns in-kernel aggregation and programmability; perf owns sampling/counting; they overlap at the shared trace-event substrate.
What a current_tracer Write Actually Does
The second half of the pluggable-tracer model is the validation path, and it is worth spelling out because three of its failure modes are things people hit and misdiagnose. Reading tracing_set_tracer() (trace.c:6244) line by line, a write to current_tracer:
Step
Code
Consequence for you
1
mutex_lock(&trace_types_lock)
Tracer switches are serialised globally
2
if !tr->ring_buffer_expanded, __tracing_resize_ring_buffer(tr, trace_buf_size, RING_BUFFER_ALL_CPUS)
The first tracer you ever select allocates the full ring buffer. Until then the kernel keeps a minimal one, which is why the first echo into current_tracer after boot is slower than later ones
3
for (t = trace_types; t; t = t->next) if (strcmp(t->name, buf) == 0) break; then if (!t) ret = -EINVAL
An unrecognised name fails with -EINVAL. This is the linked-list walk the framework is built around
4
if (t == tr->current_trace) goto out
Re-selecting the current tracer is a no-op — notably it does not clear the buffer
5
if t->use_max_tr and tr->cond_snapshot, ret = -EBUSY
A conditional snapshot armed by a trigger blocks switching to a latency tracer
6
if (system_state < SYSTEM_RUNNING && t->noboot) → warn and ignore
Some tracers are refused from the kernel command line: “Tracer ‘%s’ is not allowed on command line, ignored”
7
if (!trace_ok_for_array(t, tr)) ret = -EINVAL
“Some tracers are only allowed for the top level buffer” — they cannot be set inside an instances/ directory
8
if (tr->trace_ref) ret = -EBUSY
“If trace pipe files are being read, we can’t change the tracer.” A cat trace_pipe left running in another terminal makes every current_tracer write fail with -EBUSY
9
tr->current_trace->reset(tr), then tr->current_trace = &nop_trace
The old tracer is reset and nop is installed transiently before the new tracer is armed, so nothing races against a half-configured tracer
The current_tracer write path in v6.12. What it shows: four distinct rejection paths — two -EINVAL, two -EBUSY — plus a lazily expanded ring buffer. The insight to take:-EBUSY from step 8 is the one that wastes people’s afternoons. echo function_graph > current_tracer returning “Device or resource busy” almost always means a forgotten trace_pipe reader, not a broken kernel. Kill it and the write succeeds.
Production Notes
ftrace is the first thing reached for during a live kernel investigation precisely because it needs nothing installed. A typical triage is: echo function_graph > current_tracer, narrow with echo <fn> > set_ftrace_filter, watch trace_pipe. For anything beyond ad-hoc poking, engineers use trace-cmd record/report so the capture is a portable .dat file analyzable later in KernelShark (trace-cmd and KernelShark).
The clearest published account of why this matters in production is Brendan Gregg’s “Ftrace: The hidden light switch” (LWN, August 2014), written from a Netflix disk-I/O inflation incident on a Cassandra cluster. His framing is the one to internalise: “You may think, as I did, that analyzing the Linux kernel is like venturing through a dark dungeon: without the addition of advanced tracers like SystemTap, there’s much that can’t be seen, and can only be inferred. However, I’ve recently found hidden switches that turn on some bright lights, strategically placed by Steven Rostedt and others since the 2.6.27 release.”
The operational argument is the load-bearing part. His tools “use the /sys/kernel/debug ftrace facilities” and “work on Netflix’s existing servers (which often run Linux 3.2 with security patches) without any other addition to the system, and without requiring kernel debuginfo.” That is the real production case for ftrace and it has not changed: it is the tracer that is already there, on a machine you are not allowed to install packages on, running a kernel too old or too locked down for anything else. In that incident the decisive step was ftrace’s in-kernel stack-trace-on-tracepoint capability — tracing block:block_rq_insert with stacks and a filter, which surfaced a call chain running filemap_fault → ra_submit → __do_page_cache_readahead and identified readahead inflation, not a cache-hit regression.
Two operational habits follow from the mechanism section. First, narrow before you enable: writing function as current_tracer patches all ~64,000 call sites and will overrun a default-sized buffer on a busy machine in milliseconds; setting set_ftrace_filter first means only the sites you asked for get patched, and Rostedt notes that “just having a couple of hundred functions enabled is hardly noticeable” (LWN, January 2010). Second, arm traceoff rather than watching: echo '<fn>:traceoff' > set_ftrace_filter freezes the buffer at the moment a condition is hit, which is the only workable approach for events that are rare, fast, or fatal.
See Also
The tracefs Filesystem — the file-based control surface that is ftrace’s user interface (current_tracer, trace, events/, instances/)
The Function Tracer — the original tracer: entry tracing of every kernel function
uprobes — the userspace counterpart, registered through this framework’s uprobe_events file; its int3-and-single-step trick is the mirror image of the __fentry__ patching here
kprobes — the kernel dynamic probe, which under CONFIG_KPROBES_ON_FTRACE reuses the very __fentry__ call sites described above instead of planting its own breakpoint