The tracefs Filesystem
tracefs is the special-purpose pseudo-filesystem that is the user interface to the kernel’s ftrace tracing infrastructure. It is a small, RAM-backed virtual filesystem — modelled on
debugfsand written by Steven Rostedt in 2014 — whose files are not real data but control knobs and live views into the in-kernel tracer (fs/tracefs/inode.c, v6.12). Its top-level comment states it plainly: “tracefs is the file system that is used by the tracing infrastructure.” You configure tracing by writing to files (echo function > current_tracer), and you observe results by reading files (cat trace). The canonical mount point is/sys/kernel/tracing— a tracefs instance the kernel auto-creates whenever any ftrace option is configured in; the older/sys/kernel/debug/tracingpath (underdebugfs) is a legacy compatibility location, not the modern home (ftrace.rst, v6.12). This file-based design is what makes ftrace usable from a bare shell over SSH with no tooling installed:echoandcatare a complete client.
This note covers the control surface — why the filesystem was split out of debugfs in 2015, where it mounts, the everything-is-a-file model, how the inodes are actually manufactured (including the just-in-time eventfs layer behind events/), the file taxonomy, the destructive-vs-non-destructive read distinction, the ring buffer the files drain, per-instance tracing, and the permission model. The framework it controls is ftrace Framework; the static instrumentation it exposes under events/ is Tracepoints; the individual tracers and the filtering/trigger machinery live in The Function Tracer and ftrace Filtering and Triggers.
Version pin. Facts are pinned to Linux 6.12, a maintained long-term-support (LTS) release — the VERSION = 6 / PATCHLEVEL = 12 tree per the v6.12 Makefile. Mainline has since moved into the 7.x series (the machine these commands were run on is a Fedora 44 box on 7.1.8-200.fc44.x86_64), so anything that changed after v6.12 is dated explicitly where it appears. All source quotations below were read with curl from the v6.12 tag during this write-up on 2026-09-04; every cross-version claim names the tags that were compared.
Why tracefs Exists — The 2015 Split from debugfs
For its first six years ftrace had no filesystem of its own. Steven Rostedt’s own 2009 LWN introduction to the framework opens by telling readers exactly where to go: “Currently the API to interface with Ftrace is located in the Debugfs file system. Typically, that is mounted at /sys/kernel/debug” (Rostedt, Debugging the kernel using Ftrace — part 1, LWN, 2009). debugfs — Greg Kroah-Hartman’s general-purpose “dump whatever you like here” pseudo-filesystem — was the path of least resistance: it already existed, it already had debugfs_create_file() helpers, and ftrace needed somewhere to hang a few dozen control knobs.
That arrangement had two distinct problems, and Rostedt named both of them in the commit message of tracefs: Add new tracefs file system (4282d606, authored 2015-01-20, merged for v4.1). The first is a security and blast-radius problem:
“One thing is that in order to access the tracing infrastructure, you need to mount debugfs. As that includes debugging from all sorts of sub systems in the kernel, it is not considered advisable to mount such an all encompassing debugging system. Having the tracing system in its own file systems gives access to the tracing sub system without needing to include all other systems.” — commit 4282d60689d4, torvalds/linux
This is the argument that matters most in production. debugfs is an unstructured grab-bag: every subsystem that wants a debugging knob puts one there, with no review of what it exposes, and much of it is raw hardware register access or driver-internal state that has repeatedly been the subject of security advisories. Mounting debugfs to get at ftrace meant mounting all of it. The directory itself is mode 0700 and root-owned, so this is not a “any user can read it” hole; it is a privilege-amplification and attack-surface hole. A process or container that you want to give tracing to — a monitoring agent, a delegated group, a debug sidecar — could not be given tracing without being given the entire kernel debugging surface. Splitting tracefs out means the administrator can mount exactly the tracing tree and nothing else.
The second problem is mechanical, and is the reason instances/ exists in the shape it does:
“Another problem with tracing using the debugfs system is that the instances use mkdir to create sub buffers. debugfs does not support mkdir from userspace so to implement it, special hacks were used. By controlling the file system that the tracing infrastructure uses, this can be properly done without hacks.”
(Provenance note: both quotations are verbatim from the commit’s own message, read via GitHub’s .patch endpoint. The commit list, SHAs and dates were independently cross-checked against https://github.com/torvalds/linux/commits/v4.1/fs/tracefs.atom, which returns the same three fs/tracefs commits on 2015-02-03 with the same 40-character SHAs.)
debugfs is a filesystem that the kernel populates; userspace cannot create anything in it. But ftrace’s per-instance buffers are created by mkdir, which is a userspace mkdir(2) syscall reaching a filesystem’s ->mkdir inode operation. Owning the filesystem meant tracefs could implement ->mkdir and ->rmdir honestly (see the instances/ section below), instead of bolting a synthetic directory handler onto debugfs.
timeline title tracefs, from debugfs hack to its own filesystem 2008 : ftrace merged (v2.6.27), control files created in debugfs 2009 : Rostedt LWN series documents /sys/kernel/debug/tracing as THE interface 2015-01-20 : commit 4282d606 adds fs/tracefs — 570 insertions in total : commit 8434dc93 converts the tracing facility over to tracefs : commit f76180bc auto-mounts tracefs on debugfs/tracing for old tools : commit cc31004a adds the /sys/kernel/tracing mount point : commit eae47358 moves instance mkdir/rmdir into tracefs 2015-04 : all of the above ship in Linux v4.1 2023 : eventfs merged - events/ inodes become just-in-time, ~9 MB to ~4.5 MB 2024-11 : v6.12 LTS - the version this note is pinned to; debugfs automount still unconditional 2025 : v6.17 adds CONFIG_TRACEFS_AUTOMOUNT_DEPRECATED and a removed-in-2030 boot warning
The evolution of the tracing control surface. What it shows: tracefs was not designed up front — it was extracted from debugfs in a single January 2015 patch series (five commits, all merged for v4.1), and has been slowly shedding its debugfs compatibility ever since. The insight to take: the debugfs path is a compatibility shim with an announced end date, not an alternative interface; anything you write today should target /sys/kernel/tracing.
The whole extraction is remarkably small. fs/tracefs/inode.c arrived at 522 lines, plus a 41-line include/linux/tracefs.h and two lines in include/uapi/linux/magic.h — 570 insertions total, per the commit’s own diffstat. That is the entire cost of giving tracing its own filesystem. fs/Makefile gained one line, and note which symbol gates it:
obj-$(CONFIG_DEBUG_FS) += debugfs/
obj-$(CONFIG_TRACING) += tracefs/tracefs is built whenever CONFIG_TRACING is set — which any ftrace option selects — and is entirely independent of CONFIG_DEBUG_FS (fs/Makefile, v6.12, lines 119–120). That independence is the security win: you can build a kernel with tracing and without debugfs.
The filesystem’s magic number, also added by that commit, is a small piece of self-documentation:
#define TRACEFS_MAGIC 0x74726163Read as ASCII, 0x74 0x72 0x61 0x63 is t r a c (include/uapi/linux/magic.h, v6.12, line 77). This is what statfs(2) returns in f_type, and it is how perf and trace-cmd tell a genuine tracefs mount from a debugfs one — a distinction they had to learn in the same patch series (perf tools: Do not check debugfs MAGIC for tracing files, 5693c926).
Resolving the Canonical Mount — /sys/kernel/tracing, Not Debugfs
This point is worth settling first because it is routinely confused. The canonical, modern mount is /sys/kernel/tracing, a tracefs filesystem. The kernel creates the mount point itself at boot: tracefs_init() in fs/tracefs/inode.c (v6.12) calls sysfs_create_mount_point(kernel_kobj, "tracing") — that is, it asks sysfs to reserve a directory named tracing under the kernel kobject (/sys/kernel/), giving you /sys/kernel/tracing — and then registers the filesystem type with register_filesystem(&trace_fs_type). The ftrace.rst documentation confirms the userspace view: “When tracefs is configured into the kernel (which selecting any ftrace option will do) the directory /sys/kernel/tracing will be created,” and the explicit mount command is:
mount -t tracefs nodev /sys/kernel/tracingOn a modern systemd distribution this mount already exists, so you rarely run that command by hand. systemd ships a dedicated unit for it, and reading it on the machine this note was written on shows the real-world mount options:
# systemctl cat sys-kernel-tracing.mount (systemd, Fedora 44)
[Unit]
Description=Kernel Trace File System
Documentation=https://docs.kernel.org/trace/ftrace.html
DefaultDependencies=no
ConditionVirtualization=!lxc # never inside an LXC container
ConditionPathExists=/sys/kernel/tracing
ConditionCapability=CAP_SYS_RAWIO # skip if we lack raw-I/O privilege
Before=sysinit.target
[Mount]
What=tracefs
Where=/sys/kernel/tracing
Type=tracefs
Options=nosuid,nodev,noexecThree of those lines are worth reading closely. ConditionPathExists=/sys/kernel/tracing means systemd does not create the directory — it checks that the kernel already made it (which tracefs_init() does, below) and mounts only if so, so a kernel built without tracing simply skips the unit. ConditionCapability=CAP_SYS_RAWIO encodes the “this is privileged” judgement at the init-system level. And Options=nosuid,nodev,noexec is defence in depth against a pseudo-filesystem that has no setuid binaries, device nodes, or executables to begin with. The live mount on that same machine matches exactly:
$ findmnt /sys/kernel/tracing
TARGET SOURCE FSTYPE OPTIONS
/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime,seclabel
$ grep -E 'tracefs|debugfs' /proc/mounts
debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0Note what /proc/mounts does not show: there is no separate line for /sys/kernel/debug/tracing. The debugfs compatibility path is an automount — a lazily-triggered submount that does not materialise in the mount table until something walks into it.
The /sys/kernel/debug/tracing path you may have seen in older tutorials is the legacy compatibility location. The ftrace.rst text spells out the history: “Before 4.1, all ftrace tracing control files were within the debugfs file system, which is typically located at /sys/kernel/debug/tracing. For backward compatibility, when mounting the debugfs file system, the tracefs file system will be automatically mounted at: /sys/kernel/debug/tracing.” In other words, tracefs was split out of debugfs in the v4.1 kernel (January 2015) so that tracing could be used without compiling in the security-sensitive debugfs; and to avoid breaking the mountain of tooling that had /sys/kernel/debug/tracing hard-coded, the kernel auto-mounts the same tracefs instance under debugfs’s tracing/ directory when debugfs is mounted. So the same files appear at both paths — but /sys/kernel/tracing is the real, canonical home and /sys/kernel/debug/tracing is a backward-compatibility mirror.
This resolves the open uncertainty flag carried by the Linux Tracing and Observability MOC header: the canonical tracefs mount is /sys/kernel/tracing, verified directly against the v6.12 tracefs_init() source and ftrace.rst.
How the debugfs mirror is actually produced
The mirror is not a bind mount and not a symlink. tracing_init_dentry() in kernel/trace/trace.c registers an automount point:
/* kernel/trace/trace.c, v6.12, tracing_init_dentry() */
if (security_locked_down(LOCKDOWN_TRACEFS)) {
pr_warn("Tracing disabled due to lockdown\n");
return -EPERM;
}
...
/* ... it is still possible for tools to expect the tracing
* files to exist in debugfs/tracing, we must automount ... */
tr->dir = debugfs_create_automount("tracing", NULL,
trace_automount, NULL);debugfs_create_automount() creates a dentry with DCACHE_NEED_AUTOMOUNT set. The first time any process resolves a path through /sys/kernel/debug/tracing, the VFS calls trace_automount(), which does get_fs_type("tracefs"), builds a submount context with fs_context_for_submount(), and returns a fresh vfsmount of the same tracefs superblock. So the files are genuinely the same objects seen through a second mount, not copies — which is why edits made through one path are instantly visible through the other, and why /proc/mounts shows nothing until the automount fires.
Verified against v6.12 and v6.17 during this write-up:
- In v6.12 the automount is unconditional.
grep -i automount kernel/trace/Kconfigat thev6.12tag returns nothing: there is no config symbol guarding it, anddebugfs_create_automount()is compiled in wheneverCONFIG_TRACINGis set. - In v6.17 it became optional and started warning.
CONFIG_TRACEFS_AUTOMOUNT_DEPRECATEDfirst appears at kernel/trace/Kconfig, v6.17 (line 203) asbool "Automount tracefs on debugfs [DEPRECATED]",depends on TRACING,default y. Its help text states the timeline in the kernel’s own words: “The tracing interface was moved from /sys/kernel/debug/tracing to /sys/kernel/tracing in 2015, but the tracing file system was still automounted in /sys/kernel/debug for backward compatibility with tooling. The new interface has been around for more than 10 years and the old debug mount will soon be removed.” The v6.17trace_automount()is wrapped in#ifdef CONFIG_TRACEFS_AUTOMOUNT_DEPRECATEDand now prints, on first use:NOTICE: Automounting of tracing to debugfs is deprecated and will be removed in 2030.
This corrects a hedge the earlier draft of this note carried, which suspected CONFIG_TRACEFS_AUTOMOUNT_DEPRECATED might already exist in v6.12. It does not. If you are on a 6.12 LTS kernel, the debugfs mirror is unconditional and silent; if you are on 6.17 or later you will see the deprecation warning in dmesg the first time anything touches /sys/kernel/debug/tracing, which is a useful way to find the tooling in your fleet that still hard-codes the old path.
flowchart TB A["A tool opens a tracing file"] --> B{"Which path?"} B -->|"/sys/kernel/tracing/<file>"| C["tracefs mount<br/>created by systemd or fstab<br/>at the kernel-provided mount point"] B -->|"/sys/kernel/debug/tracing/<file>"| D["debugfs must be mounted first"] D --> E["dentry has DCACHE_NEED_AUTOMOUNT<br/>VFS calls trace_automount()"] E --> F["fs_context_for_submount(tracefs)<br/>returns a vfsmount of the<br/>SAME tracefs superblock"] C --> G["struct super_block (TRACEFS_MAGIC 0x74726163)<br/>one set of inodes, one set of buffers"] F --> G G --> H["kernel/trace read/write callbacks"] E -.->|"v6.17+ only"| W["pr_warn: deprecated,<br/>will be removed in 2030"]
Mount resolution for the two tracing paths. What it shows: both paths land on one super_block; the debugfs path just takes a detour through an automount that materialises lazily. The insight to take: there is no “debugfs copy” of your tracing state to get out of sync — but the debugfs route has an extra dependency (debugfs must be mounted, and from v6.17 it warns), so it is strictly worse and is on a removal path.
Mental Model — Everything Is a File, the Directory Is the API
The organizing idea is the classic Unix one taken to its logical end: the entire tracing API is a directory tree of files, with no dedicated syscalls or library. A file’s contents on read are a live view of kernel state; writing to it reconfigures the tracer. There is no ftrace_set_tracer() syscall — there is echo function > /sys/kernel/tracing/current_tracer. This is why the same shell commands work in a Dockerfile, an Ansible playbook, a trace-cmd invocation, or by hand.
flowchart TB ROOT["/sys/kernel/tracing/ (tracefs root)"] subgraph CTRL["Control files (write to configure)"] CT["current_tracer"] TON["tracing_on"] BSK["buffer_size_kb"] SFF["set_ftrace_filter"] SFP["set_ftrace_pid"] end subgraph VIEW["View files (read to observe)"] TR["trace (snapshot)"] TP["trace_pipe (consuming stream)"] AT["available_tracers"] AE["available_events"] end subgraph DIRS["Sub-directories"] EV["events/ (per-tracepoint dirs)"] OPT["options/ (one file per option)"] PCPU["per_cpu/ (one dir per CPU)"] INST["instances/ (independent buffers)"] end ROOT --> CTRL & VIEW & DIRS INST -->|each instance mirrors| ROOT
The tracefs control surface. What it shows: the tracefs root holds control files you write (tracer selection, on/off, buffer size, filters), view files you read (the trace, the live pipe, the capability lists), and sub-directories that group related knobs (events/, options/, per_cpu/, instances/). The insight to take: the directory layout is the API — and instances/ is recursive: every sub-directory created there is itself a near-complete copy of the root with its own buffer, tracer, and events, which is how you run several independent traces at once.
Spelled out as an actual tree, with the files grouped by the job they do, the layout looks like this. Every entry below was cross-checked against the trace_create_file() calls in init_tracer_tracefs() and tracing_init_tracefs_percpu() in kernel/trace/trace.c, v6.12, so it is the shape of the tree the code actually builds, not an idealised sketch:
/sys/kernel/tracing/ drwx------ root:root (mode 0700)
│
├── current_tracer 0640 W select the pluggable tracer; write CLEARS the buffer
├── available_tracers 0440 R the tracers compiled into this kernel
├── tracing_on 0640 W master record gate: 1 = record, 0 = freeze (non-destructive)
├── trace 0640 R human-readable snapshot; NON-consuming
├── trace_pipe 0440 R same text, CONSUMING and blocking
├── trace_options 0640 W bulk option setter (mirror of options/)
├── trace_clock 0640 W [local] global counter x86-tsc; write CLEARS the buffer
├── timestamp_mode 0440 R [delta] absolute
├── buffer_size_kb 0640 W per-CPU ring buffer size; "X" if CPUs differ
├── buffer_total_size_kb 0440 R sum across all CPUs
├── buffer_subbuf_size_kb 0640 W sub-buffer (page) size; write DISCARDS the buffer
├── buffer_percent 0640 W wake-up watermark for blocking readers (default 50)
├── free_buffer 0200 W shrink buffer to minimum when this fd is closed
├── trace_marker 0220 W userspace writes text INTO the kernel buffer
├── trace_marker_raw 0220 W same, binary payload
├── tracing_cpumask 0640 W hex mask of CPUs to trace
├── snapshot 0640 W swap in / display the secondary "max" buffer
├── error_log 0640 RW why your last bad write was rejected (write clears)
├── set_event 0640 W enable events by "subsys:event" name
├── available_events 0440 R every static tracepoint, "subsys:event" per line
│
├── events/ ← eventfs: inodes created just-in-time (see below)
│ ├── enable ← "1" here enables EVERYTHING
│ ├── header_page header_event ← ring-buffer record layout, for binary parsers
│ └── sched/
│ ├── enable ← whole subsystem
│ ├── filter ← subsystem-wide filter
│ └── sched_switch/
│ ├── enable ← this one event
│ ├── filter ← "prev_comm ~ \"*sh\" && next_pid != 0"
│ ├── trigger ← stacktrace / traceoff / hist / enable_event ...
│ ├── format ← self-describing field layout (see below)
│ ├── id ← numeric event id, used by perf_event_open()
│ └── hist hist_debug ← histogram trigger output
│
├── options/ ← one file per boolean trace option
│ ├── overwrite ← 1 (default): drop OLDEST. 0: drop NEWEST
│ ├── pause-on-trace ← 0 (default) in v6.12; see the trace/trace_pipe section
│ ├── record-cmd record-tgid irq-info markers hash-ptr event-fork ...
│
├── per_cpu/
│ ├── cpu0/
│ │ ├── trace 0640 R this CPU only, non-consuming
│ │ ├── trace_pipe 0440 R this CPU only, consuming
│ │ ├── trace_pipe_raw 0440 R BINARY ring-buffer sub-buffers; splice(2)-able
│ │ ├── stats 0440 R entries / overrun / dropped events / bytes / ts
│ │ ├── buffer_size_kb 0440 R ← note: read-only mode bits (see permission model)
│ │ └── snapshot snapshot_raw
│ └── cpu1/ ... cpuN/
│
├── instances/ ← mkdir here allocates a whole new trace_array
│ └── my_probe/ ← own buffer, own events/, own tracer, own tracing_on
│
├── set_ftrace_filter set_ftrace_notrace set_ftrace_pid set_ftrace_notrace_pid
├── set_event_pid set_event_notrace_pid set_graph_function set_graph_notrace
├── available_filter_functions available_filter_functions_addrs enabled_functions
├── kprobe_events kprobe_profile uprobe_events uprobe_profile ← dynamic probes
├── printk_formats saved_cmdlines saved_cmdlines_size saved_tgids
├── stack_trace stack_max_size stack_trace_filter
└── trace_stat/ hwlat_detector/ synthetic_events dynamic_eventsThe real tracefs layout, annotated with each file’s mode and job. ASCII rather than mermaid because this is a filesystem tree with three annotation columns per row — a mermaid flowchart would either lose the columns or become unreadable at this width. What it shows: roughly forty top-level entries falling into six families — tracer selection, buffer sizing, output views, event control, per-CPU views, and dynamic-probe registration — plus four sub-trees. The format file is the subject of Trace Event Format Files; kprobe_events and uprobe_events register the dynamic probes covered in kprobes and uprobes. The insight to take: the mode column is the API contract. 0440 files are pure views, 0220 files (trace_marker) are write-only injection points, and the 0640 files are the actual knobs; the three files whose comments say CLEARS/DISCARDS are the ones that will silently destroy a capture you have not read yet.
How the Filesystem Itself Works
tracefs is a tiny kernel module. In fs/tracefs/inode.c (v6.12) the filesystem is declared as a struct file_system_type named "tracefs":
static struct file_system_type trace_fs_type = {
.owner = THIS_MODULE,
.name = "tracefs",
.init_fs_context = tracefs_init_fs_context,
.parameters = tracefs_param_specs,
.kill_sb = kill_litter_super,
};
MODULE_ALIAS_FS("tracefs");The .name = "tracefs" is the type you pass to mount -t tracefs. Note the modern VFS idiom: there is no .mount callback, only .init_fs_context plus a .parameters table — tracefs was converted to the fs_context mount API, so options are parsed by tracefs_parse_param() into a struct tracefs_fs_info before the superblock is ever built. The parameter table is three entries:
enum { Opt_uid, Opt_gid, Opt_mode };
static const struct fs_parameter_spec tracefs_param_specs[] = {
fsparam_gid ("gid", Opt_gid),
fsparam_u32oct ("mode", Opt_mode),
fsparam_uid ("uid", Opt_uid),
{}
};fsparam_u32oct means mode= is parsed in octal, as you would hope (mode=0750, not mode=488). The default is TRACEFS_DEFAULT_MODE 0700 — root-only, rwx------ — set in tracefs_init_fs_context(). Tracing exposes deep kernel internals, so it is locked to root unless an administrator deliberately opens it up.
The files themselves are not backed by storage. The ftrace core in kernel/trace/trace.c populates the tree by calling tracefs_create_dir() / tracefs_create_file() (the tracefs equivalents of debugfs’s creation helpers), wiring each file to read/write callbacks inside the tracer. So when you cat current_tracer, you are invoking a kernel function that returns the active tracer’s name; when you echo function > current_tracer, you invoke tracing_set_tracer(), which validates function against the registered trace_types list (see ftrace Framework). The comment heritage is explicit: tracefs is “Based on debugfs by: Greg Kroah-Hartman,” reusing debugfs’s pseudo-file machinery but as a separate, tracing-only filesystem.
Every inode tracefs hands out is wrapped in its own container struct so the filesystem can carry per-inode bookkeeping the VFS knows nothing about:
classDiagram direction LR class super_block { +u32 s_magic TRACEFS_MAGIC 0x74726163 +tracefs_fs_info s_fs_info } class tracefs_fs_info { +kuid_t uid +kgid_t gid +umode_t mode +uint opts } class tracefs_inode { +inode vfs_inode +list_head list +ulong flags +void_ptr private } class flags_BIT_1_to_4 { +BIT1 TRACEFS_EVENT_INODE +BIT2 TRACEFS_GID_PERM_SET +BIT3 TRACEFS_UID_PERM_SET +BIT4 TRACEFS_INSTANCE_INODE } class inode_operations { +tracefs_file_inode_operations +tracefs_dir_inode_operations +tracefs_instance_dir_inode_operations } class eventfs_inode { +list_head children +eventfs_entry_array entries +char_ptr name +eventfs_attr attr +kref kref +bit is_freed +bit is_events +uint30 nr_entries } super_block --> tracefs_fs_info : s_fs_info super_block "1" --> "many" tracefs_inode : owns tracefs_inode --> flags_BIT_1_to_4 : flags tracefs_inode --> inode_operations : vfs_inode.i_op tracefs_inode --> eventfs_inode : private, when EVENT_INODE eventfs_inode --> eventfs_inode : children
The tracefs object model, transcribed from fs/tracefs/internal.h, v6.12. What it shows: one superblock owns a tracefs_fs_info holding the parsed uid/gid/mode, and every inode is really a struct tracefs_inode whose first member is the VFS inode — so get_tracefs() is just a container_of(). The private pointer is polymorphic: on an events/ inode it points at an eventfs_inode, which is a recursive metadata tree, not a filesystem object. The insight to take: there are only four flag bits, and they carry the whole design. TRACEFS_INSTANCE_INODE marks the directory whose ownership children inherit; TRACEFS_UID_PERM_SET / TRACEFS_GID_PERM_SET remember that a chown overrode the mount defaults, so a later remount does not stomp it; and TRACEFS_EVENT_INODE marks the inodes eventfs manufactures on demand. struct eventfs_inode has no struct inode in it at all — that is the entire eventfs trick in one line.
There is one more structural detail worth knowing, because it explains a class of bug reports. Because tracefs inodes have to be re-owned when someone remounts with a different gid=, the filesystem keeps a global linked list of every live tracefs inode (static LIST_HEAD(tracefs_inodes), guarded by tracefs_inode_lock), and tracefs_alloc_inode() adds each new inode to it with list_add_rcu(). On a remount that changes uid=/gid=, tracefs_apply_options() walks that list and rewrites the ownership of every inode that has not been explicitly chowned — and for inodes flagged TRACEFS_EVENT_INODE it hands off to eventfs_remount(), because those have children that exist only as metadata. This is why mount -o remount,gid=tracing /sys/kernel/tracing retroactively fixes up a tree that already exists, rather than only affecting newly-created files.
eventfs — Why events/ Is Not Really a Directory of Files
The single most surprising thing about tracefs is that its largest sub-tree does not exist. events/ on a distribution kernel spans thousands of directories: one per tracepoint subsystem, one per event within it, each with five or six files. The kernel developers who reworked it measured ~10,000 files and directories on their test kernel. Materialising all of that as real dentries and inodes at boot is expensive, and it is nearly all waste — a typical session touches a handful of events.
So events/ is served by eventfs, a layer inside tracefs that stores only metadata and manufactures inodes and dentries just in time. Dating it by existence-check: fs/tracefs/event_inode.c returns HTTP 404 at the v6.5 tag and HTTP 200 at v6.6, so eventfs landed in Linux 6.6 (October 2023) and is present in the 6.12 LTS this note pins to. fs/tracefs/event_inode.c says so in its header comment:
“eventfs is used to dynamically create inodes and dentries based on the meta data provided by the tracing system. eventfs stores the meta-data of files/dirs and holds off on creating inodes/dentries of the files. When accessed, the eventfs will create the inodes/dentries in a just-in-time (JIT) manner. The eventfs will clean up and delete the inodes/dentries when they are no longer referenced.” — fs/tracefs/event_inode.c, v6.12
The measured payoff, from Ajay Kaher’s cover letter for the merged series: “Events Tracing infrastructure contains lot of files, directories (internally in terms of inodes, dentries). And ends up by consuming memory in MBs… Tracing events took ~9MB, with this approach it took ~4.5MB for ~10K files/dir” (tracing: introducing eventfs, PATCH v4 00/10, July 2023). Roughly a 50% cut in the memory the tracing directory costs a machine that may never trace anything — which matters most on the embedded and container-host systems that ship tracing “just in case”.
sequenceDiagram autonumber participant U as Root shell participant V as VFS path walk participant E as eventfs_root_lookup participant M as struct eventfs_inode — metadata only participant T as tracefs inode and dentry cache participant K as kernel/trace/trace_events.c U->>V: openat on events/sched/sched_switch/enable V->>E: lookup of the name "sched" in the events dir E->>M: search the children list by name M-->>E: found — a child eventfs_inode named sched E->>T: lookup_dir_entry allocates an inode and dentry NOW T-->>V: dentry for events/sched V->>E: lookup of the name "sched_switch" E->>T: allocate another inode and dentry NOW V->>E: lookup of the name "enable" E->>T: lookup_file builds an inode whose fops come from the metadata entry Note over T: inode flagged TRACEFS_EVENT_INODE, i_private points at the trace_event_file V-->>U: file descriptor U->>K: read reaches event_enable_read on that trace_event_file K-->>U: returns the text "0" Note over T: later, on dput or under memory pressure, the dentry and inode are freed — the eventfs_inode metadata survives
A just-in-time lookup through eventfs. What it shows: nothing under events/ has an inode until a path walk asks for it by name; eventfs_root_lookup() searches an in-memory metadata list and only then allocates the dentry/inode pair, wiring i_private to the trace_event_file that the read/write callbacks operate on. The insight to take: events/ is a directory-shaped query interface, not a directory. This is why ls -R /sys/kernel/tracing/events is far more expensive than it looks — you are asking the kernel to instantiate every inode in the tree — and why tools should look up the specific events they want by path rather than enumerating.
What the metadata actually looks like is instructive. The tracing core does not hand eventfs a list of files; it hands it a list of names with callbacks, and eventfs asks the callback for the mode and file_operations only at lookup time. The top-level events/ directory is registered like this:
/* kernel/trace/trace_events.c, v6.12, create_event_toplevel_files() */
static struct eventfs_entry events_entries[] = {
{ .name = "enable", .callback = events_callback },
{ .name = "header_page", .callback = events_callback },
{ .name = "header_event", .callback = events_callback },
};
e_events = eventfs_create_events_dir("events", parent, events_entries,
nr_entries, tr);and events_callback() is a strcmp ladder that fills in *mode and *fops for whichever name was looked up. Three names, three function pointers — no inodes. Everything beneath (sched/, sched/sched_switch/, its enable/filter/trigger/format/id) is registered the same way, as nested eventfs_inode metadata whose children are only ever described until somebody walks the path.
Two practical consequences follow. First, prefer available_events over walking events/ when you just want the list of names: it is one seq_file read rather than thousands of lookups. Second, inotify on events/ is unreliable by construction — a dentry that has never been looked up cannot be watched, and one that has been evicted disappears from the dcache without the underlying event going anywhere.
The Key Files — A Guided Tour
What follows are the files you will actually touch, with their documented semantics (ftrace.rst, v6.12). First, the cheat-sheet — in particular the “destroys buffer?” column, which is the single most common source of “where did my trace go?”:
| File | Mode | Direction | What it does | Destroys buffer? |
|---|---|---|---|---|
current_tracer | 0640 | write | plug in a tracer by name from available_tracers | yes — and the snapshot buffer too |
trace_clock | 0640 | write | switch timestamp source (local/global/counter/x86-tsc) | yes |
buffer_subbuf_size_kb | 0640 | write | resize the sub-buffer; lets events exceed one page | yes — stops tracing and discards |
buffer_size_kb | 0640 | write | resize the per-CPU buffer | resizing reallocates, so effectively yes |
tracing_on | 0640 | both | gate writes into the buffer | no — this is the safe pause |
trace | 0640 | read | pretty-printed snapshot | no (but open(O_TRUNC) clears it) |
trace_pipe | 0440 | read | pretty-printed stream | consumes as it reads |
per_cpu/cpuN/trace_pipe_raw | 0440 | read | raw binary sub-buffers, splice(2)-able | consumes as it reads |
snapshot | 0640 | both | swap the live and the secondary buffer | swaps, does not clear |
free_buffer | 0200 | write | shrink the buffer to minimum on close of this fd | yes, on close |
trace_marker | 0220 | write | inject a userspace string into the kernel buffer | no |
set_event / events/*/enable | 0640 | both | enable/disable static tracepoints | no |
set_ftrace_filter | 0640 | both | restrict the function tracer to named functions | no |
Control-file cheat-sheet. What it shows: which files are knobs, which are views, and — the column that actually bites — which ones throw away everything you have captured. The insight to take: there are exactly two ways to stop recording, and only one of them is non-destructive. echo 0 > tracing_on freezes the buffer and keeps it; echo nop > current_tracer stops the tracer and wipes the buffer. Always read trace out before touching current_tracer, trace_clock, or buffer_subbuf_size_kb.
trace — “holds the output of the trace in a human readable format.” This is a non-consuming snapshot: reading it does not remove records, so you can cat trace repeatedly and grep it freely. It pretty-prints with a header showing CPU, timestamp, process, and the event.
trace_pipe — the same output, but a consumer: “reading from this file causes sequential reads to display more current data. Once data is read from this file, it is consumed, and will not be read again with a sequential read.” It also blocks when no data is available, making it ideal for live streaming (cat trace_pipe until you Ctrl-C). Use trace to inspect a captured window; use trace_pipe to watch events flow in real time.
current_tracer — “set or display the current tracer.” Writing a name from available_tracers plugs in that tracer; “changing the current tracer clears the ring buffer content as well as the ‘snapshot’ buffer.” This is the single most important control file and is the subject of ftrace Framework’s pluggable-tracer model.
available_tracers — read-only list of the function-class tracers compiled into this kernel: “the different types of tracers that have been compiled into the kernel.” These are exactly the valid current_tracer values.
available_events — read-only list of every static tracepoint, in subsystem:event form (“a list of events that can be enabled in tracing”). The same events are browsable as the events/ tree.
set_ftrace_filter — restricts the function tracer to named functions: “Echoing names of functions into this file will limit the trace to only those functions.” Its inverse is set_ftrace_notrace (“Any function that is added here will not be traced”). Both draw their valid function names from available_filter_functions, which “lists the functions that ftrace has processed and can trace.” Full treatment in ftrace Filtering and Triggers.
set_ftrace_pid — “Have the function tracer only trace the threads whose PID are listed in this file.” Combined with the function-fork option, children of a traced PID are auto-added on fork() and removed on exit(), so you can follow a process tree.
tracing_on — the master record gate: “sets or displays whether writing to the trace ring buffer is enabled. Echo 0 into this file to disable … or 1 to enable it.” This is the cheap, non-destructive way to pause and resume recording without changing the tracer or losing the buffer — the correct knob for “stop recording so I can read the trace cleanly”:
echo 0 > tracing_on # freeze the buffer
cat trace > /tmp/cap # read it out at leisure
echo 1 > tracing_on # resumebuffer_size_kb — “sets or displays the number of kilobytes each CPU buffer holds.” The size is per CPU, so the total memory is roughly buffer_size_kb × num_CPUs. If individual CPUs were sized differently (via per_cpu/cpuN/buffer_size_kb), this top-level file reads X instead of a number. Raising it is the first fix for buffer overruns under high-rate tracing.
events/ — the static-tracepoint tree. It is organized into per-subsystem directories, each holding per-event directories. Per events.rst (v6.12), enabling the scheduler’s context-switch event is echo 1 > events/sched/sched_switch/enable; enabling a whole subsystem is echo 1 > events/sched/enable; enabling everything is echo 1 > events/enable. Each event directory contains enable (toggle), filter (an expression to record only matching events), format (the self-documenting field layout — see Trace Event Format Files), and id (the numeric event identifier). The set_event file is an alternative way to enable events by writing subsys:event names into it.
options/ — “a directory that has a file for every available trace option. Options may also be set or cleared by writing a ‘1’ or ‘0’ … into the corresponding file.” Options tune output verbosity and tracer behavior (e.g. function-fork, sym-offset, print-parent); they are also writable in bulk via the trace_options file.
instances/ — the per-instance buffer mechanism, covered below.
Reading the Buffer Out — trace vs trace_pipe vs trace_pipe_raw
There are three ways to get data out of a tracefs buffer, and choosing the wrong one is the most common way to lose a capture or to think tracing “isn’t working.”
trace is a non-consuming, formatted view. ftrace.rst: “this file is not a consumer. If tracing is off (no tracer running, or tracing_on is zero), it will produce the same output each time it is read. When tracing is on, it may produce inconsistent results as it tries to read the entire buffer without consuming it.” That last clause is the caveat people miss: reading trace while tracing is live is not a coherent snapshot, because the writer is racing you. The correct idiom is always echo 0 > tracing_on first. Opening trace with O_TRUNC — which is what plain > trace from a shell does — clears the buffer, and that is the documented way to reset it without disturbing anything else.
trace_pipe is a consuming, blocking stream. “Reads from this file will block until new data is retrieved. Unlike the ‘trace’ file, this file is a consumer… Once data is read from this file, it is consumed, and will not be read again with a sequential read.” This is what you want for live watching (cat trace_pipe) and for piping into a log. It is also the reason two cat trace_pipe processes on the same buffer will split the event stream between them rather than each seeing all of it — a genuinely confusing failure until you know it.
trace_pipe_raw is the binary path, and it is per-CPU only. There is no top-level trace_pipe_raw; tracing_init_tracefs_percpu() creates it once per CPU under per_cpu/cpuN/. ftrace.rst explains why it exists: “For tools that can parse the ftrace ring buffer binary format, the trace_pipe_raw file can be used to extract the data from the ring buffer directly. With the use of the splice() system call, the buffer data can be quickly transferred to a file or to the network.” splice(2) is the point — it moves whole sub-buffer pages from the ring buffer to a pipe or socket without ever copying them through userspace or running the text formatter. This is how trace-cmd achieves low-overhead capture, and it is why a trace-cmd record costs so much less than cat trace_pipe > file at the same event rate.
trace | trace_pipe | per_cpu/cpuN/trace_pipe_raw | |
|---|---|---|---|
| Format | pretty-printed text | pretty-printed text | raw ring-buffer sub-buffers |
| Consuming? | no | yes | yes |
| Blocks when empty? | no (returns EOF) | yes | yes (watermark = buffer_percent) |
| Scope | all CPUs, merged & time-sorted | all CPUs, merged | one CPU only |
| Zero-copy | no | no | yes, via splice(2) |
| Cost per event | formatting in the kernel | formatting in the kernel | none — deferred to the consumer |
| Use it for | inspecting a frozen window | live watching, ad-hoc pipelines | high-rate capture by tooling |
The three read paths out of one buffer. What it shows: the same records are reachable three ways, trading formatting convenience against throughput. The insight to take: the distinction that costs people data is consuming vs not. trace you can read a hundred times; trace_pipe and trace_pipe_raw empty the buffer as they go, so a cat trace_pipe left running in another terminal will silently steal every event you were trying to capture in the foreground.
A stale claim in the kernel's own documentation
ftrace.rst’s
trace_pipesection still asserts: “If any process opened the trace file for reading, it will actually disable tracing and prevent new entries from being added.” This is no longer true by default in v6.12. That behaviour is now behind thepause-on-traceoption, whose own entry in the same document describes it as “When set, opening the trace file for read, will pause writing to the ring buffer (as iftracing_onwas set to zero). This simulates the original behavior of the trace file.” The code agrees:tracing_open()callstracing_stop_tr(tr)onlyif (!iter->snapshot && (tr->trace_flags & TRACE_ITER_PAUSE_ON_TRACE)), andTRACE_DEFAULT_FLAGSinkernel/trace/trace.hdoes not includeTRACE_ITER_PAUSE_ON_TRACE. So on a stock 6.12 kernel, openingtracedoes not pause tracing — which is exactly why the same document warns that a live read “may produce inconsistent results.” Two sections of one file contradict each other; the code is the arbiter. This is a worked example of the rule that in-tree kernel documentation goes stale and must be checked against source.
The Ring Buffer Behind the Files
buffer_size_kb, the overrun counter in per_cpu/cpuN/stats, and the binary bytes coming out of trace_pipe_raw all only make sense once you know the shape of the thing they describe. The full design is a note of its own — The Kernel Ring Buffer Concept — but the layout matters here because tracefs exposes it directly.
Each CPU gets its own lock-free ring buffer, made of a linked list of sub-buffers. A sub-buffer is normally one architecture page (4 KiB on x86-64), and it begins with a small header defined in kernel/trace/ring_buffer.c:
struct buffer_data_page {
u64 time_stamp; /* page time stamp */
local_t commit; /* write committed index */
unsigned char data[] RB_ALIGN_DATA; /* data of buffer page */
};
#define BUF_PAGE_HDR_SIZE offsetof(struct buffer_data_page, data)On a 64-bit machine that header is 16 bytes (an 8-byte timestamp plus an 8-byte local_t commit index), so a 4096-byte page carries 4080 bytes of records — and buffer->subbuf_size = subbuf_size - BUF_PAGE_HDR_SIZE in the allocator confirms the accounting. This is precisely why ftrace.rst warns that “no event can be larger than the page size minus the sub buffer meta data”, and why buffer_subbuf_size_kb exists at all: raise the sub-buffer size and you raise the maximum single-event size. Dated by existence-check across tags, that file first appears in Linux 6.8 (grep -c buffer_subbuf_size_kb kernel/trace/trace.c returns 0 at v6.7 and 1 at v6.8), so it is present in 6.12 but absent from older LTS lines such as 6.6 and 5.15.
One sub-buffer (default: 1 page = 4096 B on x86-64)
+--------------------------------------------------------------+ offset
| u64 time_stamp — base timestamp for this whole page | 0
+--------------------------------------------------------------+
| local_t commit — bytes committed so far (write index) | 8
+--------------------------------------------------------------+ 16 = BUF_PAGE_HDR_SIZE
| struct ring_buffer_event #1 |
| +----------------------------------------------------+ |
| | type_len:5 | time_delta:27 | (one u32 header) | |
| +----------------------------------------------------+ |
| | u32 array[] — payload, 4-byte aligned | |
| +----------------------------------------------------+ |
+--------------------------------------------------------------+
| struct ring_buffer_event #2 ... |
+--------------------------------------------------------------+
| ... up to 4080 bytes of records ... |
+--------------------------------------------------------------+
| RINGBUF_TYPE_PADDING event fills any unusable tail |
+--------------------------------------------------------------+ 4096A tracefs sub-buffer as trace_pipe_raw hands it to you. ASCII rather than packet-beta because this is a variable-length record stream inside a page, not a fixed bit-field header — packet-beta would have to invent offsets that do not exist. What it shows: a 16-byte page header followed by self-delimiting records, each prefixed by a single 32-bit word. The insight to take: the timestamp is stored once per page and each record carries only a 27-bit delta from it — which is where most of ftrace’s space efficiency comes from, and why timestamp_mode defaults to delta and warns that absolute “takes up more space and is less efficient.”
Two layers of “record” are easy to confuse here, so it is worth naming them. The struct ring_buffer_event described next is the ring buffer’s envelope — length, type, and timestamp delta, agnostic to what it carries. Inside that envelope sits the trace event payload, a struct trace_entry followed by the event’s own fields, which is what the format file describes and what Tracepoints covers in detail. tracefs exposes both: header_page and header_event describe the envelope, and each event’s format file describes the payload.
The record header is a single u32 doing double duty:
struct ring_buffer_event {
u32 type_len:5, time_delta:27;
u32 array[];
};
enum ring_buffer_type {
RINGBUF_TYPE_DATA_TYPE_LEN_MAX = 28,
RINGBUF_TYPE_PADDING, /* 29 */
RINGBUF_TYPE_TIME_EXTEND, /* 30 */
RINGBUF_TYPE_TIME_STAMP, /* 31 */
};Walking it symbol by symbol: type_len is 5 bits, so it holds 0–31. Values 1..28 mean “this is a data record and its payload is type_len << 2 bytes” — i.e. lengths are stored in 4-byte units (RB_ALIGNMENT = 4), which is how 5 bits describe payloads up to RB_MAX_SMALL_DATA = 4 × 28 = 112 bytes. A type_len of 0 means “the record is bigger than that; array[0] holds the real byte length”, costing an extra 4 bytes. The three values above 28 are not data at all: PADDING fills the unusable tail of a page, TIME_EXTEND carries 32 more bits of delta when 27 bits (about 134 ms at nanosecond resolution) is not enough, and TIME_STAMP carries an absolute time. time_delta is 27 bits of nanoseconds relative to the page’s time_stamp or the previous extend.
This encoding is exposed to userspace, deliberately, through two files in the events directory — events/header_page and events/header_event, registered in create_event_toplevel_files()’s eventfs_entry[] table in kernel/trace/trace_events.c, v6.12 — which print the field offsets and sizes of exactly these two structures. That is the ABI contract that lets trace-cmd and perf parse trace_pipe_raw from a kernel they were not compiled against: the layout is described at runtime rather than assumed, in the same spirit as the per-event format files (Trace Event Format Files).
What happens when the buffer fills
stateDiagram-v2 [*] --> Recording : echo 1 > tracing_on Recording --> Recording : event fits in the current sub-buffer Recording --> Advance : sub-buffer full Advance --> Recording : next sub-buffer is free Advance --> BufferFull : next sub-buffer is the oldest unread one state BufferFull { [*] --> Choose Choose --> Overwrite : overwrite option is 1 (the default) Choose --> DropNew : overwrite option is 0 Overwrite : discard the OLDEST sub-buffer and reuse it — stats overrun++ DropNew : refuse the NEW event — stats dropped events++ } BufferFull --> Recording : a reader consumes via trace_pipe, freeing sub-buffers Recording --> Frozen : echo 0 > tracing_on Frozen --> Recording : echo 1 > tracing_on Frozen --> [*] : echo nop > current_tracer — buffer DESTROYED
Ring-buffer behaviour under pressure. What it shows: the buffer never blocks a writer — when it is full it must throw something away, and the overwrite option only chooses which end. The insight to take: the two counters in per_cpu/cpuN/stats are diagnostic opposites. A rising overrun means you are in the default overwrite mode and losing history; a rising dropped events means overwrite is off and you are losing the present. Either way the fix is the same triad: raise buffer_size_kb, narrow with event filters, or drain faster with trace_pipe_raw.
The defaults are set in code, not left to chance. TRACE_DEFAULT_FLAGS in kernel/trace/trace.h includes TRACE_ITER_OVERWRITE, and allocate_trace_buffer() translates it directly: rb_flags = tr->trace_flags & TRACE_ITER_OVERWRITE ? RB_FL_OVERWRITE : 0. ftrace.rst states the same thing from the user’s side: “If ‘1’ (default), the oldest events are discarded and overwritten. If ‘0’, then the newest events are discarded.” Overwrite-by-default is the right choice for a debugging tool — when a crash happens you want the events just before it, and a full buffer that has stopped recording would guarantee you miss them.
buffer_percent is the knob that makes the raw path efficient: it is “the watermark for how much the ring buffer needs to be filled before a waiter is woken up,” defaulting to 50 (tr->buffer_percent = 50 in init_tracer_tracefs()). At 0 a blocking reader wakes on every single event — maximum latency, minimum throughput. At 100 it blocks until the buffer is about to start overwriting. The default of 50 is a deliberate compromise that lets splice() move half a buffer at a time.
Enabling Events, Filtering Them, and Firing Triggers
events/ is where most real work happens, because static tracepoints (Tracepoints) cost nothing when off and produce structured records when on. tracefs gives you three concentric ways to switch them on, all documented in events.rst, v6.12:
echo 1 > events/sched/sched_switch/enable # one event
echo 1 > events/sched/enable # one whole subsystem
echo 1 > events/enable # everything (rarely what you want)Reading one of those enable files back gives four possible answers, which is more informative than it looks: 0 (all off), 1 (all on), X (mixed — some of the events below this point are on), and ? (this file governs no events at all).
The alternative is set_event, which takes names rather than paths — and has a footgun the documentation calls out explicitly:
echo sched_wakeup >> set_event # NOTE the '>>'
echo '!sched_wakeup' >> set_event # '!' prefix disables
echo 'irq:*' > set_event # a whole subsystem by glob
echo '*:*' > set_event # everything
echo > set_event # disable all events”.. Note:: ’>>’ is necessary, otherwise it will firstly disable all the events.” — events.rst §2.1
A single > truncates, and truncating set_event means “disable everything, then enable what I wrote.” If you are adding events to a running capture you must use >>. There is also a boot-time form, trace_event=<event-list> on the kernel command line, for instrumenting problems that happen before userspace exists.
Filters run in the kernel, at record time, on the fields listed in the event’s format file:
cd events/sched/sched_wakeup
echo 'common_preempt_count > 4' > filter
cd ../../signal/signal_generate
echo '((sig >= 10 && sig < 15) || sig == 17) && comm != bash' > filterThe grammar is predicates (field op value) joined by && and || with parentheses. Numeric fields take == != < <= > >= &; string fields take == != ~, where ~ is a glob accepting *, ? and […] character classes. Three v6.12-era extensions are worth knowing because they are genuinely hard to discover:
filename.ustring ~ "password"— the.ustringsuffix tells the kernel the field is a pointer into userspace (assys_enter_openat’sfilenameis) and that it must copy it in before comparing.call_site.function == security_prepare_creds— the.functionsuffix converts alongfield to an address range and matches if it falls inside that function. Only valid onlong-sized fields and only with==/!=.target_cpu & CPUS{17-42}— cpumask filtering in cpulist syntax, with&(intersection),==and!=.
If a filter is rejected you get a bare write error: Invalid argument from the shell — but the kernel keeps the explanation. cat filter shows the offending expression, and error_log gives the parse position. That pair of files is the difference between five seconds and five minutes of debugging.
Triggers are actions attached to an event, written to its trigger file as command[:count] [if filter] and removed by re-writing the same command prefixed with !. The mechanism behind them is worth understanding because it explains a surprising overhead:
“Event triggers are implemented on top of ‘soft’ mode, which means that whenever a trace event has one or more triggers associated with it, the event is activated even if it isn’t actually enabled, but is disabled in a ‘soft’ mode. That is, the tracepoint will be called, but just will not be traced, unless of course it’s actually enabled.” — events.rst §6
So attaching a trigger to an event turns the tracepoint on even if you never wrote 1 to its enable file. The static branch is patched live, the probe runs, the filter is evaluated, and only the recording is suppressed. That is how a trigger can fire on an event you are not recording — and also why “I only set a trigger, why is there overhead?” has an answer.
The exact order of operations is worth pinning down, because it is not the order the file names suggest. It lives in __event_trigger_test_discard() in kernel/trace/trace.h, called from trace_event_buffer_commit():
/* kernel/trace/trace.h, v6.12 — abridged */
if (eflags & EVENT_FILE_FL_TRIGGER_COND)
*tt = event_triggers_call(file, buffer, entry, event); /* 1. triggers FIRST */
if (likely(!(file->flags & (EVENT_FILE_FL_SOFT_DISABLED |
EVENT_FILE_FL_FILTERED |
EVENT_FILE_FL_PID_FILTER))))
return false; /* fast path: keep it */
if (file->flags & EVENT_FILE_FL_SOFT_DISABLED) goto discard; /* 2. soft-disabled */
if (file->flags & EVENT_FILE_FL_FILTERED &&
!filter_match_preds(file->filter, entry)) goto discard; /* 3. event filter */
if ((file->flags & EVENT_FILE_FL_PID_FILTER) &&
trace_event_ignore_this_pid(file)) goto discard; /* 4. set_event_pid */
return false;
discard:
__trace_event_discard_commit(buffer, event);
return true;flowchart TB TP["Tracepoint site in kernel code<br/>(static key: patched nop when fully off)"] TP --> ON{"enabled, or soft-enabled<br/>because a trigger is attached?"} ON -->|no| NOP["Never reached — the site is<br/>a patched-out nop"] ON -->|yes| PROBE["trace_event_raw_event_<call>()<br/>reserves buffer space and<br/>fills in the record"] PROBE --> COMMIT["trace_event_buffer_commit()"] COMMIT --> TRIG{"TRIGGER_COND set?"} TRIG -->|yes| CALL["event_triggers_call():<br/>each trigger's own 'if filter'<br/>is evaluated, then it fires"] TRIG -->|no| GATES CALL --> ACT["traceoff / traceon / stacktrace /<br/>snapshot / enable_event /<br/>disable_event / hist"] ACT --> GATES{"any discard flag set?"} GATES -->|"SOFT_DISABLED"| DROP["__trace_event_discard_commit()<br/>reserved space reclaimed"] GATES -->|"FILTERED and filter false"| DROP GATES -->|"PID_FILTER and pid excluded"| DROP GATES -->|"none — fast path"| BUF["commit into the<br/>per-CPU ring buffer"] DROP --> POST["event_triggers_post_call()<br/>runs either way"] BUF --> POST
The path of one tracepoint hit through the tracefs event machinery. What it shows: the record is built and buffer space is reserved before anything is evaluated; triggers run first, then three independent discard gates (enable-soft-disable, the filter file, and set_event_pid) decide whether the reservation is kept or reclaimed. The insight to take: two things people get backwards. Triggers fire even on events that will be discarded — that is the whole point of soft-disable, and it is why a traceoff trigger works on an event you never enabled. And a filter saves you buffer space and reader time, not probe time: by the time the filter runs, the probe has already executed and the record has already been written. If probe cost is what hurts, you need a narrower event, not a tighter filter.
The hist trigger deserves its own mention because it changes what tracefs is: with a histogram attached, the kernel aggregates in place and you read a summary out of events/<subsys>/<event>/hist rather than a stream of records out of trace. That is the same architectural move eBPF makes with maps, available without eBPF — see ftrace Filtering and Triggers for the full syntax.
Per-Instance Tracing — Independent Buffers via instances/
A single trace buffer is a bottleneck when two investigations need to run at once, or when a long-running capture must not be disturbed by an ad-hoc probe. tracefs solves this with instances: “a way to make multiple trace buffers where different events can be recorded in different buffers” (ftrace.rst, v6.12).
Creating an instance is making a directory:
cd /sys/kernel/tracing/instances
mkdir my_probe # kernel allocates a fresh trace_array + buffers
ls my_probe
# available_tracers buffer_percent buffer_size_kb buffer_subbuf_size_kb
# buffer_total_size_kb current_tracer error_log events/ free_buffer options/
# per_cpu/ set_event snapshot timestamp_mode trace trace_clock trace_marker
# trace_marker_raw trace_options trace_pipe tracing_cpumask tracing_onHow mkdir in a pseudo-filesystem works
This is the feature that could not be built on debugfs, and the plumbing is a nice worked example of why owning your filesystem matters. instances/ is created once, at init, by a dedicated helper:
/* fs/tracefs/inode.c, v6.12 */
__init struct dentry *tracefs_create_instance_dir(const char *name,
struct dentry *parent,
int (*mkdir)(const char *name),
int (*rmdir)(const char *name))
{
/* Only allow one instance of the instances directory. */
if (WARN_ON(tracefs_ops.mkdir || tracefs_ops.rmdir))
return NULL;
dentry = __create_dir(name, parent, &tracefs_instance_dir_inode_operations);
...
tracefs_ops.mkdir = mkdir;
tracefs_ops.rmdir = rmdir;That directory — and only that directory — gets tracefs_instance_dir_inode_operations, which is the ordinary directory op table plus .mkdir = tracefs_syscall_mkdir and .rmdir = tracefs_syscall_rmdir. When your shell calls mkdir(2), the VFS routes it there, and tracefs_syscall_mkdir() does three things worth noticing:
ti = get_tracefs(inode);
ti->flags |= TRACEFS_INSTANCE_INODE; /* 1. mark this as an ownership root */
ti->private = inode;
inode_unlock(inode); /* 2. drop the inode lock ... */
ret = tracefs_ops.mkdir(name); /* ... because the callback will
create files in this very dir */
inode_lock(inode); /* 3. retake it */Step 1 is the permission model: the comment says “This is a new directory that does not take the default of the rootfs. It becomes the default permissions for all the files and directories underneath it” — so an instance can be chowned to a team’s group and everything created inside it inherits that ownership, without touching the top level. Step 2 is the interesting one: the callback (instance_mkdir() in kernel/trace/trace.c) is going to call tracefs_create_file() dozens of times inside the directory currently being created, which would deadlock against the VFS’s own i_rwsem. tracefs drops the lock and hands the callback the bare directory name as a string, leaving race handling to ftrace — exactly the “properly done without hacks” that the 2015 commit message promised.
The callback chain from there is instance_mkdir() → trace_array_create() → trace_array_create_dir(), which is where the instance gets its files:
/* kernel/trace/trace.c, v6.12 */
static int trace_array_create_dir(struct trace_array *tr)
{
tr->dir = tracefs_create_dir(tr->name, trace_instance_dir);
ret = event_trace_add_tracer(tr->dir, tr); /* its own events/ + set_event */
init_tracer_tracefs(tr, tr->dir); /* the whole control surface */
__update_tracer_options(tr);
}A second stale claim in ftrace.rst — instances do have a tracer
The “Instances” section of ftrace.rst, v6.12 still says: “Notice that none of the function tracer files are there, nor is
current_tracerandavailable_tracers. This is because the buffers can currently only have events enabled for them.” That has not been true for years.trace_array_create_dir()— the instance creation path, quoted above — calls the sameinit_tracer_tracefs()that the top level uses, and the first two lines of that function aretrace_create_file("available_tracers", …)andtrace_create_file("current_tracer", …). It also callsftrace_create_function_files(tr, d_tracer), which creates the per-instanceset_ftrace_filter/set_ftrace_notrace/set_ftrace_pidfiles. So on v6.12 you can runecho function_graph > instances/foo/current_tracerwhile the top level runs something else entirely. The examplelsoutput in the docs is a fossil of the 2014 implementation. Verified by readinginit_tracer_tracefs()andtrace_array_create_dir()in kernel/trace/trace.c at v6.12.
erDiagram TRACEFS_SUPERBLOCK ||--|| GLOBAL_TRACE : "roots" GLOBAL_TRACE ||--|| INSTANCES_DIR : "owns instances/" INSTANCES_DIR ||--o{ TRACE_ARRAY : "one per mkdir" GLOBAL_TRACE ||--|| ARRAY_BUFFER_TOP : "has" TRACE_ARRAY ||--|| ARRAY_BUFFER : "has its own" ARRAY_BUFFER ||--o{ PER_CPU_RING_BUFFER : "one per CPU" TRACE_ARRAY ||--o{ TRACE_EVENT_FILE : "own enable/filter/trigger state" TRACE_ARRAY ||--|| CURRENT_TRACER : "own pluggable tracer" TRACE_ARRAY ||--|| TRACE_FLAGS : "own options, minus ZEROED_TRACE_FLAGS" TRACE_ARRAY { string name "the directory name given to mkdir" int trace_flags "copied from global minus EVENT_FORK, FUNC_FORK, TRACE_PRINTK" int buffer_percent "wakeup watermark, defaults to 50" string system_names "optional — restrict which event subsystems exist" ulong range_addr_start "optional — persistent buffer in reserved memory" } PER_CPU_RING_BUFFER { int subbuf_size "page size minus 16-byte header" flag RB_FL_OVERWRITE "from the overwrite option" counter overrun "oldest data thrown away" counter dropped_events "newest data refused" }
The instance object graph. What it shows: mkdir instances/foo allocates a whole struct trace_array — its own ring buffers (one per CPU), its own per-event enable/filter/trigger state, its own tracer, and its own copy of the options. The insight to take: an instance is not a filter or a view onto the global buffer; it is a second, fully independent tracer. That is what makes it the correct answer to “two tools are fighting over one buffer.”
There are two deliberate exceptions to the “fully independent” rule, both visible in the code. ZEROED_TRACE_FLAGS — TRACE_ITER_EVENT_FORK | TRACE_ITER_FUNC_FORK | TRACE_ITER_TRACE_PRINTK — are the options an instance does not inherit from the global trace; a new instance starts with them off regardless of the top level (tr->trace_flags = global_trace.trace_flags & ~ZEROED_TRACE_FLAGS). And TOP_LEVEL_TRACE_FLAGS (TRACE_ITER_PRINTK, PRINTK_MSGONLY, RECORD_CMD) are supported only by the global trace. ftrace.rst’s older warning that “trace_options affect all instances and the top level buffer the same” is therefore only partly true in v6.12 — options are per-trace_array in the data structure; those two lists are the carve-outs.
Removing an instance is rmdir my_probe, which tears down its buffers — and fails with EBUSY if any process still holds one of its files open, which is a feature, not a bug: it stops you from pulling the buffer out from under a running cat trace_pipe.
Instances at boot, and buffers that survive a reboot
Instances are not only a runtime facility. kernel/trace/trace.c registers __setup("trace_instance=", boot_instance), and the documented syntax (Documentation/admin-guide/kernel-parameters.txt, v6.12) is richer than most people expect:
trace_instance=<name>[^flag][^flag],<system>:<event>,<event>,<system>trace_instance=foo,sched:sched_switch,irq_handler_entry,initcallcreates instancefooearly in boot with three things enabled: one specific event, one event named without its subsystem (allowed when the name is unique), and every event under theinitcallsystem.- Flags come before the events, separated by
^:traceoffcreates the instance with recording disabled, andtraceprintkredirectstrace_printk()output into this instance instead of the global buffer.trace_instance=foo^traceoff^traceprintk,sched,irq.
The most interesting form is a 6.12-era feature: an instance can be pointed at physically reserved memory so its ring buffer survives a reboot.
reserve_mem=12M:4096:trace trace_instance=boot_map@traceThis reserves 12 MiB at 4096-byte alignment under the tag trace, and creates a boot_map instance whose per-CPU buffers are carved out of it. In the code this is the range_addr_start / range_addr_size path in allocate_trace_buffer(), which calls ring_buffer_alloc_range() instead of the ordinary ring_buffer_alloc() and then ring_buffer_last_boot_delta() to work out how far the previous boot’s kernel text was relocated — which is why such instances also grow a last_boot_info file and a per-CPU buffer_meta file. The documentation is admirably honest about the caveats: KASLR may move the reservation between boots (losing the contents), the ring-buffer layout may change between kernel versions (the validator resets the buffer if so), and you should pair it with ^traceoff so last boot’s events do not interleave with this boot’s. Used carefully, it is the closest thing ftrace has to a flight recorder that survives a panic.
This whole mechanism is the foundation that lets tools coexist: trace-cmd can run in its own instance while you poke at the top-level buffer by hand, a monitoring agent can own a private instance that no interactive session disturbs, and ftrace_dump_on_oops=foo=orig_cpu can be pointed at one specific instance so a crash dumps only the buffer you care about.
The Permission Model — Root by Default, Delegable by Design
tracefs is locked down by default and the default is not subtle. TRACEFS_DEFAULT_MODE is 0700, and on the machine this note was written on that is exactly what you see:
$ ls -ld /sys/kernel/tracing
drwx------. 10 root root 0 Aug 20 18:25 /sys/kernel/tracing/
$ id -u
1000
$ ls /sys/kernel/tracing
ls: cannot open directory '/sys/kernel/tracing': Permission deniedThat Permission denied is the whole security model in one line: the filesystem is mounted, the kernel is tracing-capable, and an unprivileged user simply cannot see inside. Note also the trailing . on the mode string — SELinux labels are in play (seclabel appears in the mount options), so on a Fedora/RHEL-family system there is a second, orthogonal gate: even a root process must have an SELinux domain permitted to access the tracefs_t type.
Inside the directory, individual files carry their own modes, and they are not uniform:
| Mode | Symbolic | Constant | Examples | Why |
|---|---|---|---|---|
0700 | rwx------ | TRACEFS_DEFAULT_MODE | the mount root, every directory | tracing exposes kernel internals |
0640 | rw-r----- | TRACE_MODE_WRITE | current_tracer, tracing_on, buffer_size_kb, trace, filter, trigger | the actual control knobs |
0440 | r--r----- | TRACE_MODE_READ | available_tracers, available_events, trace_pipe, per_cpu/cpuN/* | pure views |
0220 | -w--w---- | literal | trace_marker, trace_marker_raw | write-only injection points |
0200 | -w------- | literal | free_buffer | owner-only side effect on close |
tracefs file modes as created by init_tracer_tracefs(). What it shows: the group bit is set on most files, which is what makes delegation possible at all — a 0640 file is readable and writable by the owning group. The insight to take: trace_marker at 0220 is the one file designed to be exposed widely. It is write-only by construction, so handing an application group write access lets it annotate the kernel trace with its own markers without being able to read anything the kernel recorded.
Delegation is done at mount time, or by remount, using the three parameters from tracefs_param_specs:
groupadd tracing
usermod -aG tracing alice
mount -o remount,gid=tracing /sys/kernel/tracing
# every inode that has not been explicitly chowned is re-grouped;
# eventfs inodes are re-owned through eventfs_remount()This works because of the global inode list described earlier: tracefs_apply_options() walks tracefs_inodes and rewrites i_gid on every inode whose TRACEFS_GID_PERM_SET flag is clear. A file you had previously chowned by hand keeps its ownership, because tracefs_setattr() sets that flag on any ATTR_UID/ATTR_GID change. The same mechanism, at a finer grain, is what TRACEFS_INSTANCE_INODE provides: chown -R alice:tracing /sys/kernel/tracing/instances/alice_debug gives one team its own buffer without exposing the top level, and instance_inode() makes every file created inside inherit from that directory rather than from the mount root.
One inconsistency is worth flagging because it will surprise anyone doing delegation. The top-level buffer_size_kb is created TRACE_MODE_WRITE (0640), but the per-CPU per_cpu/cpuN/buffer_size_kb is created TRACE_MODE_READ (0440) — even though its file_operations has a working .write = tracing_entries_write. Root never notices, because CAP_DAC_OVERRIDE bypasses the mode bits and the documented echo 10000 > per_cpu/cpu0/buffer_size_kb works fine. A delegated non-root group, however, can resize the whole buffer but not an individual CPU’s. Verified across tags: the per-CPU file was 0444 at v4.19 and v5.10, and TRACE_MODE_READ at v6.6, v6.12 and v6.17 — so this is long-standing behaviour, not a regression.
Two further gates sit above the mode bits:
- Kernel lockdown.
tracing_init_dentry()callssecurity_locked_down(LOCKDOWN_TRACEFS)and, if the kernel is locked down (typically under UEFI Secure Boot withlockdown=confidentiality), bails out with-EPERMafter printingTracing disabled due to lockdown.tracefs_create_file()performs the same check on every file creation and simply returnsNULL. So on a locked-down machine the directory can exist and be empty — which looks exactly like “ftrace is not compiled in” unless you checkdmesg. - Namespacing: there is none. tracefs is host-global. There is no per-network-namespace or per-PID-namespace view of the trace buffers, by design — tracing is a system-wide, whole-machine facility, and pretending otherwise would be a containment lie. This is why
systemd’s unit carriesConditionVirtualization=!lxc, and why a container that mounts tracefs sees (and can perturb) the host’s tracing state.
A Worked Session, Start to Finish
sequenceDiagram autonumber participant S as Root shell participant F as tracefs participant T as ftrace core in kernel/trace/trace.c participant R as per-CPU ring buffers S->>F: echo 0 > tracing_on F->>T: rb_simple_write then tracer_tracing_off Note over R: writers now skip the buffer — nothing already captured is lost S->>F: echo 1 > events/sched/sched_switch/enable F->>T: event_enable_write then ftrace_event_enable_disable T->>T: static key patched — the nop becomes a call S->>F: echo function_graph > current_tracer F->>T: tracing_set_tracer validates against the trace_types list Note over R: BUFFER CLEARED here S->>F: echo schedule > set_ftrace_filter F->>T: ftrace_filter_write recompiles the ftrace hash S->>F: echo 8192 > buffer_size_kb F->>R: ring_buffer_resize, per CPU S->>F: echo 1 > tracing_on Note over R: recording S->>F: cat trace_pipe F->>R: consuming, blocking read — drains as it goes R-->>S: live event stream until Ctrl-C S->>F: echo 0 > tracing_on S->>F: cat trace > /tmp/capture.txt F->>R: non-consuming read of whatever is left S->>F: echo nop > current_tracer Note over R: BUFFER CLEARED again — read it out first S->>F: echo 0 > events/sched/sched_switch/enable T->>T: static key patched back to a nop
One complete tracing session as a message exchange. What it shows: every step is a file write that lands in a specific kernel function, and two of them (current_tracer, twice) destroy the buffer. The insight to take: the ordering is not arbitrary. tracing_on brackets the noisy setup so your capture window is clean; the cat trace must come before echo nop > current_tracer; and disabling the event last is what patches the tracepoint back to a nop so you stop paying for it.
cd /sys/kernel/tracing # 1. canonical mount
echo 0 > tracing_on # 2. freeze while we set up
echo 1 > events/sched/sched_switch/enable # 3. enable one static event
echo function_graph > current_tracer # 4. plug in the call-graph tracer
echo schedule > set_ftrace_filter # 5. narrow function tracing to schedule()
echo 8192 > buffer_size_kb # 6. 8 MB per CPU, avoid overruns
echo 1 > tracing_on # 7. start recording
cat trace_pipe # 8. watch live; Ctrl-C to stop
echo 0 > tracing_on # 9. freeze
cat trace > /tmp/capture.txt # 10. read the snapshot out
echo nop > current_tracer # 11. unplug tracer (clears buffer)
echo 0 > events/sched/sched_switch/enable # 12. disable the eventSteps 2 and 7/9 use tracing_on to bracket setup and capture without destroying the buffer; step 3 enables an orthogonal static event that records alongside whatever the tracer does; step 4 selects a tracer; step 5 filters function tracing (see ftrace Filtering and Triggers); step 6 sizes the per-CPU buffer; step 8 streams consumingly while step 10 reads non-consumingly; step 11’s nop clears the buffer, so the read in step 10 must come first.
Failure Modes and Common Misunderstandings
“/sys/kernel/tracing doesn’t exist / is empty.” Either no ftrace option is compiled in (no CONFIG_FTRACE family), or tracefs isn’t mounted. Mount it: mount -t tracefs nodev /sys/kernel/tracing. On older systems you may only find /sys/kernel/debug/tracing — that is the legacy mirror, functionally identical.
“Permission denied.” tracefs defaults to mode 0700 (root only). Use sudo, or remount with a gid= to grant a group.
“I cat trace and it’s huge and never changes.” trace is a non-consuming snapshot of a possibly-large buffer. To watch new events live, read trace_pipe (consuming, blocking) instead.
“My events vanished after I switched tracers.” Writing current_tracer clears the buffer. Read trace before changing tracers, and prefer tracing_on (which doesn’t clear) for pausing.
“Events stop appearing under load.” The per-CPU buffer overran (it overwrites oldest-first by default). Raise buffer_size_kb, narrow with filters, or stream via trace_pipe.
“Two tools are clobbering each other’s config.” They are sharing the top-level buffer. Give each its own instances/<name>/ directory.
“/sys/kernel/tracing exists but is completely empty.” Two very different causes with the same symptom. Either tracefs is not mounted (the kernel creates the directory as a sysfs mount point at core_initcall time whether or not anything mounts it — so an empty directory is the normal pre-mount state), or the kernel is locked down. Distinguish them with findmnt /sys/kernel/tracing (empty output means not mounted) and dmesg | grep -i lockdown (look for Tracing disabled due to lockdown).
“My set_event write disabled everything I had already enabled.” You used > instead of >>. echo foo > set_event means “the enabled set is now exactly {foo}”; echo foo >> set_event means “add foo”. events.rst calls this out explicitly and it still catches people.
“write error: Invalid argument and no explanation.” Your filter or trigger expression failed to parse. cat the file you just wrote to (it echoes the rejected text back) and read error_log, which records the command, the error type, and the character position.
“Events I enabled are producing nothing.” Check three things in order: cat tracing_on (is the master gate open?), cat events/<sys>/<event>/filter (is a leftover filter rejecting everything?), and cat set_event_pid / set_ftrace_pid (is a stale PID filter excluding your workload?). All three are sticky — nothing resets them when you change tracers.
“rmdir instances/foo fails with EBUSY.” Some process still has a file inside that instance open — very often a backgrounded cat trace_pipe you forgot about. lsof +D /sys/kernel/tracing/instances/foo finds it. This is deliberate protection, not a leak.
“Only some of my events appear, and trace shows fewer than I expected.” Look at per_cpu/cpuN/stats for every CPU, not just CPU 0. A single hot CPU can be overrunning while the others are idle, and the merged trace view gives no hint of which CPU lost data. Rising overrun versus rising dropped events tells you which end you are losing (see the ring-buffer state diagram above).
“buffer_size_kb won’t take the number I gave it.” ftrace.rst shows the failure verbatim: echo 1000000000000 > buffer_size_kb yields -bash: echo: write error: Cannot allocate memory, and — the nasty part — the buffer is left at whatever the allocator could manage (85 in their example), not at its previous size. Always read the file back after writing it.
“An event is too big for the buffer.” No record can exceed one sub-buffer minus its 16-byte header. Raise buffer_subbuf_size_kb (Linux 6.8+), remembering that doing so discards the current buffer and the snapshot.
Alternatives and When to Choose Them
| Interface | Talks to the kernel via | Data comes back as | Aggregation | Needs installing? | Choose it when |
|---|---|---|---|---|---|
tracefs by hand (echo/cat) | file reads/writes on tracefs | formatted text | none (or hist triggers) | nothing | you are on an unfamiliar box over SSH with no tooling |
trace-cmd | the same tracefs files + splice() on trace_pipe_raw | binary .dat, replayable | offline | a package | repeatable captures, sharing traces, KernelShark |
perf | perf_event_open(2) | perf.data / live counters | sampling + counting | a package | PMU counters, sampling profiles, mixed kernel/user stacks |
bpftrace / BCC | eBPF programs attached to the same probes | whatever your program prints | in-kernel, in maps | a package + BTF/kernel headers | you need to summarise millions of events without shipping them |
| LTTng | its own kernel modules reading trace events | CTF trace files | offline | out-of-tree modules | very high throughput, multi-host correlation |
Front-ends onto overlapping data. What it shows: four of the five reach the same underlying probes; what differs is where the data is reduced and what has to be installed first. The insight to take: the column that decides it in an incident is “needs installing?” — tracefs is the only row that is guaranteed present, which is why it is worth knowing by hand even though every other row is more ergonomic.
Uncertain
Verify: the LTTng row. Reason: the first four rows were checked against kernel source read during this write-up, but LTTng is an out-of-tree project and no LTTng documentation was consulted for this note — the characterisation of it as out-of-tree kernel modules producing Common Trace Format (CTF) output is from background knowledge, not a source fetched here. To resolve: read
lttng.org/docs/for the currentlttng-modulesarchitecture and confirm whether it still hooks kernel tracepoints directly rather than going through tracefs. Everything about the other four rows stands on the primary sources cited elsewhere in this note. uncertain
tracefs is the only native control surface for ftrace, so the real “alternative” question is whether to drive it by hand or through a tool. For one-off investigation, raw echo/cat on tracefs is unbeatable for availability. For repeatable captures, trace-cmd wraps these exact files and produces a portable binary .dat (it even uses a private instance to avoid disturbing your manual session). For programmatic, aggregating tracing, bpftrace/BCC bypass tracefs entirely, attaching eBPF to the same underlying probes and aggregating in maps — the right choice when shipping every event through a tracefs buffer would be too costly. And perf reads the same trace events through perf_event_open rather than tracefs files. The tracefs surface is therefore the human/shell interface; perf and eBPF are the syscall/programmatic interfaces to overlapping data.
Production Notes
Containers. /sys/kernel/tracing is typically not mounted inside a container, and it is host-global rather than namespaced — tracing is system-wide, a deliberate boundary, not an oversight. Tracing from inside a container therefore means either running in the host’s mount namespace (a privileged debug pod with hostPID and the host /sys bind-mounted) or not at all. Do not “fix” this by bind-mounting tracefs into an application container: a container with write access to the top-level buffer can turn off tracing for the whole host, resize buffers to exhaust memory, or read trace_pipe and observe every other tenant’s syscall activity. If a container genuinely needs tracing, give it its own instances/<name>/ directory chowned to its user, which is exactly the delegation case the TRACEFS_INSTANCE_INODE flag was built for.
Hardened systems. Kernel lockdown (LOCKDOWN_TRACEFS) makes tracefs come up empty with a single dmesg line. Removing CONFIG_DEBUG_FS no longer costs you tracing at all — that is the whole point of the 2015 split, and obj-$(CONFIG_TRACING) += tracefs/ in fs/Makefile is the proof. A hardened kernel build with CONFIG_TRACING=y and CONFIG_DEBUG_FS=n is a genuinely reasonable configuration and one the split made possible.
Path choice. Target /sys/kernel/tracing directly. The /sys/kernel/debug/tracing mirror is on a published removal path: CONFIG_TRACEFS_AUTOMOUNT_DEPRECATED (v6.17+) is default y today but its help text says the old mount “will soon be removed,” and the runtime warning names 2030. Anything you write today will outlive that. A useful audit trick on a 6.17-or-later fleet: dmesg | grep -i 'Automounting of tracing' fires the first time anything walks the debugfs path, so it identifies which of your tools still hard-code it.
Cost when idle. tracefs is not free even with nothing enabled — that is what motivated eventfs. On a kernel with ~10,000 event files, materialising every inode cost roughly 9 MB of unswappable kernel memory; eventfs halved it to ~4.5 MB by keeping metadata and creating inodes on demand. On a fleet of small nodes that is worth knowing, and it is another reason to prefer available_events over find /sys/kernel/tracing/events in monitoring scripts: the find instantiates the entire tree.
Sizing. buffer_size_kb is per CPU. On a 128-core machine, echo 65536 > buffer_size_kb asks for 8 GiB, and it will try. Read buffer_total_size_kb after every resize, and use tracing_cpumask to narrow the CPU set before reaching for a bigger buffer.
Getting data off the box. For anything sustained, do not cat trace_pipe > file. Use trace-cmd, or read per_cpu/cpuN/trace_pipe_raw with splice(2) yourself: it moves whole sub-buffer pages without formatting them or copying them through userspace, and buffer_percent (default 50) controls how full the buffer must be before a blocking reader is woken, so you get large batched transfers rather than a wakeup per event.
Crash forensics. Two features pair well here and are under-used. ftrace_dump_on_oops=<instance> dumps one named instance’s buffer to the console on a panic, and reserve_mem=…:trace trace_instance=boot_map@trace places an instance’s ring buffer in reserved physical memory so it survives into the next boot. Together they turn tracefs into a flight recorder — with the caveats the kernel documentation is candid about (KASLR may relocate the reservation, and a kernel version change may invalidate the layout, in which case the validator resets the buffer).
See Also
The framework this filesystem drives
- ftrace Framework — the tracer framework this filesystem controls (
current_tracer, the pluggable-tracer model, the ring buffer) - The Function Tracer — the original tracer whose filters (
set_ftrace_filter) live here - The Function Graph Tracer — the call-graph tracer selected by
echo function_graph > current_tracer - ftrace Filtering and Triggers —
set_ftrace_filter,set_ftrace_notrace, event filters, and triggers in depth, including thehisttrigger
What lives under events/
- Tracepoints — the static instrumentation points that
events/exposes; the sibling note to this one - Static Keys and Tracepoint Patching — why a disabled tracepoint is a
nop, and whatecho 1 > enableactually patches - The Trace Event Subsystem — the machinery that turns a tracepoint into an
events/<subsys>/<event>/directory - Trace Event Format Files — the
formatfile inside each event directory, and how tools parse binary records with it
The buffer behind the files
- The Kernel Ring Buffer Concept — the per-CPU buffer that
buffer_size_kbsizes,overwritegoverns, andtrace/trace_pipe/trace_pipe_rawdrain
Dynamic probes registered through tracefs
- kprobes — the mechanism behind the
kprobe_eventsanddynamic_eventsfiles - uprobes — the userspace equivalent, behind
uprobe_events
Neighbouring pseudo-filesystems and gates
- sysfs and the Kernel Object Hierarchy — sibling pseudo-filesystem;
sysfs_create_mount_point(kernel_kobj, "tracing")is what reserves the/sys/kernel/tracingdirectory - Kernel Lockdown Mode — the
LOCKDOWN_TRACEFSgate that can make this whole tree come up empty
Other front-ends onto the same data
- trace-cmd and KernelShark — the front-end that drives these files, uses a private instance, and
splice()strace_pipe_raw - perf Profiling Tool / The perf_event_open Syscall — the syscall-based route to the same trace events
- bpftrace / BCC BPF Compiler Collection — the eBPF route, which bypasses tracefs buffers entirely and aggregates in maps
Parent
- Linux Tracing and Observability MOC — the parent map (whose tracefs-location uncertainty flag this note resolves)