vault

trace-cmd and KernelShark

15 min read

trace-cmd and KernelShark

trace-cmd is the command-line front-end to the kernel’s ftrace tracer: instead of echo-ing values into tracefs control files by hand, you run trace-cmd record to capture a trace into a binary trace.dat file, then trace-cmd report to render it as readable text — the man page calls it “a front-end for Ftrace” that “interacts with the Ftrace tracer that is built inside the Linux kernel” (trace-cmd(1)). KernelShark is the Qt graphical viewer that loads the same trace.dat file and visualizes it — per-CPU and per-task timelines, an event list, filtering, and time-measurement markers — turning a wall of text records into an interactive picture of scheduling and latency. Together they form the standard high-level ftrace workflow: trace-cmd record → trace.dat → trace-cmd report or KernelShark. Both are built on the shared libraries libtracefs (drives the tracefs interface), libtraceevent (parses the binary event records), and libtracecmd (reads trace.dat).

This note covers the tooling that sits on top of ftrace — why you want it, the trace-cmd subcommand surface, the trace.dat capture model, and the KernelShark GUI. The ftrace machinery itself (tracers, ring buffer, control files) lives in ftrace Framework and The tracefs Filesystem; the binary record format these tools parse is described in Trace Event Format Files and The Trace Event Subsystem. The kernel-side facts are pinned to Linux 6.12 LTS; trace-cmd and KernelShark are independent userspace projects (the project site lists trace-cmd v3.4 and KernelShark 2.1 as recent releases — trace-cmd.org, kernelshark.org).

Mental Model — A Driver and a Viewer Over One File Format

The right way to think about these two tools is as a producer/viewer pair bound by a file. trace-cmd is the producer side: it configures ftrace, drains the per-CPU ring buffers, and serializes everything — the events, their format descriptions, the kernel symbol map, and the per-CPU record streams — into a single self-describing trace.dat archive. KernelShark (and trace-cmd report) are the consumer side: they open trace.dat, use its embedded format descriptions to decode the raw binary records, and present them. The trace.dat file is the contract: because it carries the event format definitions inside it, a trace captured on one machine can be read and decoded on a completely different machine that never saw those events.

flowchart LR
  subgraph KERNEL["Kernel (ftrace)"]
    RB["per-CPU ring buffers<br/>(events, function trace)"]
    FMT["event format files<br/>(/sys/kernel/tracing/events/)"]
  end
  TC["trace-cmd record<br/>(libtracefs configures,<br/>drains per-CPU buffers)"]
  DAT["trace.dat<br/>(records + format defs<br/>+ kallsyms)"]
  RPT["trace-cmd report<br/>(ASCII text)"]
  KS["KernelShark (Qt GUI)<br/>(timeline + list + filters)"]
  RB --> TC
  FMT --> TC
  TC --> DAT
  DAT --> RPT
  DAT --> KS

The trace-cmd / KernelShark workflow. What it shows: trace-cmd record reads both the live per-CPU ring buffers and the event format descriptions, packaging them into one trace.dat file; that file is then decoded either as text by trace-cmd report or visually by KernelShark. The insight to take: trace.dat is self-describing — it bundles the format definitions and symbol map alongside the raw records, so decoding is fully portable and does not depend on the analysis machine’s running kernel.

Why You Want trace-cmd Instead of Raw tracefs

You can drive ftrace by hand — echo function_graph > current_tracer, echo 1 > tracing_on, cat trace_pipe — and the The tracefs Filesystem note covers exactly that. But raw tracefs has real friction that trace-cmd removes. First, per-CPU buffer management: at high event rates the in-kernel ring buffers overflow and drop events; trace-cmd record spawns a per-CPU reader that continuously drains each buffer to disk so nothing is lost. The man page for record describes establishing “per-CPU tracing processes that record from the kernel ring buffer into temporary files, then combines them into a trace.dat file” (trace-cmd-record(1)). Second, a binary, compact, portable artifact: trace.dat is far more efficient than text and, crucially, self-describing — you can hand it to a colleague who decodes it without your kernel. Third, a single coherent command line instead of a dozen file writes you must remember to undo. Fourth, graphical analysis: KernelShark consumes trace.dat directly, which raw tracefs text cannot feed.

The trace-cmd Subcommand Surface

trace-cmd is a multiplexing command: trace-cmd <subcommand> [options]. The man page enumerates the full set; the load-bearing ones are below (trace-cmd(1)).

  • record — “record a live trace and write a trace.dat file to the local disk or to the network.” This is the workhorse: it enables the requested events/tracer, runs (optionally wrapping a command), drains the buffers, and writes trace.dat.
  • report — “reads a trace.dat file and converts the binary data to a ASCII text readable format.” The standard offline-analysis path.
  • start — “start the tracing without recording to a trace.dat file.” Arms ftrace using trace-cmd’s convenient option syntax but leaves the data in the kernel buffers (you read it later with show/extract, or via tracefs).
  • stop — “stop tracing (only disables recording, overhead of tracer is still in effect).” Note the precise wording: stopping merely halts recording into the buffer; the instrumentation hooks are still armed and still cost cycles.
  • reset — “disables all tracing and gives back the system performance.” This is the real cleanup — it tears down the tracer and gives you back the CPU. Forgetting reset after start/stop is the classic “why is my box still slow” mistake.
  • extract — “extract the data from the kernel buffer and create a trace.dat file.” Used after a start to pull what is currently buffered into a file.
  • stream — “Start tracing and read the output directly,” i.e. live, like cat trace_pipe but with trace-cmd’s event-selection ergonomics.
  • list — “list the available plugins or events that can be recorded.” Your discovery tool: trace-cmd list -e shows recordable events.
  • show — “display the contents of one of the Ftrace Linux kernel tracing files,” a convenience over cat-ing tracefs.
  • stat — “show tracing (ftrace) status of the running system,” useful to see what is currently armed.
  • snapshot — “take snapshot of running trace,” exposing ftrace’s snapshot buffer.
  • hist — “show a histogram of the events,” surfacing ftrace’s hist-trigger aggregation.

Additional subcommands exist for distributed and specialized use — listen/agent (record from a remote host over the network), profile, split, restore, convert, attach, dump, check-events, and more (trace-cmd(1)).

trace-cmd record — Capturing into trace.dat

record is where most of the option surface lives. The mechanism: trace-cmd record “sets up the Linux Ftrace kernel tracer to capture events during command execution,” creating “per-CPU tracing processes that record from the kernel ring buffer into temporary files, then combines them into a trace.dat file” (trace-cmd-record(1)). The most-used flags, quoted from the man page:

# Record the sched_switch tracepoint while a workload runs, then write trace.dat
trace-cmd record -e sched:sched_switch -F ./my_workload
  • -e <event> — “Specify an event to trace,” using subsystem:event-name form (e.g. sched:sched_switch) or wildcards. Repeatable; -e sched enables a whole subsystem.
  • -p <tracer> — “Specify a tracer” such as function, function_graph, irqsoff, preemptoff, or wakeup. This selects an ftrace tracer (the current_tracer) rather than discrete events.
  • -F — “Filter only the executable that is given on the command line.” Trace only the command you wrap, not the whole system — the single most useful scoping flag.
  • -P <pid> — “Similar to -F but lets you specify a process ID to trace,” for attaching to an already-running process.
  • -c — “Used with either -F (or -P) to trace the process’ children too,” so a fork/exec workload is fully captured.
  • -l <function-name> — “Limit the function and function_graph tracers to only trace the given function name” (sets set_ftrace_filter under the hood). Repeatable; accepts wildcards.
  • -g <function-name> — “Graph the given function … only trace the function and all functions that it calls,” for a focused call-graph with function_graph.
  • -n <function-name> — “Opposite effect of -l … will not be traced,” to exclude noisy functions.
  • -o <output-file> — change the output path; “By default, trace-cmd report will create a trace.dat file.”
  • -N <host:port> — send data via UDP to a remote trace-cmd listen/agent instead of a local file, for tracing across machines.
  • --date — “Map the timestamp to gettimeofday which will allow wall time output,” so trace timestamps line up with the wall clock.

Worked example with a function-graph call tree of one function and its callees:

trace-cmd record -p function_graph -g vfs_read -F cat /etc/hostname

This selects the function_graph tracer (-p), restricts the graph to vfs_read and everything it calls (-g), and traces only the wrapped cat process (-F). The result is a trace.dat containing the precise kernel call tree taken by that one read.

trace-cmd report — Decoding to Text

report is the inverse of record. It “converts binary kernel trace files into human-readable ASCII format,” parsing trace.dat “using format definitions stored within it” — those embedded format specs “enable the program to interpret raw event data and present it in a structured, readable manner with timestamps, task names, CPU numbers, and event-specific fields” (trace-cmd-report(1)). This is the key point connecting back to Trace Event Format Files: each event in the kernel publishes a format file describing its field layout, trace-cmd record captures those format descriptions into trace.dat, and report replays them through libtraceevent to decode the raw records. Useful options:

  • -i <input-file> — “Opens a specified trace file instead of the default trace.dat.”
  • -F <filter> — “Applies complex event filters using field comparisons, logical operators, and regex patterns,” e.g. sched:next_pid==123, evaluated at report time over the captured records.
  • -v — “Inverts filter logic (excludes matches).”
  • --cpu <cpu-list> — “Restricts output to specified CPUs,” e.g. --cpu 0,3 or --cpu 2-4.
  • -t — “Shows full nanosecond timestamp precision.”
  • -l — “latency output format displaying interrupt state, scheduling flags, preemption depth, and lock depth” in a compact field — the same latency-format column you see in raw ftrace.
  • -w — “Reports wakeup latencies between task wake and scheduling events,” a built-in latency analysis.
  • -R — “Prints raw field values without event-specific formatting,” for when you want the unformatted numbers.

A typical first look at a capture:

trace-cmd report -i trace.dat --cpu 0 -F 'sched:next_comm ~ "ngin*"'

This decodes only CPU 0’s records and shows only sched_switch events scheduling a task whose next_comm matches ngin* — narrowing a busy trace down to the process of interest without re-recording.

KernelShark — The Graphical Viewer

KernelShark is “a graphical viewer application that displays kernel trace data. It reads trace.dat files created by trace-cmd and presents the captured event information through visual and tabular interfaces” (KernelShark documentation). Where trace-cmd report gives you a text stream, KernelShark gives you a picture of time, which is far better for understanding scheduling behavior and chasing latency.

The window has two coordinated panes. The graph (timeline) pane at the top plots events against a horizontal time axis in two modes: per-CPU plots, showing “the timeline of events that happened on a given CPU,” and per-task plots, showing “the timeline of events … for the given task regardless of what CPU it was on at the time” (KernelShark documentation). The per-task view is what makes KernelShark shine for scheduling analysis: you can literally watch a task migrate between CPUs, sit on a run queue, get preempted, and resume. You zoom into a narrow time window to inspect individual events, and place two markers on the graph to measure the exact time delta between them — the natural way to measure a wakeup-to-run latency or the duration of an off-CPU stall.

The list pane below is the tabular event view, “organized by timestamp, task name, CPU, and event type, with columns displaying interruption state, scheduling status, IRQ context, and preemption information” (KernelShark documentation) — essentially the trace-cmd report rows, but synchronized with the graph: click an event in the timeline and the list scrolls to it, and vice versa.

Filtering narrows what is shown without re-recording: event filters, task filters, and advanced field-based filters that “support regular expressions for pattern matching across event content.” KernelShark also has a plugin system (for example, plotting specific subsystems) and session management that “preserves application state between sessions, including zoom level, marker positions, and active filters,” restorable via File → Session → Restore Last Session (KernelShark documentation).

Architecturally, KernelShark is built on the same library stack as trace-cmd — it links libtracecmd to read trace.dat, libtraceevent to decode records, and its own libkshark for the visualization model; the project describes trace-cmd as “The back-end application to KernelShark” (trace-cmd.org).

Uncertain

Verify: the exact KernelShark library decomposition (that it consumes libtracecmd/libkshark specifically, and the current set of bundled plugins) on the 2.1 release. Reason: the kernelshark.org home page rendered thinly when fetched and the precise library/plugin list was assembled from the project description plus the trace-cmd site rather than a single authoritative API page. To resolve: read the KernelShark source README and src/libkshark.h for the v2.1 tag. uncertain

The Library Foundation — libtracefs, libtraceevent, libtracecmd

Both tools are thin clients over three libraries that the trace-cmd project ships, each with a single clear job (trace-cmd.org):

  • libtracefs “interfaces with the ftrace/tracefs kernel subsystem” — it is the programmatic equivalent of writing the tracefs control files, used by trace-cmd record/start to configure tracing (enable events, set the tracer, install filters).
  • libtraceevent “parses binary trace event data and formats” — it consumes the format descriptions and turns raw ring-buffer bytes into typed fields. This is the same parsing layer the kernel’s own pretty-printing and perf use, factored into userspace.
  • libtracecmd “reads and processes trace.dat files” — the file-format layer that report and KernelShark use to open and iterate a capture.

Understanding this split clarifies why a trace is portable: libtracefs (configuration) runs only at record time on the traced host, while libtraceevent + libtracecmd (decoding) run at analysis time anywhere, because the format definitions travel inside trace.dat.

The End-to-End Workflow

The canonical session ties it together:

# 1. Discover what you can trace
trace-cmd list -e | grep sched
 
# 2. Record a workload (only this command and its children), capture to trace.dat
trace-cmd record -e sched:sched_switch -e sched:sched_wakeup -c -F ./my_server
 
# 3a. Quick text triage
trace-cmd report -i trace.dat --cpu 0 | head
 
# 3b. Or open it graphically to study scheduling/latency
kernelshark trace.dat

Step 2 produces the portable trace.dat; steps 3a/3b are two views of the same artifact. A common pattern is to triage in report to confirm the events are there, then switch to KernelShark to see the timing relationships — for instance, the gap between a sched_wakeup and the matching sched_switch that put the task on-CPU, which is a scheduling latency you can measure with the two markers.

Failure Modes and Common Misunderstandings

  • stop does not give back performance. The man page is explicit: stop “only disables recording, overhead of tracer is still in effect.” You must run trace-cmd reset to actually disarm the tracer and reclaim CPU (trace-cmd(1)). Leaving a function tracer armed after a start/stop is a frequent “the box is mysteriously slow” cause.
  • Dropped events at high rate. Even with per-CPU drainers, an extreme event rate (e.g. tracing every function call system-wide) can overrun the buffers; scope with -F/-P, narrow with -l/-g, or raise the buffer size rather than trying to capture everything.
  • KernelShark cannot open a raw tracefs trace dump. It needs a real trace.dat produced by trace-cmd record/extract (which embeds the format definitions and symbol map); a hand-cat-ed text trace is not a valid input.
  • Reading a trace.dat on a host that never saw the events. This works precisely because the format descriptions are embedded — but only if trace-cmd record captured them. Stripped or partial captures can fail to decode some events; record the format files you need.

Alternatives and When to Choose Them

  • Raw tracefs (The tracefs Filesystem) — when you want zero binaries beyond a shell, or to script a tiny one-off; trace-cmd is the upgrade when you need reliable capture, a portable artifact, or graphical analysis.
  • perf (perf Profiling Tool) — overlapping but oriented toward sampling and PMU counting and flame graphs; perf sched covers some of the same scheduling-latency ground, but KernelShark’s interactive per-CPU/per-task timeline is uniquely good for seeing scheduling.
  • bpftrace / BCC (bpftrace vs BCC vs ftrace) — when you want in-kernel aggregation (a histogram, a count) rather than a recorded event timeline; trace-cmd+KernelShark is the better choice when you specifically need the timeline reconstruction of who ran when.
  • LTTng (LTTng) with its babeltrace/Trace Compass viewers — a separate high-throughput tracer with its own Common Trace Format and GUI, favored for continuous production tracing at very high event rates.

Production Notes

trace-cmd’s record -N plus listen/agent make it usable across machines and into virtual guests (the agent/setup-guest subcommands target host/guest tracing) — you can record on a fleet of hosts to a central collector. In practice, scheduler and real-time engineers reach for the trace-cmd record -e sched:* … → KernelShark loop when they need to visually understand why a task missed a deadline or migrated unexpectedly, because the per-task timeline answers “where was this task, and what was on its CPU instead?” at a glance — a question that is painful to answer from text alone. For pure latency numbers (worst-case wakeup, IRQs-off duration) the ftrace Latency Tracers and trace-cmd report -w/-l are faster; KernelShark earns its keep when the relationships between events matter.

See Also

Graph View

Backlinks