vault

Linux Observability Overview

13 min read

Linux Observability Overview

Linux ships with an unusually deep, built-in instrumentation toolbox that lets you see what the kernel — and every process running on it — is actually doing, without recompiling anything or attaching a debugger. The clarifying mental model, due to Julia Evans, is that the toolbox factors into three independent layers: data sources (where an event originates), in-kernel mechanisms (which collect and process those events), and front-end tools (which you actually type commands into) (Evans 2017). The single most important insight is that these layers are decoupled: one probe — say, a kprobe on tcp_sendmsg — can be consumed three completely different ways, written to an ftrace ring buffer, sampled by perf, or processed by an eBPF program. Learn the sources once and every front-end unlocks. This note is the orientation map for the whole subsystem; it names each piece and links out to the mechanism leaves that explain it in depth.

Mental Model — Source, Mechanism, Interface, Front-end

The cleanest way to hold the Linux observability stack in your head is as a pipeline with four stages. At the back, instrumentation sources are the places an event can be born: a static hook a kernel developer compiled in, a dynamic probe you patch into running code at an arbitrary address, or a hardware counter inside the CPU. In the middle, in-kernel mechanisms turn a fired source into something durable: ftrace’s ring buffers, the trace-event subsystem, the perf_events core, or an eBPF program running in-kernel. Those mechanisms surface their data through file interfaces — the tracefs filesystem, the /proc and /sys pseudo-filesystems, and the printk ring buffer behind dmesg. Finally, front-end tools (trace-cmd, perf, bpftrace, BCC) read those interfaces and present results to you.

The decoupling between stages is the whole point. The same sched_switch tracepoint can be enabled by writing to a tracefs file and opened by perf record and attached to by an eBPF program — three mechanisms, one source. Likewise, a single front-end like perf straddles several sources: it counts PMU hardware events, samples on them, and records tracepoints, all through one syscall. Julia Evans’ framing makes this explicit: “you can split linux tracing systems into data sources (where the tracing data comes from), mechanisms for collecting data (like ‘ftrace’) and tracing frontends (the tool you actually interact with)” (Evans 2017).

flowchart TB
  subgraph SRC["Sources — where an event is born"]
    TP["Tracepoints<br/>(static, compiled in)"]
    KP["kprobes / kretprobes<br/>(dynamic, any kernel insn)"]
    UP["uprobes / USDT<br/>(dynamic / static, userspace)"]
    PMU["PMU counters<br/>+ software events"]
  end
  subgraph MECH["In-kernel mechanisms"]
    FT["ftrace<br/>(function + event tracer)"]
    TE["trace-event subsystem<br/>(ring buffers, format files)"]
    PE["perf_events core<br/>(perf_event_open)"]
    BPF["eBPF programs<br/>(run in-kernel)"]
  end
  subgraph IFACE["File interfaces"]
    TF["tracefs<br/>(/sys/kernel/tracing)"]
    PROC["/proc + /sys<br/>(state as files)"]
    PK["printk ring buffer<br/>(dmesg, /dev/kmsg)"]
  end
  subgraph FE["Front-end tools"]
    TC["trace-cmd / KernelShark"]
    PERF["perf"]
    BT["bpftrace / BCC"]
  end
  TP --> FT
  TP --> TE
  TP --> PE
  TP --> BPF
  KP --> TE
  KP --> BPF
  UP --> TE
  UP --> BPF
  PMU --> PE
  PMU --> BPF
  FT --> TF
  TE --> TF
  TE --> PE
  PE --> PERF
  TF --> TC
  BPF --> BT
  PROC --> PERF
  PK --> PROC

The Linux observability pipeline. What it shows: a handful of sources (top) feed a handful of in-kernel mechanisms (second row), which surface through file interfaces (third row) to front-end tools (bottom). The insight to take: the arrows fan out — one tracepoint reaches ftrace, the trace-event buffers, perf, and eBPF — so the same probe is consumable four ways. The layers are independently swappable, which is why “learn the sources once” pays off across every tool.

The Sources — Where Events Originate

There are four families of sources, and the deepest organizing distinction between them is static versus dynamic, covered in its own leaf, Static vs Dynamic Tracing.

Static tracepoints are hooks that kernel developers deliberately placed at semantically meaningful spots — a scheduler context switch, a block-I/O completion, a syscall entry — using the TRACE_EVENT family of macros. The kernel documentation describes a tracepoint as code that “provides a hook to call a function (probe) that you can provide at runtime” (tracepoints.rst, v6.12). Their defining property is that they are nearly free when disabled: “When a tracepoint is ‘off’ it has no effect, except for adding a tiny time penalty (checking a condition for a branch) and space penalty” (ibid.), achieved with static keys / jump labels so a disabled tracepoint compiles down to a nop. The trade-off is coverage: a tracepoint only exists where a developer put one. The full mechanism lives in Tracepoints and Static Keys and Tracepoint Patching.

Dynamic kernel probes — kprobes and kretprobes — let you instrument almost any kernel instruction the developers never anticipated you’d care about. The kprobes documentation says a kprobe lets you “break into any kernel routine and collect debugging and performance information non-disruptively,” trapping “at almost any kernel code address” (kprobes.rst, v6.12). It works by copying the target instruction aside and overwriting it with a breakpoint (int3 on x86), so coverage is enormous but the probe is tied to a volatile internal symbol. See kprobes and kretprobes.

Dynamic userspace probes — uprobes and USDT — do the same trick in user space. A uprobe patches the text of a target binary or shared library; USDT (“Userspace Statically Defined Tracing”) is the static counterpart, a compiled-in nop plus an ELF note the developer placed. The uprobe tracer doc notes that “Uprobe based trace events are similar to kprobe based trace events,” specified by binary PATH:OFFSET (uprobetracer.rst, v6.12). See uprobes and USDT Userspace Statically Defined Tracing.

PMU hardware counters and software events are a different kind of source: not a code hook but a counter. The CPU’s Performance Monitoring Unit (PMU) tallies low-level events — cycles, instructions retired, cache misses, branch mispredictions. The perf wiki describes accessing “CPU hardware registers that track events like instructions executed, cache-misses suffered, or branches mispredicted” (perf wiki). Alongside the hardware PMU sit kernel software events (page faults, context switches, CPU-clock ticks) that need no hardware support. These feed perf Profiling Tool and the PMU leaf.

The In-Kernel Mechanisms

ftrace (“function tracer”) is the kernel’s native tracing framework. Its own documentation insists it is more than a function tracer: it “is actually a framework of several assorted tracing utilities,” including latency tracing and event tracing (ftrace.rst, v6.12). It can trace every kernel function entry (via the compiler’s -pg/mcount/fentry hooks), draw a call graph with the function_graph tracer, and measure latency. It is driven entirely through files. See ftrace Framework and The Function Tracer.

The trace-event subsystem is the shared substrate that ftrace, perf, and eBPF all read. When a TRACE_EVENT macro defines a tracepoint, it simultaneously generates the ring-buffer record layout and a self-documenting format file. Steven Rostedt, who wrote the macro, notes its design goal was deliberately tool-agnostic: “TRACE_EVENT() is also used by perf, LTTng and SystemTap” (Rostedt, LWN 379903). See The Trace Event Pipeline and The TRACE_EVENT Macro.

The perf_events core is the in-kernel subsystem behind the perf command and the perf_event_open(2) syscall. That syscall “returns a file descriptor, for use in subsequent system calls” and takes a perf_event_attr whose type field selects the source: PERF_TYPE_HARDWARE, PERF_TYPE_SOFTWARE, PERF_TYPE_TRACEPOINT, or PERF_TYPE_HW_CACHE (perf_event_open(2)). It multiplexes the PMU, software events, and every tracepoint, in either counting or sampling mode. See The perf_event_open Syscall.

eBPF programs are the modern convergence point. Instead of shipping every event to user space, an eBPF program runs inside the kernel — attached to a tracepoint, kprobe, uprobe, or perf event — and aggregates on the spot (histograms, counts) under the verifier’s safety guarantees. The engine itself (verifier, JIT, maps, CO-RE/BTF) is owned by Linux eBPF MOC; this subsystem treats eBPF as a consumer of the same sources. See bpftrace and BCC BPF Compiler Collection.

The File Interfaces

Long before dedicated tracers existed, Linux exposed state as files, and those interfaces are still the always-available, zero-tooling observability surfaces.

tracefs is the control surface for ftrace and the trace-event subsystem. The canonical mount is /sys/kernel/tracing; for backward compatibility with systems predating kernel 4.1 it also appears at /sys/kernel/debug/tracing when debugfs is mounted (ftrace.rst, v6.12). Key files include current_tracer (select the active tracer), trace (non-consuming snapshot of the buffer), trace_pipe (a consuming, blocking stream), and set_ftrace_filter (limit which functions are traced) (ibid.). Trace events live under events/<subsystem>/<event>/, each with an enable, format, and filter file (events.rst, v6.12). See The tracefs Filesystem.

/proc and /sys expose per-process and system state as files: /proc/<pid>/, /proc/meminfo, /proc/stat for the proc filesystem; /sys for the device model and kernel-object attributes. The existing vault note procfs and the proc Filesystem covers this surface, and the tracing-side leaf is The proc Filesystem.

The printk ring buffer is the oldest observability surface of all. printk() is “the standard tool we have for printing messages and usually the most basic way of tracing and debugging,” writing into “a ring buffer exported to userspace through /dev/kmsg” and read by dmesg (printk-basics.rst, v6.12). Its eight log levels run from KERN_EMERG (0) to KERN_DEBUG (7) (ibid.). See printk and Log Levels and The Kernel Ring Buffer and dmesg.

The Front-end Tools

Front-ends are where you actually type commands, and each is bound to a mechanism more than to a source. perf (resident in the kernel tree at tools/perf) drives the perf_events core: perf stat counts events, perf record samples, perf report analyses (perf wiki). trace-cmd and the KernelShark GUI drive ftrace through tracefs without you having to echo into files by hand. bpftrace is a high-level, awk-like language for eBPF one-liners, and BCC (BPF Compiler Collection) is the Python/C toolkit behind tools like execsnoop and biolatency. Brendan Gregg’s tracing-tools map shows the same point from the practitioner’s side: BPF “can run custom programs on other events, including kernel- and user-level dynamic tracing (kprobes and uprobes), static tracing (tracepoints), and hardware events” (Gregg, SCALE 2017).

A Worked Example — One Probe, Three Consumers

To make the decoupling concrete, take the do_sys_openat2 path (the kernel side of open). All three of the following observe the same underlying instrumentation:

# (1) ftrace via tracefs: enable the openat syscall tracepoint, read the buffer.
cd /sys/kernel/tracing
echo 1 > events/syscalls/sys_enter_openat/enable
cat trace_pipe                       # consuming stream of open() calls
 
# (2) perf: record the same tracepoint, then summarise.
perf record -e syscalls:sys_enter_openat -a sleep 5
perf report
 
# (3) bpftrace: attach an eBPF program to a kprobe on the function and count in-kernel.
bpftrace -e 'kprobe:do_sys_openat2 { @[comm] = count(); }'

Line-by-line: in (1), echo 1 > .../enable flips the static key behind the sys_enter_openat tracepoint, after which records flow into the trace-event ring buffer; trace_pipe drains them (events.rst, v6.12). In (2), perf record -e syscalls:sys_enter_openat opens that same tracepoint through perf_event_open(2) with type = PERF_TYPE_TRACEPOINT, but routes the data through the perf ring buffer instead of tracefs. In (3), bpftrace attaches an eBPF program to a kprobe on the underlying function and aggregates per-process counts in a BPF map, never shipping individual events to user space. Three mechanisms, three front-ends, one observable behaviour — the pipeline’s defining property.

Tracing versus Profiling versus Logging

A frequent confusion is treating all of this as one thing. They are distinct modes, covered in Tracing vs Profiling vs Logging. Tracing records a stream of discrete events (every open, every context switch) — high fidelity, potentially high volume. Profiling samples periodically (every N cycles, grab the stack) or counts in aggregate (total cache misses) — bounded overhead, statistical picture, the natural mode for the PMU. Logging (printk, application logs) is human-readable text emitted at chosen points — low volume, narrative, not structured for analysis. The perf_event_open man page draws the same line internally between “counting” events read with read(2) and “sampling” events delivered to an mmap ring buffer (perf_event_open(2)).

Overhead and Safety — Why This Is Usable in Production

The reason a production kernel can ship thousands of always-compiled-in tracepoints is that a disabled one costs essentially nothing: static-key patching makes it a literal nop until enabled (tracepoints.rst, v6.12). Dynamic probes cost more — the kprobes doc cites roughly 0.5 microseconds for an unoptimised kprobe hit, dropping to 0.07–0.1 microseconds when jump-optimised (kprobes.rst, v6.12) — so on a hot path the volume of hits, not the per-hit cost, is what bites. eBPF’s in-kernel aggregation is the modern answer: summarise where the data is rather than copying it out. The general treatment is in Observability Overhead and Safety.

Common Misunderstandings

The first is believing you must choose a tool up front. You don’t choose a tool, you choose a source; the front-end is almost an afterthought because the same source is reachable from all of them. The second is conflating tracepoints with ftrace — tracepoints are a source that predates and is independent of any single mechanism; ftrace, perf, and eBPF all read them. The third is assuming /proc and dmesg are “not real observability.” For “what is the system doing right now, cheaply?” they are the correct first stop, before any tracer. The fourth is treating dynamic and static probes as interchangeable; they have genuinely different stability and coverage trade-offs, which is exactly why Static vs Dynamic Tracing is a note of its own.

Decision Sketch — Which Surface First

For “what is my system doing right now, cheaply?” reach for /proc and /sys (procfs and the proc Filesystem) and then perf top. For “why is this slow / where are the cycles?” go to perf record → flame graph → drill with bpftrace. For “what is this one function doing / who calls it?” use the function_graph tracer (kernel) or a uprobe/kprobe one-liner in bpftrace. For “I need a histogram without per-event overhead,” aggregate in a BPF map via bpftrace/BCC rather than raw ftrace. For “the kernel hung or panicked,” that is a different class of tool — magic SysRq, kdump, kgdb — covered in the parent MOC’s §8.

See Also

Graph View