vault

The Trace Event Pipeline

12 min read

The Trace Event Pipeline

Every Linux tracing tool, despite wildly different front-ends, walks the same three-stage path: a source fires, the record is written into a per-CPU lockless ring buffer, and a consumer reads or aggregates it. A source is any instrumentation point — a static tracepoint, a dynamic kprobe/uprobe, or a PMU hardware event. The buffer is a finite, per-CPU region that the firing context writes into without taking any cross-CPU lock — the property that makes tracing’s hot path cheap enough to leave running. The consumer is whichever tool drains it: ftrace via tracefs files, perf via the perf_event_open(2) mmap’d buffer, or an eBPF program that aggregates in-kernel. The decisive architectural choice is decoupling: a source does not know or care which consumer reads it, so one source can feed many consumers and learning the sources once unlocks every front-end (Linux v6.12 ring-buffer-design.rst).

This note traces that pipeline end to end as a mechanism, pinned to the 6.12/6.18 LTS kernels. The ring buffer’s internals — the page-list, the reader-page swap, the head/tail/commit pointers — are deep enough to warrant their own treatment in The Kernel Ring Buffer Concept; here the focus is the flow (how a record gets from source to consumer) and the decoupling that flow enables. The three-modes framing of why you’d consume the same source differently lives in Tracing vs Profiling vs Logging.

Mental Model — A Producer/Consumer Conveyor, One Per CPU

Think of the pipeline as a conveyor belt with three stations, and crucially one independent belt per CPU so producers never collide across cores.

flowchart LR
  subgraph SRC["SOURCES (fire in the running context)"]
    TP["tracepoint<br/>(static, compiled-in)"]
    KP["kprobe / uprobe<br/>(dynamic, patched-in)"]
    PMU["PMU event<br/>(hardware counter overflow)"]
  end
  subgraph BUF["PER-CPU RING BUFFER (lockless)"]
    direction TB
    C0["CPU0 buffer<br/>pages + reader_page"]
    C1["CPU1 buffer"]
    Cdots["..."]
  end
  subgraph CON["CONSUMERS (drain the buffer)"]
    FT["ftrace / trace-cmd<br/>(tracefs files)"]
    PF["perf<br/>(perf_event_open mmap)"]
    BPF["eBPF program<br/>(in-kernel aggregate)"]
  end
  TP --> C0
  KP --> C0
  PMU --> C0
  TP --> C1
  C0 --> FT
  C0 --> PF
  C0 --> BPF
  C1 --> FT

The source→buffer→consumer pipeline, with a separate ring buffer per CPU. What it shows: a writer always writes to the buffer of the CPU it is running on (no cross-CPU contention), while any consumer can drain any CPU’s buffer. The insight to take: the per-CPU split is what removes locking from the hot path, and the source-to-consumer fan-out (one source, three consumers shown) is what lets ftrace, perf, and eBPF all read the same instrumentation without coordinating.

The two load-bearing properties are visible in this picture. First, per-CPU buffers mean a tracepoint firing on CPU 3 writes to CPU 3’s buffer with no lock that CPU 0 could contend — the source comment in ring_buffer.c states it directly: “A writer may only write to a buffer that is associated with the CPU it is currently executing on. A reader may read from any per cpu buffer” (ring_buffer.c). Second, decoupling: the arrows from a source fan out to multiple consumers because the source writes a self-describing record and walks away — it has no handle on, and no dependency on, the consumer.

Stage 1 — The Source Fires

A source is a point in code (kernel or userspace) that, when reached and enabled, produces a record. The archetype is the static tracepoint, declared with the TRACE_EVENT/DECLARE_TRACE machinery and placed by kernel developers at semantically meaningful spots. Its defining virtue is that when off it costs almost nothing: the tracepoints.rst documentation explains an off tracepoint “has no effect, except for adding a tiny time penalty (checking a condition for a branch) and space penalty” (Linux v6.12 tracepoints.rst). When on, “the function you provide is called each time the tracepoint is executed, in the execution context of the caller” — i.e. the record is produced synchronously, in the firing context, not handed to a worker thread. (In production this “tiny time penalty” is itself elided to a literal nop via static keys; see Static Keys and Tracepoint Patching.)

Dynamic sources — kprobes and uprobes — reach the same buffer by a different door: they patch a breakpoint (or optimized jump) into an arbitrary instruction, and the trap handler produces the record. PMU sources are different again: a hardware counter overflows after N events and raises an interrupt whose handler captures a sample. What unifies all three is the next stage: they all reserve space in the ring buffer and commit a record there. The source’s only job is to call into the buffer’s reserve/commit API with its payload.

Stage 2 — The Per-CPU Lockless Ring Buffer

This is the heart of the pipeline and the reason tracing is affordable. Each CPU owns a struct ring_buffer_per_cpu — a linked list of pages (sub-buffers) plus a dedicated reader_page, with head_page (where reads happen), tail_page (where writes happen), and commit_page (last finished write) pointers (ring_buffer.c). A write proceeds by reserving space at the tail and later committing it; the public API is ring_buffer_lock_reserve() / ring_buffer_unlock_commit() (ring_buffer.h). Despite the name “lock_reserve”, the fast path takes no spinlock that another CPU could contend — it disables preemption and uses atomic cmpxchg operations to advance the tail. The design doc’s terminology section defines cmpxchg as the “hardware-assisted atomic transaction” A = B if previous A == C, and the file notes “We will be using cmpxchg soon to make all this lockless” (ring-buffer-design.rst).

The subtle part is nested writers. A write can be interrupted by an IRQ, NMI, or softirq whose handler also traces — so multiple writes can be in flight on one CPU. The design doc captures the invariant: “No two writers can write at the same time (on the same per-cpu buffer), but a writer may interrupt another writer … The writers act like a ‘stack’” — writer1 starts, is preempted by writer2, which is preempted by writer3; writer3 finishes first, then writer2, then writer1 (ring-buffer-design.rst). The implementation tracks exactly which interrupt level is writing via a current_context bitmask over five contexts — RB_CTX_TRANSITION, RB_CTX_NMI, RB_CTX_IRQ, RB_CTX_SOFTIRQ, RB_CTX_NORMAL — capped at MAX_NEST of 5 (ring_buffer.c). This is why no cross-CPU lock is needed: on a single CPU the natural stacking of interrupt contexts already serializes writers, and cmpxchg handles the page-advance races between the interrupted and interrupting writers.

The record itself is compact. Each entry begins with a struct ring_buffer_event whose header packs type_len:5, time_delta:27 into a single 32-bit word — a 5-bit length/type field and a 27-bit time delta from the previous event (ring_buffer.h). Storing a delta rather than a full timestamp is the space optimization that makes per-event tracing cheap; when 27 bits is not enough, a special RINGBUF_TYPE_TIME_EXTEND record is inserted. The ring_buffer_print_entry_header() function emits exactly this layout for consumers to parse: “type_len : 5 bits, time_delta : 27 bits, array : 32 bits” (ring_buffer.c).

Overwrite vs Producer/Consumer Mode

A finite buffer must decide what to do when full, and the ring buffer supports two modes. The design doc: “Producer/consumer mode is where if the producer were to fill up the buffer before the consumer could free up anything, the producer will stop writing … This will lose most recent events. Overwrite mode is where … the producer will overwrite the older data … This will lose the oldest events” (ring-buffer-design.rst). For ftrace, the choice is the overwrite trace option, default 1 (overwrite): “If ‘1’ (default), the oldest events are discarded and overwritten. If ‘0’, then the newest events are discarded” (ftrace.rst). The default makes sense for “show me what just happened”: you want the most recent history when you stop tracing, so dropping the oldest is correct. Producer/consumer (blocking) mode suits a consumer that must not miss events and is willing to apply backpressure.

Stage 3 — The Consumer Drains the Buffer

Consumers read the same buffer through different interfaces, and the consuming vs non-consuming distinction matters. Via tracefs, the trace file is a non-consuming snapshot — “this file is not a consumer … it will produce the same output each time it is read” — whereas trace_pipe is a consuming stream: “this file is a consumer. This means reading from this file causes sequential reads to display more current data. Once data is read from this file, it is consumed, and will not be read again” (ftrace.rst). For zero-copy bulk extraction there is trace_pipe_raw, drained with splice(2), gated by buffer_percent (a watermark controlling how full a per-CPU buffer must be before a blocked reader is woken) (ftrace.rst). The per-CPU structure is exposed directly: per_cpu/cpu0/trace, per_cpu/cpu0/trace_pipe, and per_cpu/cpu0/stats (with the overrun/dropped counters) each address a single CPU’s buffer.

The perf consumer takes a different door to the same kind of data. perf_event_open(2) “returns a file descriptor, for use in subsequent system calls (read, mmap, …)”; sampling events “periodically write measurements to a buffer that can then be accessed via mmap(2),” structured as “1+2^n pages, where the first page is a metadata page (struct perf_event_mmap_page)” with data_head and data_tail pointers the consumer advances (perf_event_open(2)). To attach to a tracepoint, perf uses PERF_TYPE_TRACEPOINT with the event’s numeric ID — and that ID comes from the source’s own metadata, the bridge to the next section. The eBPF consumer is different again: instead of streaming records out, an eBPF program attached to the source aggregates in-kernel into a map, so the “consumer” reads a summary, not a record stream — the lowest-overhead drain.

Self-Describing Records — The format Metadata

Decoupling only works if a consumer that knows nothing about a source can still parse its records. That is what the format file provides. Every trace event has one: “Each trace event has a ‘format’ file associated with it that contains a description of each field in a logged event. This information can be used to parse the binary trace stream” (Linux v6.12 events.rst). Each field is described as field:field-type field-name; offset:N; size:N;, and the file also carries the event’s numeric ID and a print fmt. For sched_wakeup the format declares five common_ fields followed by comm, pid, prio, success, cpu, each with its byte offset and size, plus ID: 60 (events.rst).

This metadata is the linchpin of the whole pipeline’s flexibility. A consumer reads the binary record (compact, fast to write) and uses the format file to decode it (offsets, sizes, types) into named fields — and perf uses the same file’s ID to select the event in perf_event_open. So the source ships a tiny binary record on the hot path and a one-time text schema out of band; the consumer reconciles them. No consumer needs compiled-in knowledge of any source’s layout, which is precisely why a tracepoint added in a new kernel is immediately usable by an old perf or trace-cmd binary.

A Concrete Walk-through — sched_switch to a Flame of Off-CPU Time

Tie it together with one event followed through all three stages. (1) Source: echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable arms the tracepoint; now every context switch on every CPU produces a record in that CPU’s buffer, synchronously in the scheduler’s context. (2) Buffer: each record is a ring_buffer_event with the 27-bit delta header plus the sched_switch payload (prev/next pid, comm, priority, state), reserved-and-committed via cmpxchg at the per-CPU tail, with overwrite mode dropping oldest history if you let it run. (3) Consumer, three ways: cat /sys/kernel/tracing/trace gives a non-consuming human snapshot; cat trace_pipe streams it live and consuming; or perf record -e sched:sched_switch mmaps the records out via perf_event_open using the format file’s ID; or a bpftrace script on tracepoint:sched:sched_switch accumulates off-CPU durations into an in-kernel histogram, emitting only the summary. Same source, four consumers, zero changes to the source — the decoupling in action.

Failure Modes and Common Misunderstandings

Lost events from overwrite. Leave a high-frequency source armed and the per-CPU buffer wraps; in the default overwrite mode the oldest records vanish silently. The symptom is in per_cpu/cpuN/stats — overrun (overwritten) and dropped_events counters (ftrace.rst, via the ring_buffer_per_cpu overrun/dropped_events fields). A trace with non-zero overrun is incomplete; either enlarge buffer_size_kb, narrow the source with filters, or switch to in-kernel aggregation. See Observability Overhead and Safety.

The sub-buffer size caps event size. “An event can not be bigger than the size of the sub buffer,” normally one page (4 KB on x86) minus per-sub-buffer metadata (ftrace.rst). An over-large record (e.g. a deep stack trace) is dropped, not truncated-then-recovered. buffer_subbuf_size_kb can raise the cap, but resizing it discards existing buffer contents.

Reading trace expecting a live stream. Because trace is non-consuming, polling it in a loop re-reads the same window and misses events that scroll past between reads; for live capture you must use the consuming trace_pipe/trace_pipe_raw. Conversely, reading trace_pipe consumes the data, so two readers split the stream rather than each seeing all of it.

Cross-CPU ordering is not global. Because each CPU buffers independently, merging per-CPU streams into one timeline requires the per-event timestamps; the buffers themselves impose no global order. Tools reconcile via the time_delta/timestamp fields, but a naive concatenation of per-CPU buffers is not chronological.

Alternatives and Boundaries

The pipeline described is the ftrace/trace-event ring buffer (kernel/trace/ring_buffer.c). The perf subsystem has its own ring-buffer implementation (the mmap’d perf_event_mmap_page buffer), and LTTng uses a third (its own lockless per-CPU buffers in a separate framework). The shape — per-CPU, lockless, source→buffer→consumer — is identical across all three; the code is not shared. eBPF programs can write to yet another structure, the BPF ring buffer (BPF_MAP_TYPE_RINGBUF), or aggregate into maps and skip streaming entirely. The takeaway is that “per-CPU lockless ring buffer fed by a decoupled source and drained by an independent consumer” is a pattern the kernel re-implements wherever low-overhead instrumentation is needed, not a single shared object — and understanding the pattern once transfers across all of them.

See Also

Graph View

Backlinks