vault

perf Profiling Tool

15 min read

perf Profiling Tool

perf is the Linux kernel’s official userspace performance-analysis suite — a single multi-tool front-end to the kernel’s perf_events subsystem. Its own man page frames the scope precisely: “Performance counters for Linux are a new kernel-based subsystem that provide a framework for all things performance analysis. It covers hardware level (CPU/PMU, Performance Monitoring Unit) features and software features (software counters, tracepoints) as well” (perf.txt, v6.12). perf is invoked as perf <subcommand> — perf stat to count events, perf record/perf report to sample into a perf.data file and analyze it, perf top for a live top-style profile, perf trace for an strace-like syscall view, perf probe to define dynamic kprobes/uprobes, and perf list to browse the event catalog. Every one of these rests on a single kernel entry point — the perf_event_open(2) syscall — which returns a file descriptor representing a counter (perf_event_open(2)); the syscall’s mechanics are covered in The perf_event_open Syscall. The grand organizing split, which the rest of this section details, is counting (aggregate totals — how many cache misses?) versus sampling (periodic snapshots — where are the cache misses?).

Mental Model

The right way to picture perf is as a thin, scriptable shell over a hardware-and-kernel measurement engine you can’t touch directly. The actual measuring is done by two things: the CPU’s Performance Monitoring Unit (PMU) — a small bank of hardware registers that silently tally micro-architectural events like CPU cycles, retired instructions, cache misses, and branch mispredictions — and the kernel’s perf_events core, which multiplexes those scarce PMU registers across competing users, adds software events (context switches, page faults, a software clock) and every static tracepoint, and exposes the lot through perf_event_open(2). The perf binary itself holds almost no measurement logic; it parses your event spec, opens the right counters via the syscall, reads them back, and formats the result. Learn the engine (PMU + perf_events) once and every subcommand becomes a different presentation of the same underlying counters.

The second half of the model is the counting-vs-sampling duality. A PMU counter can be used two ways. Counting: start it at zero, run the workload, read the total. You learn how much (3.1 billion cycles, 2% cache-miss rate) but nothing about where. Sampling: program the counter to overflow every N events and, on each overflow, have the kernel snapshot the instruction pointer (and optionally the whole call stack). Collect millions of those snapshots and the histogram of where they land is your profile — the hot functions are simply the addresses that show up most. perf stat is the counting tool; perf record/perf top are the sampling tools. This distinction is developed fully in Counting vs Sampling Mode.

flowchart TB
  subgraph HW["Hardware"]
    PMU["PMU counters<br/>cycles, instructions,<br/>cache-misses, branches"]
  end
  subgraph KERN["Kernel: perf_events core"]
    SW["software events<br/>(context-switches, faults,<br/>cpu-clock)"]
    TP["tracepoints / k+uprobes"]
    SYS["perf_event_open(2)<br/>returns an fd per counter"]
  end
  subgraph TOOL["perf userspace suite"]
    LIST["perf list<br/>(catalog)"]
    STAT["perf stat<br/>(COUNT totals)"]
    REC["perf record -&gt; perf.data<br/>(SAMPLE)"]
    REP["perf report / perf annotate<br/>(analyze perf.data)"]
    TOP["perf top<br/>(live sample)"]
    TR["perf trace<br/>(strace-like)"]
    PROBE["perf probe<br/>(define k/uprobes)"]
  end
  PMU --> SYS
  SW --> SYS
  TP --> SYS
  SYS --> STAT
  SYS --> REC
  SYS --> TOP
  SYS --> TR
  REC --> REP
  LIST -.->|"names feed -e"| STAT
  PROBE -.->|"creates events"| LIST

The perf stack. What it shows: hardware PMU counters, kernel software events, and tracepoints/probes all funnel through the single perf_event_open(2) syscall, which every perf subcommand calls; counting (perf stat) and sampling (perf record/top) are two uses of the same counters. The insight to take: perf list names the events, perf probe can create new ones, and the rest of the subcommands are just different views — counting vs sampling vs live vs strace-like — over the identical event machinery.

The Subcommand Families

perf ships dozens of subcommands (perf --list-cmds enumerates them). They cluster into a handful of families. The names below come from command-list.txt, v6.12, each marked mainporcelain (a primary, user-facing command).

Counting — perf stat. Runs a command (or attaches to a PID/CPU) and prints aggregate counter totals: “This command runs a command and gathers performance counter statistics from it” (perf-stat.txt, v6.12). With no -e, it prints a default set (task-clock, context-switches, cycles, instructions, branches, branch-misses). This is the first thing to run — it tells you whether you are CPU-bound, memory-bound, or branch-bound before you invest in a full profile.

Sampling — perf record + perf report (+ perf annotate). perf record “runs a command and gathers a performance counter profile from it, into perf.data - without displaying anything” (perf-record.txt, v6.12); perf report then reads perf.data and presents the hot functions, and perf annotate drills into a single function showing per-instruction sample counts against the disassembly/source. The pair is the workhorse for “where are my cycles going?” — covered in perf record and perf report.

Live — perf top. “This command generates and displays a performance counter profile in real time” (perf-top.txt, v6.12) — a continuously updating, top-like view of the hottest functions on the running system, with no perf.data file. Use it for “what is hot right now.” Paired with perf stat in perf stat and perf top.

strace-like — perf trace. Shows “the events associated with the target, initially syscalls, but other system events like pagefaults, task lifetime events, scheduling events, etc.” (perf-trace.txt, v6.12). It is a lower-overhead, broader-scope strace: built on the raw_syscalls syscall tracepoints rather than ptrace, so it can watch syscalls and pagefaults and scheduling in one stream, system-wide if you ask.

Dynamic probes — perf probe. “Defines dynamic tracepoint events, by symbol and registers without debuginfo, or by C expressions (C line numbers, C function names, and C local variables) with debuginfo” (perf-probe.txt, v6.12). It is the debuginfo-aware front-end to the uprobe event interface: you name a function and a variable, perf probe consults DWARF to compute the register/offset, and installs the probe via the tracefs kprobe_events/uprobe_events files. -x PATH targets a userspace binary or library (uprobes) instead of the kernel.

The catalog — perf list. “Displays the symbolic event types which can be selected in the various perf commands with the -e option” (perf-list.txt, v6.12). It is the dictionary of every event name perf understands on this machine — hardware events, software events, hardware cache events, tracepoints, and CPU-specific “Kernel PMU events.” Always start here when you don’t know an event’s name.

Beyond these, command-list.txt lists specialized porcelain: perf sched (scheduler latency analysis, built on scheduling tracepoints), perf lock (lock contention), perf mem and perf c2c (memory access and cache-line false-sharing analysis), perf kvm (guest profiling), perf ftrace (a wrapper over ftrace), perf script (post-process perf.data programmatically), and perf bench (micro-benchmarks). These are the same engine pointed at narrower questions.

The Event-Spec Syntax (-e)

Every sampling/counting subcommand takes -e <events>, a comma-separated list. This grammar is the connective tissue across the whole suite, so it is worth learning once. The forms, drawn from perf-list.txt, v6.12 and perf-stat.txt, v6.12:

# Symbolic hardware/software events, comma-separated:
perf stat -e cycles,instructions,cache-misses,branch-misses ./myapp
 
# A tracepoint, using subsystem:event notation:
perf record -e sched:sched_switch -a sleep 5
 
# Event modifiers (colon-suffix) to scope the count:
perf stat -e cycles:u ./myapp        # :u = user-space only
perf stat -e cycles:k ./myapp        # :k = kernel only
perf record -e cycles:p ./myapp      # :p = increase sampling precision (PEBS/IBS)
 
# A raw PMU encoding, when no symbolic name exists:
perf stat -e r1a8 -a sleep 1
 
# A fully-qualified PMU event with parameters:
perf record -e 'cpu/event=0xa8,umask=0x1,name=LSD.UOPS_CYCLES,cmask=0x1/' ./myapp

Walking the pieces:

  • Symbolic names — cycles, instructions, cache-misses, branch-misses, context-switches, page-faults, cpu-clock. These are the portable aliases perf list prints; the kernel maps them onto whatever raw PMU encoding the current CPU uses, so cycles works on Intel, AMD, and ARM alike.
  • subsystem:event — selects a static tracepoint (sched:sched_switch, block:block_rq_issue, syscalls:sys_enter_openat). The : separates the tracepoint subsystem from the event name; this is the same namespace exposed under events/ in tracefs.
  • Modifiers — a colon-suffix restricts where the event is counted: u user-space, k kernel, h hypervisor, G guest (KVM), H host (perf-list.txt, v6.12). The p modifier (precision 0–3) requests precise event-based sampling so the reported instruction pointer is the actual faulting/retiring instruction rather than a skidded-forward one — see Precise Event-Based Sampling.
  • Raw encodings — rNNN (e.g. r1a8) feeds a literal hex event-select to the PMU when no symbolic name exists; the cpu/event=...,umask=.../ form spells out the individual register fields. The doc notes that on x86 “only the following bit fields can be set in x86 counter registers: event, umask, edge, inv, cmask” (perf-list.txt, v6.12). You reach for this only for esoteric counters your perf list doesn’t name.

Because every subcommand shares this -e grammar, the same event spec works whether you are counting (perf stat -e ...), sampling (perf record -e ...), or watching live (perf top -e ...).

Reading a perf stat Report

A counting run is the densest single screen of performance information Linux gives you, so it pays to read one carefully. perf stat can target a command, a set of PIDs (-p), or all CPUs system-wide (-a, “the default if no target is specified” per perf-stat.txt, v6.12); -r N repeats the run up to 100 times and reports mean and standard deviation; -I N prints rolling deltas every N milliseconds (minimum 1 ms) for a time series. A default invocation produces something like:

$ perf stat ./myapp
          1,234.56 msec task-clock        #    0.998 CPUs utilized
                42      context-switches  #   34.018 /sec
                 3      cpu-migrations    #    2.430 /sec
             1,024      page-faults       #  829.6  /sec
     4,102,553,001      cycles            #    3.323 GHz
     2,051,276,500      instructions      #    0.50  insn per cycle
       410,255,300      branches          #  332.3  M/sec
        20,512,765      branch-misses     #    5.00% of all branches
       1.236 seconds time elapsed

The counts on the left are raw totals; the # annotations on the right are the derived ratios that actually diagnose. insn per cycle (IPC) of 0.50 is low — a modern core can retire 3–4 instructions per cycle, so 0.5 says the CPU is stalling most of the time, usually waiting on memory; that is the cue to perf record cache-related events next. task-clock divided by wall time gives CPUs utilized (here ~1.0, i.e. effectively single-threaded). branch-misses at 5% is high and would itself cost cycles. The whole point of perf stat is this triage: the ratios tell you which kind of profile to record next, so you don’t blindly sample.

When you ask for more events than the PMU has physical counters, the kernel time-multiplexes them — each event runs on a counter for a slice, and perf scales the partial count up to a full-run estimate. Multiplexed events are flagged in the output (a trailing percentage like (80.00%) showing the fraction of time the event was actually scheduled), and the resulting numbers are estimates, not exact totals. The fix is to request fewer events per run, or to use --repeat and accept the averaged estimate. This is why perf stat -e a,b,c,d,e,f,g,h on a 4-counter PMU is less trustworthy than two runs of four events each.

A Typical Investigation Flow

The subcommands compose into a standard top-down workflow:

# 1. Triage: is this even CPU-bound? Count the big-picture totals.
perf stat ./myapp
#    -> low IPC (instructions-per-cycle) + high cache-misses = memory-bound.
 
# 2. Localize: sample to find the hot functions.
perf record -F 999 -g ./myapp        # -F 999 = ~999 Hz; -g = capture call stacks
perf report                          # interactive: hottest functions, expandable callers
 
# 3. Zoom: per-instruction heat inside the hot function.
perf annotate <function>
 
# 4. Or watch it live, system-wide, without a perf.data file:
perf top -g

-F 999 sets the sampling frequency (≈999 samples/second; the odd number avoids lock-step with periodic kernel activity at round Hz). -g enables call-graph capture, which is what turns a flat function list into a tree you can later fold into a flame graph (Flame Graphs and Stack Sampling). The unwinding method behind -g defaults to frame pointers (fp) and can be switched to dwarf or hardware lbr (perf-record.txt, v6.12).

Resting on perf_event_open(2)

Everything above bottoms out at one syscall. perf_event_open() “returns a file descriptor, for use in subsequent system calls (read(2), mmap(2), prctl(2), fcntl(2), etc.)” (perf_event_open(2)) — that fd is a counter. perf stat opens fds and read(2)s the totals; perf record/perf top mmap(2) a ring buffer onto the fd and drain samples as the counter overflows. The syscall takes a large struct perf_event_attr describing the event type, the sample period/frequency, what to record on each sample, and the target (which PID, which CPU). The struct’s full layout — type, config, sample_type, the exclude_* bits that the :u/:k modifiers set — is intricate, and is deferred to The perf_event_open Syscall. The takeaway here is only the boundary: perf is the ergonomic surface; perf_event_open(2) is the load-bearing kernel API beneath it, and you could call it directly (some profilers and language runtimes do) but rarely want to.

Uncertain

Verify: that perf stat with no -e uses exactly the default event set listed above (task-clock, context-switches, cpu-migrations, page-faults, cycles, instructions, branches, branch-misses), and that perf top’s implicit default event is cycles. Reason: the v6.12 perf-stat.txt/perf-top.txt man pages describe the behavior but do not pin the exact default-event list, which also varies by CPU/PMU availability and perf version. To resolve: run perf stat true and perf top (then check the header) on a 6.12/6.18 box, or read tools/perf/builtin-stat.c / builtin-top.c for the hardcoded default event arrays. uncertain

Failure Modes and Common Misunderstandings

  • perf stat ran but events show <not supported> or <not counted>. Inside VMs/containers the PMU is often unavailable or virtualized away, so hardware events silently degrade. <not counted> also appears when more events were requested than physical PMU counters exist and multiplexing kicked in (the count is then scaled/estimated). Software events (cpu-clock, context-switches) always work because they don’t need the PMU.
  • Need root, or not. Many operations require CAP_PERFMON/root depending on kernel.perf_event_paranoid (sysctl). A high paranoid level blocks kernel sampling and tracepoints for unprivileged users; symptoms are permission errors from perf record -a or empty kernel stacks.
  • Flat-looking profiles with -g. If the target was built without frame pointers and you used the default fp unwinding, call stacks collapse to one frame. Switch to -g dwarf (needs debuginfo) or -g lbr (Intel), or rebuild with -fno-omit-frame-pointer. See Flame Graphs and Stack Sampling.
  • perf trace is not strace. It does not use ptrace and won’t stop the process per-syscall; semantics and overhead differ. It can miss/merge fast events and reports system-wide differently. For exact per-process syscall fidelity, strace is still the reference; for low-overhead and broader scope, perf trace.
  • Stale perf.data. perf report resolves symbols against the binaries/debuginfo present now; if you rebuilt or stripped the binary after recording, symbolization breaks. Keep the build artifacts, or use perf archive to bundle them.

Alternatives and When to Choose Them

perf is the kernel-native default; its main peers occupy adjacent niches. eBPF tools (bpftrace, BCC) attach to the same events but aggregate in-kernel (histograms, per-key counts) instead of shipping every sample to userspace — better when raw event volume would be overwhelming or when you want custom logic, but they need the BPF toolchain. ftrace/trace-cmd is better for function-call tracing (who calls whom, latency between events) than for PMU sampling. LTTng targets very-low-overhead, high-rate production tracing with a different buffering model. Language-runtime profilers (Go pprof, py-spy) understand runtime-specific frames perf can’t unwind without help. The rule of thumb: PMU counters and CPU profiling → perf; in-kernel aggregation / custom logic → eBPF; function-flow and latency tracing → ftrace; runtime-aware profiling → the language’s own tool.

Production Notes

perf is the foundation of Brendan Gregg’s widely used CPU flame-graph methodology: perf record -F 99 -a -g for a fixed interval, then fold the stacks into an SVG (Flame Graphs and Stack Sampling). In production the practical constraints are sampling frequency (99–999 Hz is the usual band — high enough to localize hot code, low enough to keep overhead near 1%), unwinding cost (-g dwarf is far heavier than fp or lbr because it copies stack memory on every sample), and the perf_event_paranoid policy your fleet enforces. A recurring gotcha is profiling JIT/interpreted runtimes: perf needs a /tmp/perf-<pid>.map symbol file (emitted by the JVM/V8/etc.) to name dynamically generated code, or the profile is a sea of unnamed addresses. For steady-state, low-overhead production observability that survives high event rates, teams increasingly pair perf stat-style counting with eBPF aggregation rather than continuous perf record.

See Also

Graph View

Backlinks