perf Profiling Tool

perf is the Linux kernel’s official userspace performance-analysis suite — a single multi-tool front-end to the kernel’s perf_events subsystem. Its own man page frames the scope precisely: “Performance counters for Linux are a new kernel-based subsystem that provide a framework for all things performance analysis. It covers hardware level (CPU/PMU, Performance Monitoring Unit) features and software features (software counters, tracepoints) as well” (perf.txt, v6.12). perf is invoked as perf <subcommand> — perf stat to count events, perf record/perf report to sample into a perf.data file and analyze it, perf top for a live top-style profile, perf trace for an strace-like syscall view, perf probe to define dynamic kprobes/uprobes, and perf list to browse the event catalog. Every one of these rests on a single kernel entry point — the perf_event_open(2) syscall — which returns a file descriptor representing a counter (perf_event_open(2)); the syscall’s mechanics are covered in The perf_event_open Syscall. The grand organizing split, which the rest of this section details, is counting (aggregate totals — how many cache misses?) versus sampling (periodic snapshots — where are the cache misses?).

Version pin and provenance

Every kernel-source claim below is read from the v6.12 tag of the mainline tree — a maintained long-term-support (LTS) series, chosen so the note stays checkable; mainline itself is on the 7.x series as of 2026-09-04, and anything that changed after v6.12 is dated explicitly where it appears. Every command output shown as “measured” was produced on the machine this note was written on: Fedora Linux 44, kernel 7.1.8-200.fc44.x86_64, AMD Ryzen AI MAX+ 395 (Zen 5, family 0x1A), 16 cores / 32 threads, on 2026-09-04. That machine has no perf binary installed and no root, which turned out to be a feature: every measurement here was taken by calling perf_event_open(2) directly from a short C program, so what you are reading is the kernel interface’s own behaviour rather than a tool’s rendering of it.

Mental Model

The right way to picture perf is as a thin, scriptable shell over a hardware-and-kernel measurement engine you can’t touch directly. The actual measuring is done by two things: the CPU’s Performance Monitoring Unit (PMU) — a small bank of hardware registers that silently tally micro-architectural events like CPU cycles, retired instructions, cache misses, and branch mispredictions — and the kernel’s perf_events core, which multiplexes those scarce PMU registers across competing users, adds software events (context switches, page faults, a software clock) and every static tracepoint, and exposes the lot through perf_event_open(2). The perf binary itself holds almost no measurement logic; it parses your event spec, opens the right counters via the syscall, reads them back, and formats the result. Learn the engine (PMU + perf_events) once and every subcommand becomes a different presentation of the same underlying counters.

The second half of the model is the counting-vs-sampling duality. A PMU counter can be used two ways. Counting: start it at zero, run the workload, read the total. You learn how much (3.1 billion cycles, 2% cache-miss rate) but nothing about where. Sampling: program the counter to overflow every N events and, on each overflow, have the kernel snapshot the instruction pointer (and optionally the whole call stack). Collect millions of those snapshots and the histogram of where they land is your profile — the hot functions are simply the addresses that show up most. perf stat is the counting tool; perf record/perf top are the sampling tools. This distinction is developed fully in Counting vs Sampling Mode.

flowchart TB
  subgraph HW["Hardware"]
    PMU["PMU counters<br/>cycles, instructions,<br/>cache-misses, branches"]
  end
  subgraph KERN["Kernel: perf_events core"]
    SW["software events<br/>(context-switches, faults,<br/>cpu-clock)"]
    TP["tracepoints / k+uprobes"]
    SYS["perf_event_open(2)<br/>returns an fd per counter"]
  end
  subgraph TOOL["perf userspace suite"]
    LIST["perf list<br/>(catalog)"]
    STAT["perf stat<br/>(COUNT totals)"]
    REC["perf record -&gt; perf.data<br/>(SAMPLE)"]
    REP["perf report / perf annotate<br/>(analyze perf.data)"]
    TOP["perf top<br/>(live sample)"]
    TR["perf trace<br/>(strace-like)"]
    PROBE["perf probe<br/>(define k/uprobes)"]
  end
  PMU --> SYS
  SW --> SYS
  TP --> SYS
  SYS --> STAT
  SYS --> REC
  SYS --> TOP
  SYS --> TR
  REC --> REP
  LIST -.->|"names feed -e"| STAT
  PROBE -.->|"creates events"| LIST

The perf stack. What it shows: hardware PMU counters, kernel software events, and tracepoints/probes all funnel through the single perf_event_open(2) syscall, which every perf subcommand calls; counting (perf stat) and sampling (perf record/top) are two uses of the same counters. The insight to take: perf list names the events, perf probe can create new ones, and the rest of the subcommands are just different views — counting vs sampling vs live vs strace-like — over the identical event machinery.

The half of that model people skip is what physically happens on one sample, because it is the step that explains almost every surprise later in this note — the skid, the throttling, the lost records, the cost of --call-graph dwarf. A hardware counter does not politely hand the kernel a number when it feels like it. It is loaded with a negative value, decrements once per event, and overflows, and that overflow is wired to a Non-Maskable Interrupt (NMI) — an interrupt that cannot be deferred by the ordinary interrupt-disable that kernel critical sections use. The whole sample is assembled inside that NMI handler.

sequenceDiagram
    autonumber
    participant PMU as PMU counter<br/>(hardware register)
    participant NMI as NMI handler<br/>(perf_event_overflow)
    participant RB as mmap ring buffer<br/>(kernel writes head)
    participant U as perf record<br/>(userspace, reads tail)
    Note over PMU: counter preloaded with -period<br/>(e.g. -2,000,000 cycles)
    PMU->>PMU: decrement once per event
    PMU-->>NMI: overflow -> NMI raised
    Note over NMI: runs in NMI context:<br/>no sleeping, no locks that<br/>ordinary code may hold
    NMI->>NMI: capture pt_regs (IP, SP, BP)
    NMI->>NMI: optional: walk call stack<br/>(fp) or copy stack+regs (dwarf)
    NMI->>NMI: charge duration via<br/>perf_sample_event_took()
    NMI->>RB: perf_output_begin(): reserve bytes
    alt space available
        NMI->>RB: write PERF_RECORD_SAMPLE, publish head
    else buffer full
        NMI->>RB: increment lost count, emit PERF_RECORD_LOST
    end
    NMI->>PMU: reload counter with -period, re-enable
    U->>RB: poll(fd) wakes at wakeup_events / watermark
    U->>RB: read records from tail up to head
    U->>RB: publish new data_tail (kernel may now reuse)
    U->>U: append to perf.data

One sample, end to end. What it shows: the counter overflow raises an NMI; the entire sample — register capture, stack walk, and ring-buffer write — happens inside that NMI handler, and only then does userspace asynchronously drain the buffer. The insight to take: three consequences fall straight out of this picture. (1) The interrupt does not fire on the instruction that caused the event, it fires some cycles later, which is skid (step 4 records where the CPU is, not where it was) — hence PEBS/IBS. (2) Everything in steps 4–7 is charged against the CPU, which is why the kernel times itself in step 6 and will lower your sample rate if it is too slow. (3) The kernel writes data_head and userspace writes data_tail; if userspace falls behind, samples are dropped, not queued, and you get a PERF_RECORD_LOST record instead of data.

The Subcommand Families

perf ships dozens of subcommands (perf --list-cmds enumerates them). They cluster into a handful of families. The names below come from command-list.txt, v6.12, each marked mainporcelain (a primary, user-facing command).

Counting — perf stat. Runs a command (or attaches to a PID/CPU) and prints aggregate counter totals: “This command runs a command and gathers performance counter statistics from it” (perf-stat.txt, v6.12). With no -e, it prints a default set — the exact list is assembled by add_default_attributes() in tools/perf/builtin-stat.c, v6.12 and is spelled out in the section on reading a perf stat report below. This is the first thing to run — it tells you whether you are CPU-bound, memory-bound, or branch-bound before you invest in a full profile.

Sampling — perf record + perf report (+ perf annotate). perf record “runs a command and gathers a performance counter profile from it, into perf.data - without displaying anything” (perf-record.txt, v6.12); perf report then reads perf.data and presents the hot functions, and perf annotate drills into a single function showing per-instruction sample counts against the disassembly/source. The pair is the workhorse for “where are my cycles going?” — covered in perf record and perf report.

Live — perf top. “This command generates and displays a performance counter profile in real time” (perf-top.txt, v6.12) — a continuously updating, top-like view of the hottest functions on the running system, with no perf.data file. Use it for “what is hot right now.” Paired with perf stat in perf stat and perf top.

strace-like — perf trace. Shows “the events associated with the target, initially syscalls, but other system events like pagefaults, task lifetime events, scheduling events, etc.” (perf-trace.txt, v6.12). It is a lower-overhead, broader-scope strace: built on the raw_syscalls syscall tracepoints rather than ptrace, so it can watch syscalls and pagefaults and scheduling in one stream, system-wide if you ask.

Dynamic probes — perf probe. “Defines dynamic tracepoint events, by symbol and registers without debuginfo, or by C expressions (C line numbers, C function names, and C local variables) with debuginfo” (perf-probe.txt, v6.12). It is the debuginfo-aware front-end to the uprobe event interface: you name a function and a variable, perf probe consults DWARF to compute the register/offset, and installs the probe via the tracefs kprobe_events/uprobe_events files. -x PATH targets a userspace binary or library (uprobes) instead of the kernel.

The catalog — perf list. “Displays the symbolic event types which can be selected in the various perf commands with the -e option” (perf-list.txt, v6.12). It is the dictionary of every event name perf understands on this machine — hardware events, software events, hardware cache events, tracepoints, and CPU-specific “Kernel PMU events.” Always start here when you don’t know an event’s name.

Beyond these, command-list.txt lists specialized porcelain: perf sched (scheduler latency analysis, built on scheduling tracepoints), perf lock (lock contention), perf mem and perf c2c (memory access and cache-line false-sharing analysis), perf kvm (guest profiling), perf ftrace (a wrapper over ftrace), perf script (post-process perf.data programmatically), and perf bench (micro-benchmarks). These are the same engine pointed at narrower questions.

Subcommand	Mode	What it actually measures	Underlying event source	Output
perf stat	counting	totals over a whole run — how much	any event; PMU counters read with read(2)	text table of counts + derived ratios
perf record	sampling	where the events land — where	any event, sample_period/sample_freq set	perf.data (binary)
perf report	analysis	reads perf.data, aggregates by symbol/DSO/call path	— (offline)	TUI or text histogram
perf annotate	analysis	per-instruction sample counts against disassembly	— (offline)	annotated disassembly
perf script	analysis	one text line per raw sample record	— (offline)	text; the flame-graph feed
perf top	sampling, live	hottest symbols right now; default event cycles:P	cycles at max precision	refreshing TUI, no file
perf trace	sampling	syscall enter/exit, pagefaults, task lifetime	raw_syscalls tracepoints, not ptrace	live text stream
perf probe	setup only	nothing — it creates events	writes tracefs kprobe_events/uprobe_events	new event names for -e
perf list	catalog	nothing — enumerates event names on this CPU	queries every registered PMU under /sys/bus/event_source/devices/	event-name listing
perf sched	sampling	run-queue latency, scheduler timeline	sched:* tracepoints	latency report / timehist
perf lock	sampling	lock contention and hold times	lock:* tracepoints or BPF	contention report
perf mem	sampling	load/store latency and memory hierarchy level per access	precise memory events (Intel PEBS mem-loads, AMD IBS op)	per-access latency histogram
perf c2c	sampling	cache-line false sharing — which line, which two CPUs	the same precise memory events, keyed by cache line	HITM report per cache line
perf kvm	sampling	guest-side profile from the host	kvm:* tracepoints + guest symbol maps	guest-symbol report
perf ftrace	wrapper	function-call flow, not PMU sampling	ftrace via tracefs	ftrace trace output
perf bench	synthetic	nothing about your workload — runs microbenchmarks	—	benchmark timings

The perf subcommand families, mapped to what they physically measure (names from command-list.txt, v6.12; per-command behaviour from the matching perf-*.txt in tools/perf/Documentation/). What it shows: only three of these are measurement modes — counting, sampling, and live sampling; the rest are either offline analysis of a perf.data file, setup that creates events, or catalog queries. The insight to take: the column that decides which command you want is Mode, not the topic in the name. perf mem and perf c2c are not separate subsystems; they are perf record with a precise memory event and a different aggregation key. And perf probe, perf list, and perf bench measure nothing at all — mistaking them for profilers is a common beginner detour.

The Event-Spec Syntax (-e)

Every sampling/counting subcommand takes -e <events>, a comma-separated list. This grammar is the connective tissue across the whole suite, so it is worth learning once. The forms, drawn from perf-list.txt, v6.12 and perf-stat.txt, v6.12:

# Symbolic hardware/software events, comma-separated:
perf stat -e cycles,instructions,cache-misses,branch-misses ./myapp
 
# A tracepoint, using subsystem:event notation:
perf record -e sched:sched_switch -a sleep 5
 
# Event modifiers (colon-suffix) to scope the count:
perf stat -e cycles:u ./myapp        # :u = user-space only
perf stat -e cycles:k ./myapp        # :k = kernel only
perf record -e cycles:p ./myapp      # :p = increase sampling precision (PEBS/IBS)
 
# A raw PMU encoding, when no symbolic name exists:
perf stat -e r1a8 -a sleep 1
 
# A fully-qualified PMU event with parameters:
perf record -e 'cpu/event=0xa8,umask=0x1,name=LSD.UOPS_CYCLES,cmask=0x1/' ./myapp

Walking the pieces:

• Symbolic names — cycles, instructions, cache-misses, branch-misses, context-switches, page-faults, cpu-clock. These are the portable aliases perf list prints; the kernel maps them onto whatever raw PMU encoding the current CPU uses, so cycles works on Intel, AMD, and ARM alike.
• subsystem:event — selects a static tracepoint (sched:sched_switch, block:block_rq_issue, syscalls:sys_enter_openat). The : separates the tracepoint subsystem from the event name; this is the same namespace exposed under events/ in tracefs.
• Modifiers — a colon-suffix restricts where the event is counted: u user-space, k kernel, h hypervisor, G guest (KVM), H host (perf-list.txt, v6.12). The p modifier (precision 0–3) requests precise event-based sampling so the reported instruction pointer is the actual faulting/retiring instruction rather than a skidded-forward one — see Precise Event-Based Sampling.
• Raw encodings — rNNN (e.g. r1a8) feeds a literal hex event-select to the PMU when no symbolic name exists; the cpu/event=...,umask=.../ form spells out the individual register fields. The doc notes that on x86 “only the following bit fields can be set in x86 counter registers: event, umask, edge, inv, cmask” (perf-list.txt, v6.12). You reach for this only for esoteric counters your perf list doesn’t name.

Because every subcommand shares this -e grammar, the same event spec works whether you are counting (perf stat -e ...), sampling (perf record -e ...), or watching live (perf top -e ...).

The six event types, and what the kernel does with each

Underneath the friendly names, every event is a (type, config) pair in struct perf_event_attr. There are exactly six types, and they are not variations on one theme — they are routed to genuinely different producers inside the kernel. enum perf_type_id in include/uapi/linux/perf_event.h, v6.12 fixes their numbering as part of the ABI:

attr.type	Value	attr.config means	Who increments it	Needs PMU hardware?	Example -e spelling
PERF_TYPE_HARDWARE	0	one of enum perf_hw_id — a portable alias the arch PMU driver translates	the CPU’s PMU, in silicon	yes	cycles, instructions, branch-misses
PERF_TYPE_SOFTWARE	1	one of enum perf_sw_ids	kernel C code, at the relevant call site	no	context-switches, page-faults, cpu-clock
PERF_TYPE_TRACEPOINT	2	the numeric event id read from tracefs events/<sub>/<evt>/id	the static tracepoint call site	no	sched:sched_switch
PERF_TYPE_HW_CACHE	3	a packed triple: cache level | op << 8 | result << 16	the PMU, via a per-arch lookup table	yes	L1-dcache-load-misses
PERF_TYPE_RAW	4	the literal event-select/umask bits for this CPU model	the PMU, unmediated	yes	r1a8
PERF_TYPE_BREAKPOINT	5	encoded in bp_type/bp_addr/bp_len, not config	the CPU debug registers	(debug regs)	a perf probe-style data watchpoint

The six perf_event_attr.type values (v6.12 UAPI). What it shows: the type field decides which producer the kernel wires the file descriptor to; only types 0, 3, 4 and 5 touch hardware at all. The insight to take: this is why cycles degrades to <not supported> inside a VM while context-switches never does — type 1 events are ordinary kernel counters incremented by ordinary kernel code, with no register behind them. It is also why cycles is portable and r1a8 is not: PERF_TYPE_HARDWARE is an alias namespace that each architecture’s PMU driver maps onto its own encodings, whereas PERF_TYPE_RAW hands your bits straight to the register.

Any value of attr.type above PERF_TYPE_MAX selects a dynamic PMU — a driver that registered itself at runtime and published its own numeric type under sysfs. This is how uncore counters, tracepoints, kprobes, and vendor precise-sampling units all coexist. On the measurement machine (Fedora 44, Zen 5), the registered PMUs are:

$ ls /sys/bus/event_source/devices/
amd_df       amd_umc_0 .. amd_umc_15   cpu          kprobe     power_core
amd_iommu_0  breakpoint                ibs_fetch    msr        software
amd_l3       ibs_op                    power        tracepoint uprobe
 
$ cat /sys/bus/event_source/devices/ibs_op/type
11
$ cat /sys/bus/event_source/devices/cpu/caps/branches
16
$ cat /sys/bus/event_source/devices/cpu/caps/max_precise
0

Measured PMU inventory, 2026-09-04. What it shows: cpu is only one of ~25 registered event sources; amd_df (Data Fabric), amd_l3, and amd_umc_* (Unified Memory Controller) are uncore PMUs that count outside any core, ibs_fetch/ibs_op are AMD’s Instruction Based Sampling units, and kprobe/uprobe/tracepoint/breakpoint/software are the software producers. The insight to take: the two caps/ files are the ones that decide whether a profile will work. branches=16 is the Last Branch Record depth on this chip — 16 entries, so --call-graph lbr can never show a stack deeper than 16 frames here. max_precise=0 says the core PMU supports no precise-sampling levels at all, which is why cycles:p fails on this machine (see the precise-sampling section below). Read those two files before you plan a profiling strategy; they are cheap and they are the ground truth.

Reading a perf stat Report

A counting run is the densest single screen of performance information Linux gives you, so it pays to read one carefully. perf stat can target a command, a set of PIDs (-p), or all CPUs system-wide (-a, “the default if no target is specified” per perf-stat.txt, v6.12); -r N repeats the run up to 100 times and reports mean and standard deviation; -I N prints rolling deltas every N milliseconds (minimum 1 ms) for a time series. A default invocation produces something like:

$ perf stat ./myapp
          1,234.56 msec task-clock        #    0.998 CPUs utilized
                42      context-switches  #   34.018 /sec
                 3      cpu-migrations    #    2.430 /sec
             1,024      page-faults       #  829.6  /sec
     4,102,553,001      cycles            #    3.323 GHz
     2,051,276,500      instructions      #    0.50  insn per cycle
       410,255,300      branches          #  332.3  M/sec
        20,512,765      branch-misses     #    5.00% of all branches
       1.236 seconds time elapsed

The counts on the left are raw totals; the # annotations on the right are the derived ratios that actually diagnose. insn per cycle (IPC) of 0.50 is low — a modern core can retire 3–4 instructions per cycle, so 0.5 says the CPU is stalling most of the time, usually waiting on memory; that is the cue to perf record cache-related events next. task-clock divided by wall time gives CPUs utilized (here ~1.0, i.e. effectively single-threaded). branch-misses at 5% is high and would itself cost cycles. The whole point of perf stat is this triage: the ratios tell you which kind of profile to record next, so you don’t blindly sample.

When you ask for more events than the PMU has physical counters, the kernel time-multiplexes them — each event runs on a counter for a slice, and perf scales the partial count up to a full-run estimate. Multiplexed events are flagged in the output (a trailing percentage like (80.00%) showing the fraction of time the event was actually scheduled), and the resulting numbers are estimates, not exact totals. The fix is to request fewer events per run, or to use --repeat and accept the averaged estimate. This is why perf stat -e a,b,c,d,e,f,g,h on a 4-counter PMU is less trustworthy than two runs of four events each.

What the default event set actually is

Documentation does not pin the default list, so read the code. add_default_attributes() in tools/perf/builtin-stat.c, v6.12 builds it from four static arrays, conditionally:

struct perf_event_attr default_attrs0[] = {          /* always added */
  { .type = PERF_TYPE_SOFTWARE, .config = PERF_COUNT_SW_TASK_CLOCK       },
  { .type = PERF_TYPE_SOFTWARE, .config = PERF_COUNT_SW_CONTEXT_SWITCHES },
  { .type = PERF_TYPE_SOFTWARE, .config = PERF_COUNT_SW_CPU_MIGRATIONS   },
  { .type = PERF_TYPE_SOFTWARE, .config = PERF_COUNT_SW_PAGE_FAULTS      },
  { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_CPU_CYCLES       },
};
struct perf_event_attr frontend_attrs[] = {          /* only if the PMU has it */
  { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_STALLED_CYCLES_FRONTEND },
};
struct perf_event_attr backend_attrs[] = {           /* only if the PMU has it */
  { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_STALLED_CYCLES_BACKEND  },
};
struct perf_event_attr default_attrs1[] = {          /* always added */
  { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_INSTRUCTIONS        },
  { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_BRANCH_INSTRUCTIONS },
  { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_BRANCH_MISSES       },
};

and then assembles them:

if (!evsel_list->core.nr_entries) {           /* the user gave no -e at all */
        if (target__has_cpu(&target))
                default_attrs0[0].config = PERF_COUNT_SW_CPU_CLOCK;   /* (1) */
        evlist__add_default_attrs(evsel_list, default_attrs0);
        if (perf_pmus__have_event("cpu", "stalled-cycles-frontend"))  /* (2) */
                evlist__add_default_attrs(evsel_list, frontend_attrs);
        if (perf_pmus__have_event("cpu", "stalled-cycles-backend"))
                evlist__add_default_attrs(evsel_list, backend_attrs);
        evlist__add_default_attrs(evsel_list, default_attrs1);
        if (metricgroup__has_metric(pmu, "Default")) { ... }          /* (3) */
}

Three details that only the source gives you. (1) In system-wide mode (-a, i.e. target__has_cpu() true) the first event silently changes from task-clock to cpu-clock — the same clock measured per-task versus per-CPU, so the “CPUs utilized” ratio means something different in the two modes. (2) stalled-cycles-frontend/-backend appear only if the current PMU exposes them, which is why the same perf stat command prints eight rows on one machine and ten on another; it is not a version difference. (3) On PMUs that ship a metric group named Default (Intel’s TopdownL1, for example) perf stat additionally schedules the topdown metric events, which is a large part of why a bare perf stat can start multiplexing on its own. Note also what is not in the default set: cache-misses is not a default event. If you want the memory-boundedness signal you must ask for it.

Counting, measured directly through perf_event_open(2)

Because the measurement machine has no perf binary, the following was produced by opening seven counters with perf_event_open(2), enabling them with ioctl(fd, PERF_EVENT_IOC_ENABLE), running a fixed floating-point loop, and read(2)ing each fd back — which is precisely what perf stat does. read_format was set to PERF_FORMAT_TOTAL_TIME_ENABLED | PERF_FORMAT_TOTAL_TIME_RUNNING, so each read(2) returns three u64s: the count, the nanoseconds the event was enabled, and the nanoseconds it was actually running on a counter.

=== COUNTING (perf_event_open + read(2)), exclude_kernel=1 ===
           240696289  cycles              enabled=48144476 ns running=48144476 ns  (100.00% scheduled)
           480000481  instructions        enabled=48143754 ns running=48143754 ns  (100.00% scheduled)
            60000153  branches            enabled=48143734 ns running=48143734 ns  (100.00% scheduled)
                4034  branch-misses       enabled=48143874 ns running=48143874 ns  (100.00% scheduled)
                 275  cache-misses        enabled=48144616 ns running=48144616 ns  (100.00% scheduled)
                   0  page-faults         enabled=48143484 ns running=48143484 ns  (100.00% scheduled)
                   0  context-switches    enabled=48143654 ns running=48143654 ns  (100.00% scheduled)
                   #  1.99 insn per cycle
                   #  0.01% of all branches

Measured counting run: 60 million iterations of s += 1.0/(double)i, Fedora 44 / Zen 5, 2026-09-04. What it shows: 480,000,481 instructions retired against 240,696,289 cycles — an instructions-per-cycle (IPC) of 1.99, exactly two instructions retired per cycle. Branch misses are 4,034 out of 60,000,153 branches (0.01%), because the loop’s back-edge is trivially predictable, and cache misses are 275 in total because the working set is three registers. The insight to take: this is what a healthy perf stat looks like, and it is the calibration you need to read an unhealthy one: the loop is dominated by a serially-dependent double-precision divide, so IPC of 2.0 is near the practical ceiling for this code, not near the 4–6 the core can retire. “Low IPC” is never an absolute threshold — it is low relative to what this code could achieve. Note also running == enabled on every line: nothing was multiplexed, so every number is an exact total rather than a scaled estimate.

The read_format fields are what make that last sentence checkable, and they are the single most under-used part of the interface. When running < enabled, the count you just read is a partial observation and the honest value is count * enabled / running. perf stat does that scaling for you and prints the ratio as a trailing percentage; if you call the syscall yourself and ignore those two fields, you will silently under-report.

Multiplexing, measured

You do not have to guess how many hardware counters you have. Open N copies of one event and watch time_running / time_enabled for the point at which the kernel starts rotating them — the last N that stays at 100% is the number of usable general-purpose counters:

 1 identical 'cycles' events : worst time-scheduled = 100.00%
 2 identical 'cycles' events : worst time-scheduled = 100.00%
 3 identical 'cycles' events : worst time-scheduled = 100.00%
 4 identical 'cycles' events : worst time-scheduled = 100.00%
 5 identical 'cycles' events : worst time-scheduled = 100.00%     <-- ceiling
 6 identical 'cycles' events : worst time-scheduled =  80.96%
 7 identical 'cycles' events : worst time-scheduled =  68.78%
 8 identical 'cycles' events : worst time-scheduled =  61.98%
 9 identical 'cycles' events : worst time-scheduled =  49.67%
10 identical 'cycles' events : worst time-scheduled =  47.32%

Measured counter-count probe, Fedora 44 / Zen 5, 2026-09-04. What it shows: exactly five simultaneous hardware events stay resident for the whole run; the sixth forces rotation, and from there the worst-scheduled fraction falls roughly as 5/N. The insight to take: the cliff is sharp, and it is discoverable in about a second without root or perf. The likely arithmetic is six general-purpose core PMCs on this Zen part minus one permanently held by the NMI hard-lockup watchdog, which is enabled here (/proc/sys/kernel/nmi_watchdog reads 1). The practical rule: keep your hardware event count at or below this measured ceiling. If you need more events, take more runs; do not take one run with more events.

That “six minus the watchdog” is not a guess, and the chain of evidence is worth following because it is reusable on any machine. amd_core_pmu_init() in arch/x86/events/amd/core.c, v6.12 starts from AMD64_NUM_COUNTERS_CORE, which arch/x86/include/asm/perf_event.h defines as 6, and then — when the CPU advertises Performance Monitoring v2 — replaces it with a value read straight out of CPUID:

if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) {
        ebx.full = cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES);   /* leaf 0x80000022 */
        x86_pmu.version = 2;
        /* Find the number of available Core PMCs */
        x86_pmu.cntr_mask64 = GENMASK_ULL(ebx.split.num_core_pmc - 1, 0);
}

Executing that CPUID leaf on the measurement machine:

CPUID 0x80000022: eax=0x00000007 ebx=0x00402106 ecx=0x0000ffff edx=0x00000000
  eax.PerfMonV2  (bit 0)      = 1
  ebx.NumCorePmc  (bits 3:0)  = 6
  ebx.NumLbrStack (bits 9:4)  = 16

The hardware’s own answer, read with CPUID leaf 0x80000022, 2026-09-04. What it shows: six core PMCs and a 16-entry LBR stack, which the kernel then publishes as caps/branches = 16 and which the counter probe above sees as a ceiling of five once the NMI watchdog has taken one. The insight to take: the whole chain closes — hardware register count, kernel source, sysfs capability file, and observed multiplexing behaviour all agree, and any of the four can be checked against the others in seconds. When a profile looks wrong, this is the loop to run first: the number of counters you have is a fact you can read, not a thing to guess from the vendor’s marketing name.

Requesting different events makes it worse, because some events have restricted counter assignments. The same probe with four distinct events cycled through twelve descriptors gave:

 4 hw events open  : cycles(ev0)=160094612     worst time-scheduled=100.00%
 8 hw events open  : cycles(ev0)=100285964     worst time-scheduled=62.29%
12 hw events open  : cycles(ev0)=54875940      worst time-scheduled=33.83%

Measured multiplexing with mixed events, identical workload each time. What it shows: the raw cycles count collapses from 160M to 55M for exactly the same work — it is a raw partial, not a total. The insight to take: multiplexing does not degrade gracefully into “slightly noisier”; it turns each count into a sample of a rotation schedule. Scaling by enabled/running recovers the expectation but not the variance, and a short or bursty run can land entirely inside a window where the event you cared about was descheduled. A perf stat line ending in (33.83%) is not a measurement with an error bar — it is an extrapolation from a third of the data.

A Typical Investigation Flow

The subcommands compose into a standard top-down workflow:

# 1. Triage: is this even CPU-bound? Count the big-picture totals.
perf stat ./myapp
#    -> low IPC (instructions-per-cycle) + high cache-misses = memory-bound.
 
# 2. Localize: sample to find the hot functions.
perf record -F 999 -g ./myapp        # -F 999 = ~999 Hz; -g = capture call stacks
perf report                          # interactive: hottest functions, expandable callers
 
# 3. Zoom: per-instruction heat inside the hot function.
perf annotate <function>
 
# 4. Or watch it live, system-wide, without a perf.data file:
perf top -g

-F 999 sets the sampling frequency (≈999 samples/second; the odd number avoids lock-step with periodic kernel activity at round Hz). -g enables call-graph capture, which is what turns a flat function list into a tree you can later fold into a flame graph (Flame Graphs and Stack Sampling). The unwinding method behind -g defaults to frame pointers (fp) and can be switched to dwarf or hardware lbr (perf-record.txt, v6.12).

Resting on perf_event_open(2)

Everything above bottoms out at one syscall. perf_event_open() “returns a file descriptor, for use in subsequent system calls (read(2), mmap(2), prctl(2), fcntl(2), etc.)” (perf_event_open(2)) — that fd is a counter. perf stat opens fds and read(2)s the totals; perf record/perf top mmap(2) a ring buffer onto the fd and drain samples as the counter overflows. The syscall takes a large struct perf_event_attr describing the event type, the sample period/frequency, what to record on each sample, and the target (which PID, which CPU). The struct’s full layout — type, config, sample_type, the exclude_* bits that the :u/:k modifiers set — is intricate, and is deferred to The perf_event_open Syscall. The takeaway here is only the boundary: perf is the ergonomic surface; perf_event_open(2) is the load-bearing kernel API beneath it, and you could call it directly (some profilers and language runtimes do) but rarely want to.

Resolved (2026-09-04)

An earlier revision of this note flagged the default event sets as unverified. Both are now read from v6.12 source. perf stat’s defaults come from add_default_attributes() in tools/perf/builtin-stat.c and are listed in full above — including the two conditional stalled-cycles-* events and the task-clock → cpu-clock swap in system-wide mode. perf top’s default is not plain cycles: cmd_top() in tools/perf/builtin-top.c, v6.12 does parse_event(top.evlist, can_profile_kernel ? "cycles:P" : "cycles:Pu"), where can_profile_kernel = perf_event_paranoid_check(1). So perf top asks for maximum precision (:P) by default, and silently falls back to user-space-only (:Pu) when perf_event_paranoid forbids kernel profiling — which is why an unprivileged perf top shows no kernel symbols at all rather than showing them as unresolved addresses.

The mmap’d Ring Buffer

perf stat gets its numbers with read(2). Sampling cannot work that way — samples arrive asynchronously from an NMI, thousands per second, and a syscall per sample would cost more than the thing being measured. So perf record and perf top mmap(2) the event file descriptor, and the kernel gives back a shared-memory ring buffer that the NMI handler writes into and userspace drains without ever entering the kernel to read.

The mapping has a fixed shape, mandated by the perf_event_open(2) ABI: one control page followed by 2ⁿ data pages. The first page is struct perf_event_mmap_page, whose tail (after 116×8 bytes of reserved space that pad it “to 1k”, per the comment in include/uapi/linux/perf_event.h, v6.12) carries the four fields that make the whole thing work:

__u64   data_head;   /* head in the data section  — written by the KERNEL   */
__u64   data_tail;   /* user-space written tail   — written by USERSPACE    */
__u64   data_offset; /* where the record buffer starts within the mapping   */
__u64   data_size;   /* size of the record buffer                           */

Mermaid cannot express a memory layout with live pointer positions, so this one is an ASCII/box diagram in the RFC style; the offsets below are the measured values from a real mapping of nine pages on the measurement machine.

   mmap(NULL, (1 + 8) * 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)
   ├───────────────────────── 36,864 bytes total ──────────────────────────┤

   +==================+========================================================+
   | control page     |             data area (data_size = 32,768)             |
   | perf_event_mmap_ |                                                        |
   | page, 4096 bytes |                                                        |
   +==================+========================================================+
   ^                  ^                                                        ^
   0            data_offset = 4096                              4096 + 32768

   The data area is a RING. Two monotonically increasing byte counters index
   into it modulo data_size:

        data_tail                              data_head
            |                                      |
            v                                      v
   +--------+======================================+------------------------+
   | free   |  unread records (head - tail bytes)  |  free (kernel may fill)|
   +--------+======================================+------------------------+
    <------ kernel writes here ------------------------------------------->
            <--- userspace reads and then advances data_tail ---->

   Invariant:  0 <= data_head - data_tail <= data_size
   If a record would push (head - tail) past data_size, the kernel does NOT
   overwrite: it drops the sample and emits PERF_RECORD_LOST instead.

The perf ring buffer, drawn from a measured 9-page mapping. What it shows: one control page plus a power-of-two data area; data_head and data_tail are absolute byte counters that never wrap (they are 64-bit and indexed modulo data_size), with the kernel owning one and userspace the other. The insight to take: this is a single-producer / single-consumer lock-free queue with no locks anywhere, which is the only design that can be written from an NMI handler — an NMI cannot take a lock that ordinary kernel code might already hold. The price is that back-pressure is impossible: a full buffer means dropped samples, never a blocked producer. PERF_RECORD_LOST is the receipt.

Here is the sequence, measured. A cycles counter was programmed with sample_period = 2,000,000, the ring buffer mapped, the same floating-point workload run, and the buffer drained by hand:

=== SAMPLING (mmap ring buffer), precise_ip=0 ===
mmap: 9 pages = 36864 bytes  (1 control page + 8 data pages)
before: data_head=0 data_tail=0 data_offset=4096 data_size=32768
after : data_head=3840 data_tail=0  -> 3840 bytes of unread records
  PERF_RECORD_SAMPLE size=32 ip=0x400b59 pid=3354932 tid=3354932 time=1291922495253897
  PERF_RECORD_SAMPLE size=32 ip=0x400b5d pid=3354932 tid=3354932 time=1291922495657778
  PERF_RECORD_SAMPLE size=32 ip=0x400b59 pid=3354932 tid=3354932 time=1291922496057211
  PERF_RECORD_SAMPLE size=32 ip=0x400b59 pid=3354932 tid=3354932 time=1291922496463466
  PERF_RECORD_SAMPLE size=32 ip=0x400b5d pid=3354932 tid=3354932 time=1291922496863750
  PERF_RECORD_SAMPLE size=32 ip=0x400b59 pid=3354932 tid=3354932 time=1291922497270747
drained: 120 PERF_RECORD_SAMPLE, 0 PERF_RECORD_LOST, 0 THROTTLE/UNTHROTTLE
after publishing tail: data_head=3840 data_tail=3840 (buffer drained)

A real drain of the perf ring buffer, 2026-09-04. What it shows: 120 sample records × 32 bytes = 3,840 bytes, matching data_head exactly. Each record is a 8-byte struct perf_event_header (type=9 for PERF_RECORD_SAMPLE, misc, size) followed by exactly the fields the sample_type bitmask asked for, in the fixed order the enum declares them — here PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_TIME, so 8 (IP) + 8 (pid,tid as two u32) + 8 (time) = 24 payload bytes. The insight to take: two arithmetic checks you can do on any profile. First, 120 samples × 2,000,000 cycles/sample = 240 million cycles, which matches the 240,696,289 the counting run measured for the identical workload to within 0.3% — sampling and counting agree, as they must, because they are the same counter used two ways. Second, the record size is entirely determined by sample_type: turning on PERF_SAMPLE_CALLCHAIN adds 8 bytes per stack frame and turning on PERF_SAMPLE_STACK_USER (what --call-graph dwarf does) adds 8 KiB by default, i.e. a 256× inflation of every record. That single fact explains most of the cost difference between the call-graph modes.

The two ip values in the trace — 0x400b59 and 0x400b5d — are four bytes apart, both inside the same tight loop body. That is the sampling profiler working exactly as designed: it is not tracking function calls, it is photographing the instruction pointer at pseudo-random moments and letting the law of large numbers do the rest.

The ordering protocol matters and is easy to get wrong. The header comment in perf_event.h states it: “User-space reading the data_head value should issue an smp_rmb(), after reading this value. When the mapping is PROT_WRITE the data_tail value should be written by userspace to reflect the last read data, after issuing an smp_mb() to separate the data read from the ->data_tail store. In this case the kernel will not over-write unread data.” The last sentence is the important one: mapping the buffer PROT_WRITE is what opts you into back-pressure semantics. A read-only mapping cannot publish a tail, so the kernel treats the buffer as overwritable and you get the newest samples at the cost of losing the oldest — which is exactly what perf record --overwrite wants for a flight-recorder capture, and exactly what you do not want for a normal profile.

Buffer size is capped, per user and per CPU, by kernel.perf_event_mlock_kb. The default is not a round number, and the source says why: sysctl_perf_event_mlock = 512 + (PAGE_SIZE / 1024) in kernel/events/core.c, v6.12 — 512 KiB of data plus one control page. On the measurement machine /proc/sys/kernel/perf_event_mlock_kb reads 516, matching exactly. Documentation/admin-guide/perf-security.rst spells out the consequence: “if a machine has eight cores and perf_event_mlock_kb limit is set to 516 KiB, then a user process is provided with 516 KiB * 8 = 4128 KiB of memory above the RLIMIT_MEMLOCK limit”, and “if the user wants to start two or more performance monitoring processes, the user is required to manually distribute the available 4128 KiB between the monitoring processes, for example, using the --mmap-pages Perf record mode option. Otherwise, the first started performance monitoring process allocates all available 4128 KiB and the other processes will fail to proceed due to the lack of memory.” That is the real explanation behind the otherwise baffling “perf record fails with ENOMEM while another perf is running” report.

Sampling Frequency, Sampling Period, and Throttling

There are two ways to ask for samples, and they are not interchangeable. Setting attr.sample_period asks for one sample every N events — every 2,000,000 cycles, say. Setting attr.freq = 1 and attr.sample_freq = N asks for N samples per second, and the kernel then continuously adjusts the period to hit that rate; perf record -F 99 and perf top -F 99 take this path. Frequency mode is what you almost always want, because it makes the sample count predictable regardless of how fast the workload is running, and it automatically compensates for a CPU that drops to a low P-state.

Frequency mode is not free, though, and the adjustment loop is visible in the source. __perf_event_account_interrupt() in kernel/events/core.c runs on every single overflow and, for a frequency-mode event, recalculates:

if (event->attr.freq) {
        u64 now = perf_clock();
        s64 delta = now - hwc->freq_time_stamp;
        hwc->freq_time_stamp = now;
        if (delta > 0 && delta < 2*TICK_NSEC)
                perf_adjust_period(event, delta, hwc->last_period, true);
}

so the effective period is a feedback-controlled quantity that changes throughout the run. This is why a perf.data from a frequency-mode capture carries PERF_SAMPLE_PERIOD per record: each sample is worth a different number of events, and a consumer that assumes uniform weight will produce a skewed profile.

The rate ceiling, measured

kernel.perf_event_max_sample_rate is a hard ceiling that perf_event_open(2) enforces at open time, not a soft target. Bisecting sample_freq against EINVAL on the measurement machine finds the boundary precisely:

max accepted sample_freq = 50000 Hz
$ cat /proc/sys/kernel/perf_event_max_sample_rate
50000
$ cat /proc/sys/kernel/perf_cpu_time_max_percent
25

The interesting part is that 50,000 is not the default. kernel/events/core.c sets #define DEFAULT_MAX_SAMPLE_RATE 100000 and #define DEFAULT_CPU_TIME_MAX_PERCENT 25. The machine’s ceiling is half the default because the kernel lowered it at runtime, three separate times, and left the receipts in the kernel log:

$ journalctl -k | grep 'perf:'
Aug 20 18:25:54 kernel: perf: AMD IBS detected (0x00081bff)
Aug 20 20:41:15 kernel: perf: interrupt took too long (2540 > 2500), lowering kernel.perf_event_max_sample_rate to 78000
Aug 21 16:05:15 kernel: perf: interrupt took too long (3180 > 3175), lowering kernel.perf_event_max_sample_rate to 62000
Aug 28 22:44:57 kernel: perf: interrupt took too long (3994 > 3975), lowering kernel.perf_event_max_sample_rate to 50000

Measured dynamic throttling on the note’s own machine, 2026-08-20 to 2026-08-28. What it shows: the ceiling ratcheted 100,000 → 78,000 → 62,000 → 50,000 over eight days, each step triggered by the average NMI handler duration exceeding the allowance. The insight to take: perf_event_max_sample_rate is a dynamic, monotonically decreasing, machine-lifetime value on a busy box. Two profiles taken a week apart on the same host can be at different frequencies without anyone changing a setting, and perf record -F max will silently give you different sample counts. If you are comparing profiles over time, pin -F to an explicit number and check the sysctl.

The mechanism is perf_sample_event_took(), which the PMU code calls with the measured duration of each sample. It maintains a decaying average over the last NR_ACCUMULATED_SAMPLES (128) samples and, when that average exceeds the allowance, ratchets everything down:

avg_len = running_len / NR_ACCUMULATED_SAMPLES;
if (avg_len <= max_len)
        return;
/* Compute a throttle threshold 25% below the current duration. */
avg_len += avg_len / 4;
max = (TICK_NSEC / 100) * sysctl_perf_cpu_time_max_percent;
if (avg_len < max)  max /= (u32)avg_len;
else                max = 1;
WRITE_ONCE(perf_sample_allowed_ns, avg_len);
WRITE_ONCE(max_samples_per_tick, max);
sysctl_perf_event_sample_rate = max * HZ;

The comment above the function is the design rationale, stated plainly: “perf samples are done in some very critical code paths (NMIs). If they take too much CPU time, the system can lock up and not get any real work done. This will drop the sample rate when we detect that events are taking too long.” And perf_cpu_time_max_percent_handler() will let you disable the whole safety net by writing 0 or 100 to perf_cpu_time_max_percent, but prints "perf: Dynamic interrupt throttling disabled, can hang your system!" when you do. That is not hyperbole: a deep --call-graph dwarf capture at a high frequency really can spend more time in NMI handlers than in your program.

Per-event throttling, which is a different mechanism

The sysctl ratchet above is global and permanent-ish. There is a second, per-event, per-tick throttle that fires far more often and is the one you actually see in profiles. __perf_event_account_interrupt() counts interrupts within the current tick and, if one event exceeds max_samples_per_tick, stops that event until the next timer tick:

stateDiagram-v2
    [*] --> Running
    Running --> Running : overflow — interrupts counter incremented<br/>still at or under max_samples_per_tick
    Running --> Throttled : interrupts counter exceeds max_samples_per_tick<br/>counter pinned to MAX_INTERRUPTS<br/>perf_log_throttle(event, 0)
    Throttled --> Throttled : further overflows produce NO sample
    Throttled --> Running : next timer tick — perf_adjust_freq_unthr_events<br/>sees MAX_INTERRUPTS, resets counter to 0,<br/>perf_log_throttle(event, 1), restarts the PMU
    Running --> [*] : event disabled or closed

    note right of Throttled
        Emits PERF_RECORD_THROTTLE (type 5)
        into the ring buffer; the matching
        PERF_RECORD_UNTHROTTLE (type 6)
        is emitted on the way out.
        Samples in this window are LOST,
        not delayed.
    end note

The per-event throttle state machine, from __perf_event_account_interrupt() and perf_adjust_freq_unthr_events() in kernel/events/core.c, v6.12. What it shows: an event that overflows more than max_samples_per_tick times inside one timer tick is stopped for the remainder of that tick, and both the entry and the exit are recorded in the data stream as PERF_RECORD_THROTTLE/PERF_RECORD_UNTHROTTLE. The insight to take: throttling is silently biasing, not merely lossy. It removes samples preferentially from the busiest moments — exactly the moments you were profiling — so a throttled profile systematically under-represents its own hot spots. This is why perf report warns about throttling, and why the honest first step when reading any unfamiliar perf.data is to check for throttle records (perf script --show-throttle-events, or just count PERF_RECORD_THROTTLE in a hand-written reader as the measured run above does).

Call-Graph Recording: fp vs dwarf vs lbr

A flat profile tells you malloc is hot. It cannot tell you which of the forty call paths into malloc is responsible, and that is almost always the question. -g (or --call-graph <method>) turns on stack capture by setting PERF_SAMPLE_CALLCHAIN in sample_type, and then the interesting question is how the kernel produces that array of return addresses.

The man page draws the boundary precisely: “The unwinding method used for kernel space is dependent on the unwinder used by the active kernel configuration, i.e. CONFIG_UNWINDER_FRAME_POINTER (fp) or CONFIG_UNWINDER_ORC (orc). Any option specified here controls the method used for user space.” (perf-record.txt, v6.12). So a mixed kernel+user stack is always assembled by two unwinders, and --call-graph only chooses the second.

flowchart TB
    S["PMU overflow -&gt; NMI<br/>PERF_SAMPLE_CALLCHAIN requested"] --> M{"where did the<br/>sample land?"}
    M -->|kernel mode| K["kernel unwinder<br/>chosen at BUILD time by<br/>CONFIG_UNWINDER_*"]
    K --> ORC["ORC tables (.orc_unwind)<br/>x86-64 Kconfig default<br/>since v4.15"]
    K --> KFP["kernel frame pointers<br/>CONFIG_UNWINDER_FRAME_POINTER"]
    ORC --> CTX["PERF_CONTEXT_USER marker<br/>pushed into the callchain"]
    KFP --> CTX
    M -->|user mode| CTX
    CTX --> U{"--call-graph<br/>method?"}
    U -->|"fp (default)"| FP["walk %rbp chain IN THE NMI<br/>arch/x86: perf_callchain_user()<br/>cheap; needs frame pointers"]
    U -->|dwarf| DW["copy regs + 8 KiB of stack<br/>PERF_SAMPLE_REGS_USER +<br/>PERF_SAMPLE_STACK_USER<br/>unwind LATER in userspace"]
    U -->|lbr| LBR["read hardware branch ring<br/>no compiler support needed<br/>depth-capped by the CPU"]
    FP --> OUT["callchain array in<br/>PERF_RECORD_SAMPLE"]
    DW --> OUT2["perf.data grows 256x per sample;<br/>libunwind/libdw replays CFI offline"]
    LBR --> OUT
    OUT2 --> OUT

How one call-graph sample is produced. What it shows: the kernel half of the stack is decided when the kernel was compiled and you cannot change it from the command line; the user half is decided per-invocation by --call-graph, and the three methods differ in where the work happens — in the NMI (fp, lbr) or offline in userspace after copying raw stack memory (dwarf). The insight to take: the PERF_CONTEXT_USER marker in the middle is why a stack can be perfectly good above the line and garbage below it. A profile with clean kernel symbols and a wall of unknown user frames is not “broken perf” — it is ORC working and frame pointers missing.

	fp (default)	dwarf	lbr
What is captured per sample	the return-address chain, already walked	user registers + 8,192 bytes of stack (default)	the CPU’s last-branch ring
Where the unwinding happens	in the NMI handler, in-kernel	offline, in perf, via libunwind or libdw	in hardware; perf just reads it
Per-sample record growth	~8 bytes per frame	~8 KiB flat, tunable via --call-graph dwarf,4096	~16 bytes per branch entry
Compiler requirement	target and every library built -fno-omit-frame-pointer	DWARF CFI present (-g, or -debuginfo packages)	none
Depth limit	kernel.perf_event_max_stack (127 on the test machine); --call-graph fp,32 to cap	limited by the dumped stack window — silently truncates deeper stacks	the hardware LBR depth (16 on the test machine, per caps/branches)
Covers kernel stacks?	yes (kernel half uses ORC regardless)	user only; kernel half still ORC	“It can only get user call chain”
Availability	everywhere	everywhere perf was linked against libunwind/libdw	“only available on new Intel platforms, such as Haswell”; AMD needs LBR v2
Typical failure	[unknown] frames on stripped/optimized binaries	huge perf.data, measurable workload perturbation	stacks silently cut off at the hardware depth

The three user-space unwinding methods (quotations from perf-record(1)/perf-record.txt, v6.12; depth limits measured on Fedora 44 / Zen 5, 2026-09-04). What it shows: the three methods trade compiler cooperation against per-sample cost against depth. The insight to take: the row that decides it in practice is “per-sample record growth”. dwarf mode multiplies your perf.data by roughly the ratio of 8 KiB to your normal record size, and it copies that memory inside the NMI handler, which feeds straight back into the throttling machinery above — a --call-graph dwarf -F 999 capture is a realistic way to make the kernel ratchet your own sample rate down mid-run. Use fp where you can, dwarf where you must and then tune the window down (dwarf,4096 or lower), and lbr only when you know your stacks are shallow.

The frame-pointer availability question — which is what actually decides whether fp works — is a long story with a datable resolution (Fedora 38, Ubuntu 24.04) and is told with measurements in Flame Graphs and Stack Sampling, which owns the stack-capture-and-visualisation layer. What belongs here is the perf-side consequence: --call-graph is the single option most likely to make the difference between a useful and a useless profile, and its right value is a property of how your distribution built its binaries, not a matter of taste.

perf_event_paranoid and What Each Level Permits

Almost every “perf doesn’t work” report is this sysctl. kernel.perf_event_paranoid is a single integer that gates what an unprivileged process may ask perf_event_open(2) for, and int sysctl_perf_event_paranoid __read_mostly = 2; in kernel/events/core.c makes 2 the kernel’s own default. Distributions vary, and container runtimes frequently inherit the host value while removing the capability that would bypass it.

Level	Kernel profiling (exclude_kernel=0)	System-wide / per-CPU events (pid = -1)	Raw tracepoints & ftrace function tracepoints	perf_event_mlock_kb limit
-1	allowed	allowed	allowed	ignored
0	allowed	allowed	denied without CAP_PERFMON	enforced (unless CAP_IPC_LOCK)
1	allowed	denied without CAP_PERFMON	denied	enforced
2 (default)	denied without CAP_PERFMON	denied	denied	enforced

What each perf_event_paranoid level permits, from Documentation/admin-guide/perf-security.rst and Documentation/admin-guide/sysctl/kernel.rst, both v6.12. What it shows: the levels are cumulative restrictions, and each one removes a scope, not a feature. The insight to take: the capability that lifts these is CAP_PERFMON, not CAP_SYS_ADMIN. kernel.rst is explicit that CAP_SYS_ADMIN still works “for backward compatibility reasons” but that its “usage for secure system performance monitoring and observability operations is discouraged”. Granting CAP_PERFMON to a profiling agent — via setcap cap_perfmon+ep, or a container’s --cap-add PERFMON — is the correct fix, and it is dramatically narrower than making the process root.

The failure signature is measurable and unambiguous. At the default level 2, opening the two forbidden scopes returns EACCES:

cycles, exclude_kernel=0, own pid : fd=-1 errno=Permission denied
cycles, system-wide pid=-1 cpu=0 : fd=-1 errno=Permission denied

Measured at perf_event_paranoid = 2, unprivileged, 2026-09-04. What it shows: it is the perf_event_open(2) call itself that fails with EACCES — before any counting starts. The insight to take: EACCES from perf_event_open is a scope problem, and reading it that way tells you which knob to turn. EACCES only when kernel symbols are involved means level 2; EACCES only when you add -a means level ≥ 1; EACCES even on a per-process, user-space-only event means the sysctl is set above anything mainline defines (some distributions and hardened kernels carry an extra, stricter level) or that seccomp is blocking the syscall outright. Contrast EINVAL, which is never a permission problem — it means the PMU cannot express what you asked for.

Precise Sampling, Skid, and Why :p Can Simply Refuse

The overflow sequence at the top of this note has a built-in inaccuracy: by the time the NMI is delivered and the handler reads pt_regs, the CPU has moved on from the instruction that caused the overflow. The gap is skid, and on a deeply out-of-order core it is routinely tens of instructions. For counting this does not matter at all. For perf annotate, where the entire point is attributing cycles to specific instructions, it is fatal — skid systematically pushes samples onto whatever instruction follows a long-latency one, so the load that actually missed cache appears free and the innocent instruction after it appears expensive.

The hardware answer is to have the PMU itself record the architectural state at the offending instruction, into a memory buffer, without involving an interrupt at all: Intel calls this PEBS (Precise Event-Based Sampling), AMD calls its equivalent IBS (Instruction Based Sampling). perf exposes it through the :p modifier, one p per precision level (:p, :pp, :ppp, or :P for “maximum available”), which sets attr.precise_ip to 0–3.

The crucial practical point is that this is a per-PMU capability that can be entirely absent, and the kernel tells you so in a file:

$ cat /sys/bus/event_source/devices/cpu/caps/max_precise
0
 
cycles sampling, precise_ip=0    : fd=3   (ok)
cycles sampling, precise_ip=1    : fd=-1  errno=Invalid argument
cycles sampling, precise_ip=2    : fd=-1  errno=Invalid argument
cycles sampling, precise_ip=3    : fd=-1  errno=Operation not supported
 
ibs_op PMU type=11
ibs_op exclude_kernel=1                 fd=-1 errno=Invalid argument
ibs_op exclude_kernel=0                 fd=-1 errno=Permission denied

Measured precise-sampling capability, AMD Zen 5, Fedora 44, unprivileged, 2026-09-04. What it shows: caps/max_precise reads 0, so the core PMU on this chip offers no precise levels through the ordinary cpu PMU at all, and precise_ip ≥ 1 is rejected with EINVAL. AMD’s precise sampling lives on a separate PMU (ibs_op, dynamic type 11), which in turn refuses exclude_kernel=1 with EINVAL — IBS tags instructions in hardware and cannot be told to ignore kernel ones — and therefore requires exclude_kernel=0, which at perf_event_paranoid=2 is exactly the thing the previous section showed is denied. The insight to take: this is a two-step trap that produces a confusing error. On AMD, precise sampling is not merely “less good than Intel’s”; it is reached through a different PMU and it is structurally impossible without privilege, because the only mode it supports is the one paranoid level 2 forbids. perf record -e cycles:pp failing on an AMD box is not a bug and not a missing package — check caps/max_precise first, then perf_event_paranoid.

The consequence for reading profiles is worth stating flatly: on a machine where precise sampling is unavailable, perf annotate output must be read with a skid allowance. Attribute a hot instruction to “somewhere in the preceding handful of instructions”, not to the exact line highlighted. The mechanism, the PEBS/IBS record formats, and the :ppp semantics are the subject of Precise Event-Based Sampling.

Per-cgroup and Container Profiling

Profiling “the container” rather than “the machine” is a scope problem, and perf solves it with -G/--cgroup. The semantics are unusual enough that the man page’s own wording is the clearest statement: “monitor only in the container (cgroup) called ‘name’. This option is available only in per-cpu mode. The cgroup filesystem must be mounted. All threads belonging to container ‘name’ are monitored when they run on the monitored CPUs. Multiple cgroups can be provided. Each cgroup is applied to the corresponding event, i.e., first cgroup to first event, second cgroup to second event and so on. It is possible to provide an empty cgroup (monitor all the time) using, e.g., -G foo,,bar” (perf-record.txt, v6.12).

Three things fall out of that, and all three surprise people:

1. It is per-CPU mode only. -G attaches a cgroup filter to a CPU-wide event; there is no “follow this cgroup wherever it goes” per-task form. Combined with perf_event_paranoid ≥ 1 denying per-CPU events to unprivileged users, this means cgroup profiling is a privileged operation, full stop.
2. The cgroup list is positional against the event list, not a global filter. -e cycles,instructions -G foo filters cycles to foo and leaves instructions system-wide. The doc gives the idiom for the common intent: “If wanting to monitor, say, ‘cycles’ for a cgroup and also for system wide, this command line can be used: perf stat -e cycles -G cgroup_name -a -e cycles” — repeat the event.
3. Filtering is not the same as attribution. A cgroup-filtered event counts only while a task of that cgroup is on the monitored CPU. Work the container caused but that runs elsewhere — kernel worker threads doing writeback for its dirty pages, softirq processing for its packets — is outside the filter. For “what did this container cost the machine”, cgroup filtering undercounts by construction.

For attributing samples after the fact rather than filtering them during capture, perf record --all-cgroups records PERF_RECORD_CGROUP events and enables the cgroup sort key in perf report, and --namespaces records PERF_RECORD_NAMESPACES and enables the cgroup_id sort key. That is usually the better shape for a shared host: capture everything once, slice by cgroup in the report.

The related container gotcha has nothing to do with -G at all. Inside a VM or a nested virtualization environment the PMU is frequently not virtualized, so PERF_TYPE_HARDWARE events return <not supported> while PERF_TYPE_SOFTWARE events keep working — the split explained in the event-type table above. And in a container, /proc/sys/kernel/perf_event_paranoid is not namespaced: it is the host’s value, which is why raising it inside the container has no effect and why the fix is a capability (--cap-add PERFMON) or a host-side sysctl.

Failure Modes and Common Misunderstandings

• perf stat ran but events show <not supported> or <not counted>. Inside VMs/containers the PMU is often unavailable or virtualized away, so hardware events silently degrade. <not counted> also appears when more events were requested than physical PMU counters exist and multiplexing kicked in (the count is then scaled/estimated). Software events (cpu-clock, context-switches) always work because they don’t need the PMU.
• Need root, or not. Many operations require CAP_PERFMON/root depending on kernel.perf_event_paranoid (sysctl). A high paranoid level blocks kernel sampling and tracepoints for unprivileged users; symptoms are permission errors from perf record -a or empty kernel stacks.
• Flat-looking profiles with -g. If the target was built without frame pointers and you used the default fp unwinding, call stacks collapse to one frame. Switch to -g dwarf (needs debuginfo) or -g lbr (Intel), or rebuild with -fno-omit-frame-pointer. See Flame Graphs and Stack Sampling.
• perf trace is not strace. It does not use ptrace and won’t stop the process per-syscall; semantics and overhead differ. It can miss/merge fast events and reports system-wide differently. For exact per-process syscall fidelity, strace is still the reference; for low-overhead and broader scope, perf trace.
• Stale perf.data. perf report resolves symbols against the binaries/debuginfo present now; if you rebuilt or stripped the binary after recording, symbolization breaks. Keep the build artifacts, or use perf archive to bundle them.
• EINVAL and EACCES mean different things, and the difference is diagnostic. EACCES is always a scope refusal — perf_event_paranoid (or a container missing CAP_PERFMON, or seccomp) is denying the scope you asked for; the measurements above show it for kernel profiling and for system-wide mode. EINVAL is always a capability refusal — the PMU cannot express the request. precise_ip=1 on a PMU whose caps/max_precise is 0, sample_freq above perf_event_max_sample_rate, and exclude_kernel=1 on AMD ibs_op all produce EINVAL. Reaching for sudo on an EINVAL wastes time; reaching for a different event on an EACCES does too.
• The unnamed third failure: silent throttling. A profile can be complete, symbolized, deep-stacked, and still wrong because the kernel stopped sampling during the busiest windows. PERF_RECORD_THROTTLE records in the stream are the only evidence, and most tutorials never mention checking for them. Treat a profile with throttle records the way you would treat a benchmark that swapped.
• perf.data records the machine, not just the program. A perf record capture includes PERF_RECORD_MMAP events describing every mapped executable and library, PERF_RECORD_COMM/FORK/EXIT for process lifetime, and (with the right flags) PERF_RECORD_CGROUP/NAMESPACES. This is how perf report resolves an address to libc.so.6+0x1234 at all, and it is why a perf.data from a container can be unreadable on the host — the paths it recorded do not exist outside the container’s mount namespace.
• Profiling a short-lived process misses the start. perf record ./app forks and execs the target with the counters already open, so this case is fine. Attaching with -p <pid> to an already-running process is not: you get nothing from before the attach, and for a program whose interesting behaviour is startup that is everything. perf record -e ... -a plus filtering after the fact avoids the race.

Alternatives and When to Choose Them

perf is the kernel-native default; its main peers occupy adjacent niches. eBPF tools (bpftrace, BCC) attach to the same events but aggregate in-kernel (histograms, per-key counts) instead of shipping every sample to userspace — better when raw event volume would be overwhelming or when you want custom logic, but they need the BPF toolchain. ftrace/trace-cmd is better for function-call tracing (who calls whom, latency between events) than for PMU sampling. LTTng targets very-low-overhead, high-rate production tracing with a different buffering model. Language-runtime profilers (Go pprof, py-spy) understand runtime-specific frames perf can’t unwind without help. The rule of thumb: PMU counters and CPU profiling → perf; in-kernel aggregation / custom logic → eBPF; function-flow and latency tracing → ftrace; runtime-aware profiling → the language’s own tool.

	perf	bpftrace / BCC	ftrace via tracefs	LTTng
Kernel entry point	perf_event_open(2)	bpf(2), then attach to a perf event / kprobe / tracepoint	write(2) to files under /sys/kernel/tracing	its own kernel modules + a userspace daemon
Where data is reduced	userspace, after the fact	in-kernel, in BPF maps	in-kernel filters/triggers; formatting in-kernel	userspace daemon, from per-CPU buffers
Can read PMU counters	yes — this is its unique job	yes, via hardware:/software: providers, but no counting mode	no	no
Can sample a call stack	yes (fp/dwarf/lbr)	yes (ustack/kstack, frame pointers only)	function graph tracer, not sampling	yes, with its own unwinder
Cost model at high event rate	one record per event to userspace	O(1) per event, one summary at the end	one record per event to a ring buffer	one record per event, but a very cheap one
Best at	“where are my cycles going”, IPC/cache/branch ratios, flame graphs	“give me a latency histogram keyed by X, cheaply”	“what functions ran, in what order, with what latency”	high-rate production tracing with long retention
Worst at	multi-million-events-per-second sources	anything needing exact counter totals	PMU-based CPU profiling	ad-hoc one-liners

The four Linux observability stacks, compared on the axis that actually decides between them. What it shows: they are not competing implementations of one idea; they differ in where the data is reduced, which sets everything else. The insight to take: perf’s distinguishing capability is the PMU — nothing else in this table can tell you your IPC or your cache-miss rate — and its distinguishing weakness is that reduction happens after the data has already been shipped to userspace. Those are two faces of the same design. When the question is “how much and where”, use perf. When the question is “summarize a million events per second by key”, the answer is in-kernel aggregation, and the deep treatments live in bpftrace and In-Kernel Aggregation with BPF Maps. The probe mechanisms all four share are covered in kprobes and Tracepoints; the ftrace control surface in The tracefs Filesystem.

Production Notes

perf is the foundation of Brendan Gregg’s widely used CPU flame-graph methodology: perf record -F 99 -a -g for a fixed interval, then fold the stacks into an SVG (Flame Graphs and Stack Sampling). In production the practical constraints are sampling frequency (99–999 Hz is the usual band — high enough to localize hot code, low enough to keep overhead near 1%), unwinding cost (-g dwarf is far heavier than fp or lbr because it copies stack memory on every sample), and the perf_event_paranoid policy your fleet enforces. A recurring gotcha is profiling JIT/interpreted runtimes: perf needs a /tmp/perf-<pid>.map symbol file (emitted by the JVM/V8/etc.) to name dynamically generated code, or the profile is a sea of unnamed addresses. For steady-state, low-overhead production observability that survives high event rates, teams increasingly pair perf stat-style counting with eBPF aggregation rather than continuous perf record.

The JIT case has a documented, boringly simple contract that is worth knowing exactly, because it is easy to satisfy from any runtime you control. tools/perf/Documentation/jit-interface.txt, v6.12 is 447 bytes long and specifies the whole thing: “The JIT has to write a /tmp/perf-%d.map (%d = pid of process) file. This is a text file. Each line has the following format, fields separated with spaces: START SIZE symbolname. START and SIZE are hex numbers without 0x. symbolname is the rest of the line, so it could contain special characters. The ownership of the file has to match the process.” That is the entire interface — no library, no protocol, no daemon. The JVM’s -XX:+PreserveFramePointer (JDK 8u60 and JDK 9 onward) plus a map-emitting agent, V8’s --perf-basic-prof, and every perf-map-agent-style tool exist only to produce that file. The two failure modes both come straight from the spec: a map written by a different UID is ignored (the ownership rule), and a PID-namespaced container writes /tmp/perf-<inner pid>.map while perf on the host looks for the outer PID.

A checklist for planning a production profiling capability, in the order the constraints actually bite:

1. Decide the privilege story first. perf_event_paranoid at the default 2 permits per-process, user-space-only profiling and nothing else. If your agent needs kernel stacks or system-wide scope, it needs CAP_PERFMON — grant that, not root, and not perf_event_paranoid=-1 fleet-wide.
2. Check caps/max_precise and caps/branches per CPU model before standardizing on :pp or --call-graph lbr. A fleet with mixed Intel and AMD hosts does not have a single right answer, as the measurements above show.
3. Pin -F to a literal number, not max. The measured ratchet on this machine — 100,000 down to 50,000 over eight days — is what max follows, and it makes captures incomparable over time.
4. Budget the ring buffer. perf_event_mlock_kb is per-user, per-CPU; two profilers on one host will fight for it, and the loser gets ENOMEM. Size with --mmap-pages deliberately rather than letting the first process take everything.
5. Verify against throttle and lost records, not against the absence of an error message. A profile with PERF_RECORD_THROTTLE or PERF_RECORD_LOST in it is biased toward the quiet periods.
6. Keep the build artifacts, or archive them with the profile. perf archive bundles the DSOs referenced by a perf.data into a tarball keyed by build-id; without it, a profile taken on a host that has since been redeployed is unsymbolizable.

See Also

• The perf_event_open Syscall — the single kernel syscall every perf subcommand rests on
• The Performance Monitoring Unit — the hardware counters perf reads
• Hardware vs Software Performance Events — what cycles vs context-switches actually measure
• Counting vs Sampling Mode — the central duality framing perf stat vs perf record
• perf stat and perf top — the counting and live-profiling subcommands in depth
• perf record and perf report — the sampling-to-perf.data workflow in depth
• Flame Graphs and Stack Sampling — turning perf record -g output into flame graphs
• Precise Event-Based Sampling — what the :p modifier and PEBS/IBS buy you
• kprobe and uprobe Event Interface — the tracefs probe files that perf probe writes to
• Syscall Tracepoints — the events behind perf trace
• kprobes — the dynamic kernel probe mechanism perf probe installs, in full
• Tracepoints — the static instrumentation behind every subsystem:event name in -e
• The tracefs Filesystem — the file-based control surface perf bypasses by using perf_event_open(2) instead
• bpftrace — the in-kernel-aggregation alternative, for when the event rate is too high to ship every sample
• The Kernel Ring Buffer Concept — the ftrace ring buffer, a different design from the perf mmap buffer described here
• Stack Unwinding and ORC — the kernel-side unwinder that produces the kernel half of every -g stack
• Linux Tracing and Observability MOC — the parent map