Counting vs Sampling Mode

Once you have chosen which event to measure — see Hardware vs Software Performance Events — there is a second, orthogonal decision: how perf uses it. There are exactly two modes, and they answer different questions. In counting mode (the perf stat mode) the counter simply accumulates; you read the running total over an interval and learn how much happened — an exact aggregate at near-zero overhead, but no notion of where. In sampling mode (the perf record mode) you configure the event to overflow every N occurrences; on each overflow the hardware raises an interrupt and the kernel records a sample — the instruction pointer, optionally the call stack, registers, and a timestamp — into a memory-mapped ring buffer. Sampling is statistical (it sees one event in N, not all of them) but it tells you where the cycles or cache misses are concentrated. The single switch that selects the mode is sample_period / sample_freq in struct perf_event_attr: leave it zero and the event counts; set it positive and the event samples. The man page makes the rule explicit — “A sampling event has sample_period > 0” (perf_event_open(2)).

This note covers the two modes, the period-versus-frequency choice within sampling, why frequency mode is usually preferred, and skid — the reason a sampled instruction pointer can lag the event that triggered it, and the motivation for IBS. Every numeric claim about limits, multiplexing, and throttling in this note was measured on a live machine rather than quoted; the measurement section near the end shows the programs and their raw output.

Versions, and the measurement box

Kernel source claims are pinned to Linux v6.12, read from raw.githubusercontent.com/torvalds/linux/v6.12/.... v6.12 is this vault’s pin, not the only longterm branch — as of 2026-09-11 kernel.org/releases.json reports mainline 7.3-rc2, stable 7.2.5, and longterm 6.18.51 / 6.12.109 / 6.6.156 / 6.1.187 / 5.15.220 / 5.10.269. The measurements were taken on Fedora 44, Linux 7.1.13-200.fc44.x86_64, an AMD Ryzen AI Max+ 395 (Zen 5), 32 logical CPUs, gcc 16.2.1, as an unprivileged user at kernel.perf_event_paranoid = 2 — which means every event opened here carries exclude_kernel = 1. Neither perf nor bpftool is installed on that box, so the demonstrations call perf_event_open(2) directly; that is a feature, not a workaround, because it makes the mode switch visible as a struct field rather than a command-line flag.

Mental Model — Aggregate vs Spotlight

flowchart TB
  subgraph COUNT["Counting (perf stat) — sample_period = 0"]
    EV1["event fires"] --> ACC["counter += 1<br/>(silently accumulates)"]
    ACC --> RD["userspace read():<br/>exact total over interval"]
  end
  subgraph SAMPLE["Sampling (perf record) — sample_period &gt; 0"]
    EV2["event fires"] --> DEC["counter toward overflow"]
    DEC -->|"every Nth event"| OVF["overflow → interrupt (NMI)"]
    OVF --> REC["record sample:<br/>IP, callchain, regs, time"]
    REC --> RING["mmap ring buffer"]
    RING --> RT["perf reads asynchronously<br/>→ perf.data"]
  end

The two ways to use one event. What it shows: counting keeps a single growing number and hands you the exact total when you ask; sampling configures the counter to interrupt every N events and snapshots program state at each interrupt. The insight to take: counting answers “how many?” precisely and cheaply but blindly; sampling answers “where?” approximately but with overhead proportional to the sample rate. They are not competitors — the standard workflow is perf stat first to quantify, then perf record to localize.

Laid out dimension by dimension, the two modes differ in almost every property that matters operationally:

Dimension	Counting mode	Sampling mode
Selected by	sample_period == 0 and sample_freq == 0	sample_period > 0, or freq = 1 with sample_freq > 0
Front-end	perf stat, perf top -e ... --stdio intervals	perf record, perf top
Question answered	how many?	where?
Data path	counter register → read(fd) on demand	counter overflow → PMU interrupt (NMI on x86) → handler builds a record → mmap ring buffer → reader drains it
Kernel work per event	none	none until overflow; then one interrupt + optional stack walk + ring-buffer write
Result type	exact total (unless multiplexed)	statistical estimate, ≈ 1 observation per period
Overhead driver	number of events × read frequency	sample rate × cost per sample
Needs mmap?	no	yes — the ring buffer is the only delivery channel
Needs overflow-interrupt support in the PMU?	no	yes — this is why sampling fails in many VMs while counting works
Degrades under pressure by	multiplexing — each event runs only part of the time, counts are scaled	throttling — samples are dropped for the rest of the tick, and PERF_RECORD_THROTTLE is emitted
Silent-failure mode	scaled counts presented as if exact	throttled profile presented as if representative
Typical perf_event_attr	read_format = TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING	sample_type = IP|TID|TIME|PERIOD[|CALLCHAIN], wakeup_events, precise_ip

Counting and sampling as two configurations of one interface. What it shows: the modes are not two tools but two settings of struct perf_event_attr, and almost every downstream difference — ring buffer, NMI, statistical error, VM support — follows from that one field being zero or non-zero. The insight to take: look at the last two rows. Both modes have a degradation path that produces plausible-looking wrong numbers rather than an error, and in both cases the kernel does tell you (through time_enabled/time_running, and through PERF_RECORD_THROTTLE records) — but only if you read the field. Most confusion about perf output is a failure to read the degradation signal, not a failure to understand the mode.

Counting Mode — perf stat

In counting mode the event’s counter just runs. For a hardware event that is a literal PMU register incrementing in silicon; for a software event it is a kernel variable being bumped. Userspace obtains the value by read()-ing the perf file descriptor (or letting the kernel scale and aggregate across CPUs). perf stat wraps this: it opens the events, runs the workload, and on completion prints the totals. The man page describes it as gathering “performance counter statistics” over the command’s execution, accumulating “over its entire execution” (perf-stat(1)).

Three properties make counting the right first tool:

1. Exactness. You see every event, not a sampled fraction. cycles and instructions from perf stat are the true totals, so IPC (instructions ÷ cycles) is exact.
2. Near-zero overhead. Nothing is recorded per event — the counter increments for free in hardware, and the only software cost is reading the total at the end (or at each interval). Brendan Gregg’s advice is to “start by counting events using the perf stat command… This subcommand costs the least overhead” (brendangregg.com/perf.html).
3. No localization. A total cannot tell you which function spent the cycles. For that you must sample.

perf stat -I 1000 prints “count deltas every N milliseconds” (per the man page) — counting per interval rather than once at the end, the basis of live counter dashboards. When you request more hardware events than the PMU has counters, the kernel multiplexes and scales the result by the fraction of time each event was actually scheduled, surfaced through the TOTAL_TIME_ENABLED / TOTAL_TIME_RUNNING read fields — exactness then becomes an estimate. (See perf stat and perf top and The Performance Monitoring Unit.)

Multiplexing — when the exact total quietly becomes an estimate

Property 1 above (“exactness”) carries a large asterisk, and it is the single most-missed caveat in counting mode. A CPU has a fixed, small number of general-purpose performance counters — typically four to eight per core. Ask for more simultaneous hardware events than there are counters and the kernel does not fail; it time-slices them. A per-CPU hrtimer (perf_mux_hrtimer_handler() → perf_rotate_context(), kernel/events/core.c, v6.12) fires every perf_event_mux_interval_ms (1 ms by default, readable at /sys/bus/event_source/devices/cpu/perf_event_mux_interval_ms) and rotates which flexible events currently own hardware.

Each event therefore observes only a fraction of the measurement window, and the kernel reports that fraction so userspace can correct for it. Setting read_format to include PERF_FORMAT_TOTAL_TIME_ENABLED | PERF_FORMAT_TOTAL_TIME_RUNNING makes read(fd) return three u64s instead of one, and the correction is:

$\text{scaled\_count}=\text{raw\_count}\times \frac{\text{time\_enabled}}{\text{time\_running}}$

walked symbol by symbol: raw_count is what the counter actually accumulated; time_enabled is the total nanoseconds the event was supposed to be measuring (roughly, how long your workload ran); time_running is the nanoseconds it actually held a hardware counter. Their ratio is ≥ 1, and multiplying by it extrapolates “what the count would have been had the event run the whole time.” This is exactly the number perf stat prints, and the percentage it shows in brackets — [ 66.67% ] — is 100 × time_running / time_enabled. A bracketed percentage below 100 is perf stat telling you this number is an extrapolation, not a measurement.

The extrapolation assumes the event’s rate is stationary over the window. Measured on the test box (a deterministic 16 MiB shuffled-sum workload, four runs, exclude_kernel = 1):

Events requested	time_running / time_enabled	instructions raw	instructions scaled	Error vs. unmultiplexed truth
4	100.00 %	144,709,558	144,709,558	— (this is the truth)
5	100.00 %	144,709,623	144,709,623	+0.00004 %
6	85.03 %	122,958,988	144,604,550	−0.07 %
10	44.73 %	70,556,521	157,722,617	+9.0 %

Measured with a 60-line perf_event_open() program, not perf stat (which is not installed on that box); full output in the measurement section below. What it shows: the workload is deterministic enough that instructions is reproducible to five significant figures when it is not multiplexed, which makes it a usable ground truth. At 6 events the scaled estimate is still excellent (−0.07 %); at 10 events, where each event holds a counter under half the time, the estimate is 9 % high. The insight to take: scaling is not “wrong”, it is noisy, and the noise grows as the duty cycle falls. A 9 % error in instructions is a 9 % error in IPC, which is enough to invert a conclusion about whether a workload is front-end or back-end bound. If the bracketed percentage is low, either drop events and re-run, or run the same workload several times with different event subsets.

The interesting detail is where the threshold sits. Five events ran at 100 %; six did not. AMD Zen-generation cores expose six general-purpose core PMU counters, and this box has the NMI hard-lockup watchdog enabled (/proc/sys/kernel/nmi_watchdog reads 1), which permanently occupies one of them — leaving five for everyone else, exactly matching the observed cliff.

Uncertain

The explanation for the 5-vs-6 threshold. Verify: that this Zen 5 core has six general-purpose core PMCs and that nmi_watchdog=1 consumes exactly one of them, leaving five. Reason: the counter count was inferred from the measured cliff plus the watchdog sysctl, not read from a primary source — /sys/bus/event_source/devices/cpu/caps/ on this box exposes only branches: 16 and max_precise: 0, not a counter count, and the AMD PPR for this part was not consulted. To resolve: read the “Performance Monitor Counters” section of the AMD Processor Programming Reference for family 1Ah, or set nmi_watchdog=0 (needs root) and re-run the measurement — if the cliff moves from 6 to 7, the watchdog explanation is confirmed. #uncertain

Three practical consequences follow. First, fixed-function counters do not multiplex the same way: on Intel, instructions, cycles, and ref-cycles have dedicated counters and are usually schedulable alongside a full set of programmable events, so an event list’s composition matters as much as its length. Second, grouping changes the failure mode: events opened as a group (passing the group leader’s fd to perf_event_open) are scheduled all-or-nothing, so a group that does not fit is never scheduled at all rather than being scaled — which is what you want when you are computing a ratio between two events and need them to have observed the same window. Third, perf stat --no-scale exists precisely to show the raw, unscaled counts when you would rather see the honest partial measurement than an extrapolation. The front-end ergonomics are covered in perf stat and perf top; the hardware constraint behind all of it is in The Performance Monitoring Unit.

Sampling Mode — perf record

Sampling inverts the relationship. Instead of reading a total, you tell the counter to overflow after a fixed number of events, and you treat each overflow as a sampling tick. The man page: an overflow notification is generated “every N events, where N is given by sample_period.” Mechanically, the counter is loaded with a negative starting value of -N; it counts up; when it wraps past zero the PMU raises a performance-monitoring interrupt — on x86 a non-maskable interrupt (NMI), so it can fire even in code that has disabled normal interrupts. The kernel’s overflow handler (__perf_event_overflow() in kernel/events/core.c) then assembles a sample and writes it into the per-CPU mmap ring buffer that perf set up; the perf record process drains that buffer asynchronously into perf.data. Gregg describes exactly this: sampling “writes event data to a kernel buffer, which is read at a gentle asynchronous rate by the perf command.”

Traced end to end against the v6.12 source, one sample looks like this:

sequenceDiagram
    autonumber
    participant PMU as PMU counter<br/>(loaded with -N)
    participant NMI as x86_pmu_handle_irq()<br/>arch/x86/events/core.c:1670
    participant OVF as __perf_event_overflow()<br/>core.c:9805
    participant ACC as __perf_event_account_interrupt()<br/>core.c:9654
    participant OUT as perf_event_output()<br/>core.c:8068
    participant RB as mmap ring buffer<br/>(data_head / data_tail)
    participant USR as perf record / reader
    PMU->>PMU: counter increments per event
    PMU->>NMI: wraps past 0 → raise PMI (an NMI on x86)
    Note over NMI: fires even with IRQs off,<br/>so kernel critical sections are visible
    NMI->>OVF: perf_event_overflow(event, data, regs)
    OVF->>ACC: account this interrupt
    ACC-->>OVF: interrupts > max_samples_per_tick ?<br/>→ set MAX_INTERRUPTS, log THROTTLE, return 1
    ACC->>ACC: if attr.freq, call perf_adjust_period()<br/>to re-derive the period
    OVF->>OUT: overflow_handler(event, data, regs)
    OUT->>RB: perf_output_begin() reserves space,<br/>copies IP / TID / TIME / CALLCHAIN,<br/>advances data_head
    OUT-->>RB: no space → increment lost count,<br/>emit PERF_RECORD_LOST when room appears
    OVF->>USR: irq_work_queue(pending_irq)<br/>→ wake poll()/SIGIO out of NMI context
    USR->>RB: read records, advance data_tail
    USR->>USR: append to perf.data
    PMU->>PMU: counter reloaded with -N, repeat

One sample, from counter wrap to perf.data. The x86 entry point is perf_event_nmi_handler() → x86_pmu_handle_irq() (arch/x86/events/core.c:1670), whose loop calls perf_event_overflow() at line 1713 and whose comment at line 1729 states the policy plainly: “Always use NMI for PMU”. What it shows: the overflow is delivered as a non-maskable interrupt, so the handler runs in NMI context and cannot take sleeping locks or wake a process directly — which is why the wakeup is deferred through irq_work_queue() (core.c:9882) instead of calling perf_event_wakeup() inline. The insight to take: two lossy steps hide in this path and neither raises an error. Step 6 can decide this sample is over budget and discard the rest of the tick’s samples (throttling); step 10’s ring buffer can be full, in which case records are counted as lost and a PERF_RECORD_LOST is emitted later. A profile is a best-effort stream, and perf report will tell you about lost events only if you look.

The ring buffer is the mode’s real cost

Counting needs no memory mapping at all; sampling cannot work without one, and the buffer’s size and drain rate are what actually bound a profiler’s overhead. The layout is fixed by the UAPI (include/uapi/linux/perf_event.h, v6.12): you mmap (1 + 2^n) pages on the event fd; page 0 is a struct perf_event_mmap_page control page and the remaining 2^n pages are the data area.

flowchart LR
  subgraph MAP["mmap(fd, (1 + 2^n) pages)"]
    direction TB
    CTRL["page 0 — struct perf_event_mmap_page<br/>data_head (kernel writes)<br/>data_tail (userspace writes)<br/>data_offset, data_size<br/>time_mult/time_shift for rdpmc"]
    DATA["pages 1..2^n — record area<br/>PERF_RECORD_SAMPLE (9)<br/>PERF_RECORD_LOST (2)<br/>PERF_RECORD_THROTTLE (5)<br/>PERF_RECORD_UNTHROTTLE (6)"]
  end
  K["NMI handler<br/>perf_output_begin()"] -->|"reserve, copy,<br/>advance data_head"| DATA
  U["reader (perf record,<br/>or your own loop)"] -->|"read from data_tail<br/>to data_head, then<br/>store new data_tail"| DATA
  CTRL -.->|"smp_rmb() after reading data_head"| U
  U -.->|"smp_mb() before storing data_tail"| CTRL
  FULL["data_head - data_tail == data_size"] -->|"non-overwrite mode"| LOST["kernel drops the sample,<br/>counts it, emits<br/>PERF_RECORD_LOST later"]
  FULL -->|"overwrite mode (perf record --overwrite)"| OVER["kernel overwrites the oldest<br/>records; reader keeps only<br/>the most recent window"]

The perf ring buffer, from the v6.12 UAPI definitions. What it shows: a single-producer/single-consumer ring with the producer in NMI context, synchronised by nothing more than two counters and a pair of memory barriers that the header itself documents (smp_rmb() after reading data_head, smp_mb() before storing data_tail, lines 711–716). The insight to take: the buffer is the pressure valve for the whole mode. Too small, or a reader too slow, and samples are lost rather than delayed — and the loss is reported as a separate PERF_RECORD_LOST record carrying a count, which is easy to never look at. That is why perf record -m (mmap pages) is a real tuning knob and not an obscure one, and why --overwrite exists: for “capture the last N milliseconds when something goes wrong” you would rather lose old samples than new ones.

What goes into each sample is chosen by the sample_type bitmask in perf_event_attr. From the v6.12 UAPI header, the relevant flags include PERF_SAMPLE_IP (the instruction pointer — the most basic, “where was the CPU”), PERF_SAMPLE_CALLCHAIN (the call stack, which is what makes flame graphs possible), PERF_SAMPLE_TID, PERF_SAMPLE_TIME, PERF_SAMPLE_ADDR, PERF_SAMPLE_CPU, and PERF_SAMPLE_REGS_USER/_STACK_USER (for DWARF stack unwinding). Capturing PERF_SAMPLE_IP plus PERF_SAMPLE_CALLCHAIN over thousands of samples and bucketing by stack gives you a statistical profile: the functions that appear in the most samples are where the event is concentrated.

The trade-off is the mirror of counting: sampling tells you where but is statistical (you see ~1 event in N, so rare hot spots can be missed and the numbers carry sampling noise) and has overhead proportional to the sample rate (each sample is an interrupt plus a stack walk plus a ring-buffer write). The whole game is choosing a rate high enough to be representative but low enough not to perturb the workload.

Stack unwinding makes samples expensive

A bare-IP sample is cheap; a sample with a call stack is not, and how the stack is captured matters. perf record -g (--call-graph) offers three methods (per perf-record(1)): fp (walk frame pointers — fast but needs frame-pointer-compiled binaries), dwarf (copy a chunk of the user stack into each sample and unwind it later with DWARF CFI — works without frame pointers but bloats every sample and raises overhead), and lbr (use the CPU’s hardware Last Branch Record — low overhead but limited depth and user-space only on the listed Intel parts). The choice is a recurring source of “my profile has no stacks” confusion and is detailed in Flame Graphs and Stack Sampling.

Period vs Frequency — Two Ways to Set the Rate

Within sampling there is a sub-choice: hold the period fixed, or hold the frequency (samples per second) fixed. They share one union in the struct:

union {
	__u64 sample_period;   /* sample every N events     */
	__u64 sample_freq;     /* with freq=1: target Hz    */
};
...
__u64 freq : 1,            /* use freq, not period      */

(from the v6.12 UAPI header). Whether the field is read as a period or a frequency is decided by the single-bit freq flag.

Because both modes and both rate settings are the same struct, it is worth writing out what each front-end command actually asks for. These are the fields that differ; everything else (type, config, exclude_*, inherit) is orthogonal:

Command	sample_period	sample_freq	freq	read_format	sample_type	Needs mmap?
perf stat -e cycles	0	0	0	TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING	0	no
perf stat -e '{cycles,instructions}'	0	0	0	above + PERF_FORMAT_GROUP	0	no
perf stat -I 1000	0	0	0	as above, read() every 1000 ms	0	no
perf record (defaults)	—	4000	1	—	IP|TID|TIME|PERIOD (+CALLCHAIN with -g)	yes
perf record -F 99	—	99	1	—	as above	yes
perf record -c 10000	10000	—	0	—	as above	yes
perf top	—	4000	1	—	as above	yes

The two front-ends as perf_event_attr configurations. What it shows: perf stat and perf record differ in three fields and one syscall (mmap). The -F/-c choice is one bit. The insight to take: everything this note says about the two modes is a statement about struct perf_event_attr, which is why it applies equally to anything that calls perf_event_open(2) directly — a language runtime’s built-in profiler, a continuous-profiling agent, or the 60-line C program used for the measurements below. The mode is not a perf feature.

The choice itself is a short decision:

flowchart TD
  A["I want to sample event E"] --> B{"Does N have physical<br/>meaning for E?<br/>e.g. one sample per<br/>10,000 cache misses"}
  B -->|yes| C["Fixed period<br/>attr.freq = 0<br/>attr.sample_period = N<br/>(perf record -c N)"]
  B -->|no, I want a<br/>steady rate| D{"Do I need a bounded,<br/>predictable overhead<br/>budget?"}
  D -->|yes| E["Fixed frequency<br/>attr.freq = 1<br/>attr.sample_freq = H<br/>(perf record -F H)"]
  D -->|"no, I need exact<br/>reproducibility of<br/>which events are sampled"| C
  E --> F{"Is H a round number<br/>like 100 or 1000?"}
  F -->|yes| G["Use an odd nearby value:<br/>99, 999, 4001 —<br/>avoids lockstep with<br/>periodic timers"]
  F -->|no| H["OK"]
  G --> H
  H --> I{"H &gt; kernel.perf_event_max<br/>_sample_rate?"}
  I -->|yes| J["perf_event_open returns EINVAL;<br/>perf silently clamps unless<br/>--strict-freq"]
  I -->|no| K["Kernel runs a feedback loop:<br/>perf_adjust_period() re-derives<br/>the period every tick"]
  C --> L["Period is constant;<br/>sample rate varies with<br/>the workload"]

Choosing between -c and -F. What it shows: fixed period is the right answer only when N itself is the quantity of interest or when exact reproducibility matters; otherwise fixed frequency wins because it makes overhead the controlled variable. The insight to take: the two branches differ in which quantity is held constant and therefore which one is allowed to move. Under -c, overhead moves with the workload’s event rate; under -F, the per-sample event weight moves instead — which is exactly why PERF_SAMPLE_PERIOD exists, so the reader can weight each sample by the period in force when it was taken rather than counting samples equally.

Fixed period (perf record -c N, “Event period to sample”): overflow exactly every N events. The man page sets the rule and the front-end follows it. Fixed period is deterministic and the right choice when N has physical meaning — -e cache-misses -c 10000 samples one in every 10 000 cache misses regardless of how fast they occur. Its weakness: the temporal sample rate then depends entirely on how busy the event is. Pick N for cache-misses and run a cache-light workload and you may get a handful of samples all hour; pick it for a cache-heavy workload and you may be flooded.

Fixed frequency (perf record -F H, “Profile at this frequency”): you ask for roughly H samples per second and let the kernel find the period that achieves it. This is the default — perf record’s record_opts initializes .freq = 4000 in tools/perf/builtin-record.c (v6.12), so an unqualified perf record samples at 4000 Hz. Frequency mode is preferred for most profiling because it makes the sample budget (samples/second, which bounds overhead) the thing you control directly, rather than a derived quantity that swings with workload behavior.

How the kernel turns a target frequency into a period

Frequency mode is implemented entirely in software — the PMU only knows about periods, so the kernel must continuously re-derive the period that yields the requested Hz, because the event’s rate drifts as the workload changes phase. Two functions in kernel/events/core.c (v6.12) do this. perf_calculate_period() inverts the relationship “samples per second = event-rate ÷ period”: given how many events were observed over a known nanosecond interval and the target frequency, it solves for the period that would have produced the desired number of samples. perf_adjust_period() then nudges the live period toward that target rather than snapping to it, through a low-pass filter:

delta = (s64)(period - hwc->sample_period);
if (delta >= 0)
	delta += 7;
else
	delta -= 7;
delta /= 8;                    /* low pass filter */
sample_period = hwc->sample_period + delta;

(quoted from the v6.12-era fix on LKML, tip perf/core 2409; the delta /= 8 is the damping factor, and the directional +7/-7 is rounding so that small adjustments are not lost to integer truncation). The adjustment runs both when the event overflows and on every scheduler tick while the event is active, so the period chases the target frequency smoothly instead of oscillating. The insight: frequency mode is the kernel running a feedback loop on the period; a sudden phase change in the workload causes a brief transient over- or under-sampling until the filter re-converges.

Why not just sample as fast as possible?

Because sampling overhead is real and the kernel caps it. A global ceiling, kernel.perf_event_max_sample_rate, defaults to DEFAULT_MAX_SAMPLE_RATE = 100000 (100 000 samples/s per CPU) in kernel/events/core.c. Request more and perf “throttle[s] down to the currently maximum allowed frequency” (per perf-record(1)), unless you pass --strict-freq to fail instead. Beyond the static ceiling there is a dynamic throttle: perf_sample_event_took() tracks how long sample handling takes (an exponential moving average) and, because “perf samples are done in some very critical code paths (NMIs)… if they take too much CPU time, the system can lock up,” it auto-lowers the effective rate to keep handler time under kernel.perf_cpu_time_max_percent of CPU. This is why a profile can silently contain fewer samples than -F requested.

That dynamic lowering is not hypothetical, and it leaves a fingerprint you can read. On the test box:

$ cat /proc/sys/kernel/perf_event_max_sample_rate
62000
$ cat /proc/sys/kernel/perf_cpu_time_max_percent
25

62000, not the DEFAULT_MAX_SAMPLE_RATE of 100000. The mechanism is in perf_sample_event_took() (core.c:554): it keeps a decaying per-CPU average of how long sample handling takes, and when that average exceeds the allowance it recomputes

max = (TICK_NSEC / 100) * sysctl_perf_cpu_time_max_percent;
if (avg_len < max)
        max /= (u32)avg_len;
else
        max = 1;
WRITE_ONCE(max_samples_per_tick, max);
sysctl_perf_event_sample_rate = max * HZ;

This kernel is CONFIG_HZ=1000, so 62000 = 62 × 1000 means max_samples_per_tick was driven down to 62 samples per 1 ms tick. The kernel also logs perf: interrupt took too long (... > ...), lowering kernel.perf_event_max_sample_rate to 62000 — though kernel.dmesg_restrict = 1 on this box hid it from an unprivileged shell, so the sysctl value was the only visible evidence.

Two consequences worth internalising. First, the ceiling is per-machine and changes over a machine’s uptime — a -F value that worked yesterday can be rejected today, and the same profiling script can produce different sample densities on two supposedly identical hosts. Second, the ceiling is enforced at perf_event_open() time: attr.sample_freq > sysctl_perf_event_sample_rate returns EINVAL (core.c:12701), which the measurement section below demonstrates landing on the exact boundary. Whether you see an error or a silently reduced rate depends on whether your tool clamps before calling: perf record clamps unless --strict-freq, while a program calling perf_event_open() directly simply gets EINVAL.

Why Frequency Mode Is Usually Preferred

• Overhead is bounded and predictable. At a fixed Hz you know roughly how many interrupts per second you are paying for, independent of the workload. With fixed period, a sudden burst of the event can spike the sample rate and the overhead with it.
• Comparable profiles across runs and workloads. Two runs at -F 999 are directly comparable; two runs at -c N produce sample counts that depend on each run’s event volume.
• Avoiding lockstep aliasing. Gregg recommends an odd frequency such as 99 Hz over a round 100 Hz: “The choice of 99 Hertz, instead of 100 Hertz, is to avoid accidentally sampling in lockstep with some periodic activity, which would produce skewed results” — e.g. a 100 Hz timer task being caught (or missed) every single time. -F 99 is the canonical low-overhead production rate; the 4000 Hz default is fine for short interactive profiling but “higher frequencies means higher overhead.”

Fixed period still wins when you want a per-event sampling ratio with physical meaning (one sample per N cache misses for a memory study), or when reproducibility of the exact sampled events matters more than a steady temporal rate.

Skid — Why the Sampled IP Lies, and What Fixes It

The deepest accuracy gotcha in sampling is skid. The man page defines it precisely: skid “is how many instructions execute between an event of interest happening and the kernel being able to stop and record the event.” The chain from PMU overflow to recorded IP is not instantaneous: the counter wraps, the interrupt is raised, the pipeline drains, the NMI handler runs and reads the current instruction pointer — and by then the CPU has retired several more instructions past the one that actually caused the overflow. So PERF_SAMPLE_IP points near the culprit, not at it. Gregg: “if you’re trying to capture the IP on some PMC event, and there’s a delay between the PMC overflow and capturing the IP, then the IP will point to the wrong address. This is skew” (brendangregg.com/perf.html). The visible symptom is a profile that blames the instruction after a slow load, or smears a hot loop across nearby lines.

perf exposes a request for less skid through precise_ip, a 2-bit field whose levels the UAPI header documents verbatim:

precise_ip : 2,   /* skid constraint */
/*
 *  0 - SAMPLE_IP can have arbitrary skid
 *  1 - SAMPLE_IP must have constant skid
 *  2 - SAMPLE_IP requested to have 0 skid
 *  3 - SAMPLE_IP must have 0 skid
 */

On the command line these map to the :p event modifiers: :p (precise level 1), :pp (level 2), :ppp (level 3) — e.g. perf record -e cycles:pp. Honoring them requires hardware support, because the only way to truly eliminate skid is for the silicon to record the precise architectural state at the moment of the event rather than letting an interrupt arrive late. On Intel that mechanism is PEBS (Precise Event-Based Sampling); on AMD it is IBS (Instruction-Based Sampling). Gregg: PEBS uses “CPU hardware support to capture the real state of the CPU at the time of the event.” This is the entire motivation for Precise Event-Based Sampling — covered in depth there, including which events support it, the “one-off” PEBS IP correction, and why some events still cannot be made precise.

Measured on a Real Box

Everything above can be demonstrated on an ordinary Linux machine by an ordinary user. This section shows the measurements behind the numbers already quoted, because the interesting results are the ones that contradict the tidy story.

The constraint that shapes all of it. kernel.perf_event_paranoid = 2 on the test box. Per Documentation/admin-guide/perf-security.rst (v6.12), level >= 2 means “scope includes per-process performance monitoring only. CPU and system events happened when executing in user space only can be monitored.” Concretely: an unprivileged perf_event_open(2) succeeds if and only if it targets a process (not pid = -1 system-wide) and sets exclude_kernel = 1. That is a real restriction, not a formality — it means every count and every sample below is user-space only, so kernel time in these workloads is invisible. It is also the reason unprivileged profiling of a syscall-heavy workload produces a profile that looks strangely empty.

perf_event_paranoid	What an unprivileged process may do	Counting works?	Sampling works?
-1	everything, and perf_event_mlock_kb is ignored	yes	yes, system-wide
>= 0	per-process and system-wide, but no raw tracepoints and no ftrace function tracepoints without CAP_PERFMON	yes	yes
>= 1	per-process only — no CPU-wide events (pid = -1) without CAP_PERFMON	yes, per-process	yes, per-process
>= 2 (the default)	per-process only, and user-space events only — every event must set exclude_kernel = 1	yes, user-only	yes, user-only

From Documentation/admin-guide/sysctl/kernel.rst (v6.12, lines 929–955) and perf-security.rst. What it shows: the levels are cumulative restrictions, and the capability that lifts them is CAP_PERFMON, not CAP_SYS_ADMIN — the docs explicitly discourage CAP_SYS_ADMIN for this. The insight to take: paranoid level restricts scope, never mode. Both counting and sampling work at every level ≥ 0; what shrinks is what you are allowed to point them at. So “I can’t profile without root” is usually false — what is true is “I can’t profile the kernel, or the whole machine, without CAP_PERFMON.”

Two pieces of history make this table make sense. First, CAP_PERFMON exists because perf_event_open() used to be gated on CAP_SYS_ADMIN, which LWN describes as having become “the new root” — granting it to let someone profile also let them “mount filesystems, access PCI configuration spaces, tune memory-management policies, load BPF programs, and more” (LWN, “CAP_PERFMON — and new capabilities in general”, 2020). Splitting it out was the whole point. Second, you may encounter perf_event_paranoid = 3 on Debian- or Android-derived kernels: that level restricts perf_event_open() to capability-holders entirely and is a downstream patch that was rejected upstream, with Peter Zijlstra objecting that “we have bugs we fix them, we don’t kill complete infrastructure because of them” (LWN, “Disallowing perf_event_open()”, 2016). So a 3 in that file means a vendor patch, not a mainline level — and at 3, both counting and sampling are unavailable to an unprivileged user regardless of exclude_kernel.

One related limit shows up the moment you try to mmap a ring buffer: /proc/sys/kernel/perf_event_mlock_kb read 516 on the test box, matching the source default 512 + (PAGE_SIZE / 1024) (core.c:456) exactly — 512 KiB of buffer plus one 4 KiB control page. That is the per-user-per-CPU allowance that does not count against RLIMIT_MEMLOCK; the 64 data pages (256 KiB) used in the sampling program below sit comfortably inside it.

Counting mode, and the multiplexing cliff. The program opens N PERF_TYPE_HARDWARE/PERF_TYPE_HW_CACHE events as independent (ungrouped) events with read_format = PERF_FORMAT_TOTAL_TIME_ENABLED | PERF_FORMAT_TOTAL_TIME_RUNNING, runs a fixed workload (build a 16 MiB unsigned array, then sum it four times in a stride-17 shuffled order), and prints raw, enabled, running, and scaled values.

$ ./count 5
event                         raw_count   time_enabled   time_running      pct       scaled_count
cpu-cycles                     56182450       13861758       13861758  100.00%           56182450
instructions                  144709623       13892024       13892024  100.00%          144709623
branch-instructions            17830264       13885101       13885101  100.00%           17830264
branch-misses                      5436       13881865       13881865  100.00%               5436
cache-references               35767115       13879972       13879972  100.00%           35767115

$ ./count 10
event                         raw_count   time_enabled   time_running      pct       scaled_count
cpu-cycles                     25417435       13772150        5133883   37.28%           68184789
instructions                   70556521       13819219        6181967   44.73%          157722617
branch-instructions             9411539       13812887        7248154   52.47%           17935674
branch-misses                      5110       13810612        8163637   59.11%               8645
cache-references               15240681       13808549        8818174   63.86%           23865677
cache-misses                     261441       13806584        8672701   62.82%             416203
L1d-read-access                 8809741       13803990        7644806   55.38%           15907477
L1d-read-miss                   8211982       13800443        6572157   47.62%           17243804
dTLB-read-access                 136210       13796997        5654339   40.98%             332362
dTLB-read-miss                    65006       13794222        4990375   36.18%             179687

Three readings. (a) At five events every pct is exactly 100 % and instructions = 144,709,623 — reproducible to five significant figures across runs, which is what makes it usable as ground truth. (b) At ten events the duty cycles spread from 36.18 % to 63.86 % and are markedly unequal, so two events in the same perf stat run did not observe the same window; comparing them is comparing two different measurements. (The observed pattern is a hump — lowest at both ends of the list, highest in the middle — which this harness cannot explain cleanly, since it also enables events 1..N−1 slightly before event 0; the honest claim is only that the duty cycles are unequal, not that list position predicts them.) (c) The scaled instructions is 157,722,617 against a true 144,709,623: +9.0 %. The extrapolation is doing real work and getting it visibly wrong.

A fourth reading is the most useful: branch-instructions scaled to 17,935,674 against a true 17,830,264, an error of +0.6 %, while instructions on the same run was off by 9 %. Same duty-cycle regime, wildly different accuracy — because the branch rate happened to be more stationary across the rotation slices than the instruction rate. Scaling error is not predictable from the duty cycle alone, which is why “the percentage was above 50 %, so it’s fine” is not a safe rule.

Sampling mode, and what a profile converges to. The second program mmaps a 64-page ring buffer on a cpu-cycles event with sample_type = IP | TID | TIME | PERIOD, runs three functions whose iteration counts are in a known 400 : 100 : 20 ratio (76.92 % / 19.23 % / 3.85 %), and buckets the sampled instruction pointers by function:

Mode	Samples	hot()	warm()	cold()	Mean period actually used	THROTTLE records
(ground truth)	—	76.92 %	19.23 %	3.85 %	—	—
-c 20,000,000	26	76.9 %	19.2 %	3.8 %	20,000,000 (exact)	0
-c 1,000,000	535	77.4 %	18.9 %	3.7 %	1,000,000 (exact)	0
-c 100,000	5,358	76.9 %	19.2 %	3.8 %	100,000 (exact)	0
-F 997	92	72.8 %	20.7 %	4.3 %	5,657,061	0
-F 10000	1,058	76.7 %	19.2 %	3.8 %	496,006	0
-F 61999	6,415	83.9 %	15.9 %	0.0 %	76,447	86
-F 62001	—	-	-	-	—	perf_event_open: EINVAL

Six sampling configurations against a known ground truth, same workload each time. What it shows: period mode reports the period back verbatim (PERF_SAMPLE_PERIOD equals what was asked for); frequency mode reports a derived period that scales roughly inversely with the requested rate, which is perf_adjust_period()’s feedback loop converging. Statistical convergence is visible in the sample counts — 92 samples at -F 997 is off by 4 points on hot(), 1,058 samples at -F 10000 is off by 0.2. The insight to take: the -F 61999 row is the important one and it is the opposite of what more sampling is supposed to buy you.

One row is a trap for the unwary: -c 20,000,000 produced 26 samples that landed on 20/5/1 — exactly the true ratio to one decimal place. That is coincidence, not convergence. With 26 samples the 95 % confidence interval on cold()’s 3.8 % spans roughly 0 % to 20 %; the estimate happened to be perfect, and a second run would not be. The lesson is that a profile’s plausibility says nothing about its precision — only the sample count does, and it should be reported alongside any percentage.

The -F 61999 row deserves unpacking. Asking for 61,999 Hz — just under the box’s 62,000 ceiling — produced 86 PERF_RECORD_THROTTLE / PERF_RECORD_UNTHROTTLE pairs and a profile that is materially wrong: hot() inflated from 76.9 % to 83.9 %, and cold() vanished entirely (0 of 6,415 samples, where ~247 were expected). The mechanism is __perf_event_account_interrupt() (core.c:9654): once an event exceeds max_samples_per_tick within one tick it is marked MAX_INTERRUPTS and stops sampling for the remainder of that tick, resuming only when perf_adjust_freq_unthr_context() unthrottles it on the next tick. So throttling does not thin the profile uniformly — it deletes the tail of each tick. Any function whose execution correlates with the end of a tick is under-counted, and cold(), being short and running last, was systematically the victim.

This is the single most important practical lesson in this note: turning the sample rate up can make a profile less accurate, not more. The failure is silent unless you count PERF_RECORD_THROTTLE records, which perf report will show in its event summary but which most people never look at. The canonical 99 Hz / 999 Hz advice is not only about lockstep aliasing — it is also comfortably below any throttle threshold.

The ceiling is a hard edge. -F 61999 opened; -F 62001 returned EINVAL from perf_event_open(). That is core.c:12701:

if (attr.sample_freq > sysctl_perf_event_sample_rate)
        return -EINVAL;

— checked against the current, dynamically adjusted sysctl_perf_event_sample_rate, which is why the boundary sat at 62,000 rather than the compiled-in 100,000.

Uncertain

Verify: whether perf record -F 61999 on this same box would show the same 84/16/0 distortion, or whether the perf front-end’s own clamping and buffer management would mask it. Reason: perf is not installed on the test box, so the comparison was made between two settings of a hand-written perf_event_open() program, not between perf stat and perf record. The kernel-side mechanism is the same either way (throttling happens in __perf_event_account_interrupt(), below any front-end), but perf record’s larger default ring buffer and multi-CPU aggregation could change how visible the effect is. To resolve: install perf, run perf record -F 61999 -e cycles:u ./workload and check perf report --header for throttle counts. #uncertain

Failure Modes and Misunderstandings

• “My perf record profile has zero samples.” The event may be a hardware event with no overflow-interrupt support (common in VMs — “PMU Hardware doesn’t support sampling/overflow-interrupts”), or the rate was throttled to near zero, or the workload barely ran. Counting the same event with perf stat confirms whether the event works at all.
• “perf stat is exact but perf record totals are smaller.” Expected. The “Samples” in a record run are a fraction (≈ 1/period) of the true event count; record is not trying to count everything.
• “I set -F 100000 and got fewer samples.” You hit perf_event_max_sample_rate and/or the dynamic CPU-time throttle. Check cat /proc/sys/kernel/perf_event_max_sample_rate and look for “throttle” lines.
• “The hot line in my flame graph is one instruction off.” Classic skid. Re-run with :pp / PEBS — see Precise Event-Based Sampling.
• “Profiling at exactly 1000 Hz gave weird results.” Possible lockstep with a periodic timer; use 99/999/4001 Hz instead.
• “I raised -F to get a better profile and the answer changed.” Measured above: at 61,999 Hz on a box whose ceiling is 62,000, throttling deleted the tail of every tick and erased a function that genuinely consumed 3.8 % of cycles. Throttling biases which code is sampled, not just how much. Check the PERF_RECORD_THROTTLE count before trusting a high-rate profile.
• “perf_event_max_sample_rate is 100000, so -F 50000 is safe.” Only until the dynamic throttle lowers it. The measured box read 62000, driven down by perf_sample_event_took(). The ceiling is machine- and uptime-dependent; read it, do not assume the compiled-in default.
• “perf stat printed a number, so it is the count.” Only if the bracketed percentage is 100 %. Below that it is raw × enabled / running, and the measured error on instructions at a 45 % duty cycle was 9 % — enough to move an IPC figure across a decision boundary.
• “My unprivileged profile shows almost no time anywhere.” At perf_event_paranoid = 2 every event must set exclude_kernel = 1, so all kernel time is invisible by construction. A syscall-bound or page-fault-bound workload will look nearly idle. This is a scope restriction, not a broken profiler.
• “Counting works in my VM but perf record returns no samples.” Counting needs only a counter; sampling additionally needs overflow-interrupt support, which many hypervisors’ virtual PMUs do not expose. The two modes have genuinely different hardware requirements.

See Also

• Hardware vs Software Performance Events — the orthogonal axis: which event you measure (a count/sample can be taken on any of these types).
• Precise Event-Based Sampling — PEBS/IBS: the hardware that drives skid to zero, the resolution to the skid problem raised here.
• Flame Graphs and Stack Sampling — what you build from sampled call stacks, and the fp/dwarf/lbr unwinding choice.
• perf record and perf report — the sampling front-end and how perf.data is consumed.
• perf stat and perf top — the counting front-end, intervals, and multiplexing/scaling.
• The perf_event_open Syscall — the perf_event_attr fields (sample_period, sample_freq, freq, sample_type, precise_ip) this note configures.
• The Performance Monitoring Unit — the counter overflow and PMU-interrupt machinery sampling rests on.
• perf record and perf report — the sampling front-end, and where PERF_RECORD_LOST / PERF_RECORD_THROTTLE counts surface in a report.
• Linux Tracing and Observability MOC — parent map (§5, perf and the PMU).
• Performance Engineering MOC — parent map: counting, sampling, and tracing as the three measurement modes.