Counting vs Sampling Mode

Once you have chosen which event to measure — see Hardware vs Software Performance Events — there is a second, orthogonal decision: how perf uses it. There are exactly two modes, and they answer different questions. In counting mode (the perf stat mode) the counter simply accumulates; you read the running total over an interval and learn how much happened — an exact aggregate at near-zero overhead, but no notion of where. In sampling mode (the perf record mode) you configure the event to overflow every N occurrences; on each overflow the hardware raises an interrupt and the kernel records a sample — the instruction pointer, optionally the call stack, registers, and a timestamp — into a memory-mapped ring buffer. Sampling is statistical (it sees one event in N, not all of them) but it tells you where the cycles or cache misses are concentrated. The single switch that selects the mode is sample_period / sample_freq in struct perf_event_attr: leave it zero and the event counts; set it positive and the event samples. The man page makes the rule explicit — “A sampling event has sample_period > 0” (perf_event_open(2)).

This note covers the two modes, the period-versus-frequency choice within sampling, why frequency mode is usually preferred, and skid — the reason a sampled instruction pointer can lag the event that triggered it, and the motivation for IBS.

Mental Model — Aggregate vs Spotlight

flowchart TB
  subgraph COUNT["Counting (perf stat) — sample_period = 0"]
    EV1["event fires"] --> ACC["counter += 1<br/>(silently accumulates)"]
    ACC --> RD["userspace read():<br/>exact total over interval"]
  end
  subgraph SAMPLE["Sampling (perf record) — sample_period &gt; 0"]
    EV2["event fires"] --> DEC["counter toward overflow"]
    DEC -->|"every Nth event"| OVF["overflow → interrupt (NMI)"]
    OVF --> REC["record sample:<br/>IP, callchain, regs, time"]
    REC --> RING["mmap ring buffer"]
    RING --> RT["perf reads asynchronously<br/>→ perf.data"]
  end

The two ways to use one event. What it shows: counting keeps a single growing number and hands you the exact total when you ask; sampling configures the counter to interrupt every N events and snapshots program state at each interrupt. The insight to take: counting answers “how many?” precisely and cheaply but blindly; sampling answers “where?” approximately but with overhead proportional to the sample rate. They are not competitors — the standard workflow is perf stat first to quantify, then perf record to localize.

Counting Mode — perf stat

In counting mode the event’s counter just runs. For a hardware event that is a literal PMU register incrementing in silicon; for a software event it is a kernel variable being bumped. Userspace obtains the value by read()-ing the perf file descriptor (or letting the kernel scale and aggregate across CPUs). perf stat wraps this: it opens the events, runs the workload, and on completion prints the totals. The man page describes it as gathering “performance counter statistics” over the command’s execution, accumulating “over its entire execution” (perf-stat(1)).

Three properties make counting the right first tool:

1. Exactness. You see every event, not a sampled fraction. cycles and instructions from perf stat are the true totals, so IPC (instructions ÷ cycles) is exact.
2. Near-zero overhead. Nothing is recorded per event — the counter increments for free in hardware, and the only software cost is reading the total at the end (or at each interval). Brendan Gregg’s advice is to “start by counting events using the perf stat command… This subcommand costs the least overhead” (brendangregg.com/perf.html).
3. No localization. A total cannot tell you which function spent the cycles. For that you must sample.

perf stat -I 1000 prints “count deltas every N milliseconds” (per the man page) — counting per interval rather than once at the end, the basis of live counter dashboards. When you request more hardware events than the PMU has counters, the kernel multiplexes and scales the result by the fraction of time each event was actually scheduled, surfaced through the TOTAL_TIME_ENABLED / TOTAL_TIME_RUNNING read fields — exactness then becomes an estimate. (See perf stat and perf top and The Performance Monitoring Unit.)

Sampling Mode — perf record

Sampling inverts the relationship. Instead of reading a total, you tell the counter to overflow after a fixed number of events, and you treat each overflow as a sampling tick. The man page: an overflow notification is generated “every N events, where N is given by sample_period.” Mechanically, the counter is loaded with a negative starting value of -N; it counts up; when it wraps past zero the PMU raises a performance-monitoring interrupt — on x86 a non-maskable interrupt (NMI), so it can fire even in code that has disabled normal interrupts. The kernel’s overflow handler (__perf_event_overflow() in kernel/events/core.c) then assembles a sample and writes it into the per-CPU mmap ring buffer that perf set up; the perf record process drains that buffer asynchronously into perf.data. Gregg describes exactly this: sampling “writes event data to a kernel buffer, which is read at a gentle asynchronous rate by the perf command.”

What goes into each sample is chosen by the sample_type bitmask in perf_event_attr. From the v6.12 UAPI header, the relevant flags include PERF_SAMPLE_IP (the instruction pointer — the most basic, “where was the CPU”), PERF_SAMPLE_CALLCHAIN (the call stack, which is what makes flame graphs possible), PERF_SAMPLE_TID, PERF_SAMPLE_TIME, PERF_SAMPLE_ADDR, PERF_SAMPLE_CPU, and PERF_SAMPLE_REGS_USER/_STACK_USER (for DWARF stack unwinding). Capturing PERF_SAMPLE_IP plus PERF_SAMPLE_CALLCHAIN over thousands of samples and bucketing by stack gives you a statistical profile: the functions that appear in the most samples are where the event is concentrated.

The trade-off is the mirror of counting: sampling tells you where but is statistical (you see ~1 event in N, so rare hot spots can be missed and the numbers carry sampling noise) and has overhead proportional to the sample rate (each sample is an interrupt plus a stack walk plus a ring-buffer write). The whole game is choosing a rate high enough to be representative but low enough not to perturb the workload.

Stack unwinding makes samples expensive

A bare-IP sample is cheap; a sample with a call stack is not, and how the stack is captured matters. perf record -g (--call-graph) offers three methods (per perf-record(1)): fp (walk frame pointers — fast but needs frame-pointer-compiled binaries), dwarf (copy a chunk of the user stack into each sample and unwind it later with DWARF CFI — works without frame pointers but bloats every sample and raises overhead), and lbr (use the CPU’s hardware Last Branch Record — low overhead but limited depth and user-space only on the listed Intel parts). The choice is a recurring source of “my profile has no stacks” confusion and is detailed in Flame Graphs and Stack Sampling.

Period vs Frequency — Two Ways to Set the Rate

Within sampling there is a sub-choice: hold the period fixed, or hold the frequency (samples per second) fixed. They share one union in the struct:

union {
	__u64 sample_period;   /* sample every N events     */
	__u64 sample_freq;     /* with freq=1: target Hz    */
};
...
__u64 freq : 1,            /* use freq, not period      */

(from the v6.12 UAPI header). Whether the field is read as a period or a frequency is decided by the single-bit freq flag.

Fixed period (perf record -c N, “Event period to sample”): overflow exactly every N events. The man page sets the rule and the front-end follows it. Fixed period is deterministic and the right choice when N has physical meaning — -e cache-misses -c 10000 samples one in every 10 000 cache misses regardless of how fast they occur. Its weakness: the temporal sample rate then depends entirely on how busy the event is. Pick N for cache-misses and run a cache-light workload and you may get a handful of samples all hour; pick it for a cache-heavy workload and you may be flooded.

Fixed frequency (perf record -F H, “Profile at this frequency”): you ask for roughly H samples per second and let the kernel find the period that achieves it. This is the default — perf record’s record_opts initializes .freq = 4000 in tools/perf/builtin-record.c (v6.12), so an unqualified perf record samples at 4000 Hz. Frequency mode is preferred for most profiling because it makes the sample budget (samples/second, which bounds overhead) the thing you control directly, rather than a derived quantity that swings with workload behavior.

How the kernel turns a target frequency into a period

Frequency mode is implemented entirely in software — the PMU only knows about periods, so the kernel must continuously re-derive the period that yields the requested Hz, because the event’s rate drifts as the workload changes phase. Two functions in kernel/events/core.c (v6.12) do this. perf_calculate_period() inverts the relationship “samples per second = event-rate ÷ period”: given how many events were observed over a known nanosecond interval and the target frequency, it solves for the period that would have produced the desired number of samples. perf_adjust_period() then nudges the live period toward that target rather than snapping to it, through a low-pass filter:

delta = (s64)(period - hwc->sample_period);
if (delta >= 0)
	delta += 7;
else
	delta -= 7;
delta /= 8;                    /* low pass filter */
sample_period = hwc->sample_period + delta;

(quoted from the v6.12-era fix on LKML, tip perf/core 2409; the delta /= 8 is the damping factor, and the directional +7/-7 is rounding so that small adjustments are not lost to integer truncation). The adjustment runs both when the event overflows and on every scheduler tick while the event is active, so the period chases the target frequency smoothly instead of oscillating. The insight: frequency mode is the kernel running a feedback loop on the period; a sudden phase change in the workload causes a brief transient over- or under-sampling until the filter re-converges.

Why not just sample as fast as possible?

Because sampling overhead is real and the kernel caps it. A global ceiling, kernel.perf_event_max_sample_rate, defaults to DEFAULT_MAX_SAMPLE_RATE = 100000 (100 000 samples/s per CPU) in kernel/events/core.c. Request more and perf “throttle[s] down to the currently maximum allowed frequency” (per perf-record(1)), unless you pass --strict-freq to fail instead. Beyond the static ceiling there is a dynamic throttle: perf_sample_event_took() tracks how long sample handling takes (an exponential moving average) and, because “perf samples are done in some very critical code paths (NMIs)… if they take too much CPU time, the system can lock up,” it auto-lowers the effective rate to keep handler time under kernel.perf_cpu_time_max_percent of CPU. This is why a profile can silently contain fewer samples than -F requested.

Why Frequency Mode Is Usually Preferred

• Overhead is bounded and predictable. At a fixed Hz you know roughly how many interrupts per second you are paying for, independent of the workload. With fixed period, a sudden burst of the event can spike the sample rate and the overhead with it.
• Comparable profiles across runs and workloads. Two runs at -F 999 are directly comparable; two runs at -c N produce sample counts that depend on each run’s event volume.
• Avoiding lockstep aliasing. Gregg recommends an odd frequency such as 99 Hz over a round 100 Hz: “The choice of 99 Hertz, instead of 100 Hertz, is to avoid accidentally sampling in lockstep with some periodic activity, which would produce skewed results” — e.g. a 100 Hz timer task being caught (or missed) every single time. -F 99 is the canonical low-overhead production rate; the 4000 Hz default is fine for short interactive profiling but “higher frequencies means higher overhead.”

Fixed period still wins when you want a per-event sampling ratio with physical meaning (one sample per N cache misses for a memory study), or when reproducibility of the exact sampled events matters more than a steady temporal rate.

Skid — Why the Sampled IP Lies, and What Fixes It

The deepest accuracy gotcha in sampling is skid. The man page defines it precisely: skid “is how many instructions execute between an event of interest happening and the kernel being able to stop and record the event.” The chain from PMU overflow to recorded IP is not instantaneous: the counter wraps, the interrupt is raised, the pipeline drains, the NMI handler runs and reads the current instruction pointer — and by then the CPU has retired several more instructions past the one that actually caused the overflow. So PERF_SAMPLE_IP points near the culprit, not at it. Gregg: “if you’re trying to capture the IP on some PMC event, and there’s a delay between the PMC overflow and capturing the IP, then the IP will point to the wrong address. This is skew” (brendangregg.com/perf.html). The visible symptom is a profile that blames the instruction after a slow load, or smears a hot loop across nearby lines.

perf exposes a request for less skid through precise_ip, a 2-bit field whose levels the UAPI header documents verbatim:

precise_ip : 2,   /* skid constraint */
/*
 *  0 - SAMPLE_IP can have arbitrary skid
 *  1 - SAMPLE_IP must have constant skid
 *  2 - SAMPLE_IP requested to have 0 skid
 *  3 - SAMPLE_IP must have 0 skid
 */

On the command line these map to the :p event modifiers: :p (precise level 1), :pp (level 2), :ppp (level 3) — e.g. perf record -e cycles:pp. Honoring them requires hardware support, because the only way to truly eliminate skid is for the silicon to record the precise architectural state at the moment of the event rather than letting an interrupt arrive late. On Intel that mechanism is PEBS (Precise Event-Based Sampling); on AMD it is IBS (Instruction-Based Sampling). Gregg: PEBS uses “CPU hardware support to capture the real state of the CPU at the time of the event.” This is the entire motivation for Precise Event-Based Sampling — covered in depth there, including which events support it, the “one-off” PEBS IP correction, and why some events still cannot be made precise.

Failure Modes and Misunderstandings

• “My perf record profile has zero samples.” The event may be a hardware event with no overflow-interrupt support (common in VMs — “PMU Hardware doesn’t support sampling/overflow-interrupts”), or the rate was throttled to near zero, or the workload barely ran. Counting the same event with perf stat confirms whether the event works at all.
• “perf stat is exact but perf record totals are smaller.” Expected. The “Samples” in a record run are a fraction (≈ 1/period) of the true event count; record is not trying to count everything.
• “I set -F 100000 and got fewer samples.” You hit perf_event_max_sample_rate and/or the dynamic CPU-time throttle. Check cat /proc/sys/kernel/perf_event_max_sample_rate and look for “throttle” lines.
• “The hot line in my flame graph is one instruction off.” Classic skid. Re-run with :pp / PEBS — see Precise Event-Based Sampling.
• “Profiling at exactly 1000 Hz gave weird results.” Possible lockstep with a periodic timer; use 99/999/4001 Hz instead.

See Also

• Hardware vs Software Performance Events — the orthogonal axis: which event you measure (a count/sample can be taken on any of these types).
• Precise Event-Based Sampling — PEBS/IBS: the hardware that drives skid to zero, the resolution to the skid problem raised here.
• Flame Graphs and Stack Sampling — what you build from sampled call stacks, and the fp/dwarf/lbr unwinding choice.
• perf record and perf report — the sampling front-end and how perf.data is consumed.
• perf stat and perf top — the counting front-end, intervals, and multiplexing/scaling.
• The perf_event_open Syscall — the perf_event_attr fields (sample_period, sample_freq, freq, sample_type, precise_ip) this note configures.
• The Performance Monitoring Unit — the counter overflow and PMU-interrupt machinery sampling rests on.
• Linux Tracing and Observability MOC — parent map (§5, perf and the PMU).