vault

Precise Event-Based Sampling

14 min read

Precise Event-Based Sampling

When an ordinary PMU counter overflows and raises an interrupt, the instruction pointer the kernel records is not the instruction that caused the event — it is wherever the CPU happened to be when the interrupt was finally delivered, typically several instructions later. This gap is called skid, and it is fatal for instruction-level profiling: it smears a cache-miss or a branch-mispredict across the wrong source lines. Precise Event-Based Sampling (PEBS) on Intel, and the analogous Instruction-Based Sampling (IBS) on AMD, fix this in hardware: the processor itself writes a precise record — the exact instruction pointer, the architectural registers, and often the data address and access latency — into a memory buffer on the Nth occurrence of the event, with no interrupt latency in the loop. In perf, you ask for this with a precise-level suffix on the event (cycles:pp), which sets the precise_ip field of perf_event_attr. This note explains why skid happens, what PEBS and IBS capture, how the kernel turns the :p/:pp/:ppp modifiers into hardware behavior, and why precise sampling is mandatory for any instruction-level or memory-access profiling.

This note assumes the counting-vs-sampling mechanics of Counting vs Sampling Mode and the counter hardware of The Performance Monitoring Unit. Skid is a property of the sampling path specifically; counting mode is unaffected.

Mental Model — The Interrupt Always Arrives Late

A modern out-of-order CPU has hundreds of instructions in flight. When a performance event fires (a cache miss retires, a counter wraps), the CPU does not stop on the spot — it raises a Performance Monitoring Interrupt (PMI), which has to drain the pipeline, take the interrupt, and run the kernel handler before anyone can read the instruction pointer. By then the program counter has moved on. The IP the handler reads “designates the place where the program was interrupted to process the PMU interrupt, not the place where the counter actually overflows” (Bakhvalov, easyperf). The cause: “latency in the microarchitecture between the generation of events and the generation of interrupts on overflow,” compounded by out-of-order execution speculatively running far past the event point.

flowchart LR
  subgraph SKID["Ordinary PMI sampling (has skid)"]
    E1["Event-causing<br/>instruction I_n"] --> D1["pipeline drain +<br/>PMI delivery latency"]
    D1 --> H1["handler reads IP<br/>= I_n+k (k = skid)"]
  end
  subgraph PEBS["PEBS / IBS (no skid)"]
    E2["Nth event tags<br/>instruction I_n"] --> HW["hardware writes<br/>precise record"]
    HW --> R2["record holds exact<br/>IP of I_n + regs + data"]
  end

Why precise sampling exists. What it shows (top): in ordinary sampling the IP is captured by software after an interrupt-delivery delay, so it lands k instructions past the real cause. (Bottom): PEBS/IBS have the hardware tag the offending instruction and write the record directly, so the IP is exact. The insight to take: skid is not a software bug to be fixed in the kernel — it is inherent to interrupt-driven sampling on out-of-order cores, and only hardware support can remove it.

The size of k is not small. Without precise sampling the IP can be off by dozens of instructions; with PEBS “the instruction pointer is off only by a single instruction, at most” (easyperf), and on Haswell-and-later hardware even that single-instruction off-by-one is corrected (see below). This is the entire reason the feature exists: for cycles profiling at function granularity skid is tolerable, but the moment you want to know which load missed the cache or which branch mispredicted, an attribution that is off by dozens of instructions points at the wrong source line and the profile is worthless.

What the Hardware Captures

Intel PEBS

PEBS works through the Debug Store (DS) area, a software-allocated memory region whose base address the kernel programs into the MSR_IA32_DS_AREA model-specific register (ds.c). The Intel SDM describes the loop: a counter “is configured to overflow after counting a preset number of events”; when “a counter is enabled to capture machine state, the processor will write machine state information to a memory buffer,” and on the next qualifying event “the PEBS hardware triggers an assist and causes a PEBS record to be written” (Intel SDM §18.15.7). A PEBS record contains, at minimum, “the architectural state of the processor (state of the general purpose registers, EIP register, and EFLAGS register)” (easyperf, Intel SDM).

The Linux kernel models the evolving record layouts directly. In ds.c, pebs_record_core is the base format (flags, IP, general-purpose registers AX–SP and R8–R15); pebs_record_nhm (Nehalem) adds the Data Linear Address (DLA), the Data Source Encoding (DSE), and latency (LAT); pebs_record_hsw (Haswell) adds real_ip (the eventing IP) and TSX transaction data; pebs_record_skl (Skylake) adds the TSC timestamp; and format-4-and-later (Ice Lake “Adaptive PEBS”) uses a pebs_basic header with a format_size field selecting which optional groups are present (ds.c). The progression matters: it is exactly the extra fields — data address, data source, latency — that make PEBS the substrate for memory profiling. When you run perf mem or sample mem-loads, perf requests PERF_SAMPLE_ADDR, PERF_SAMPLE_DATA_SRC, PERF_SAMPLE_WEIGHT, and PERF_SAMPLE_PHYS_ADDR (perf_event.h), and the kernel fills them from these PEBS record fields — the access’s virtual/physical address, whether it hit L1/L2/L3/DRAM, and how many cycles it took. None of that is possible with ordinary skidded sampling.

A subtle terminology note: PEBS originally stood for Precise Event-Based Sampling, but when Intel updated the SDM for Goldmont in 2016 it was renamed Processor Event-Based Sampling, because Goldmont extended PEBS to support all events, not just the precise at-retirement ones (WebSearch corroboration, Intel SDM history). Both expansions refer to the same DS-based mechanism; perf and most literature still say “precise.”

The eventing IP and the kernel’s fixup

The most important detail for accuracy lives in setup_pebs_fixed_sample_data(). Even PEBS originally had a residual one-instruction off-by-one: the record’s IP pointed just past the instruction that caused the event. Haswell fixed this in hardware with the eventing IP (Intel’s “real IP”). The kernel comment states it plainly: “Haswell and later processors have an ‘eventing IP’ (real IP) which fixes the off-by-one skid in hardware. Use it when precise_ip >= 2” (ds.c). For precise_ip < 2 on pre-Haswell parts, the kernel must correct the off-by-one in software: intel_pmu_pebs_fixup_ip() uses the Last Branch Records (LBR) to find the start of the basic block and then decodes instructions forward to rewind from the off-by-one IP to the true instruction boundary. This is why precise_ip levels map onto real hardware behavior rather than being arbitrary knobs — level 2 is the threshold at which the hardware eventing IP is trusted directly.

AMD IBS

AMD’s mechanism, Instruction-Based Sampling (IBS), is architecturally different but solves the same problem. Instead of arming on a counted event, IBS tags an instruction and follows it through the pipeline, recording everything about it when it completes. There are two flavors: IBS Fetch samples instruction-fetch operations (via MSR_AMD64_IBSFETCHCTL), and IBS Op samples instruction-execution/retirement (via MSR_AMD64_IBSOPCTL) (ibs.c). Because the tagged op is tracked by hardware, its IP is exact. The kernel comment is explicit: “The rip of IBS samples has skid 0. Thus, IBS supports precise levels 1 and 2 and the PERF_EFLAGS_EXACT is set. In rare cases the rip is invalid when IBS was not able to record the rip correctly. We clear PERF_EFLAGS_EXACT and take the rip from pt_regs then” (ibs.c). IBS Op captures the precise IP (IbsOpRip), the data linear address for memory ops (IbsDcLinAd, read from MSR_AMD64_IBSDCLINAD), and cache-miss latency (op_data3.dc_miss_lat) — the AMD counterparts of PEBS’s DLA and LAT fields. IBS validation rejects anything outside levels 1–2: if (!event->attr.precise_ip || event->attr.precise_ip > 2) return -EOPNOTSUPP; (ibs.c). The pt_regs register frame the fallback path reads from is the same structure described in The pt_regs Register Frame.

One conceptual difference worth holding onto: IBS Op uses a counter that can count cycles or micro-ops (selected by the IbsOpCntCtl field, bit 19) and tags an op when it overflows, optionally with randomization. So when you ask for cycles:p on AMD, perf maps PERF_COUNT_HW_CPU_CYCLES onto “ibs op counting cycle count” (ibs.c) — a single IBS Op sampler stands in for the precise cycles event rather than one PEBS-enabled counter per event as on Intel.

The precise_ip Field and the :p Modifiers

The user-facing control is the precise-level modifier appended to an event name. perf-list(1) documents the four levels of the precise_ip field verbatim (perf-list(1)):

0 - SAMPLE_IP can have arbitrary skid
1 - SAMPLE_IP must have constant skid
2 - SAMPLE_IP requested to have 0 skid
3 - SAMPLE_IP must have 0 skid, or uses randomization to avoid sample shadowing effects.

These map onto the perf_event_attr bitfield exactly. In the UAPI header precise_ip is a 2-bit field with the matching comment (perf_event.h):

precise_ip     :  2, /* skid constraint */
   /* 0 - SAMPLE_IP can have arbitrary skid
    * 1 - SAMPLE_IP must have constant skid
    * 2 - SAMPLE_IP requested to have 0 skid
    * 3 - SAMPLE_IP must have 0 skid
    * See also PERF_RECORD_MISC_EXACT_IP */

Two bits, four values — one per precise level. The :p modifier sets it: each p you append raises the level by one, so event:p → precise_ip=1, event:pp → precise_ip=2, event:ppp → precise_ip=3 (perf-list(1)). There is also :P (“use maximum detected precise level”), which asks perf to probe the highest level the hardware accepts and use that. When a sample’s IP is genuinely exact, the kernel marks the record with PERF_RECORD_MISC_EXACT_IP (bit 14 of the record’s misc field), which signals that “the content of PERF_SAMPLE_IP points to the actual instruction that triggered the event” (perf_event.h).

The hardware-support summary from perf-list(1) is the practical key: “For Intel systems precise event sampling is implemented with PEBS which supports up to precise-level 2, and precise level 3 for some special cases” and “On AMD systems it is implemented using IBS OP (up to precise-level 2)” (perf-list(1)). So on both vendors :pp (level 2, zero requested skid) is the workhorse; level 3 is Intel-only and only for certain events.

Worked examples

# Precise cycles profile — the default reach-for. ':pp' = precise_ip=2.
$ perf record -e cycles:pp -g -- ./myapp
#                     │
#                     └─ zero-skid IPs, so per-instruction attribution is trustworthy
 
# Precise memory-load profile: PEBS data-source + latency fields drive perf mem.
$ perf mem record -- ./myapp        # internally uses a precise mem-loads event
$ perf mem report --sort=mem        # buckets by L1/L2/L3/DRAM hit, by latency
 
# Sample retired loads above a latency threshold (memory-latency hunting).
$ perf record -e cpu/mem-loads,ldlat=30/pp -- ./myapp
#                                       └─ only loads costing ≥30 cycles

The cycles:pp form is the one to internalize: it is the precise analogue of plain cycles and the right default whenever you intend to drill below the function level. For the stack side of a profile (which call path, not which instruction) see Flame Graphs and Stack Sampling — precise IP and stack unwinding are orthogonal and you usually want both (cycles:pp -g).

Failure Modes and Common Misunderstandings

“Precise” only fixes the IP, not the stack. PEBS/IBS make the instruction pointer exact; they do nothing for call-stack accuracy, which still depends on frame pointers, DWARF, or LBR (Flame Graphs and Stack Sampling). A flame graph with broken stacks is broken whether or not you used :pp.

Asking for a level the hardware can’t give. perf_event_open returns -EOPNOTSUPP if you request a precise level the event or CPU doesn’t support — e.g. :ppp on AMD (IBS caps at 2) or a precise modifier on an event that has no PEBS counterpart. The IBS validation if (!event->attr.precise_ip || event->attr.precise_ip > 2) return -EOPNOTSUPP; is the literal guard (ibs.c). Using :P (auto-detect) sidesteps this.

Not every event is PEBS-capable. Only specific at-retirement events have PEBS support; on Sandy Bridge “only 7 events support PEBS” (easyperf). Adding :pp to an arbitrary raw event will fail. perf list annotates which events are precise-capable.

Sample shadowing / blind spots. Because PEBS arms on the next event after a counter reset, instructions immediately after the reset point are slightly under-sampled — a systematic blind spot. Level 3’s “uses randomization to avoid sample shadowing effects” (perf-list(1)) jitters the reset value to spread samples and remove the shadow, which is why level 3 exists as more than just “level 2 but stricter.”

PMI-storm vs PEBS buffering. A hidden benefit, not just accuracy: because the hardware writes records straight to the DS buffer and only interrupts when the buffer is near full, PEBS slashes interrupt overhead — “the Linux kernel is only involved when the PEBS buffer fills up … there is no interrupt until a lot of samples are available” (easyperf). Mapping that DS buffer into the per-CPU entry area requires a cross-CPU TLB shootdown, which ds.c notes explicitly (“we must shoot down all TLB entries for it”) — a real but one-time setup cost.

Uncertain

Verify: the claim that precise level 3 is Intel-only and unavailable on AMD IBS. Reason: perf-list(1) (fetched 2026-06-20) states Intel supports “precise level 3 for some special cases” and AMD “up to precise-level 2,” and ibs.c rejects precise_ip > 2, which together strongly imply level 3 is Intel-only — but the man page does not say so in exactly those words. To resolve: this is well-corroborated by the ibs.c guard; treat as high-confidence. uncertain

Alternatives and When to Choose Them

Plain (skidded) sampling is the right choice for coarse, function-level CPU profiling where dozens of instructions of skid wash out across a function body — and it works on events and CPUs with no PEBS/IBS support. LBR-based sampling is the third precise-ish tool: rather than a precise IP, LBR gives a precise branch history, which perf can use both to reconstruct call stacks without frame pointers and to pin down which branch a sample’s basic block came from (Flame Graphs and Stack Sampling covers LBR call-graph mode). PEBS/IBS and LBR are complementary — Intel’s intel_pmu_pebs_fixup_ip() literally uses LBR to correct pre-Haswell PEBS IPs (ds.c). perf mem and perf c2c are higher-level front-ends built entirely on precise memory sampling: c2c (“cache-to-cache”) uses PEBS/IBS data-address and data-source records to find false-sharing hotspots, which is impossible without precise data-address capture. Reach for precise sampling specifically when you need instruction-level or memory-access truth; reach for plain sampling when you only need to know which function is hot.

Production Notes

Two practices follow from the mechanism. First, default to :pp for any drill-down: the cost over plain sampling is negligible and the attribution is dramatically better, so experienced practitioners reach for cycles:pp reflexively when they care about source-line accuracy. Second, precise memory profiling is the killer app: diagnosing NUMA-remote accesses, false sharing across cores, and which specific load is causing LLC misses all depend on PEBS’s data-address/data-source/latency fields (or their IBS equivalents). Tools like perf c2c turned previously near-undiagnosable cache-contention bugs into routine investigations precisely because the hardware now records the data address of the offending access. The portability wrinkle to remember in production: a profiling recipe that works on Intel with :ppp may need :pp on AMD, and :P (auto-detect) is the safe cross-vendor default in scripts that must run on both.

See Also

Graph View

Backlinks