The Performance Monitoring Unit

The Performance Monitoring Unit (PMU) is the block of CPU hardware that makes performance analysis possible at all: a small bank of dedicated registers — performance-monitoring counters (PMCs) — that tally micro-architectural events the rest of the machine can’t see, such as CPU cycles, retired instructions, last-level-cache misses, and branch mispredictions (Gregg, perf Examples). Software can program a counter to count a chosen event, read its running value, or arm it to fire an interrupt when it overflows — which is precisely how a sampling profiler captures where the cycles and the cache misses are going. The PMU is a fixed, scarce hardware resource: a typical Intel core exposes a handful of fixed-function counters plus four or so general-purpose ones, so when you ask for more events than there are physical counters, the kernel must multiplex them in time and scale the results (Gregg). Linux drives this hardware through per-architecture PMU drivers under arch/<arch>/events/; on x86 that is arch/x86/events/core.c plus a vendor file like arch/x86/events/intel/core.c (v6.12 source).

This is a §5 leaf of the Linux Tracing and Observability MOC. The PMU is the hardware behind a PERF_TYPE_HARDWARE event; the syscall that programs it is The perf_event_open Syscall; the distinction between PMU-backed and kernel-software events is Hardware vs Software Performance Events; the count-vs-overflow duality is Counting vs Sampling Mode; and the most precise way to attribute an overflow to an instruction is Precise Event-Based Sampling.

Mental Model — A Bank of Programmable Tally Registers

Picture the PMU as a small set of hardware counters wired directly into the core’s pipeline, each able to be told “increment yourself every time this event happens.” A counter has two halves: a control register that selects the event and toggles counting (on x86, an IA32_PERFEVTSEL model-specific register), and a count register that holds the running tally (IA32_PMC). You count by programming the control register; you read by reading the count register; and you sample by pre-loading the count register with a value chosen so it overflows after exactly N events, at which point the hardware raises an interrupt and the kernel records a sample.

flowchart LR
  subgraph CORE["CPU core pipeline"]
    EV["micro-arch events:<br/>cycles, insns retired,<br/>cache misses, branch mispredicts"]
  end
  subgraph PMU["Performance Monitoring Unit"]
    SEL["IA32_PERFEVTSEL0..N<br/>(control: pick event,<br/>USR/OS, INT, ENABLE)"]
    CNT["IA32_PMC0..N<br/>(general-purpose counters)"]
    FIX["fixed counters<br/>(cycles / insns / ref-cycles)"]
    GCTRL["GLOBAL_CTRL<br/>(enable all at once)"]
  end
  EV --> CNT
  EV --> FIX
  SEL -->|programs| CNT
  GCTRL -->|gates| CNT
  GCTRL -->|gates| FIX
  CNT -->|overflow| PMI["overflow -> NMI/PMI<br/>-> perf_event_overflow()<br/>-> a sample"]
  FIX -->|overflow| PMI

The PMU as Linux sees it on x86. What it shows: micro-architectural events feed both general-purpose counters (steered by per-counter PERFEVTSEL control registers) and fixed-function counters (hardwired to common events); a global control register arms them all together; and when a counter overflows it raises a performance-monitoring interrupt that the kernel turns into a sample. The insight: counting and sampling are the same hardware — sampling is just counting with the overflow line wired to an interrupt and the counter pre-loaded to overflow at the chosen period.

Why You Need the PMU — Events Software Cannot See

A counter like PERF_COUNT_SW_PAGE_FAULTS is maintained by the kernel in software: the page-fault handler increments a variable. But there is no software path that runs on every cache miss or every branch misprediction — those happen deep inside the silicon, billions of times a second, with no trap to the kernel. The only way to count them is for the hardware itself to keep the tally. That is the PMU’s reason to exist, and it is why metrics like IPC (instructions per cycle) are impossible without it: IPC is instructions_retired / cpu_cycles, and both of those are PMU counters. As Brendan Gregg puts it, hardware events “instrument low-level processor activity, for example, CPU cycles, instructions retired, memory stall cycles, level 2 cache misses, etc.” — and IPC “requires simultaneous measurement of two distinct PMU counters” (Gregg). The same goes for cache-miss-rate (cache-misses / cache-references) and branch-misprediction-rate (branch-misses / branch-instructions): each is a ratio of two PMU counters, and each is the first thing you reach for when a workload is mysteriously slow despite low apparent CPU usage — a high stall rate or miss rate points straight at the micro-architectural bottleneck the CPU-time number alone hides.

Fixed-Function versus General-Purpose Counters

The PMU’s counters come in two flavours, and the difference shapes everything the kernel does with them.

General-purpose counters are fully programmable: each has its own event-select register and can count any event the chip supports. On x86 these are the IA32_PMC0..N count registers paired with IA32_PERFEVTSEL0..N control registers, at MSR addresses MSR_ARCH_PERFMON_PERFCTR0 = 0xc1 and MSR_ARCH_PERFMON_EVENTSEL0 = 0x186 respectively (v6.12 arch/x86/include/asm/perf_event.h). A modern Intel core typically exposes four general-purpose counters per logical CPU (and historically only three were available when hyper-threading split the bank between two threads, though contemporary parts widen this) (v6.12 arch/x86/events/intel/core.c).

Fixed-function counters trade flexibility for not consuming a scarce general-purpose slot: each is hardwired to one specific, extremely common event, so measuring cycles and instructions — which almost everyone always wants — doesn’t burn two of your four programmable counters. Intel defines three classic fixed counters (v6.12 intel/core.c):

• Fixed counter 0 → INST_RETIRED.ANY (event code 0x00c0), mapping PERF_COUNT_HW_INSTRUCTIONS.
• Fixed counter 1 → CPU_CLK_UNHALTED.CORE (0x003c), mapping PERF_COUNT_HW_CPU_CYCLES.
• Fixed counter 2 → CPU_CLK_UNHALTED.REF (0x0300), mapping PERF_COUNT_HW_REF_CPU_CYCLES.

Newer micro-architectures add a fourth fixed counter for TOPDOWN.SLOTS (PERF_COUNT_HW_SLOTS, code 0x0400), the basis of the “top-down” pipeline-utilisation methodology (v6.12 intel/core.c). Fixed counters are programmed differently from general-purpose ones: instead of a per-counter event-select MSR, they share a single control register, MSR_CORE_PERF_FIXED_CTR_CTRL (0x38d), with a 4-bit field per counter; the count registers are MSR_CORE_PERF_FIXED_CTR0..2 at 0x309, 0x30a, 0x30b (v6.12 arch/x86/include/asm/msr-index.h). The per-counter control bits are INTEL_FIXED_0_KERNEL (bit 0), INTEL_FIXED_0_USER (bit 1), INTEL_FIXED_0_ANYTHREAD (bit 2), and INTEL_FIXED_0_ENABLE_PMI (bit 3) (v6.12 perf_event.h).

Uncertain

Verify: the exact fixed/general-purpose counter counts on a specific modern part (e.g. “4 fixed + 8 general-purpose on Golden Cove” or the hybrid P-core/E-core asymmetry). Reason: the v6.12 Intel driver gates counter counts on PMU version and the CPUID 0x0A leaf at runtime rather than hard-coding them, and the numbers differ per micro-architecture and with hyper-threading; the figures above (“typically four general-purpose, three fixed”) are the common case, not a universal constant. To resolve: read dmesg | grep -i pmu or /proc/cpuinfo, or decode CPUID leaf 0x0A (union cpuid10_eax: version_id:8; num_counters:8; bit_width:8) on the target box. uncertain

How the kernel discovers the counters — CPUID leaf 0x0A

The kernel does not hard-code counter counts; it asks the CPU. Intel’s architectural performance monitoring is reported through CPUID leaf 0x0A, whose EAX register the kernel decodes via union cpuid10_eax with bitfields version_id:8; num_counters:8; bit_width:8; mask_length:8 (v6.12 perf_event.h). The version_id (PMU “version”) gates feature availability — version ≥ 2 is when MSR_CORE_PERF_GLOBAL_CTRL and fixed counters become usable (v6.12 intel/core.c) — num_counters is how many general-purpose counters exist, and bit_width is the counter width (commonly 48 bits) used by the kernel as cntval_bits/cntval_mask. This is why the same kernel binary correctly drives PMUs from a dozen CPU generations: it reads the capabilities at boot.

The MSRs — Programming and Reading a Counter

The control register IA32_PERFEVTSEL packs the event selection and the counting mode into one 64-bit word. The v6.12 header lays out the bits (v6.12 perf_event.h):

#define ARCH_PERFMON_EVENTSEL_EVENT   0x000000FFULL  /* event-select code   */
#define ARCH_PERFMON_EVENTSEL_UMASK   0x0000FF00ULL  /* unit mask qualifier */
#define ARCH_PERFMON_EVENTSEL_USR     (1ULL << 16)   /* count user (ring 3) */
#define ARCH_PERFMON_EVENTSEL_OS      (1ULL << 17)   /* count kernel (ring 0)*/
#define ARCH_PERFMON_EVENTSEL_EDGE    (1ULL << 18)
#define ARCH_PERFMON_EVENTSEL_INT     (1ULL << 20)   /* PMI on overflow     */
#define ARCH_PERFMON_EVENTSEL_ENABLE  (1ULL << 22)   /* start this counter  */
#define ARCH_PERFMON_EVENTSEL_INV     (1ULL << 23)
#define ARCH_PERFMON_EVENTSEL_CMASK   0xFF000000ULL  /* counter-mask threshold */

The low two bytes (EVENT and UMASK) name the micro-architectural event — e.g. an L1-D load miss has its own (event, umask) pair. The USR and OS bits implement the exclude_user/exclude_kernel attributes from The perf_event_open Syscall: the kernel sets ARCH_PERFMON_EVENTSEL_USR unless exclude_user is requested, and ARCH_PERFMON_EVENTSEL_OS unless exclude_kernel is (v6.12 arch/x86/events/core.c):

event->hw.config = ARCH_PERFMON_EVENTSEL_INT;
if (!event->attr.exclude_user)
    event->hw.config |= ARCH_PERFMON_EVENTSEL_USR;
if (!event->attr.exclude_kernel)
    event->hw.config |= ARCH_PERFMON_EVENTSEL_OS;

The INT bit is the one that turns a counting event into a sampling event — it tells the PMU to raise an interrupt on overflow. The driver reads and writes these MSRs through wrmsrl()/rdmsrl_safe() (and reads counts fast via the RDPMC instruction, rdpmcl()), with x86_pmu_config_addr(idx) and x86_pmu_event_addr(idx) returning the PERFEVTSEL and PMC MSR addresses for counter idx (v6.12 core.c).

A global control register, MSR_CORE_PERF_GLOBAL_CTRL (0x38f), can enable or disable all counters in one write — essential so that a group of counters starts and stops atomically and stays coherent. Its companions are MSR_CORE_PERF_GLOBAL_STATUS (0x38e), whose bits say which counter overflowed, and MSR_CORE_PERF_GLOBAL_OVF_CTRL (0x390), which acknowledges an overflow (v6.12 msr-index.h; v6.12 intel/core.c).

Counter Multiplexing — Time-Sharing a Scarce Resource

Because there are only a handful of physical counters, the central tension of PMU use is over-subscription. Brendan Gregg states it plainly: PMCs “are a fixed hardware resource on the processor (a limited number of registers),” and “only a few or several can be recorded at the same time, from the many thousands that are available” (Gregg). When perf is asked to count more events than there are counter slots, the kernel multiplexes: it runs a subset for a slice of time, rotates to the next subset, and so on, then scales each event’s partial count up to an estimate of what it would have been had it run the whole time.

The rotation is driven by a per-context high-resolution timer: perf_mux_hrtimer_handler() fires and calls perf_rotate_context(), which schedules one group out (ctx_sched_out()) and the next in (ctx_sched_in()) (v6.12 kernel/events/core.c). To make the scaling honest, the kernel tracks two times per event: time_enabled (total wall-time the event was enabled) and time_running (the subset of that during which it was actually on a physical counter). The accounting is in __perf_update_times() (v6.12 core.c):

*enabled = event->total_time_enabled;
if (state >= PERF_EVENT_STATE_INACTIVE)
    *enabled += delta;
*running = event->total_time_running;
if (state >= PERF_EVENT_STATE_ACTIVE)
    *running += delta;

When the event was time-shared, time_running < time_enabled, and userspace recovers an estimate by scaling: final_count ≈ raw_count × (time_enabled / time_running). The man page describes exactly this: the two times “can be used to scale an estimated value for the count” (man page). perf stat surfaces the multiplexing fraction directly — an annotation like [ 75.00% ] next to a number means “this counter was actually on the hardware only 75% of the time, and the value you see has been scaled up accordingly” (Gregg). The catch: scaling assumes the event rate was uniform across the measurement; for a bursty workload, the slice you missed may have been the interesting one, so multiplexed numbers carry sampling error. The way to avoid multiplexing entirely is to put related counters in one event group small enough to fit, so they are always on the hardware together — which is why coherent IPC requires grouping, not just two independent counters.

PMU Interrupts — From Overflow to Sample

Sampling is the PMU’s second mode, and it hinges on counter overflow. The kernel doesn’t wait for a counter to climb to its natural maximum; it pre-loads the counter with a negative starting value so it overflows after exactly the desired number of events. x86_perf_event_set_period() does this (v6.12 arch/x86/events/core.c):

local64_set(&hwc->prev_count, (u64)-left);
wrmsrl(hwc->event_base, (u64)(-left) & x86_pmu.cntval_mask);

Here left is the remaining period; the counter is loaded with -left (masked to the counter width), so after left more events it wraps through zero and trips the overflow bit. The hardware then raises a PMI (performance-monitoring interrupt), delivered as an NMI (non-maskable interrupt) so it can fire even in code that has disabled normal interrupts — important, because much of what you want to profile (locks held, IRQs off) runs with interrupts disabled. The NMI lands in perf_event_nmi_handler(), which dispatches to the PMU driver’s x86_pmu_handle_irq() (v6.12 core.c):

for_each_set_bit(idx, x86_pmu.cntr_mask, X86_PMC_IDX_MAX) {
    val = static_call(x86_pmu_update)(event);
    if (val & (1ULL << (x86_pmu.cntval_bits - 1)))
        continue;                  /* sign bit clear -> this one didn't overflow */
    handled++;
    if (!static_call(x86_pmu_set_period)(event))
        continue;
    perf_sample_data_init(&data, 0, event->hw.last_period);
    if (perf_event_overflow(event, &data, regs))   /* -> records a sample      */
        x86_pmu_stop(event, 0);
}

The handler walks every active counter, uses the sign bit of the counter value to detect which one overflowed (a counter pre-loaded negative is “still negative” until it crosses zero), re-arms the period, and calls perf_event_overflow(). On Intel, intel_pmu_handle_irq() first reads MSR_CORE_PERF_GLOBAL_STATUS to learn which counters overflowed before iterating (v6.12 intel/core.c). perf_event_overflow() is the generic core path: it captures the interrupted register state (pt_regs, hence the instruction pointer of whatever was running), assembles a PERF_RECORD_SAMPLE, and writes it to the event’s mmap ring buffer — or runs an attached BPF program (v6.12 kernel/events/core.c). To protect the system from a runaway sampling rate, the core enforces a throttle: when an event hits MAX_INTERRUPTS in a tick it is logged via perf_log_throttle() and quieted (v6.12 core.c).

One important imprecision lives here: by the time the NMI is taken and the kernel reads the interrupted IP, the pipeline has often moved on — the recorded IP “skids” past the instruction that actually caused the event. Eliminating that skid is the job of Precise Event-Based Sampling (PEBS on Intel), where the hardware itself stamps a precise record into a debug-store buffer at the time of the event; the v6.12 Intel driver maintains exactly such a struct debug_store *ds per CPU and drains it via x86_pmu.drain_pebs() (v6.12 intel/core.c).

The Per-Architecture Driver Abstraction

Linux confines the chip-specific knowledge to a driver structure, struct x86_pmu, holding function pointers (handle_irq, enable, disable, enable_all, disable_all) and capability fields (cntr_mask64 for the general-purpose counters, fixed_cntr_mask for fixed, cntval_bits/cntval_mask for width, event_map() to translate the portable PERF_COUNT_HW_* ids into the chip’s real event codes) (v6.12 arch/x86/events/core.c). The generic arch/x86/events/core.c implements the scheduling, period, and IRQ machinery against these pointers, while arch/x86/events/intel/core.c, .../amd/, and others fill in the vendor specifics — and entirely separate trees, arch/arm64/kernel/perf_event.c and friends, provide the ARM PMU driver. This is why the same perf_event_open ABI and the same perf tool work across vendors: the portability seam is the (type, config) encoding from The perf_event_open Syscall, and below it each struct x86_pmu-like driver translates to MSRs (or ARM system registers) for its silicon.

The general-purpose-counter scarcity also shows up as a scheduling problem inside the driver. Because some events can only run on certain counters (constraints), x86_schedule_events() runs a constraint solver (perf_assign_events() with a perf_sched state machine) to assign each requested event to a physical counter, and refuses the assignment — bubbling up an error or forcing multiplexing — when it can’t fit them all; hyper-threading further halves the available pool via the is_ht_workaround_enabled() path (v6.12 core.c).

Failure Modes and Common Misunderstandings

“Why does adding events change my IPC?” Because each new event competes for the same counters, pushing you into multiplexing; once time_running < time_enabled, every value is a scaled estimate. The fix is to measure fewer events at once, or group the related ones so they stay co-resident.

Empty or wrong counts inside a VM. The PMU is physical hardware; a guest only sees it if the hypervisor virtualises or passes through the vPMU. Without it, PERF_TYPE_HARDWARE events return zero or ENOENT, while PERF_TYPE_SOFTWARE events still work — a frequent source of “perf works on the host but not the guest” confusion, and the practical reason cloud profilers lean on software events plus whatever vPMU the provider exposes (Hardware vs Software Performance Events).

Skid surprises. A non-precise sample’s IP can land several instructions past the true culprit, so a hot line in a flame graph may be the instruction after the expensive one. This is inherent to NMI-based sampling; use precise_ip / PEBS for instruction-accurate attribution (Precise Event-Based Sampling).

Counter aliasing across siblings. With hyper-threading, two logical CPUs share one physical PMU; certain events behave per-core, not per-thread, so attributing them to a single hyper-thread can be misleading. The driver’s HT workarounds exist precisely because of this sharing.

Production Notes

The PMU is the backbone of every credible CPU-performance investigation. perf stat on a workload that shows low CPU% but is slow will often reveal a high cache-misses or stalled-cycles-frontend rate — the PMU exposing a memory or front-end bottleneck the scheduler’s CPU-time accounting cannot. Top-down micro-architecture analysis (Intel’s TOPDOWN.SLOTS fixed counter, surfaced as perf stat --topdown / the toplev tool) decomposes every pipeline slot into retiring / bad-speculation / front-end-bound / back-end-bound, turning the PMU’s raw counters into an actionable bottleneck classification. Continuous fleet profilers keep PMU sampling at low frequencies (tens of Hz) to bound the PMI overhead, and they care about the throttle path: a misconfigured high-frequency event triggers perf_log_throttle and silently drops samples. The recurring operational reality is counter scarcity — teams that try to collect a dozen events in one perf stat get heavily-multiplexed, error-prone numbers, and the discipline is to measure a focused, fitting set, validating that the multiplexing fraction stays at or near 100%.

See Also

• The perf_event_open Syscall — the syscall that programs the PMU; PERF_TYPE_HARDWARE config maps to event-select MSRs
• Hardware vs Software Performance Events — PMU-backed events vs kernel-software events (and why VMs differ)
• Counting vs Sampling Mode — counting (read the PMC) vs sampling (overflow → PMI → sample), and the multiplexing/scaling math
• Precise Event-Based Sampling — PEBS: eliminating the NMI skid for instruction-accurate samples
• perf Profiling Tool — the userspace tool that drives PMU counters via perf stat/record
• Flame Graphs and Stack Sampling — what PMU overflow samples become after stack unwinding
• Linux Tracing and Observability MOC — the parent map (§5, perf and the PMU)