vault

Per-CPU Variables

15 min read

Per-CPU Variables

A per-CPU variable is a variable for which the kernel keeps a separate, private copy for every processor, so that the common code path on a given CPU touches only that CPU’s copy and never a cache line shared with another CPU. This is the kernel’s purest share-nothing scaling tool: where a global counter forces every CPU that increments it to fight over one cache line — bouncing it through the cache-coherence protocol and serializing on a lock — a per-CPU counter lets each CPU bump its own copy with no lock, no atomic, and no contention. The mechanism rests on a single trick: each variable is laid out once in a special .data..percpu linker section, and the address of CPU n’s copy is computed at runtime by adding a per-CPU offset to that base address (__per_cpu_offset[n]). Accessors come in two families: per_cpu_ptr(ptr, cpu) reaches any CPU’s copy by offset, while this_cpu_ptr(ptr) reaches the current CPU’s copy using a cached offset — on x86 a segment-register prefix that makes the access a single instruction with no explicit address arithmetic (percpu-defs.h v6.12; asm-generic/percpu.h v6.12). The price is paid on the read-the-whole-thing path: summing a per-CPU counter means walking every CPU’s copy.

This note owns the generic per-CPU mechanism. The specific kernel subsystems built on it — the scheduler’s Per-CPU Run Queues and struct rq, the page allocator’s Per-CPU Page Lists, and BPF’s Per-CPU Maps — are covered in their own notes and cross-linked rather than re-derived here. The locking primitive that protects per-CPU state against preemption and interrupts on the same CPU lives in local_lock and this_cpu Operations.

Mental Model: One Slab, Many Offsets

The central idea is that a per-CPU variable is not an array of NR_CPUS separate objects scattered across memory. It is a single template laid down once in a dedicated section of the kernel image, and the kernel allocates one contiguous replica of that whole section for each online CPU. To find a particular CPU’s instance of a particular variable, you take the variable’s address within the template and add a fixed, per-CPU base offset.

flowchart LR
  subgraph TPL[".data..percpu template (in kernel image)"]
    V1["var A<br/>(at template offset 0x10)"]
    V2["var B<br/>(at template offset 0x40)"]
  end
  TPL -->|"copied at boot"| C0["CPU0 replica<br/>base = __per_cpu_offset[0]"]
  TPL -->|"copied at boot"| C1["CPU1 replica<br/>base = __per_cpu_offset[1]"]
  TPL -->|"copied at boot"| C2["CPU2 replica<br/>base = __per_cpu_offset[2]"]
  C0 -->|"A on CPU0 = base0 + 0x10"| A0(("addr"))
  C1 -->|"A on CPU1 = base1 + 0x10"| A1(("addr"))

The per-CPU layout. What it shows: every variable has one offset within the template section, fixed at link time. Each CPU gets a private copy of the whole section; the start of CPU n’s copy is recorded in __per_cpu_offset[n]. The address of variable A on CPU n is simply __per_cpu_offset[n] + offset_of(A). The insight to take: the “variable” you declare is really just a symbolic offset; the real storage is one big block per CPU, so accessing the current CPU’s copy is one add (or, on x86, a segment-prefixed instruction that does the add implicitly). No pointer chasing, no array indexing, no shared cache line.

This layout is why per-CPU access is cheap. A naive “array of pointers” design — which is what the kernel used before 2007 — required two memory loads (load the pointer for this CPU, then dereference it) and wasted space when NR_CPUS was set far above the real processor count, with multiple CPUs’ pointers sharing one cache line and bouncing it (Corbet, Better per-CPU variables, LWN 2007). The single-block design replaces the double indirection with one offset add and gives each CPU its own cache-isolated region.

Mechanical Walk-through

Declaration and definition

A static per-CPU variable is declared and defined with macros that place it in the per-CPU section. From percpu-defs.h:

#define DECLARE_PER_CPU(type, name)  DECLARE_PER_CPU_SECTION(type, name, "")
#define DEFINE_PER_CPU(type, name)   DEFINE_PER_CPU_SECTION(type, name, "")

DEFINE_PER_CPU(int, counter) does not allocate NR_CPUS integers. It places a single int counter symbol into the .data..percpu section. The linker collects all such symbols into one section; that section is the template the kernel replicates per CPU at boot. Because the symbol lives in a special section, you cannot read or write it directly as counter — the address you get is a template offset, not a usable pointer. The accessor macros exist precisely to turn that offset into a real address (percpu-defs.h v6.12).

A common variant aligns the variable to its own cache line so that one CPU’s hot writes never falsely-share a line with another CPU’s data:

#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)                    \
    DEFINE_PER_CPU_SECTION(type, name, PER_CPU_SHARED_ALIGNED_SECTION) \
    ____cacheline_aligned_in_smp

____cacheline_aligned_in_smp forces the object onto a fresh cache line under SMP. This matters when a per-CPU object is occasionally read by other CPUs (e.g. statistics aggregation): without alignment, a remote read of CPU 0’s copy could pull in CPU 1’s copy on the same line and cause that line to ping-pong.

The offset mechanism

The runtime translation from template offset to real address is the heart of the system. The generic implementation in asm-generic/percpu.h defines:

extern unsigned long __per_cpu_offset[NR_CPUS];
#define per_cpu_offset(x) (__per_cpu_offset[x])

with the comment “per_cpu_offset() is the offset that has to be added to a percpu variable to get to the instance for a certain processor.” So per_cpu_ptr(&counter, 3) computes &counter + __per_cpu_offset[3] and yields a pointer to CPU 3’s counter. The macro from percpu-defs.h:

#define per_cpu_ptr(ptr, cpu)                  \
({                                             \
    __verify_pcpu_ptr(ptr);                    \
    SHIFT_PERCPU_PTR((ptr), per_cpu_offset((cpu))); \
})

__verify_pcpu_ptr is a compile-time-only check that ptr was declared __percpu; it never touches memory. SHIFT_PERCPU_PTR does the actual add:

/* Add an offset to a pointer but keep the pointer as-is.  Use RELOC_HIDE()
 * to prevent the compiler from making incorrect assumptions about the
 * pointer value. */
#define SHIFT_PERCPU_PTR(__p, __offset)        \
    RELOC_HIDE((typeof(*(__p)) __kernel __force *)(__p), (__offset))

RELOC_HIDE is an arithmetic add wrapped to defeat the compiler’s pointer-provenance reasoning: because the __percpu symbol’s apparent address (its template offset) bears no relation to the real per-CPU address, the compiler must be stopped from “optimizing” based on the symbol’s nominal value (percpu-defs.h v6.12).

For the current CPU, the kernel avoids even looking up smp_processor_id() in the general case. __my_cpu_offset is the offset for the running CPU:

#define __my_cpu_offset per_cpu_offset(raw_smp_processor_id())
#define arch_raw_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset)

An architecture may override __my_cpu_offset with something faster. x86 does exactly this: it keeps the current CPU’s offset in a per-CPU variable this_cpu_off reachable through a segment register, so __my_cpu_offset is this_cpu_read(this_cpu_off) and individual accesses fold into a single instruction. From arch/x86/include/asm/percpu.h:

#ifdef CONFIG_X86_64
# define __percpu_seg  gs
#else
# define __percpu_seg  fs
#endif

On x86-64 the %gs segment base points at the start of the current CPU’s per-CPU block. An access like this_cpu_read(this_cpu_off) becomes a mov with a %%gs: prefix — the CPU adds the segment base to the template offset in hardware, so no explicit add instruction is emitted at all (arch/x86/include/asm/percpu.h v6.12). This is why per-CPU access on x86 is genuinely free relative to a normal global load.

this_cpu_ptr versus raw_cpu_ptr

There are two ways to get the current CPU’s pointer, and the difference is debug-checking, not behaviour:

#define this_cpu_ptr(ptr)                      \
({                                             \
    __verify_pcpu_ptr(ptr);                    \
    SHIFT_PERCPU_PTR(ptr, my_cpu_offset);      \
})
 
#define raw_cpu_ptr(ptr)                       \
({                                             \
    __verify_pcpu_ptr(ptr);                    \
    arch_raw_cpu_ptr(ptr);                     \
})

The split lives in my_cpu_offset:

#ifdef CONFIG_DEBUG_PREEMPT
#define my_cpu_offset per_cpu_offset(smp_processor_id())
#else
#define my_cpu_offset __my_cpu_offset
#endif

When CONFIG_DEBUG_PREEMPT is on, this_cpu_ptr resolves the CPU number through smp_processor_id(), which warns if called with preemption enabled — catching the bug of computing a “current CPU” pointer that could become stale the instant the task migrates. raw_cpu_ptr skips that check. So: use this_cpu_ptr by default (it will tell you if you forgot to disable preemption); use raw_cpu_ptr only when you know preemption is already disabled or migration is genuinely harmless (asm-generic/percpu.h v6.12).

get_cpu_var / put_cpu_var: bundling preemption disabling

Computing the current CPU’s pointer is only safe while the task stays on that CPU. If the kernel preempts the task and reschedules it on a different CPU between the offset lookup and the memory access, the pointer now refers to the wrong CPU’s copy. get_cpu_var makes the safe pattern a one-liner by disabling preemption around the access:

#define get_cpu_var(var)        \
(*({                            \
    preempt_disable();          \
    this_cpu_ptr(&var);         \
}))
 
#define put_cpu_var(var)        \
do {                            \
    (void)&(var);               \
    preempt_enable();           \
} while (0)

So the canonical use is:

int *p = &get_cpu_var(counter);   /* preempt_disable() + this_cpu_ptr(&counter) */
(*p)++;
put_cpu_var(counter);             /* preempt_enable() */

get_cpu_var returns an lvalue (the dereferenced pointer) with preemption already disabled; put_cpu_var re-enables it. The matching pointer-flavoured pair get_cpu_ptr(var) / put_cpu_ptr(var) does the same for a dynamically allocated per-CPU pointer. The (void)-cast of the argument in put_cpu_var exists only to evaluate var for the compiler without using its value (percpu-defs.h v6.12). The deeper point: preemption disabling here is load-bearing locking of per-CPU state — a contract made explicit and PREEMPT_RT-safe by local_lock.

Dynamic allocation

Static DEFINE_PER_CPU is for variables known at compile time. For per-CPU storage attached to a runtime object (e.g. one per network device, one per cgroup), the kernel has a dedicated per-CPU allocator. From percpu.h:

#define alloc_percpu(type)                                    \
    (typeof(type) __percpu *)__alloc_percpu(sizeof(type),     \
                                            __alignof__(type))
extern void free_percpu(void __percpu *__pdata);

alloc_percpu(struct mystat) returns a struct mystat __percpu * — a single pointer that, fed to per_cpu_ptr(p, cpu) or this_cpu_ptr(p), yields each CPU’s copy via the same offset mechanism as static variables. Under the hood it calls pcpu_alloc_noprof(size, align, reserved, gfp). There is a _gfp variant (alloc_percpu_gfp) to control allocation flags (e.g. GFP_ATOMIC in atomic context). The per-CPU allocator is special enough that it must work before the slab allocator exists: the header notes “Percpu allocator can serve percpu allocations before slab is initialized which allows slab to depend on the percpu allocator.” The first chunk reserves space — PERCPU_DYNAMIC_RESERVE — so early-boot allocations succeed before the full machinery is up (percpu.h v6.12). Always pair alloc_percpu with free_percpu; the returned __percpu pointer is not an ordinary kernel pointer and must not be kfree’d.

Reading the whole counter: the cost side

The asymmetry that defines per-CPU data: writes are local and cheap; a global read is expensive. A per-CPU counter has no single authoritative value — the logical value is the sum across all CPUs. To read it you must walk every CPU:

unsigned long total = 0;
int cpu;
for_each_possible_cpu(cpu)
    total += *per_cpu_ptr(my_counter, cpu);

This loop reaches into every CPU’s cache line (each per_cpu_ptr is a remote access), so a frequent global read can itself become a scalability problem — and worse, the snapshot is never perfectly consistent, because CPUs you have already summed may keep incrementing while you read the later ones. This is the fundamental trade-off: per-CPU variables make the hot path (per-CPU writes) free at the cost of making the rare path (global aggregation) O(number of CPUs) and only approximately consistent. It is the right trade exactly when writes vastly outnumber whole-counter reads — which is why per-CPU counters back statistics, and why the kernel’s percpu_counter type adds an approximate global cache on top to bound read cost.

Worked Example: a Per-CPU Statistics Counter

#include <linux/percpu.h>
#include <linux/cpumask.h>
 
/* 1. One copy of `events` per CPU, in .data..percpu */
static DEFINE_PER_CPU(unsigned long, events);
 
/* 2. Hot path: bump the *current* CPU's copy, no lock, no atomic */
void note_event(void)
{
    /* this_cpu_inc disables-nothing on x86 — single inc gs:[events] */
    this_cpu_inc(events);
}
 
/* 3. Rare path: sum across all CPUs for /proc output */
unsigned long total_events(void)
{
    unsigned long total = 0;
    int cpu;
 
    for_each_possible_cpu(cpu)
        total += per_cpu(events, cpu);   /* per_cpu(v,c) == *per_cpu_ptr(&v,c) */
    return total;
}

Line by line: DEFINE_PER_CPU(unsigned long, events) reserves one events template slot. this_cpu_inc(events) increments the running CPU’s copy as a single instruction (covered in local_lock and this_cpu Operations); because only this CPU touches this copy on the hot path, no LOCK prefix and no memory barrier are needed. per_cpu(events, cpu) expands to *per_cpu_ptr(&events, cpu) — it dereferences a specific CPU’s copy, which is exactly the remote read the aggregation loop needs. for_each_possible_cpu iterates every CPU the kernel could ever bring online (using possible rather than online avoids losing counts that were accumulated on a CPU before it was hot-unplugged).

Failure Modes and Common Misunderstandings

Accessing the current CPU’s copy with preemption enabled. The classic bug: int *p = this_cpu_ptr(&v); ... preemptible work ... (*p)++;. If the task is preempted and migrates between computing p and using it, p points at the old CPU’s copy and two CPUs may now race on it. The fix is to either bracket the access with preempt_disable()/preempt_enable() (or get_cpu_var/put_cpu_var), use a this_cpu_* op that does the whole read-modify-write in one preemption-safe step, or hold a local_lock. With CONFIG_DEBUG_PREEMPT, this_cpu_ptr via smp_processor_id() warns about this; raw_cpu_ptr will not.

Assuming per-CPU means atomic. A per-CPU variable is not automatically protected against an interrupt handler on the same CPU that also touches it. A plain v++ on a per-CPU variable is a non-atomic read-modify-write; if a hardirq fires mid-instruction and the handler also increments v, an update can be lost. Use the this_cpu_* operations (which are single-instruction and thus atomic against same-CPU interrupts on x86) or disable interrupts. Cross-CPU atomicity is never provided — see local_lock and this_cpu Operations.

Remote writes. Writing another CPU’s copy (per_cpu_ptr(p, other_cpu) then store) is legal but defeats the entire purpose: it forces the owning CPU’s cache line out, and if that line is also touched by the owner via this_cpu_* it reintroduces exactly the cache-line bouncing and atomicity hazards per-CPU data was meant to eliminate. The kernel’s guidance is to prefer an IPI (inter-processor interrupt) that asks the owning CPU to update its own copy (this_cpu_ops.rst v6.12).

free_percpu on a static variable, or kfree on a __percpu pointer. free_percpu is only for pointers returned by alloc_percpu; static DEFINE_PER_CPU storage lives for the lifetime of the kernel/module and is never freed. Conversely an alloc_percpu result is a special offset token, not a kernel virtual address, so passing it to kfree corrupts the heap.

Uncertain

Verify: the exact early-boot ordering claim that the per-CPU allocator’s first chunk is usable before mm_init/slab, and the precise value/meaning of PERCPU_DYNAMIC_RESERVE. Reason: stated from the percpu.h header comment, not from mm/percpu.c allocator source or Documentation/core-api/, which were not fetched for this note. To resolve: read mm/percpu.c and mm/percpu-internal.h in v6.12 for the chunk/first-chunk lifecycle. uncertain

Alternatives and When to Choose Them

Per-CPU variables sit at one end of the synchronization spectrum: zero sharing, zero contention, but no globally-consistent view. Compare:

  • Atomic global counter (atomic_t). One shared variable updated with a LOCK-prefixed instruction (see Kernel Atomic Operations and atomic_t). Gives an always-consistent value with a cheap read, but every write contends on one cache line — fine for low write rates, catastrophic for hot paths. Choose this when reads must be exact and writes are rare.
  • percpu_counter. A hybrid: per-CPU local counters plus an approximate global total, with a batch threshold that folds local deltas into the global value periodically. Bounds the O(CPUs) read cost of a raw per-CPU sum at the price of approximate reads between folds. Choose this when you have a high write rate and frequent reads that can tolerate slack.
  • Sequence Locks and seqlock or RCU. For read-mostly structured data (not just counters), these give cheap readers without per-CPU duplication of the whole structure.

The decision rule: reach for a raw per-CPU variable when the data is naturally per-CPU (a CPU’s own run queue, its own free-page cache, its own slab magazine) and the global aggregate is needed rarely or never. If you find yourself summing it on every read, you probably want percpu_counter or a different structure.

Production Notes

Per-CPU variables are everywhere in the hot paths of the kernel precisely because they remove contention. The scheduler gives each CPU its own [[Per-CPU Run Queues and struct rq|run queue (struct rq)]] so that enqueue/dequeue of a task touches no other CPU’s lock. The page allocator keeps a per-CPU page list (pcplist) so that the common single-page alloc/free never takes the zone lock. The slab allocator keeps per-CPU “magazines” of free objects. Network statistics, VM event counters (vm_event_states), and many subsystem counters are per-CPU. In BPF, per-CPU maps expose the same mechanism to userspace-loaded programs so that high-rate event counting scales across cores.

The introduction of local_lock (merged in Linux 5.8, August 2020, out of the PREEMPT_RT effort) was driven by the realization that the implicit preempt_disable()/local_irq_save() that “protected” per-CPU data was invisible to lockdep and scopeless — and unfriendly to real-time kernels, where you cannot simply disable preemption (Corbet, Local locks in the kernel, LWN 2020). That history is the bridge between this note and its sibling.

See Also

Graph View

Backlinks