vault

Memory Fences and Barriers

15 min read

Memory Fences and Barriers

A memory fence (equivalently memory barrier) is the abstract primitive that restores ordering in a world where the compiler and the CPU are both free to reorder memory operations for speed. On its own, a store or a load carries no ordering promise relative to its neighbors; a fence is an instruction (or a compiler pseudo-op) placed between memory operations that forbids a specified class of reorderings from crossing it. The taxonomy is small and universal: barriers come in four elemental flavors — LoadLoad, StoreStore, LoadStore, and StoreLoad — from which every richer notion is built (Preshing 2012). Composed, these give the familiar full (general) barrier, read (load) barrier, and write (store) barrier of the Linux kernel (kernel.org, memory-barriers.txt), and the one-way acquire and release fences of language memory models. A crucial early distinction: a compiler barrier only stops the compiler from moving accesses, while a CPU barrier emits an actual hardware instruction (mfence on x86, dmb on ARM) to stop the processor — and on a multiprocessor you almost always need both. This note develops the abstract model; the concrete kernel API lives in Memory Barriers in the Linux Kernel and the language-level pairing in Acquire Release and Fence Semantics.

Mental Model — a wall that some operations cannot climb over

Think of your thread’s memory operations as a stack of bricks laid in program order. Both the compiler (once, at build time) and the CPU (continuously, at run time) are allowed to slide bricks past each other to keep the pipeline and the store buffer busy — a later load may be hoisted above an earlier store, an earlier store may sink below a later one. A fence is a wall inserted between two bricks. But it is not always a solid wall: each fence flavor blocks only specific pairs from swapping across it. The art of lock-free programming is inserting the weakest wall that still blocks the one reordering that would break correctness — a stronger wall is correct but slower.

flowchart TD
    subgraph BEFORE["Before the fence (program order)"]
        L1["Load A"]
        S1["Store B"]
    end
    FENCE{{"FENCE<br/>(blocks selected reorderings<br/>from crossing)"}}
    subgraph AFTER["After the fence"]
        L2["Load C"]
        S2["Store D"]
    end
    L1 --> FENCE
    S1 --> FENCE
    FENCE --> L2
    FENCE --> S2

    NOTE["LoadLoad: Load A stays before Load C<br/>StoreStore: Store B stays before Store D<br/>LoadStore: Load A stays before Store D<br/>StoreLoad: Store B stays before Load C (the expensive one)"]

What the diagram shows and the insight to take: the fence sits between an earlier group (Load A, Store B) and a later group (Load C, Store D). Each of the four barrier flavors names exactly which earlier–later pair is forbidden to reorder across the wall. The insight is that “a barrier” is never one thing — it is a set of these constraints, and the single most expensive constraint to enforce is StoreLoad (keeping an earlier store ordered before a later load), because it is the one reordering that even strongly-ordered x86 hardware performs by default and the only one that requires draining the store buffer (Preshing 2012).

The four elemental barriers

Preshing’s decomposition is the cleanest way to name what a fence actually constrains, so use it as the foundation. A LoadLoad barrier prevents a load before the barrier from being reordered with a load after it — it guarantees you see earlier reads before later reads. A StoreStore barrier prevents an earlier store from being reordered past a later store — earlier writes become visible before later ones. A LoadStore barrier keeps an earlier load ordered before a later store. And a StoreLoad barrier — the heavyweight — ensures that every store before the barrier is globally visible before any load after it can take its value, so that a later load cannot “jump ahead” and read stale data before the prior store has propagated (Preshing 2012).

The reason StoreLoad is singled out as the costly one is the store buffer (see Cache Coherence and the Store Buffer). A CPU posts a store into its local buffer and lets execution race ahead; a subsequent load to a different address may complete out of the buffer’s shadow, effectively reordering “store then load” into “load then store.” This is the exact reordering behind the store-buffering litmus test in which two threads each write a flag and read the other’s, and both read zero — an outcome impossible under Sequential Consistency but permitted on x86-TSO, ARM, and POWER alike (Cox, “Hardware Memory Models”). Only a StoreLoad barrier (a full fence) forbids it; Preshing notes that instructions acting as StoreLoad tend to be more expensive than the other three flavors (Preshing 2012). The other three flavors are cheap on x86 because x86-TSO already forbids their reorderings in hardware — x86 keeps memory reordering to a minimum, permitting essentially only StoreLoad (Preshing 2012).

Composite barriers — full, read, write

Real APIs bundle the four elementals into three composites, and the Linux kernel documentation gives them their canonical definitions. A write (store) barrier guarantees that all STORE operations before it appear to happen before all STORE operations after it, with respect to the rest of the system — it is essentially a StoreStore barrier and has no required effect on loads (kernel.org). A read (load) barrier gives the symmetric guarantee for LOAD operations before and after it, and additionally implies an address-dependency barrier (kernel.org). A general (full) barrier guarantees that all LOADs and STOREs before it appear to happen before all LOADs and STOREs after it — it implies both a read and a write barrier, and can substitute for either (kernel.org). In kernel notation these are wmb(), rmb(), and mb() for the mandatory (device-facing) forms, and smp_wmb(), smp_rmb(), smp_mb() for the SMP-conditional forms used between CPUs — the full concrete API is the subject of Memory Barriers in the Linux Kernel, which this note deliberately does not duplicate.

The kernel documentation is also explicit about what a barrier does not promise, and these caveats are where real bugs live. There is no guarantee that memory accesses before a barrier are complete when the barrier instruction finishes — a barrier orders visibility, not completion (kernel.org). There is no guarantee that a barrier on one CPU has any direct effect on another CPU. And most importantly: a barrier on CPU 1 is useless unless CPU 2 executes a matching barrier — barriers must be paired to establish an ordering between two CPUs (kernel.org). A lone fence on the producer with none on the consumer orders nothing observable.

One-way fences — acquire and release

The four elemental barriers are two-sided: a StoreStore wall blocks all store–store reorderings across it, in both directions of the pairing. Language memory models and modern ISAs also provide one-way barriers, which are cheaper and map directly onto the release discipline. The kernel documentation describes an ACQUIRE operation as a one-way permeable barrier: all memory operations after the acquire appear to happen after it, but operations before the acquire may leak downward past it (kernel.org). A RELEASE operation is the mirror image: all memory operations before the release appear to happen before it, but operations after may leak upward past it (kernel.org). The “permeable” metaphor is exact — the wall lets bricks pass one way but not the other.

Preshing pins this down in the four-flavor vocabulary: a standalone acquire fence acts as a LoadLoad + LoadStore barrier (it prevents a load before it from being reordered with any load or store after it), while a standalone release fence acts as a LoadStore + StoreStore barrier (it prevents any load or store before it from being reordered with a store after it) (Preshing 2013). Note the asymmetry: acquire constrains what may move down across it, release constrains what may move up across it, and neither alone provides StoreLoad — which is why a release-then-acquire pair is famously not equivalent to a full barrier, a caveat the kernel documentation states outright (kernel.org).

Standalone fence versus fenced atomic operation

A subtle but load-bearing distinction: an ordering constraint can be attached either to a standalone fence sitting between operations, or to the atomic operation itself (a store-release, a load-acquire). These are not interchangeable, and the C++ standard is precise about why. A standalone release fence followed by a relaxed atomic store turns that store into a release operation — but, as Preshing notes, in that case it is the fence, not the store, that synchronizes-with a later acquire (Preshing 2013). The C++ [atomics.fences] clause makes the fence-to-fence rule normative: a release fence A synchronizes with an acquire fence B when there exist atomic operations X and Y on some object M such that A is sequenced before X, X modifies M, Y is sequenced before B, and Y reads the value X wrote (or a value from the release sequence X heads) (C++ draft, atomics.fences).

The practical consequence is that a standalone fence is more broadly acting than a fenced operation: a fence orders all prior/subsequent accesses of the relevant kind, whereas a release store orders only relative to that specific store’s location. std::atomic_thread_fence(std::memory_order_release) is a release fence; with memory_order_acquire an acquire fence; with memory_order_acq_rel both; with memory_order_seq_cst a sequentially-consistent full fence; with memory_order_relaxed it has no effect at all (C++ draft). Prefer the fenced operation (store-release / load-acquire) when you only need to order one location — it is what the hardware’s cheap one-way instructions implement directly; reach for a standalone fence when you must order a group of surrounding accesses, or to separate the fence from the atomic for clarity.

Compiler barriers versus CPU barriers

The single most common beginner error is conflating these two, so keep them sharply separate. A compiler barrier constrains only the compiler’s code motion; it emits no machine instruction. In the Linux kernel it is barrier(), which prevents the compiler from moving memory accesses from one side of it to the other and forces reloads of variables held in registers across loops (kernel.org). The finer-grained READ_ONCE() and WRITE_ONCE() are weak per-access compiler barriers that also prevent load/store tearing, load/store merging, and invented (speculative) loads on the specific access they wrap (kernel.org).

A compiler barrier is sufficient on a uniprocessor — a single CPU is self-consistent and never observes its own accesses out of order — which is exactly why the kernel’s SMP barriers degrade to compiler barriers when compiled for a uniprocessor: smp_mb() becomes just barrier() there (kernel.org). But on a real multiprocessor a compiler barrier is not enough: even with the compiler forbidden to reorder, the CPU’s store buffer and out-of-order engine will still reorder visible to other CPUs. That requires a genuine CPU barrier — a hardware instruction. Conversely, note that all CPU memory barriers (except the address-dependency barrier) imply a compiler barrier as well (kernel.org), so you rarely need both by hand.

Hardware instructions — x86 and ARM

On x86 / x86-64 the relevant instructions are MFENCE, SFENCE, and LFENCE. MFENCE is the full fence: every load and store before it becomes globally visible before any load or store after it — it orders both loads and stores in both directions, and is the instruction you need for the StoreLoad ordering that x86-TSO otherwise permits (felixcloutier, MFENCE). SFENCE orders stores against stores and LFENCE orders loads, but because x86-TSO already provides StoreStore, LoadLoad, and LoadStore ordering for ordinary write-back memory, plain code seldom needs them — a locked read-modify-write instruction (e.g. LOCK XADD, XCHG) also carries full-fence semantics and is often used as the StoreLoad barrier in practice (Cox).

On ARM (AArch64) the weaker memory model means fences are needed far more often. DMB (Data Memory Barrier) is the workhorse: it ensures that memory accesses before it are observed before accesses after it, without stalling the pipeline — it just tells the memory system to preserve externally-visible ordering (Microsoft, “AArch64 Barriers”; ARM Developer). DMB takes options selecting a shareability domain and access type — e.g. DMB ISH for the inner-shareable domain (the usual SMP case), and DMB ISHST to order only stores (a StoreStore barrier) rather than the full DMB ISH (Microsoft). DSB (Data Synchronization Barrier) is stronger than DMB: it additionally stalls execution until all outstanding accesses complete, used for TLB and cache-maintenance sequencing, not ordinary locking. ISB (Instruction Synchronization Barrier) flushes the instruction pipeline so subsequent instructions are re-fetched — used after self-modifying code or system-register changes, not for data ordering (Microsoft). Crucially, ARMv8 also provides one-way instructions — LDAR (load-acquire) and STLR (store-release) — that implement acquire/release semantics directly and cheaply, so a well-written ARM lock uses these rather than a two-sided DMB wherever possible (Microsoft). This is the concrete face of the two-sided-fence-versus-one-sided-operation distinction above.

Uncertain

Verify: the exact behavior of DMB ISHST versus DMB ISHLD and the full set of DMB shareability/access-type option combinations. Reason: the ARM Developer “Barriers” page is a JavaScript single-page app that returned only its title to a plain fetch, so the ARM specifics here rest on Raymond Chen’s (Microsoft) “AArch64 Barriers” write-up and a web-search digest rather than the ARM Architecture Reference Manual directly. To resolve: consult the ARM ARM (DDI 0487) section on the DMB/DSB instructions for the authoritative option semantics. #uncertain

Failure Modes and Common Misunderstandings

Forgetting the matching barrier. A fence orders your CPU’s accesses; establishing an ordering between two CPUs requires a fence on both sides — a release on the producer paired with an acquire on the consumer. A lone fence orders nothing another thread can observe (kernel.org).

Believing a compiler barrier is a CPU barrier. barrier() (or a C asm volatile("" ::: "memory")) stops compiler reordering only. It is invisible to the store buffer and does nothing to prevent hardware reordering on SMP. This bug hides on x86 (whose strong TSO forgives most reorderings) and detonates on ARM/POWER — the classic “worked on my laptop, broke on the phone.”

Assuming release+acquire equals a full fence. It does not — the pair provides no StoreLoad ordering, so a store after a release and a load after an acquire can still reorder. If you need a full fence you must use one (kernel.org).

Thinking a barrier flushes or “commits” memory. A barrier constrains relative ordering of visibility; it does not force writes to complete, does not flush caches, and does not push data to another core faster. It is a reordering constraint, nothing more (kernel.org).

Mistaking volatile (in C/C++) for a fence. volatile prevents the compiler from eliding or coalescing an access, but emits no CPU barrier and provides no cross-thread ordering — it is not a synchronization primitive. (Java’s volatile is different: it does carry acquire/release semantics — see The Java Memory Model.)

Alternatives and When to Choose Them

Standalone fences are the lowest-level ordering tool, and most code should prefer something higher. Prefer fenced atomic operations (store-release, load-acquire) when you order a single location — they map to the cheap one-way hardware instructions (STLR/LDAR on ARM) and read more clearly. Prefer a standalone fence when a group of surrounding non-atomic accesses must be ordered as a block, or when separating the fence from the atomic aids readability. Prefer locks, channels, or higher-level synchronization for ordinary application code — they insert the correct fences for you and keep you inside DRF-SC, so you never reason about fences at all. Reach for explicit fences only in lock-free data structures and the innards of synchronization primitives themselves, where the fence is the mechanism.

Production Notes

Explicit fences are concentrated in exactly two places: the implementation of synchronization primitives (mutexes, spinlocks, RCU, lock-free queues) and device drivers talking to memory-mapped hardware (where the mandatory mb()/wmb() forms order accesses against a device, not just against another CPU) (kernel.org). The dominant real-world lesson is portability: because x86-TSO forgives LoadLoad, StoreStore, and LoadStore reordering, code that is missing a fence frequently runs flawlessly on x86 for years, then fails intermittently the day it is ported to ARM or POWER, whose weak models expose the reordering (Cox). The corollary for reviewers: a fence is not “extra safety you can add later” — its absence is a latent bug that your test hardware may be structurally incapable of revealing. When in doubt about which flavor, the abstract rule is “insert the weakest barrier that blocks the one reordering that breaks you,” and the concrete rule is “if you can’t prove which, a full fence is correct and only costs performance.”

See Also

Graph View

Backlinks