vault

Futex and OS Synchronization Primitives

15 min read

Futex and OS Synchronization Primitives

Below the runtime semaphore — which parks goroutines — sits a second, lower layer that parks OS threads (the M in the G-M-P model). When an M has no goroutine to run, or when the runtime’s own internal locks are contended, the runtime must put the underlying kernel thread to sleep and later wake it. On Linux, FreeBSD, and DragonFly BSD it does this with the futex (“fast userspace mutex”) system call; on Darwin, Windows, and most other platforms it uses an OS semaphore. Go abstracts both behind a tiny note type and a mutex type, with two interchangeable implementation files — src/runtime/lock_futex.go and src/runtime/lock_sema.go — selected at build time by //go:build constraints (lock_futex.go).

Mental Model

The key distinction this note exists to make: goroutine parking and thread parking are different layers. When sync.Mutex.Lock() blocks, it parks a goroutine via gopark — the M is freed and immediately runs another goroutine. The kernel thread never sleeps. Thread parking happens one level down: when the scheduler genuinely has nothing for an M to do, it stops the M itself, and that is what reaches a futex.

A futex is a hybrid: a 32-bit integer in shared userspace memory, plus a kernel wait-queue keyed by that integer’s address. The fast path — uncontended lock and unlock — is a pure userspace atomic operation and never enters the kernel at all. The kernel is involved only when a thread must actually block (FUTEX_WAIT) or wake a blocked thread (FUTEX_WAKE) (futex(2), Drepper, “Futexes Are Tricky”).

graph TD
    subgraph "Goroutine layer (sema.go)"
      G["goroutine blocks on sync.Mutex"]
      G -->|"gopark — M stays runnable"| RUNQ["M picks next goroutine"]
    end
    subgraph "Thread layer (lock_*.go)"
      M["M has NO goroutine to run<br/>OR runtime-internal mutex contended"]
      M --> NOTE["note / mutex primitive"]
      NOTE -->|"lock_futex.go build"| FX["futexsleep → FUTEX_WAIT syscall<br/>kernel parks the OS thread"]
      NOTE -->|"lock_sema.go build"| SEM["semasleep → OS semaphore<br/>(mach semaphore / Win event)"]
    end
    caption["Two layers. Top: goroutine blocks, M keeps working — no syscall.<br/>Bottom: the M itself has nothing to do, so the kernel thread sleeps.<br/>Only the bottom layer touches a futex / OS semaphore."]

Diagram: futexes only appear at the thread layer. The insight — most “blocking” in a Go program is goroutine parking that never sleeps a thread; the futex path is the rarer, deeper case of idling an entire M.

Mechanical Walk-through

The two abstractions: note and mutex

The runtime defines two internal types, both keyed off a single machine word:

  • note — a one-shot notification. It is created cleared, one thread notesleeps on it, another thread notewakeups it exactly once. Used for events like “the new M has finished bootstrapping” or “sysmon, wake up.”
  • mutex (runtime-internal, not sync.Mutex) — a mutual-exclusion lock used to protect runtime data structures (sched.lock, the semaRoot.lock, etc.). It has a fast spinning path and falls back to the same kernel sleep mechanism via futexsleep / semasleep.

These two layers — the platform note/semaphore primitives, and the runtime mutex — are now in separate files in the current tree, and the split matters for getting the picture right. Verified against the Go 1.26.3 tag:

  • lock_futex.go (//go:build dragonfly || freebsd || linux) and lock_sema.go (//go:build aix || darwin || netbsd || openbsd || plan9 || solaris || windows) contain only the note one-shot primitives and, on the sema platforms, the per-M semacreate/semasleep/semawakeup glue. Exactly one of the two is compiled in. The split exists because Linux/FreeBSD/DragonFly expose a futex syscall (so the kernel owns the wait queue), whereas the others must build the wait list themselves on top of a per-thread kernel semaphore (lock_futex.go, lock_sema.go).
  • The runtime mutex’s lock2/unlock2 no longer live in those files. As of Go 1.24 the default implementation is the “spinbit” mutex in lock_spinbit.go (//go:build !wasm), enabled by GOEXPERIMENT=spinbitmutex (on by default; disable with GOEXPERIMENT=nospinbitmutex) — the Go 1.24 release notes describe it as “a new runtime-internal mutex implementation” (go1.24 notes, proposal 68578). lock_spinbit.go still delegates the actual blocking to the platform futexsleep / semasleep from those two files, so the layering is intact — only the lock2 spin-and-park logic moved.

The futex path (lock_futex.go)

futexsleep(addr *uint32, val uint32, ns int64) is a thin wrapper over the raw syscall:

func futexsleep(addr *uint32, val uint32, ns int64) {
    if ns < 0 {
        futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, nil, nil, 0)
        return
    }
    var ts timespec
    ts.setNsec(ns)
    futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, &ts, nil, 0)
}

The semantics of FUTEX_WAIT are the crux: it tells the kernel “put this thread to sleep, but only if *addr still equals val.” The kernel re-checks the value atomically against the wait-queue lock. If another thread already changed *addr (i.e., the event already happened), the syscall returns EAGAIN immediately and the thread does not sleep. This compare-then-sleep is what closes the lost-wakeup race: a waker that updates *addr and calls FUTEX_WAKE between the sleeper’s userspace check and its syscall is still observed, because the kernel does the value check itself under the queue lock.

_FUTEX_WAIT_PRIVATE (rather than plain FUTEX_WAIT) tells the kernel the futex is never shared across processes, letting it skip the cost of resolving the address to a global inode-based key — a measurable optimization for a single-process Go binary.

futexwakeup(addr, cnt) issues FUTEX_WAKE_PRIVATE to wake at most cnt threads queued on addr. The runtime almost always passes cnt = 1. Notably, if futexwakeup ever gets an error, the runtime deliberately crashes by writing to address 0x1006 — a futex wake failing means the runtime’s internal invariants are broken and there is no safe recovery.

notewakeup on the futex path is: atomically Xchg the note’s key from 0 to 1 (and throw on double-wakeup), then futexwakeup. notesleep spins a for atomic.Load(key) == 0 loop calling futexsleep(key, 0, ...) — the loop handles spurious wakeups, which FUTEX_WAIT is explicitly allowed to deliver.

The OS-semaphore path (lock_sema.go)

Where there is no futex, each M is given a private kernel semaphore at creation (semacreate). notesleep stores the M’s pointer into the note’s key with a CAS; if the CAS fails the wakeup already happened, so it returns. Otherwise it calls semasleep(-1) — a blocking down() on the M’s semaphore. notewakeup CASes the key to locked, reads the previous value, and if it was an M pointer, calls semawakeup(m) — an up() on that M’s semaphore.

The structural difference: the futex keys on the note’s address and the kernel maintains the wait queue; the semaphore path keys on the M and each M carries its own one-slot kernel object. Functionally identical (note semantics are preserved), but the futex version needs no per-M kernel resource.

Timed sleeps and the syscall accounting

notetsleep adds a deadline; notetsleepg is the variant called on a user goroutine’s stack rather than g0, and it wraps the sleep in entersyscallblock() / exitsyscall(). That pairing is critical: it tells the scheduler “this M is about to block in the kernel for a while” so the scheduler can hand this M’s P to another thread — otherwise a P would sit idle behind a sleeping thread. This is the same entersyscall/exitsyscall handshake used for ordinary blocking system calls.

The runtime mutex: spinning then sleeping

The runtime-internal mutex is an adaptive lock: spin briefly in userspace, then park in the kernel. The reason to spin first is that on a multiprocessor the holder may release within a few cycles, and a syscall would cost far more than the wait; the reason to eventually park is that a descheduled holder could keep a pure spinlock burning a core for a full timeslice. This cheap-spin-then-expensive-park shape is the classic Drepper futex-mutex recipe (Drepper) — but Go’s current implementation refines it in two specific ways worth getting right.

The tristate, not a boolean. A naive mutex with states {unlocked, locked} cannot tell, on unlock, whether anyone is waiting — so it would have to issue a FUTEX_WAKE on every unlock, paying a syscall even when uncontended. Go therefore splits “locked” into two values: 1 = “locked, no contention” and 2 = “locked, and a thread may be asleep.” The unlocker issues a wakeup only when it sees state 2; an uncontended unlock from state 1 is a single atomic store with no syscall at all. The locker only escalates to state 2 (and then sleeps) after it has failed to grab the lock by spinning. This three-state design — unlocked / locked-no-contention / locked-with-sleepers — exists on every multithreaded platform; proposal 68578 names it “tristate” (proposal 68578, Background). On the futex platforms the kernel owns the sleeper list, so the runtime only needs the “may be asleep” hint; on the semaphore platforms the runtime must itself track exactly which Ms are parked (the LIFO stack threaded through the state word).

The “spinning” bit — Go 1.24’s scalability fix. The tristate alone has a thundering-herd flaw: unlock2 wakes a sleeper whenever one exists, without checking whether some other waiter is already awake and spinning. Under heavy contention with a short critical section, many threads wake, all reload the mutex’s state word in a loop, and that shared cache line ping-pongs between cores — so every write to acquire or release the lock slows down, and throughput collapses as cores are added. The fix, shipped as the default in Go 1.24 under GOEXPERIMENT=spinbitmutex (go1.24 notes), adds a “spinning” flag (mutexSpinning, bit 8 in lock_spinbit.go’s uintptr word) to the state. Threads contend (non-blockingly) for that one bit; only the thread holding it is permitted to reload the state word in a loop. Every other waiter goes straight to sleep instead of busy-spinning. The unlocker, seeing the spinning bit set, knows a waiter is already awake and skips the wakeup entirely. The net effect is that at most one thread spins on the cache line at a time, cutting coherency traffic and letting the locked/unlocked transitions stay fast as the core count grows (proposal 68578, Proposal). The word also carries a mutexSleeping (bit 1) hint and a mutexStackLocked (bit 9) try-lock guarding inspection of the sleeper stack, with the upper bits holding a partial pointer to the most-recently-slept M.

Either way — tristate-only or spinbit — the kernel interaction is gated on a “someone might be asleep” bit, and that bit is only set when a thread genuinely could not get the lock by spinning. This is precisely why “uncontended lock/unlock never enters the kernel” holds.

Why a futex, and not a plain spinlock or a plain syscall

The futex exists because the two obvious alternatives are each badly wrong at one extreme. A pure spinlock never sleeps — under contention it burns a whole CPU core busy-waiting, catastrophic when the holder is descheduled and the spinner spins for a full timeslice doing nothing. A pure kernel mutex (one syscall per lock, one per unlock) is correct but pays a ~hundreds-of-nanoseconds syscall on every operation including the overwhelmingly common uncontended case. The futex is the hybrid that gets both right: uncontended operations are userspace atomics (spinlock-cheap), and genuine blocking sleeps in the kernel (mutex-correct). The runtime layers one more level — a bounded spin before the futex sleep — so brief contention is resolved by spinning and only sustained contention pays the syscall, capturing the spinlock’s best case without its worst.

Code Examples

How the scheduler idles an M

// runtime/proc.go — stopm(), simplified: an M with no work parks itself
func stopm() {
    mp := getg().m
    lock(&sched.lock)
    mput(mp)                 // put M on the idle-M list
    unlock(&sched.lock)
    mPark()                  // (1) the actual park
    acquirep(mp.nextp.ptr()) // (2) woken: a P was handed to us
    mp.nextp = 0
}
 
func mPark() {
    gp := getg()
    notesleep(&gp.m.park)    // (3) sleep the OS thread on the note
    noteclear(&gp.m.park)
}
  1. mPark is where a surplus OS thread actually goes to sleep — this is the gateway from Go scheduler logic into the kernel.
  2. When the M is later woken, a P has already been attached (mp.nextp); the M resumes by re-acquiring it and looking for goroutines.
  3. notesleep(&gp.m.park) is the platform-dependent call — on Linux it becomes futexsleep; on Darwin it becomes semasleep. The same line of Go compiles to a futex FUTEX_WAIT or a Mach semaphore wait depending only on the //go:build tag.

Waking an idle M when work appears

// runtime/proc.go — startm() wakes a parked M when a goroutine becomes runnable
func startm(pp *p, spinning, lockheld bool) {
    // ... obtain an idle M ...
    mp.nextp.set(pp)            // hand it a P to run
    notewakeup(&mp.park)        // FUTEX_WAKE / semaphore up — unparks the thread
}

notewakeup(&mp.park) is the matching wake. On Linux this is one FUTEX_WAKE_PRIVATE syscall waking exactly one thread. The note’s one-shot discipline guarantees the M was either already awake (key already non-zero, wake is a no-op-ish fast path) or asleep on the futex and gets woken — never a lost or doubled wakeup.

Why notesleep insists on g0

func notesleep(n *note) {
    gp := getg()
    if gp != gp.m.g0 {
        throw("notesleep not on g0")
    }
    // ...
}

notesleep must run on the M’s g0 (the scheduling stack), not a user goroutine, because parking the thread here means the M is not scheduling — there is no goroutine context to preserve. notetsleepg is the deliberate exception: it runs on a user g but brackets the sleep with entersyscallblock/exitsyscall so the scheduler treats it as a syscall and reclaims the P.

Failure Modes and Common Misunderstandings

  • “sync.Mutex is a futex.” No. sync.Mutex is a user-space goroutine lock built on the treap runtime semaphore; blocking on it parks a goroutine and the thread keeps working. A futex is only reached when the thread itself has nothing to do, or for the runtime’s own internal mutex. Conflating the two is the single most common error here.
  • Spurious wakeups. FUTEX_WAIT may return without the condition being true (signal interruption, EINTR). Every notesleep/semasleep caller loops and re-checks; code that assumed one wakeup per sleep would be wrong.
  • The lost-wakeup window. The whole reason FUTEX_WAIT takes an expected value is to atomically close the gap between a userspace check and the kernel sleep. If you naively if !ready { sleep() }, a waker between the check and the sleep is lost. The futex’s kernel-side value re-check eliminates this; a hand-rolled version that drops the value argument reintroduces the bug.
  • Per-process vs per-thread. _PRIVATE futexes assume single-process use. Go binaries are single-process, so this holds — but code embedding Go via cgo into shared memory must not assume Go’s futexes are visible to other processes.
  • A parked M does not hold a P. A common misconception is that an idle M wastes a P. stopm runs after the M’s P has been released; the M parks empty-handed and is handed a fresh P only on wakeup.

Alternatives and When They Apply

The futex/semaphore note is the runtime’s thread-blocking primitive. Its counterpart one layer up is the treap runtime semaphore (semacquire/semrelease), the goroutine-blocking primitive — and that is what every sync type uses. Choosing between layers is not a user decision; it is structural: user concurrency code blocks goroutines (semaphore), the scheduler blocks threads (futex).

Within the thread layer, futex versus OS semaphore is purely a platform choice, not a tuning knob — Linux gets the futex because the kernel offers it and it needs no per-thread kernel object; Darwin/Windows get the semaphore path because that is the primitive their kernels expose. For pure spin-only coordination with no kernel involvement at all, the runtime’s adaptive mutex spin phase and atomics cover the uncontended case; the futex is strictly the fallback for genuine blocking.

Production Notes

In a perf or strace trace of a healthy Go program, futex calls are normal and expected — they appear whenever the scheduler parks and unparks Ms as load fluctuates. A high and growing rate of futex syscalls, however, is a classic symptom of lock contention on a runtime-internal mutex or excessive M churn (Ms repeatedly parking and unparking because work arrives in tiny bursts). The execution tracer is the better tool here — it shows M parking events directly rather than as raw syscalls.

GODEBUG=schedtrace=1000 prints scheduler state every second, including idle-M counts; a fluctuating idle-M count corresponds to futexsleep/futexwakeup traffic. GOMAXPROCS interacts with this: more Ps means more Ms that can be running, hence more parking/unparking transitions as the runtime balances threads against the available P slots.

The high-level algorithm and the bit layout above are verified against proposal 68578 and confirmed present as the default at the Go 1.26.3 tag (lock2/unlock2 live in lock_spinbit.go; lock_futex.go/lock_sema.go carry only the note/semaphore primitives). The one detail below remains unverified against a directly quoted line of source:

Uncertain

Verify: the exact numeric spin-iteration bounds in lock_spinbit.go (the per-iteration procyield count and the number of active/passive spin rounds before parking). Reason: these constants are an implementation detail that has been tuned across releases, and they were not confirmed against a directly quoted line of the Go 1.26.3 source — only the spin-then-park structure was. To resolve: read the lock2 body and the active_spin/active_spin_cnt/passive_spin constants in src/runtime/lock_spinbit.go at the go1.26.3 tag. uncertain

See Also

Graph View

Backlinks