Kernel Mutexes

The Linux kernel struct mutex is the canonical sleeping mutual-exclusion lock for process context: at most one task may hold it, and a task that finds it held is put to sleep on the scheduler’s run queue and woken when the holder releases it. Unlike a spinlock, which burns a CPU busy-waiting, a contended mutex frees its CPU to run other work — which is why it is the right tool when the critical section may itself block (on I/O, on memory reclaim) or simply runs long. Mutexes were introduced by Ingo Molnar in late 2005/2006 as a purpose-built, strictly-enforced alternative to abusing counting semaphores for binary locking (LWN, Generic mutex subsystem); the in-kernel documentation’s standing advice is blunt: “Unless the strict semantics of mutexes are unsuitable and/or the critical region prevents the lock from being shared, always prefer them to any other locking primitive” (mutex-design.rst, v6.12). Everything below is pinned to Linux 6.12 LTS (released 2024-11-17).

The single most important idea: a mutex packs the owner into the lock word itself, takes the lock with one uncontended atomic compare-and-swap, and only descends into a slow path with a wait list and the scheduler when there is actual contention. The fast, common case touches one cache line and never enters the kernel’s locking machinery at all. And between the fast path and the sleep there is a third path — optimistic spinning, which busy-waits like a spinlock for as long as the current owner is still running on another CPU. That midpath is the design’s central bet, and it is why the in-tree documentation calls the mutex “more practically a hybrid type” rather than a pure sleeping lock.

Version pin

Every source line, macro, struct layout, and configuration default below was read from the v6.12 tag of the mainline tree on 2026-09-04. The tag object itself is dated 2024-11-17 (tagger Linus Torvalds, per the GitHub git-tag API), and kernel.org’s machine-readable release index lists the 6.12 series as longterm — a maintained LTS branch, at 6.12.108 as of 2026-09-02 — while mainline has moved on to the 7.x series (7.3-rc1, latest stable 7.2.3). So v6.12 is a deliberate LTS pin, not “current”: every claim here is “true as of v6.12,” and anything after that release date is out of scope unless explicitly noted. Where a behaviour changed in a later release, this note says so and names the tag.

---

Mental Model

Think of a mutex as a single atomic machine word — atomic_long_t owner — that means three things at once. When the word is 0, the lock is free. When it holds a struct task_struct * pointer, that task owns the lock. And because task_struct pointers are aligned to at least L1_CACHE_BYTES (typically 64 bytes on x86-64), their low bits are always zero, so the kernel steals the bottom three of them to carry lock state — “there are waiters,” “hand the lock off,” “handoff done, awaiting pickup” — without needing any separate flags field. The whole uncontended protocol is: atomically swing that word from 0 to current on lock, and from current back to 0 on unlock.

flowchart TB
  L["mutex_lock(lock)"] --> MS["might_sleep()<br/>(debug: assert process context,<br/>may schedule)"]
  MS --> FP{"fast path:<br/>cmpxchg owner<br/>0 → current ?"}
  FP -->|"success<br/>(uncontended)"| H["HOLD the lock<br/>owner == current"]
  FP -->|"failed<br/>(someone owns it)"| SLOW["__mutex_lock_slowpath"]
  SLOW --> MID["midpath: optimistic spin<br/>while owner is ON-CPU<br/>(see sibling note)"]
  MID -->|"acquired"| H
  MID -->|"owner sleeps /<br/>must reschedule"| WL["take wait_lock,<br/>add mutex_waiter to FIFO wait_list,<br/>set MUTEX_FLAG_WAITERS"]
  WL --> SL["set_current_state(TASK_UNINTERRUPTIBLE)<br/>schedule_preempt_disabled()<br/>→ task SLEEPS"]
  SL -->|"woken by unlock"| H
  H --> U["mutex_unlock(lock)"]
  U --> UFP{"fast path:<br/>cmpxchg owner<br/>current → 0 ?<br/>(no waiters, no handoff)"}
  UFP -->|"success"| DONE["done — no wakeup needed"]
  UFP -->|"WAITERS set"| UW["__mutex_unlock_slowpath:<br/>wake first waiter via wake_q"]
  UW --> DONE

The full lifecycle of a mutex acquisition and release, as of 6.12 LTS. What it shows: the two common cases — uncontended lock/unlock — are single atomic compare-and-swaps on the owner word (FP and UFP boxes); the entire wait-list, sleep, and wakeup machinery (WL, SL, UW) only runs when a second task actually contends. The MID box is the optimistic-spinning midpath, walked in full in The Midpath section below and covered exhaustively in Adaptive Mutex Spinning and Optimistic Spinning. The insight to take: a mutex is “expensive” only under contention; the design goal was to make the no-contention path as close to free as a spinlock’s while still being able to sleep when it matters.

---

Three Paths, Not Two

Documentation/locking/mutex-design.rst (v6.12) enumerates the acquisition paths explicitly, and the middle one is the part most descriptions of “a sleeping lock” leave out:

When acquiring a mutex, there are three possible paths that can be taken, depending on the state of the lock: (i) fastpath … tries to atomically acquire the lock by cmpxchg()ing the owner with the current task. (ii) midpath, aka optimistic spinning, tries to spin for acquisition while the lock owner is running and there are no other tasks ready to run that have higher priority. … (iii) slowpath: last resort, if the lock is still unable to be acquired, the task is added to the wait-queue and sleeps until woken up by the unlock path.

The rationale for the midpath is a bet about the near future, stated in the same document: “if the lock owner is running, it is likely to release the lock soon.” If that bet pays off, the waiter avoids two context switches (out and back) and the wakeup latency, at the cost of a few hundred nanoseconds of spinning — which is why the doc concludes that “[w]hile formally kernel mutexes are sleepable locks, it is path (ii) that makes them more practically a hybrid type.”

stateDiagram-v2
    [*] --> FastPath
    state "FASTPATH — __mutex_trylock_fast()" as FastPath
    state "HELD — owner == current" as Held
    state "MIDPATH — mutex_optimistic_spin()" as Mid
    state "OSQ — queue behind other spinners (osq_lock)" as Osq
    state "SPIN — while owner_on_cpu(owner) && !need_resched()" as Spin
    state "SLOWPATH — __mutex_lock_common()" as Slow
    state "WAIT LIST — FIFO, sets MUTEX_FLAG_WAITERS" as Wait
    state "ASLEEP — TASK_UNINTERRUPTIBLE / _INTERRUPTIBLE / _KILLABLE" as Sleep

    FastPath --> Held: atomic_long_try_cmpxchg_acquire<br/>0 -> current SUCCEEDS<br/>(the overwhelmingly common case)
    FastPath --> Mid: cmpxchg fails — someone owns it
    Mid --> Osq: mutex_can_spin_on_owner() says yes
    Osq --> Spin: became the single designated spinner
    Spin --> Held: owner released, __mutex_trylock_or_owner() wins
    Spin --> Slow: owner went OFF-CPU, or need_resched()
    Mid --> Slow: owner already sleeping, or<br/>!CONFIG_MUTEX_SPIN_ON_OWNER
    Slow --> Wait: raw_spin_lock(&lock->wait_lock)<br/>__mutex_add_waiter() at the TAIL
    Wait --> Sleep: set_current_state(state)<br/>schedule_preempt_disabled()
    Sleep --> Held: woken by unlock, wins trylock<br/>or receives a HANDOFF
    Sleep --> Spin: woken and now FIRST waiter —<br/>gets one more spin attempt
    Sleep --> [*]: signal arrives (interruptible/killable)<br/>returns -EINTR WITHOUT the lock
    Held --> [*]: mutex_unlock()

The three acquisition paths as a state machine, traced against kernel/locking/mutex.c (v6.12). What it shows: each path is a fallback for the previous one failing, and control can move backwards — a task that has already slept and woken as the first waiter is given another optimistic-spin attempt (if (first) { ... mutex_optimistic_spin(lock, ww_ctx, &waiter) }) rather than being forced straight back to sleep. The insight to take: the two escape edges out of the diagram are the ones that catch people. mutex_unlock() is the normal exit; the -EINTR exit exists only for mutex_lock_interruptible() and mutex_lock_killable(), and taking it means you leave the function without the lock — which is why both are marked __must_check. Plain mutex_lock() has no such edge: once it commits to sleeping in TASK_UNINTERRUPTIBLE, nothing short of the lock becoming available will wake it.

	Fastpath	Midpath (optimistic spin)	Slowpath
Entered when	lock is free	held, and owner is running on another CPU	held, and owner is not running (or spinning gave up)
Cost	one cmpxchg	busy-wait, no context switch	2 context switches + wakeup latency
CPU released to other work?	n/a	no — behaves like a spinlock	yes
Data structures touched	owner word only	owner + the osq MCS spinner queue	wait_lock, wait_list, scheduler
Config gate	always	CONFIG_MUTEX_SPIN_ON_OWNER	always
Bails out on	—	!owner_on_cpu(), need_resched(), vcpu_is_preempted()	signal (interruptible/killable variants only)

The three paths compared on what they cost and what they touch. What it shows: the midpath is deliberately a spinlock embedded inside a sleeping lock — it does not release the CPU, and it is bounded not by a spin count but by an event (the owner going off-CPU or the scheduler asking for a reschedule). The insight to take: this is why the honest answer to “mutex or spinlock?” is not “mutexes are slower under short contention.” Under short contention with a running owner, a mutex is a spinlock, and the extra cost is one osq_lock()/osq_unlock() pair plus the owner-liveness check. The mutex only pays the sleeping-lock price when sleeping is the right thing to do anyway.

The owner word and its three flag bits

The non-PREEMPT_RT struct mutex in include/linux/mutex_types.h (v6.12) is small and deliberate:

struct mutex {
	atomic_long_t		owner;
	raw_spinlock_t		wait_lock;
#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
	struct optimistic_spin_queue osq; /* Spinner MCS lock */
#endif
	struct list_head	wait_list;
#ifdef CONFIG_DEBUG_MUTEXES
	void			*magic;
#endif
#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lockdep_map	dep_map;
#endif
};

Walking the fields: owner is the atomic lock word described below. wait_lock is a raw_spinlock_t — an internal spinlock that protects the mutex’s own wait_list during the slow path (the mutex is a sleeping lock to its users, but it serializes its internal bookkeeping with a tiny held-for-nanoseconds spinlock). osq is the MCS spinner queue used by the optimistic-spinning midpath, present only when CONFIG_MUTEX_SPIN_ON_OWNER is configured — see Adaptive Mutex Spinning and Optimistic Spinning. wait_list is the doubly-linked FIFO of blocked tasks. The final two fields exist only in debug builds: magic (a CONFIG_DEBUG_MUTEXES sentinel that catches uninitialized or corrupted mutexes) and dep_map (the CONFIG_DEBUG_LOCK_ALLOC hook that lets lockdep track this lock’s acquisition orderings). Under CONFIG_PREEMPT_RT the entire body is replaced by a struct rt_mutex_base rtmutex — on the real-time kernel a “mutex” is actually a priority-inheriting RT-mutex (see Priority Inheritance and the RT-Mutex).

The cleverness lives in owner. The header comment in kernel/locking/mutex.c (v6.12) states it verbatim:

/*
 * @owner: contains: 'struct task_struct *' to the current lock owner,
 * NULL means not owned. Since task_struct pointers are aligned at
 * at least L1_CACHE_BYTES, we have low bits to store extra state.
 *
 * Bit0 indicates a non-empty waiter list; unlock must issue a wakeup.
 * Bit1 indicates unlock needs to hand the lock to the top-waiter
 * Bit2 indicates handoff has been done and we're waiting for pickup.
 */
#define MUTEX_FLAG_WAITERS	0x01
#define MUTEX_FLAG_HANDOFF	0x02
#define MUTEX_FLAG_PICKUP	0x04
 
#define MUTEX_FLAGS		0x07

packet-beta
0: "WAITERS"
1: "HANDOFF"
2: "PICKUP"
3-5: "always 0 (alignment slack, unused)"
6-63: "struct task_struct * — upper bits of the owner pointer. Whole word == 0 means FREE."

The 64-bit atomic_long_t owner word on x86-64, from the MUTEX_FLAG_* definitions in kernel/locking/mutex.c (v6.12). What it shows: the pointer and the protocol flags share one word because task_struct allocations are aligned to at least L1_CACHE_BYTES — 64 bytes on x86-64 — so the low six bits of any real task_struct * are guaranteed zero and three of them are free to steal. The insight to take: this is not merely a space optimisation. Because owner identity and unlock obligations live in the same word, a single cmpxchg can atomically test both — which is exactly what makes the unlock fast path safe: comparing against a bare curr with no flag bits set means “I own it and nobody is waiting,” verified in one instruction. Note bits 3–5: the alignment guarantees six spare bits but the protocol only needs three, so there is room to grow.

stateDiagram-v2
    direction LR
    Free: owner = 0<br/>FREE
    Owned: owner = T<br/>held, no waiters
    OwnedW: owner = T + WAITERS<br/>held, wait list non-empty
    Handoff: owner = T + WAITERS + HANDOFF<br/>top waiter demanded a handoff
    Pickup: owner = W + PICKUP (+WAITERS)<br/>lock GIVEN to waiter W, not yet claimed
    Free --> Owned: __mutex_trylock_fast()<br/>cmpxchg_acquire 0 -> current
    Owned --> Free: __mutex_unlock_fast()<br/>cmpxchg_release current -> 0
    Owned --> OwnedW: __mutex_add_waiter() sets<br/>MUTEX_FLAG_WAITERS
    OwnedW --> Handoff: __mutex_trylock_or_handoff(lock, first)<br/>sets MUTEX_FLAG_HANDOFF
    OwnedW --> Free: unlock slowpath clears owner,<br/>wakes head of wait_list
    Handoff --> Pickup: __mutex_handoff(lock, next)<br/>owner becomes next + PICKUP
    Pickup --> Owned: waiter W runs, clears PICKUP<br/>via __mutex_trylock_common()

The owner word as a state machine. What it shows: the flags are not independent booleans, they are a small protocol — WAITERS means “an unlock must wake somebody,” HANDOFF means “an unlock must give the lock directly to the top waiter rather than dropping it,” and PICKUP means “it has already been given away and is waiting to be claimed.” The insight to take: the Handoff → Pickup → Owned path is the anti-starvation mechanism. Without it, unlock would return the word to 0 and any CPU racing past could steal the lock — including, repeatedly, the task that just released it, since it has the cache line. HANDOFF converts “the lock is free, fight for it” into “the lock is now W’s,” which bounds the wait for a queued task. Note that Pickup is the one state where the pointer in the word is not the current owner-in-execution but a promise to a task that has not run yet.

So the one machine word simultaneously encodes who holds the lock (the high bits, a real pointer) and what the unlock path must do (the low three bits). To recover just the owner pointer, the kernel masks the flag bits away:

static inline struct task_struct *__mutex_owner(struct mutex *lock)
{
	return (struct task_struct *)(atomic_long_read(&lock->owner) & ~MUTEX_FLAGS);
}

MUTEX_FLAG_WAITERS (bit 0) is the one that matters most for understanding the protocol: it tells the unlocker “the wait list is non-empty, so you cannot just clear the word and walk away — you must wake somebody.” MUTEX_FLAG_HANDOFF and MUTEX_FLAG_PICKUP (bits 1 and 2) implement an anti-starvation mechanism: rather than always dropping the lock to 0 and letting any racing task grab it (which under heavy contention could starve a long-waiting task indefinitely), the unlock path can instead directly hand the lock to the top waiter, who then “picks it up.” Those two bits exist so a queued waiter is guaranteed to eventually get the lock — fairness that the bare cmpxchg fast path does not provide on its own. The protocol is walked step by step in Handoff and pickup: bounding the wait, below.

---

Mechanical walk-through: the fast path

The uncontended acquire is two lines of real work. From kernel/locking/mutex.c (v6.12):

static __always_inline bool __mutex_trylock_fast(struct mutex *lock)
{
	unsigned long curr = (unsigned long)current;
	unsigned long zero = 0UL;
 
	if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr))
		return true;
 
	return false;
}

current is the per-CPU pointer to the running task’s task_struct. atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr) is a single atomic compare-and-swap: “if owner currently equals zero (0 — free, all flag bits clear), atomically set it to curr (this task’s pointer) and return true; otherwise return false and update zero with the observed value.” The _acquire suffix gives it acquire memory-ordering semantics, which is exactly what a lock needs: no memory access inside the critical section may be reordered to before this point, so the lock genuinely fences off the protected data (see Acquire Release and Fence Semantics and Compare-and-Swap and cmpxchg in the Kernel). If the cmpxchg succeeds, the task owns the lock and mutex_lock returns immediately — no wait list touched, no spinlock taken, no scheduler involvement.

mutex_lock() itself is just a guard around that fast path:

void __sched mutex_lock(struct mutex *lock)
{
	might_sleep();
 
	if (!__mutex_trylock_fast(lock))
		__mutex_lock_slowpath(lock);
}
EXPORT_SYMBOL(mutex_lock);

might_sleep() is a debugging annotation (active under CONFIG_DEBUG_ATOMIC_SLEEP) that asserts “this function may sleep, so you had better be in a context where sleeping is legal.” If you call mutex_lock while holding a spinlock, inside an interrupt handler, or with preemption disabled, might_sleep() is what catches it and screams in the log. It is the mechanical enforcement of the process-context-only rule discussed below. Only if the fast path fails — i.e. the lock is already owned — does control fall into __mutex_lock_slowpath, the slow path.

The unlock fast path is the mirror image:

static __always_inline bool __mutex_unlock_fast(struct mutex *lock)
{
	unsigned long curr = (unsigned long)current;
 
	return atomic_long_try_cmpxchg_release(&lock->owner, &curr, 0UL);
}

“If owner currently equals exactly curr (this task, with no flag bits set), atomically reset it to 0.” The _release suffix gives release ordering — the symmetric partner to the acquire — ensuring all writes inside the critical section are globally visible before the lock appears free. Crucially, this fast unlock succeeds only when no flag bits are set: if MUTEX_FLAG_WAITERS or MUTEX_FLAG_HANDOFF is set, the compare against bare curr fails and the kernel must enter __mutex_unlock_slowpath to perform a wakeup. So waiters never get stranded: the very existence of a waiter flips bit 0, which forces the unlocker off the fast path.

---

sequenceDiagram
    autonumber
    participant T as Task T (process context)
    participant O as owner word (atomic_long_t)
    participant S as Scheduler / wait_list
    Note over O: owner = 0 (free)
    T->>T: might_sleep() — debug assert: is sleeping legal here?
    T->>O: atomic_long_try_cmpxchg_acquire(&owner, &0, current)
    O-->>T: success
    Note over O: owner = T. No wait_lock taken.<br/>No wait_list touched. Scheduler never consulted.
    rect rgba(120,160,220,0.15)
        Note over T: critical section — may kmalloc(GFP_KERNEL),<br/>may copy_to_user(), may block on I/O
    end
    T->>O: atomic_long_try_cmpxchg_release(&owner, &current, 0)
    O-->>T: success — because NO flag bits were set
    Note over O: owner = 0 (free). Total cost: two atomics.
    Note over S: S was never involved. This is the case that runs ~always.

The complete uncontended lock/unlock cycle. What it shows: two atomic compare-and-swaps on one word, and nothing else — wait_lock is never acquired, wait_list is never walked, and the scheduler is never called. The insight to take: step 8 is where the design’s safety lives. The unlock fast path compares against a bare curr with every flag bit clear, so if any waiter has arrived and set MUTEX_FLAG_WAITERS, this compare-and-swap fails and the unlocker is forced into __mutex_unlock_slowpath() to perform a wakeup. There is no separate “are there waiters?” check that could race — the same instruction that releases the lock also detects the obligation. Note step 1: might_sleep() is the only reason this path knows or cares about context, and it compiles to nothing without CONFIG_DEBUG_ATOMIC_SLEEP.

The Midpath: Optimistic Spinning and the owner_on_cpu() Bet

When the fast path fails, __mutex_lock_common() does not immediately go to sleep. It calls mutex_optimistic_spin(), and what happens there is the most consequential design decision in the whole subsystem: a sleeping lock that busy-waits.

The reasoning is a straightforward cost comparison. Going to sleep costs two context switches plus the latency of being woken and rescheduled — call it a microsecond or more. Spinning costs one CPU doing nothing, for however long you spin. If the lock will be released within that microsecond, spinning wins outright. And there is a cheap, surprisingly good predictor of “will be released soon”: is the owner currently executing on a CPU? A running owner is making progress toward its own mutex_unlock(). A sleeping owner is not, and might not for milliseconds.

That predictor is owner_on_cpu(), defined in include/linux/sched.h (v6.12) and gated on CONFIG_SMP:

static inline bool owner_on_cpu(struct task_struct *owner)
{
	/*
	 * As lock holder preemption issue, we both skip spinning if
	 * task is not on cpu or its cpu is preempted
	 */
	return READ_ONCE(owner->on_cpu) && !vcpu_is_preempted(task_cpu(owner));
}

Two conditions, and the second one matters more than it looks. owner->on_cpu is the scheduler’s flag for “this task is currently running.” vcpu_is_preempted(task_cpu(owner)) asks the hypervisor whether the virtual CPU that the owner is nominally running on has itself been descheduled by the host. Inside a VM, a task can be on_cpu from the guest’s point of view while its vCPU has not been scheduled by the hypervisor for milliseconds — spinning on such an “owner” burns guest CPU to no purpose. This is the same lock-holder-preemption hazard that makes x86’s virt_spin_lock() abandon fair queueing inside guests; see Kernel Spinlocks.

The spin loop itself, mutex_spin_on_owner(), is deliberately careful about what it is allowed to touch:

static noinline
bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner, ...)
{
	bool ret = true;
 
	lockdep_assert_preemption_disabled();
 
	while (__mutex_owner(lock) == owner) {
		barrier();
		if (!owner_on_cpu(owner) || need_resched()) {
			ret = false;
			break;
		}
		...
		cpu_relax();
	}
	return ret;
}

The comment above it is blunt about the hazard: “Look out! ‘owner’ is an entirely speculative pointer access and not reliable.” The spinner is dereferencing a task_struct it does not hold a reference to, and that task could in principle exit and have its memory freed mid-loop. The safety argument is a use of RCU-by-preemption: preemption is already disabled around the spin (asserted by lockdep_assert_preemption_disabled()), and on a non-PREEMPT_RT kernel a preempt-disabled region is an RCU read-side critical section, so task_struct freeing — which goes through call_rcu — cannot complete while the spinner is inside it. The source states exactly this: “we already disabled preemption which is equal to the RCU read-side crital section in optimistic spinning code. Thus the task_strcut structure won’t go away during the spinning period” (typos in the original). See Read-Copy-Update Fundamentals for why a preempt-disabled section is a grace-period barrier. The barrier() before the check exists to force the compiler to re-load lock->owner and to sequence the owner->on_cpu dereference after confirming lock->owner still equals owner.

The loop exits false — give up, go to sleep — on either of two conditions, and both are correctness-relevant rather than tuning knobs. !owner_on_cpu(owner) means the bet has failed: the owner has gone off-CPU, so waiting for it to release is now open-ended. need_resched() means the scheduler wants this CPU for something else, quite possibly something more important than this waiter, and continuing to spin would be a latency violation. There is no spin-count limit at all — the loop is bounded by events, not by iterations.

flowchart TB
  A["fastpath cmpxchg failed —<br/>someone owns the mutex"] --> B{"waiter == NULL?<br/>(i.e. not yet on the wait list)"}
  B -->|"yes"| C{"mutex_can_spin_on_owner():<br/>need_resched()? owner off-CPU?"}
  C -->|"no — do not spin"| Z["SLOWPATH: take wait_lock,<br/>enqueue, sleep"]
  C -->|"yes — worth spinning"| D{"osq_lock(&lock->osq)"}
  D -->|"failed (preempted out<br/>of the spinner queue)"| Z
  D -->|"won — I am THE spinner"| E
  B -->|"no — I am the first waiter,<br/>spinning after a wakeup"| E["spin loop"]
  E --> F{"__mutex_trylock_or_owner()"}
  F -->|"returns NULL — GOT THE LOCK"| W["HELD"]
  F -->|"returns owner"| G{"mutex_spin_on_owner()"}
  G -->|"lock->owner changed"| F
  G -->|"!owner_on_cpu(owner)<br/>owner descheduled or<br/>its vCPU preempted"| H["osq_unlock(); bail"]
  G -->|"need_resched()"| H
  H --> I{"need_resched()?"}
  I -->|"yes"| J["__set_current_state(TASK_RUNNING)<br/>schedule_preempt_disabled()<br/><i>yield FIRST, then retry</i>"]
  I -->|"no"| Z
  J --> Z

The optimistic-spinning midpath in kernel/locking/mutex.c (v6.12). What it shows: two gates guard entry — a cheap pre-check (mutex_can_spin_on_owner()) that avoids paying for the spinner queue when spinning is obviously pointless, and osq_lock(), an MCS queue that admits exactly one designated spinner. The insight to take: the osq (optimistic spin queue) exists to prevent a stampede. Without it, every waiter would poll lock->owner simultaneously and the mutex would reproduce precisely the cacheline-bouncing pathology that queued spinlocks were invented to eliminate; mutex-design.rst says so directly — “the spinners need to take a MCS (queued) lock first before spinning on the owner field.” Note also the bottom-right path: on bailing out because of need_resched(), the code calls schedule_preempt_disabled() before re-attempting the lock, “[t]his avoids getting scheduled out right after we obtained the mutex” — grabbing a lock and then immediately being preempted while holding it is worse than yielding first.

There is one further refinement that the MCS spinner queue gets, tailored for a sleeping lock: spinners can leave the queue. mutex-design.rst calls it out as “[a]n important feature of the customized MCS lock… that it has the extra property that spinners are able to exit the MCS spinlock queue when they need to reschedule. This further helps avoid situations where MCS spinners that need to reschedule would continue waiting to spin on mutex owner, only to go directly to slowpath upon obtaining the MCS lock.” A plain MCS lock has no cancel operation; osq_lock()/osq_unlock() in kernel/locking/osq_lock.c add one, and osq_lock() returning false is exactly “I was cancelled, go to the slowpath.”

One structural contrast with the queued spinlock is worth noticing, because it shows how much the no-interrupt-context rule buys. A qspinlock needs a per-CPU array of four queue nodes, one for each nesting level (task, softirq, hardirq, NMI), because a spinlock can be taken from any of them. The optimistic spin queue needs only one node per CPU: static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node);, justified in kernel/locking/osq_lock.c (v6.12) as “safe because sleeping locks should not be called from interrupt context and we have preemption disabled while spinning.” The tail is likewise a single atomic_t holding an encoded CPU number, with OSQ_UNLOCKED_VAL of 0 and encode_cpu(cpu_nr) returning cpu_nr + 1 — the same “+1 so that zero means empty” trick the qspinlock tail uses. The full mechanics of the queue, including the unlink protocol that makes cancellation race-free, belong to Adaptive Mutex Spinning and Optimistic Spinning, which this note defers to rather than duplicating.

Mechanical walk-through: the contended slow path

When the fast path fails, __mutex_lock_common() runs. It first gives the midpath a chance — mutex_optimistic_spin(), which busy-waits while the current owner is actively running on another CPU on the bet that it will release the lock momentarily (covered in full in Adaptive Mutex Spinning and Optimistic Spinning). If spinning does not win the lock, the task must genuinely block. Here is the core of the sleeping path (abridged from v6.12 __mutex_lock_common):

	raw_spin_lock(&lock->wait_lock);          /* protect the wait_list */
	if (__mutex_trylock(lock))                /* last chance under the lock */
		goto skip_wait;
 
	waiter.task = current;
	if (!use_ww_ctx)
		__mutex_add_waiter(lock, &waiter, &lock->wait_list);  /* FIFO tail */
 
	set_current_state(state);                 /* e.g. TASK_UNINTERRUPTIBLE */
	for (;;) {
		if (__mutex_trylock(lock))
			goto acquired;
		if (signal_pending_state(state, current)) {
			ret = -EINTR;
			goto err;
		}
		raw_spin_unlock(&lock->wait_lock);
		schedule_preempt_disabled();          /* SLEEP here */
		first = __mutex_waiter_is_first(lock, &waiter);
		set_current_state(state);
		if (__mutex_trylock_or_handoff(lock, first))
			break;
		/* first waiter may optimistically spin again ... */
		raw_spin_lock(&lock->wait_lock);
	}

Step by step. raw_spin_lock(&lock->wait_lock) takes the internal spinlock so the wait list can be mutated race-free. The blocked task’s bookkeeping — struct mutex_waiter waiter; — is declared as a local variable, so it lives on the blocking task’s own kernel stack, not in heap-allocated memory. From kernel/locking/mutex.h (v6.12):

struct mutex_waiter {
	struct list_head	list;
	struct task_struct	*task;
	struct ww_acquire_ctx	*ww_ctx;
#ifdef CONFIG_DEBUG_MUTEXES
	void			*magic;
#endif
};

This stack-residency is a quiet but important design choice: there is no allocation on the contention path, and the waiter node is automatically valid for exactly as long as the task is parked inside __mutex_lock_common (its stack frame). __mutex_add_waiter() links it to the tail of wait_list (FIFO ordering, oldest waiter at the head) and, if it is the first waiter, sets MUTEX_FLAG_WAITERS:

static void
__mutex_add_waiter(struct mutex *lock, struct mutex_waiter *waiter,
		   struct list_head *list)
{
	debug_mutex_add_waiter(lock, waiter, current);
 
	list_add_tail(&waiter->list, list);
	if (__mutex_waiter_is_first(lock, waiter))
		__mutex_set_flag(lock, MUTEX_FLAG_WAITERS);
}

Then set_current_state(state) marks the task as no longer runnable — for plain mutex_lock the state is TASK_UNINTERRUPTIBLE, meaning signals will not wake it. The loop then drops the wait_lock (raw_spin_unlock) and calls schedule_preempt_disabled(), which invokes the scheduler to context-switch away. This is the sleep. The CPU goes off to run other tasks; this task does not run again until something wakes it. When it does wake, it loops back, re-takes the lock state, and tries __mutex_trylock_or_handoff — succeeding either because it won a fresh cmpxchg or because the unlocker handed the lock directly to it. Note this is the mutex’s own wait-list-plus-set_current_state machinery; it conceptually mirrors but does not use a generic wait_queue_head_t (compare Wait Queues and Task Blocking). The mutex hand-rolls its blocking against the scheduler’s TASK_* states directly, which is why it can implement owner-tracking, optimistic spinning, and handoff that a generic wait queue could not.

The unlock slow path performs the matching wakeup:

static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigned long ip)
{
	struct task_struct *next = NULL;
	DEFINE_WAKE_Q(wake_q);
	unsigned long owner;
	...
	owner = atomic_long_read(&lock->owner);
	for (;;) {
		MUTEX_WARN_ON(__owner_task(owner) != current);   /* only owner unlocks */
		MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP);
		if (owner & MUTEX_FLAG_HANDOFF)
			break;
		if (atomic_long_try_cmpxchg_release(&lock->owner, &owner,
						    __owner_flags(owner))) {
			if (owner & MUTEX_FLAG_WAITERS)
				break;
			return;                                  /* no waiters: done */
		}
	}
 
	raw_spin_lock(&lock->wait_lock);
	debug_mutex_unlock(lock);
	if (!list_empty(&lock->wait_list)) {
		struct mutex_waiter *waiter =
			list_first_entry(&lock->wait_list,
					 struct mutex_waiter, list);
		next = waiter->task;
		wake_q_add(&wake_q, next);
	}
	if (owner & MUTEX_FLAG_HANDOFF)
		__mutex_handoff(lock, next);
	raw_spin_unlock(&lock->wait_lock);
	wake_up_q(&wake_q);
}

The cmpxchg_release loop clears the owner pointer portion of the word while preserving the flag bits (__owner_flags(owner)); the MUTEX_WARN_ON(__owner_task(owner) != current) is the runtime assertion that only the owner may unlock. Once the lock is released, the unlocker takes wait_lock, plucks the head of the FIFO with list_first_entry(&lock->wait_list, ...), and stages that task into a wake_q via wake_q_add. The actual wake_up_q(&wake_q) — which transitions the parked task back to TASK_RUNNING and puts it on a run queue — is deliberately done after releasing wait_lock. Batching wakeups into a wake_q and firing them outside the lock avoids holding the internal spinlock across the (relatively expensive) scheduler wakeup, reducing the time other CPUs spend spinning on wait_lock.

---

sequenceDiagram
    autonumber
    participant A as Task A (holder)
    participant O as owner word
    participant WL as wait_lock + wait_list
    participant B as Task B (contender)
    participant S as Scheduler
    A->>O: cmpxchg 0 -> A — fastpath, holds the lock
    B->>O: cmpxchg fails
    B->>B: mutex_optimistic_spin() — A is on-CPU, so spin
    Note over A: A blocks on I/O and goes off-CPU
    B->>B: owner_on_cpu(A) now false, so bail out of the spin
    B->>WL: raw_spin_lock(&lock->wait_lock)
    B->>WL: __mutex_add_waiter(&waiter) at the TAIL
    Note over WL: waiter lives on B's OWN KERNEL STACK — no allocation
    WL->>O: __mutex_set_flag(MUTEX_FLAG_WAITERS)
    B->>B: set_current_state(TASK_UNINTERRUPTIBLE)
    B->>WL: raw_spin_unlock(&lock->wait_lock)
    B->>S: schedule_preempt_disabled() — B SLEEPS here
    Note over A: A finishes its critical section
    A->>O: __mutex_unlock_fast(): cmpxchg A -> 0 FAILS<br/>(WAITERS bit is set, so the bare compare misses)
    A->>O: slowpath: cmpxchg_release, keeping the flag bits
    A->>WL: raw_spin_lock(&lock->wait_lock)
    A->>WL: list_first_entry(&wait_list) -> B
    A->>A: wake_q_add(&wake_q, B) — STAGED, not yet woken
    A->>WL: raw_spin_unlock(&lock->wait_lock)
    A->>S: wake_up_q(&wake_q) — the wakeup happens OUTSIDE wait_lock
    S->>B: B becomes runnable, is scheduled
    B->>O: __mutex_trylock_or_handoff() succeeds
    Note over O: owner = B

Two tasks contending, traced through __mutex_lock_common() and __mutex_unlock_slowpath() at v6.12. What it shows: the transition from spinning to sleeping is triggered by the owner’s behaviour (A going off-CPU at step 4), not by a timeout on B’s side. The insight to take: look at steps 18–21. The unlocker stages the wakeup into a wake_q while holding wait_lock, then releases wait_lock, and only then calls wake_up_q(). The wakeup — which touches run queues and can be expensive — is deliberately performed outside the mutex’s internal spinlock, so other CPUs contending for wait_lock are not stalled behind a scheduler operation. Note also step 15: the fast unlock does not “check whether there are waiters”; it simply fails, because the WAITERS bit changed the word out from under the compare value. The obligation and the release are detected by the same instruction.

Handoff and pickup: bounding the wait

The HANDOFF/PICKUP pair deserves its own explanation, because it is the mutex’s answer to a real starvation hazard. The default unlock path returns the word to 0 and wakes the head of the FIFO. But waking a task is not the same as giving it the lock: the woken task must be scheduled, and in the interval between “lock is free” and “woken task actually runs,” any other CPU — including the one that just unlocked, which has the cache line hot, and any optimistic spinner, which is by definition already running — can take it. Under sustained contention a queued waiter can lose that race indefinitely. This is lock stealing, and it is good for throughput and terrible for tail latency.

The handoff protocol converts a fair-ish FIFO into a guaranteed one when the first waiter demands it. __mutex_trylock_or_handoff(lock, first) is called by a waiter that has woken up and is the first in line; when it cannot get the lock it sets MUTEX_FLAG_HANDOFF, which is a message to whoever unlocks next. That unlocker sees the flag and takes a different exit:

	owner = atomic_long_read(&lock->owner);
	for (;;) {
		MUTEX_WARN_ON(__owner_task(owner) != current);
		MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP);
 
		if (owner & MUTEX_FLAG_HANDOFF)
			break;                        /* do NOT clear the owner */
		...
	}
	...
	if (owner & MUTEX_FLAG_HANDOFF)
		__mutex_handoff(lock, next);

and __mutex_handoff() writes the waiter’s task_struct pointer into the owner word with MUTEX_FLAG_PICKUP set:

		new = (owner & MUTEX_FLAG_WAITERS);
		new |= (unsigned long)task;
		if (task)
			new |= MUTEX_FLAG_PICKUP;
 
		if (atomic_long_try_cmpxchg_release(&lock->owner, &owner, new))
			break;

From that instant the lock is not free — it belongs to a task that has not run yet, so no spinner and no racing CPU can take it. When the designated waiter finally runs, __mutex_trylock_common() recognises the situation: it sees a non-zero owner with PICKUP set, checks task != curr and refuses if the pointer is not itself, and otherwise clears PICKUP and returns success. The function’s own comment records the memory-ordering contract: __mutex_handoff() “[p]rovides RELEASE semantics like a regular unlock, the __mutex_trylock() provides a matching ACQUIRE semantics for the handoff” — so the handoff is a proper release/acquire pair between two different tasks, exactly like a normal unlock/lock, and the critical-section writes of the previous owner are visible to the new one. See Acquire Release and Fence Semantics for why that pairing is what makes the data safe to read.

The two MUTEX_WARN_ON lines in the unlock loop are the runtime enforcement of two documented rules: __owner_task(owner) != current fires if a task tries to unlock a mutex it does not own, and owner & MUTEX_FLAG_PICKUP fires if an unlock happens while a handoff is still awaiting pickup — a state that should be unreachable.

Initialization and the API

A mutex must be initialized before use — either statically at compile time or dynamically at run time. Static initialization uses DEFINE_MUTEX, which expands through __MUTEX_INITIALIZER (v6.12 include/linux/mutex.h):

#define __MUTEX_INITIALIZER(lockname) \
		{ .owner = ATOMIC_LONG_INIT(0) \
		, .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(lockname.wait_lock) \
		, .wait_list = LIST_HEAD_INIT(lockname.wait_list) \
		__DEBUG_MUTEX_INITIALIZER(lockname) \
		__DEP_MAP_MUTEX_INITIALIZER(lockname) }
 
#define DEFINE_MUTEX(mutexname) \
	struct mutex mutexname = __MUTEX_INITIALIZER(mutexname)

So DEFINE_MUTEX(my_lock); produces a fully-initialized file-scope mutex whose owner is 0 (free), whose internal wait_lock is unlocked, and whose wait_list is an empty list head. For a mutex embedded in a dynamically allocated structure, you instead call mutex_init() at run time:

#define mutex_init(mutex)						\
do {									\
	static struct lock_class_key __key;				\
	__mutex_init((mutex), #mutex, &__key);				\
} while (0)

The static struct lock_class_key __key; is per-call-site and gives lockdep a stable identity for this class of lock; __mutex_init() zeroes the owner (atomic_long_set(&lock->owner, 0)), initializes wait_lock and wait_list, and (under CONFIG_MUTEX_SPIN_ON_OWNER) the osq spinner queue. A typical driver pattern:

struct my_device {
	struct mutex	state_lock;   /* serializes config changes */
	int		power_state;
};
 
static int my_probe(struct platform_device *pdev)
{
	struct my_device *dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
 
	mutex_init(&dev->state_lock);   /* MUST init before first lock */
	...
}
 
static int my_set_power(struct my_device *dev, int state)
{
	mutex_lock(&dev->state_lock);   /* may sleep — fine here, process context */
	dev->power_state = state;       /* the critical section */
	mutex_unlock(&dev->state_lock); /* same task that locked unlocks */
	return 0;
}

The core API, declared in include/linux/mutex.h (v6.12):

extern void mutex_lock(struct mutex *lock);
extern int __must_check mutex_lock_interruptible(struct mutex *lock);
extern int __must_check mutex_lock_killable(struct mutex *lock);
extern void mutex_lock_io(struct mutex *lock);
extern int mutex_trylock(struct mutex *lock);
extern void mutex_unlock(struct mutex *lock);
extern bool mutex_is_locked(struct mutex *lock);

The variants differ in how they sleep:

• mutex_lock() sleeps in TASK_UNINTERRUPTIBLE. Nothing — not even kill -9 — wakes the task until the lock is acquired. Use only when the wait is genuinely bounded.
• mutex_lock_interruptible() sleeps in TASK_INTERRUPTIBLE; any delivered signal aborts the wait and the function returns -EINTR without the lock. Its kerneldoc reads: “If a signal is delivered while the process is sleeping, this function will return without acquiring the mutex.” This is the correct choice when a userspace process is blocking on the lock through a syscall — a user pressing Ctrl-C should be able to bail out.
• mutex_lock_killable() is the middle ground: only a fatal signal (one that will kill the process anyway) aborts the wait, returning -EINTR. Routine signals are ignored. This lets long uninterruptible waits still be killed without exposing them to spurious EINTR from ordinary signals.
• mutex_lock_io() acquires the mutex exactly like mutex_lock(), but wraps the wait in io_schedule_prepare() / io_schedule_finish() so the scheduler accounts the blocked time as I/O wait — it shows up in iowait statistics and can influence CPU-frequency and load-balancing decisions. Use it when blocking on this mutex is effectively waiting for I/O to complete.

The two functions that do not sleep: mutex_trylock() attempts __mutex_trylock once and returns true/false immediately — useful when you have a fallback path and cannot afford to block. mutex_is_locked() reports whether the lock is currently held; it is advisory only (the answer can change the instant it returns) and is mostly for assertions and debugging.

Both mutex_lock_interruptible() and mutex_lock_killable() are marked __must_check: the compiler warns if you ignore the return value, because ignoring -EINTR means proceeding into the critical section without actually holding the lock — a classic and dangerous bug.

Entry point	Sleeps as	Aborted by	Returns	Reach for it when
mutex_lock()	TASK_UNINTERRUPTIBLE	nothing	void	the wait is short and bounded by kernel work only
mutex_lock_interruptible()	TASK_INTERRUPTIBLE	any delivered signal	0 or -EINTR, __must_check	a userspace process is blocking here through a syscall and Ctrl-C must work
mutex_lock_killable()	TASK_KILLABLE	only fatal signals	0 or -EINTR, __must_check	the wait may be long, ordinary signals must not cause spurious EINTR, but kill -9 must still work
mutex_lock_io()	TASK_UNINTERRUPTIBLE	nothing	void	blocking on this mutex is effectively waiting for I/O — wraps the wait in io_schedule_prepare()/io_schedule_finish() so the time is accounted as iowait
mutex_trylock()	does not sleep	n/a	1 on success, 0 on contention	you have a useful fallback and cannot block. Not usable from interrupt context
mutex_lock_nested(lock, subclass)	TASK_UNINTERRUPTIBLE	nothing	void	you legitimately hold two locks of the same class; tells lockdep the ordering is intentional
atomic_dec_and_mutex_lock(cnt, lock)	TASK_UNINTERRUPTIBLE	nothing	1 if it took the lock	“decrement this refcount and, only if it hit zero, take the mutex” — as one operation
mutex_is_locked()	n/a	n/a	bool, advisory only	assertions and debug output; the answer can change the instant it returns

The full struct mutex entry-point surface at v6.12, from include/linux/mutex.h and the kerneldoc in kernel/locking/mutex.c. What it shows: the variants differ in exactly one axis — what, if anything, is allowed to terminate the wait early — plus two non-blocking outliers. The insight to take: the third column is the whole design. mutex_lock() is the default because most kernel waits are bounded by other kernel work, but the moment a user process can be parked on that lock through a syscall, TASK_UNINTERRUPTIBLE becomes a process that cannot be killed and a D state in ps — which is why _interruptible and _killable exist and why both force you to handle the failure. mutex_lock_killable() is the pragmatic middle ground people reach for too rarely.

mutex_trylock() and why it is not usable from interrupt context

mutex_trylock() never sleeps — it performs one __mutex_trylock() and returns — so it is tempting to reach for it from a timer callback or an interrupt handler where a blocking acquire would be illegal. The kerneldoc in kernel/locking/mutex.c (v6.12) forbids it outright:

 * NOTE: this function follows the spin_trylock() convention, so
 * it is negated from the down_trylock() return values! Be careful
 * about this when converting semaphore users to mutexes.
 *
 * This function must not be used in interrupt context. The
 * mutex must be released by the same task that acquired it.

The two sentences are one argument. A mutex has strict owner semantics — locktypes.rst (v6.12) states the rule for the whole family: “The context (task) that acquired the lock must release it” — and the owner is recorded as current’s task_struct pointer in the lock word itself. An interrupt handler runs in the context of whatever task happened to be running; it has a current, but that task did not ask for the lock and has no relationship to the critical section. So a successful mutex_trylock() in a handler stamps an arbitrary victim task as the owner, and the corresponding mutex_unlock() — which asserts __owner_task(owner) != current — must then happen in that same interrupt-borrowed context, which is exactly the constraint that makes the pattern unusable in practice. Worse, if the handler interrupted a task that was itself in the middle of the mutex’s slow path, the lock’s internal wait_lock may already be held by the interrupted code, and the trylock’s debug and lockdep paths can deadlock against it.

locktypes.rst gives the general form of the warning, and the reason it stops short of an absolute prohibition: “Although implementations allow try_lock() from other contexts, it is necessary to carefully evaluate the safety of unlock() as well as of try_lock(). Furthermore, it is also necessary to evaluate the debugging versions of these primitives. In short, don’t acquire sleeping locks from other contexts unless there is no other option.” Treat that as “no.” If you need a lock in interrupt context, you need a spinlock.

Note also the first sentence of the kerneldoc, which is a genuine footgun during refactors: mutex_trylock() returns 1 on success, following the spin_trylock() convention, while the semaphore down_trylock() it replaced returns 0 on success. Converting a semaphore user to a mutex without inverting that test compiles cleanly and inverts the locking.

Scope-based acquisition with guard(mutex)

v6.12’s linux/cleanup.h machinery gives the mutex compiler-enforced scope-bound release. include/linux/mutex.h declares three guard classes:

DEFINE_GUARD(mutex, struct mutex *, mutex_lock(_T), mutex_unlock(_T))
DEFINE_GUARD_COND(mutex, _try, mutex_trylock(_T))
DEFINE_GUARD_COND(mutex, _intr, mutex_lock_interruptible(_T) == 0)

used as:

static int my_set_power(struct my_device *dev, int state)
{
	guard(mutex)(&dev->state_lock);      /* unlocked at end of scope */
 
	if (!dev->present)
		return -ENODEV;              /* early return still unlocks */
	dev->power_state = state;
	return 0;
}
 
static int my_try(struct my_device *dev)
{
	/* scoped_guard SILENTLY SKIPS the body if the acquire fails */
	scoped_guard(mutex_intr, &dev->state_lock) {
		dev->count++;
	}
	return 0;
}
 
static int my_try_checked(struct my_device *dev)
{
	/* scoped_cond_guard runs the _fail expression instead */
	scoped_cond_guard(mutex_intr, return -EINTR, &dev->state_lock) {
		dev->count++;
	}
	return 0;
}

This removes the single most common mutex bug — a return path that forgets mutex_unlock(). But note the difference between the last two forms, because it is a trap: include/linux/cleanup.h (v6.12) documents that for scoped_guard, “for conditional locks the loop body is skipped when the lock is not acquired,” which silently converts a signal-interrupted acquire into “the critical section did not run and nobody was told.” scoped_cond_guard(_name, _fail, args...) is the variant that “does fail when the lock acquire fails” — it evaluates the _fail expression instead. The header itself says plain guard() is “not recommended for conditional locks.” None of this changes the semantics of the lock: a guard(mutex) region is still process-context-only and still sleeps when contended.

---

ww_mutex: The Same Mutex, With a Deadlock-Avoidance Protocol Bolted On

A plain mutex is only safe under a global lock ordering that every acquirer respects. There is a class of problem where no such ordering exists, and the graphics stack is the canonical case: a GPU command submission must reserve every buffer object the batch touches, and the set and order of those buffers “is directly under control of userspace, and a result of the sequence of GL calls that an application makes” (Documentation/locking/ww-mutex-design.rst, v6.12). Two processes submitting overlapping batches in opposite orders produce an unavoidable AB-BA deadlock with ordinary mutexes.

struct ww_mutex solves it by importing a deadlock-avoidance protocol from database transaction theory. Each acquire context — one attempt to lock a whole set — draws a monotonically increasing stamp from a ww_class counter, and on contention the protocol deterministically picks a loser based on stamp age. The loser gets -EDEADLK, drops every lock it holds, and retries. Because the oldest transaction never loses, forward progress is guaranteed.

What matters here, in a note about struct mutex, is how little of the mutex changes to support it. ww_mutex “currently encapsulates a struct mutex, this means no extra overhead for normal mutex locks, which are far more common.” The entire integration is one boolean threaded through __mutex_lock_common():

	if (!use_ww_ctx) {
		/* add waiting tasks to the end of the waitqueue (FIFO): */
		__mutex_add_waiter(lock, &waiter, &lock->wait_list);
	} else {
		/*
		 * Add in stamp order, waking up waiters that must kill
		 * themselves.
		 */
		ret = __ww_mutex_add_waiter(&waiter, lock, ww_ctx);
		if (ret)
			goto err_early_kill;
	}

That is the whole structural difference: the wait list stops being FIFO and becomes stamp-ordered. The documented invariants are “(1) Waiters with an acquire context are sorted by stamp order; waiters without an acquire context are interspersed in FIFO order” and “(2) For Wait-Die, among waiters with contexts, only the first one can have other locks acquired already.” Everything else — the owner word, the fast path, the optimistic-spinning midpath, the handoff protocol — is the machinery already described above, unmodified. The two additional hooks in the slowpath (__ww_mutex_check_kill() before sleeping, __ww_mutex_check_waiters() after acquiring) are the points where a transaction discovers it has been wounded.

sequenceDiagram
    autonumber
    participant T1 as Txn 1 (stamp 100, OLDER)
    participant A as ww_mutex A
    participant B as ww_mutex B
    participant T2 as Txn 2 (stamp 200, YOUNGER)
    Note over T1,T2: ww_acquire_init() draws a stamp from the ww_class counter
    T1->>A: ww_mutex_lock(A, ctx1) — OK
    T2->>B: ww_mutex_lock(B, ctx2) — OK
    T2->>A: ww_mutex_lock(A, ctx2) — held by OLDER T1
    Note over T2: Wait-Die: younger contender DIES
    A-->>T2: -EDEADLK
    T1->>B: ww_mutex_lock(B, ctx1) — held by YOUNGER T2
    Note over T1: Wait-Die: older contender WAITS (T2 will release)
    T2->>B: ww_mutex_unlock(B) — dropping ALL held locks
    T2->>A: ww_mutex_lock_slow(A, ctx2) — block on the contending lock
    B-->>T1: acquired
    Note over T1: T1 completes its transaction, keeping stamp 100
    T1->>A: ww_mutex_unlock(A)
    T1->>B: ww_mutex_unlock(B)
    A-->>T2: acquired — T2 restarts its loop from A

A wait/wound cycle under the Wait-Die algorithm. What it shows: the AB-BA deadlock is broken not by ordering the locks but by ordering the transactions — the younger one is unconditionally sacrificed, so the cycle can never close. The insight to take: the backoff at step 10 must drop every lock the transaction holds, not just the contended one, and the retry re-uses the same stamp — ww_acquire_init() is called once, outside the retry loop. A transaction that re-drew its stamp on each retry could be starved forever by a stream of newer arrivals; keeping the stamp is what converts “eventually someone wins” into “the oldest always wins.” Step 11 uses ww_mutex_lock_slow() rather than plain ww_mutex_lock(): semantically it is the same call, but it returns void instead of __must_check int and, under full debug, verifies that all other locks really were released.

	Wait-Die (DEFINE_WD_CLASS)	Wound-Wait (DEFINE_WW_CLASS)
Contender is younger than holder	contender waits	contender wounds the holder, asking it to die
Contender is older than holder	contender dies (backs off, -EDEADLK)	contender waits
Who backs off	the contender, reactively	the holder, preemptively
Number of backoffs	typically more	typically fewer
Work per backoff	less	more
Preemptive?	no	yes — needs a reliable way to pick up the wounded condition
In-tree user	dma-resv (DEFINE_WD_CLASS(reservation_ww_class))	DRM modesetting (DEFINE_WW_CLASS(crtc_ww_class))

The two algorithms a ww_class can select, per ww-mutex-design.rst (v6.12), with the in-tree users verified by reading drivers/dma-buf/dma-resv.c and drivers/gpu/drm/drm_modeset_lock.c at that tag. What it shows: both are provably fair — “a transaction will eventually succeed” under either — and they differ only in who takes the corrective action. The insight to take: the doc’s rule of thumb is “use Wound-Wait iff you expect the number of simultaneous competing transactions to be typically small, and you want to reduce the number of rollbacks.” That the kernel’s two flagship ww_mutex users made opposite choices is the most instructive fact here: buffer reservation (many concurrent submitters, cheap retry) picks Wait-Die; modesetting (few concurrent atomic commits, expensive state recomputation) picks Wound-Wait. Wound-Wait’s preemption is implemented lazily — “[t]he wounded status of the transaction is checked only when there is contention for a new lock and hence a true chance of deadlock,” which lets the wounded transaction identify a lock worth blocking on before restarting.

The full treatment — the ww_acquire_ctx fields, the three canonical acquisition idioms from the design document, -EALREADY for duplicate entries, and the CONFIG_DEBUG_WW_MUTEX_SLOWPATH deadlock injector that randomly returns -EDEADLK to exercise your retry path — is in Wound-Wait Mutexes. This note deliberately stops at the structural relationship: a ww_mutex is a struct mutex with a stamp-ordered wait list.

lockdep Annotations and Nested Locking

Every struct mutex built with CONFIG_DEBUG_LOCK_ALLOC carries a struct lockdep_map dep_map, and mutex_init() gives it a stable identity:

#define mutex_init(mutex)						\
do {									\
	static struct lock_class_key __key;				\
	__mutex_init((mutex), #mutex, &__key);				\
} while (0)

The static struct lock_class_key __key is per call site, not per lock, and that is the whole point. Lockdep reasons about lock classes, not instances: “[a] class of locks is a group of locks that are logically the same with respect to locking rules, even if the locks may have multiple (possibly tens of thousands of) instantiations. For example a lock in the inode struct is one class, while each inode has its own instantiation of that lock class” (Documentation/locking/lockdep-design.rst, v6.12). Because every struct my_device initialised at one mutex_init() call site shares a key, an ordering violation observed between any two devices is reported, even though those two specific mutexes may never have raced.

That aggregation is what makes lockdep powerful and also what creates its one recurring false positive: code that legitimately holds two locks of the same class at once. lockdep-design.rst gives the canonical example — a whole-disk block device and one of its partitions, where “the partition is ‘part of’ the whole device and as long as one always takes the whole disk lock as a higher lock than the partition lock, the lock ordering is fully correct. The validator does not automatically detect this natural ordering, as the locking rule behind the ordering is not static.”

The fix is to tell lockdep the ordering by hand, with a subclass:

  enum bdev_bd_mutex_lock_class
  {
       BD_MUTEX_NORMAL,
       BD_MUTEX_WHOLE,
       BD_MUTEX_PARTITION
  };
 
  mutex_lock_nested(&bdev->bd_contains->bd_mutex, BD_MUTEX_PARTITION);

mutex_lock_nested(lock, subclass) is defined under CONFIG_DEBUG_LOCK_ALLOC as __mutex_lock(lock, TASK_UNINTERRUPTIBLE, subclass, NULL, _RET_IP_) — identical to mutex_lock() except for the subclass argument — and “[t]he validator treats a lock that is taken in such a nested fashion as a separate (sub)class for the purposes of validation.” Without CONFIG_DEBUG_LOCK_ALLOC the subclass argument is discarded and the call is a plain mutex_lock(), so this costs nothing in production. The same treatment exists for the other variants: mutex_lock_interruptible_nested(), mutex_lock_killable_nested(), mutex_lock_io_nested(), and _mutex_lock_nest_lock(), the last of which says “I hold this outer lock, which serialises all acquisitions of the inner class” rather than assigning a numeric level.

flowchart TB
  subgraph CLASS["one lock CLASS = one static lock_class_key at one mutex_init() call site"]
    I1["dev A -> state_lock<br/>(instance)"]
    I2["dev B -> state_lock<br/>(instance)"]
    I3["dev C -> state_lock<br/>(instance)"]
  end
  CLASS --> LD["lockdep records ordering facts<br/>against the CLASS, not the instance"]
  LD --> OK{"has the reverse order<br/>ever been observed<br/>for this class pair?"}
  OK -->|"no"| PASS["record the edge, continue"]
  OK -->|"yes"| SPLAT["SPLAT: possible circular locking dependency"]
  LD --> SAME{"same class taken twice<br/>by one task?"}
  SAME -->|"unannotated"| REC["SPLAT: possible recursive locking"]
  SAME -->|"mutex_lock_nested(l, subclass)"| SUB["treated as a distinct subclass —<br/>no splat, ordering still checked"]

How lockdep sees a mutex, and where mutex_lock_nested() fits. What it shows: the class abstraction is what lets lockdep find a deadlock from a single non-deadlocking execution — it generalises from instances to classes — and mutex_lock_nested() is the escape hatch for the cases where that generalisation is wrong. The insight to take: the escape hatch is dangerous. lockdep-design.rst warns: “When changing code to use the _nested() primitives, be careful and check really thoroughly that the hierarchy is correctly mapped; otherwise you can get false positives or false negatives.” A wrong subclass annotation does not just silence a spurious warning — it teaches the validator a false ordering rule and can hide a real deadlock. Reach for it only when you can name the invariant that makes the nesting safe.

Two assertion families complete the picture, and both are worth using liberally because they compile out without lockdep: lockdep_assert_held(&lock) documents and checks that a function’s caller holds a given mutex, turning an unwritten calling convention into a runtime-verified one; and lockdep_pin_lock()/lockdep_unpin_lock() assert that a lock is not released and re-acquired across a region, catching the subtle bug where a helper drops and re-takes a lock the caller assumed was held continuously. The validator’s own internals — the dependency graph, the chain cache, and how to read a full splat — are in lockdep Runtime Lock Validator.

Failure Modes and Common Misunderstandings

The mutex enforces a strict contract, and mutex-design.rst (v6.12) lists the rules it imposes (several checked at runtime under CONFIG_DEBUG_MUTEXES): only one task holds it at a time; only the owner may unlock; multiple or recursive unlocks are forbidden; a mutex must be initialized via the API (never by memset-to-zero alone, in debug builds); a task may not exit while holding a mutex; the memory backing a held mutex must not be freed; a held mutex must not be re-initialized; and — the rule that trips people most — mutexes cannot be used in hardware/software interrupt contexts such as tasklets and timers.

Using a mutex in atomic context. This is the single most common mistake. A mutex sleeps when contended, and sleeping means calling the scheduler. You cannot call the scheduler from an interrupt handler, a softirq, a tasklet, a timer callback, or while holding a spinlock or with preemption otherwise disabled — there is no task context to schedule away from, and doing so corrupts the kernel. might_sleep() at the top of every mutex-lock entry point is the tripwire: on a CONFIG_DEBUG_ATOMIC_SLEEP kernel it prints a loud “BUG: sleeping function called from invalid context” with a stack trace. If your code can run in interrupt context, you must use a spinlock (or an _irqsave variant), not a mutex.

Unlocking from the wrong task. Because the lock word is the owner pointer, the unlock fast path compares owner against current and the slow path asserts __owner_task(owner) != current. A task that did not lock the mutex cannot validly unlock it. This rules out the classic “use a binary semaphore as a completion signal where thread A locks and thread B unlocks” idiom — for that hand-off pattern, use a completion or a semaphore, not a mutex.

Recursive locking deadlocks. Mutexes are not recursive. If a task that already holds a mutex tries to take the same mutex again, it deadlocks against itself — the fast path fails (the lock is owned), and it descends into the slow path to wait for a release that can only come from… itself. lockdep (lockdep Runtime Lock Validator) flags this immediately as a “possible recursive locking” report.

Freeing memory while the lock is (or was just) held. This is subtle and was the subject of a 2013 linux-kernel discussion led by Linus Torvalds (LWN). The naive reference-count-inside-the-object pattern —

mutex_lock(obj->lock);
dead = !--obj->refcount;
mutex_unlock(obj->lock);
if (dead)
	free(obj);

— looks safe but is not. The problem is that mutex_unlock’s slow path may still be touching lock->wait_lock and lock->wait_list (to wake a waiter) after another CPU, which was the waiter, has acquired the lock, dropped the refcount to zero, and freed obj. CPU1 then writes into freed memory. Torvalds’ conclusion: “it’s unsafe to protect reference counts inside objects with anything but spinlocks and/or atomic refcounts. Or you have to have the lock outside the object you’re protecting.” The general rule the documentation states — the memory backing a held mutex must not be freed — is really this hazard: the unlock path is not instantaneous, so the object must outlive every possible in-flight unlock.

sequenceDiagram
    autonumber
    participant C0 as CPU 0 (last-but-one ref)
    participant M as obj->lock (inside obj)
    participant C1 as CPU 1 (last ref)
    C0->>M: mutex_lock(obj->lock)
    C1->>M: mutex_lock(obj->lock) — contended, sleeps on wait_list
    C0->>C0: dead = !--obj->refcount  (still 1, so dead == false)
    C0->>M: mutex_unlock(obj->lock) — WAITERS set, so SLOWPATH
    M->>M: cmpxchg_release clears owner. LOCK IS NOW FREE.
    M->>C1: wake_up_q() — C1 becomes runnable
    C1->>M: acquires the mutex
    C1->>C1: dead = !--obj->refcount  (now 0, so dead == true)
    C1->>M: mutex_unlock(obj->lock)
    C1->>C1: kfree(obj) — obj, INCLUDING obj->lock, is gone
    Note over C0,M: MEANWHILE, C0 is still inside __mutex_unlock_slowpath():<br/>it has not yet returned from raw_spin_unlock(&lock->wait_lock)
    C0->>M: touches lock->wait_lock / lock->wait_list
    Note over M: USE-AFTER-FREE

The reference-count-inside-the-object hazard that Torvalds diagnosed on linux-kernel in 2013 (LWN). What it shows: mutex_unlock() releases the lock at step 5 but keeps touching the mutex structure through steps 6 and beyond, so the window between “another CPU can acquire this” and “this CPU is finished with the memory” is non-empty. The insight to take: the rule from mutex-design.rst states this precisely — “mutex_unlock() may access the mutex structure even after it has internally released the lock already - so it’s not safe for another context to acquire the mutex and assume that the mutex_unlock() context is not using the structure anymore.” The same document draws the contrast that makes the fix obvious: this is “in contrast with spin_unlock() [or completion_done()], which APIs can be used to guarantee that the memory is not touched by the lock implementation after spin_unlock()… releases the lock.” So the three valid fixes are: use a spinlock for the refcount, use an atomic_t refcount with no lock at all, or — Torvalds’ own phrasing — “have the lock outside the object you’re protecting.”

Assuming a mutex always sleeps. The mirror-image misconception of “a mutex is slow.” Because of the optimistic-spinning midpath, a mutex_lock() on a lock whose owner is running does not release the CPU — it busy-waits, exactly like a spinlock, for an unbounded number of iterations. This matters in two directions. If you were counting on a contended mutex_lock() to yield the CPU promptly (say, in a latency-sensitive path), it may not. And if you are profiling and see CPU time attributed to mutex_spin_on_owner() — which is deliberately marked noinline “so that this function shows up on perf profiles” — that is not a bug, that is the design working; the question to ask is whether the critical sections it is spinning on are longer than they should be.

Assuming spin_lock and mutex_lock differ on PREEMPT_RT the way they do elsewhere. On a CONFIG_PREEMPT_RT kernel — selectable on mainline x86-64, arm64 and riscv from v6.12, verified by the appearance of ARCH_SUPPORTS_RT in those architectures’ Kconfig at that tag and its absence at v6.11 — struct mutex is replaced wholesale by struct rt_mutex_base, and spinlock_t becomes an rt_mutex too. Both are then sleeping, priority-inheriting locks; the categorical “spinlocks never sleep, mutexes always might” distinction that governs non-RT code collapses to a difference of degree. What survives unchanged is raw_spinlock_t. See Kernel Spinlocks and Priority Inheritance and the RT-Mutex.

Forgetting to check the interruptible/killable return value. As noted above, ignoring -EINTR from mutex_lock_interruptible() and barging into the critical section is a real-world bug that the __must_check annotation exists to prevent.

---

Alternatives and When to Choose Them — Mutex vs. Spinlock

The pivotal question, the way a kernel reviewer would push you, is: can the lock holder sleep, and might it run in interrupt context? That single decision — not raw performance — picks the primitive.

Use a spinlock when: the code may run in interrupt/atomic context (where sleeping is illegal); the critical section is very short (a few instructions, no blocking calls); or you genuinely cannot afford the cost of a context switch. A spinlock busy-waits, so a waiter burns its CPU — acceptable only because the wait is expected to be sub-microsecond.

Use a mutex when: the code runs in process context and the critical section may sleep (it allocates with GFP_KERNEL, does I/O, copies to/from user space, or simply holds the lock long enough that spinning would waste more CPU than a context switch costs). A contended mutex releases its CPU to do useful work — eventually; because of the optimistic-spinning midpath it first behaves like a spinlock for as long as the holder is running, and only sleeps once the holder goes off-CPU. That makes the real crossover a question about the holder’s behaviour rather than about elapsed time; see Where the crossover actually is, below.

There is also a structural constraint: a spinlock critical section may not call any function that might sleep, because preemption (and possibly interrupts) are disabled while it is held. So even a “short” critical section must become a mutex the moment it needs to, say, kmalloc(GFP_KERNEL) or copy_from_user().

flowchart TB
  Q1{"Can any code path that takes this lock<br/>run in hardirq, softirq, tasklet,<br/>or timer context?"}
  Q1 -->|"yes"| SPIN["It MUST be a spinlock.<br/>Sleeping there is illegal —<br/>there is no task to schedule away from."]
  Q1 -->|"no — process context only"| Q2{"Might the critical section sleep?<br/>GFP_KERNEL allocation, I/O,<br/>copy_to_user(), another mutex"}
  Q2 -->|"yes"| MTX["It MUST be a mutex.<br/>No further analysis needed."]
  Q2 -->|"no — it genuinely cannot sleep"| Q3{"Does the lock need to be<br/>released by a DIFFERENT task<br/>than the one that took it?"}
  Q3 -->|"yes"| SEM["Not a mutex — mutexes enforce<br/>owner semantics. Use a completion<br/>(for signalling) or a semaphore."]
  Q3 -->|"no"| Q4{"Read-mostly, with readers<br/>that may sleep?"}
  Q4 -->|"yes"| RWS["rw_semaphore"]
  Q4 -->|"no"| Q5{"Is the critical section a handful<br/>of instructions with no calls?"}
  Q5 -->|"yes"| SPIN2["A spinlock is marginally cheaper<br/>and avoids the mutex's 32 bytes"]
  Q5 -->|"no / not sure"| DEF["Default to a mutex.<br/>'Unless the strict semantics of mutexes<br/>are unsuitable... always prefer them<br/>to any other locking primitive.'"]

The decision procedure, with the constraints asked before the trade-offs. What it shows: in three of the six leaves the answer is forced by context or by semantics, and performance never enters the discussion. The insight to take: the tie-breaker at the bottom is a default, and the in-tree documentation’s default is the mutex, not the spinlock — mutex-design.rst (v6.12) closes with “[u]nless the strict semantics of mutexes are unsuitable and/or the critical region prevents the lock from being shared, always prefer them to any other locking primitive.” The intuition that a spinlock is “the cheap one you should use unless you have to sleep” gets the burden of proof backwards.

Where the crossover actually is

The folk rule is “if the wait would exceed two context switches, the mutex wins.” Optimistic spinning makes that rule mostly moot in the direction people expect, and it is worth being precise about why.

Scenario	Spinlock	Mutex	Winner
Uncontended	1 atomic + preempt_disable()	1 atomic + might_sleep()	roughly a tie
Contended, holder running, releases in ~100 ns	spin ~100 ns	spin ~100 ns via the midpath, plus one osq_lock/osq_unlock pair	spinlock, marginally
Contended, holder running, holds for ~10 µs	every waiter burns 10 µs of CPU	same — the midpath spins for the whole 10 µs	tie, and both are bad: shorten the critical section
Contended, holder goes off-CPU (blocks on I/O, is preempted)	waiters burn CPU for the entire blocked duration — potentially milliseconds	midpath detects !owner_on_cpu() and sleeps; CPU does other work	mutex, by orders of magnitude
Contended, holder is on a preempted vCPU in a guest	waiters burn guest CPU until the host reschedules that vCPU	vcpu_is_preempted() detects it and sleeps	mutex
Any case where the critical section can sleep	illegal	correct	mutex — no contest

The mutex-vs-spinlock comparison, decomposed by what the holder is doing rather than by how long the critical section is. What it shows: the crossover is not really about duration — it is about whether the holder is making progress. Optimistic spinning means a mutex behaves like a spinlock precisely while spinning is the right strategy, and stops the moment it is not. The insight to take: row four is where the real difference lives, and it is not a small effect. A spinlock cannot detect that its holder has stopped running, because on a non-RT kernel the holder cannot stop running — that invariant is exactly what the no-sleeping rule buys, and exactly what a mutex gives up in exchange for the ability to detect and adapt.

Uncertain

Verify: the “~100 ns”, “~10 µs”, and “two context switches ≈ 1 µs” figures above. Reason: none of these were measured during this task, and the kernel documentation does not state a crossover point — mutex-design.rst gives no numbers, and the only per-operation costs in the tree (Documentation/kernel-hacking/locking.rst) are from a 700 MHz Pentium III. They are order-of-magnitude placeholders to make the shape of the comparison concrete. To resolve: measure with perf lock contention -Y mutex,spinlock on the workload in question, or run CONFIG_LOCK_STAT and read the wait/hold distributions from /proc/lock_stat. The rows’ relative ordering is derived from the code paths and is solid; the absolute numbers are not. uncertain

struct mutex under PREEMPT_RT

One row of the comparison changes entirely on a real-time kernel. include/linux/mutex_types.h replaces the whole struct:

#ifndef CONFIG_PREEMPT_RT
struct mutex {
	atomic_long_t		owner;
	raw_spinlock_t		wait_lock;
	...
};
#else /* !CONFIG_PREEMPT_RT */
/*
 * Preempt-RT variant based on rtmutexes.
 */
#include <linux/rtmutex.h>
 
struct mutex {
	struct rt_mutex_base	rtmutex;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lockdep_map	dep_map;
#endif
};
#endif /* CONFIG_PREEMPT_RT */

so on RT a “mutex” is a priority-inheriting RT-mutex, and locktypes.rst groups it with the other sleeping locks that RT leaves sleeping. The consequential change is not to the mutex — it already slept — but to everything around it: spinlock_t becomes a sleeping rt_mutex too, so the category boundary that this whole section is organised around dissolves. What remains sharp is raw_spinlock_t, which “is a strict spinning lock implementation in all kernels, including PREEMPT_RT kernels.” Notably, RT does not change struct semaphore, and locktypes.rst explains why in a way that doubles as an argument for mutexes: “PREEMPT_RT does not change the semaphore implementation because counting semaphores have no concept of owners, thus preventing PREEMPT_RT from providing priority inheritance for semaphores. After all, an unknown owner cannot be boosted. As a consequence, blocking on semaphores can result in priority inversion.” The mutex’s owner field — introduced for correctness checking and reused for optimistic spinning — turns out to also be the thing that makes priority inheritance possible at all. See Priority Inversion and Priority Inheritance.

Among the sleeping locks themselves: a mutex is the binary, single-owner case and is what you should reach for by default (“always prefer them,” per the docs). A semaphore is a counting primitive (allows N holders) and, in its binary form, permits the lock-by-one-task/unlock-by-another hand-off that a mutex forbids — but it lacks owner tracking and the optimistic-spinning fast path, so it is slower and weaker on debugging. A rw_semaphore is the right call when reads vastly outnumber writes and reads can proceed concurrently. The RT-mutex adds priority inheritance for real-time workloads (and is what struct mutex becomes under PREEMPT_RT). One size disadvantage worth noting: per mutex-design.rst, on x86-64 struct mutex is 32 bytes versus 24 for a semaphore and 40 for an rw_semaphore — a consideration only when embedding many of them.

Uncertain

Verify: the “32 bytes (x86-64)” figure for struct mutex. Reason: the size is taken from mutex-design.rst prose, which is not regenerated per release; the actual sizeof(struct mutex) depends on config options (CONFIG_MUTEX_SPIN_ON_OWNER, debug options) and could differ on a real 6.12 build. To resolve: build a 6.12 kernel with a standard defconfig and check pahole -C mutex or a BUILD_BUG_ON(sizeof(struct mutex) != 32) probe. uncertain

---

Production Notes

Mutexes are everywhere in the kernel — they are the default lock for any per-object or per-subsystem state that is touched only from process context. A device driver’s per-device “don’t reconfigure while I’m reconfiguring” lock, filesystem inode mutexes (i_rwsem is the rw variant; many subsystems still use plain mutexes for metadata), and config/sysfs handlers all use them. The reason is exactly the docs’ guidance: process-context code that might allocate memory or do I/O under the lock must be able to sleep, and a mutex is the cheapest way to do that safely while retaining owner tracking and lockdep coverage.

The historical performance story validates the design, and it is worth quoting precisely because the numbers are often repeated without their workload. Ingo Molnar’s original 2005 posting (LWN, Generic Mutex Subsystem) benchmarked creat+unlink+close of separate per-task files in /tmp with 16 parallel tasks on an 8-way x86 machine:

	Semaphores	Mutexes
avg loops/sec	34,713	84,153
CPU utilisation	63%	22%
ops/sec per 1% CPU	551	3,825

The measurement that justified adding a whole new locking primitive, from Molnar’s 2005 patch posting. What it shows: 2.4× the throughput at 2.8× less CPU — the third row is the honest headline, a 6.9× improvement in work per CPU cycle. The insight to take: the win came from a contended VFS workload, not from the uncontended path; Molnar’s own claim about the fast path was the opposite — “there are no fastpath tradeoffs, the mutex fastpath is just as tight as the semaphore fastpath,” at two x86 instructions each. On a 2-way P4 HT box the same test showed a smaller 41% throughput gain but still 4.1× the per-cycle efficiency, which is the signature of a scalability fix rather than a constant-factor one.

Two historical details are easy to get backwards. First, the original mutex fast path was not the cmpxchg-on-owner design described earlier in this note — it was a counting-style lock decl/lock incl pair on an integer count word, exactly like the semaphore it replaced (Molnar’s posting shows the two-instruction disassembly: lock decl (%eax) / js for lock, lock incl (%eax) / jle for unlock); the owner-pointer word and the optimistic-spinning midpath were both added later. Second, the size argument has inverted. In 2005 Molnar’s fourth-listed reason was that “‘struct mutex’ is smaller: on x86, ‘struct semaphore’ is 20 bytes, ‘struct mutex’ is 16 bytes.” Today mutex-design.rst (v6.12) opens its Disadvantages section with “[u]nlike its original design and purpose, ‘struct mutex’ is among the largest locks in the kernel. E.g: on x86-64 it is 32 bytes, where ‘struct semaphore’ is 24 bytes and rw_semaphore is 40 bytes.” Owner tracking, the osq spinner queue, and lockdep support all cost space. Do not cite the 2005 size claim about the modern struct.

Measuring and debugging a mutex in production

Three tools, in increasing order of intrusiveness.

Tracepoints, already compiled in. __mutex_lock_common() brackets its contended path with trace_contention_begin(lock, LCB_F_MUTEX) / trace_contention_end(lock, ret), and — importantly — re-emits trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN) when the first waiter drops into an optimistic-spin attempt. So the lock:contention_begin/lock:contention_end tracepoints distinguish sleeping contention from spinning contention on the same mutex, without a rebuild.

perf lock contention. Built on those tracepoints, or on BPF with -b. perf lock contention -a -b -Y mutex restricts the report to mutexes; -Y accepts mutex, rtmutex, semaphore, rwsem, rwsem:R, rwsem:W, spinlock, rwlock, pcpu-sem, and the RT variants (tools/perf/Documentation/perf-lock.txt, v6.12). -l/--lock-addr aggregates per lock instance rather than per call site, which is how you tell “one hot mutex” from “a hot mutex class spread across thousands of objects” — the distinction lockdep deliberately erases. -o/--lock-owner attributes the wait to the holder rather than the waiter and requires --use-bpf; for a mutex this actually works, because the owner is recorded in the lock word.

CONFIG_LOCK_STAT and /proc/lock_stat. A rebuild, but it gives per-class wait-time and hold-time distributions (min/max/total/avg, microseconds) plus the four hottest contention points as symbolisable instruction pointers, and con-bounces/acq-bounces counters that measure cross-CPU cache-line traffic directly. Enable with echo 1 > /proc/sys/kernel/lock_stat, clear with echo 0 > /proc/lock_stat. The number to look at first is hold time, not contention count: a mutex with a long average hold is a design problem that no amount of lock-splitting will fix. See Lock Statistics and Contention Analysis.

When the system is already stuck. echo w > /proc/sysrq-trigger dumps the stack of every task in uninterruptible sleep; tasks blocked on a mutex show __mutex_lock_slowpath / schedule_preempt_disabled in their traces, and because the mutex records its owner, CONFIG_DEBUG_MUTEXES and lockdep can name the holder rather than leaving you to guess. For a test or CI kernel the pairing to enable is CONFIG_DEBUG_MUTEXES (which “fully enforce[s]” all nine documented semantic rules at runtime) plus CONFIG_PROVE_LOCKING. Between them, recursive locking, wrong-context use, unlock-by-non-owner, and bad ordering all become loud failures long before they become a hang.

Those gains came partly from the lean fast path and partly from the later-added optimistic spinning, which lets a mutex behave almost like a spinlock under brief contention (when the owner is still running) and only fall back to sleeping when the owner itself blocks — the best of both worlds, detailed in Adaptive Mutex Spinning and Optimistic Spinning.

A practical debugging note: when a system hangs with tasks stuck, echo w > /proc/sysrq-trigger (or cat /proc/<pid>/stack) shows tasks blocked in __mutex_lock_slowpath / schedule_preempt_disabled, and the owner-tracking field means tools and lockdep can name the holder. Enabling CONFIG_DEBUG_MUTEXES and CONFIG_PROVE_LOCKING (lockdep) on a test kernel is the standard way to catch the failure modes above — recursive locking, wrong-context use, and bad ordering — long before they deadlock in production. See Lock Statistics and Contention Analysis for measuring how often a given mutex is actually contended.

A note on relatives in other runtimes: the kernel mutex is a kernel-space sleeping lock whose blocking bottoms out in the scheduler. Its userspace cousins — the Go runtime’s sync.Mutex and pthread mutexes — are userspace locks whose uncontended path is a single atomic in user memory and whose contended path parks the thread via the kernel futex syscall (see Wait Queues and Task Blocking for the kernel side of futex parking). The shapes rhyme — fast atomic path, slow park-the-waiter path, anti-starvation handoff — but the kernel struct mutex never leaves the kernel and never touches a futex; it talks to the scheduler directly.

---

See Also

• Adaptive Mutex Spinning and Optimistic Spinning — the deferred midpath: how a mutex spins while the owner is on-CPU before sleeping (osq_lock, owner-on-CPU detection)
• Kernel Spinlocks — the spin-don’t-sleep sibling; the heart of the mutex-vs-spinlock decision
• Wait Queues and Task Blocking — how blocked tasks sleep and wake in general; the mutex uses its own wait_list, not a generic wait_queue_head_t
• Kernel Semaphores — counting/binary sleeping lock; allows lock-by-A/unlock-by-B which a mutex forbids
• Read-Write Semaphores — the reader/writer sleeping lock for read-mostly state
• Priority Inheritance and the RT-Mutex — the PI-aware mutex; what struct mutex becomes under PREEMPT_RT
• Completions — the right primitive for one-shot task hand-off / event signalling
• lockdep Runtime Lock Validator — catches recursive locking, wrong-context use, and bad lock ordering
• Acquire Release and Fence Semantics / Compare-and-Swap and cmpxchg in the Kernel — the memory-ordering and atomic primitives the fast path is built on
• sync.Mutex Internals — the Go runtime’s userspace, futex-backed mutex, for comparison (not the same as a kernel mutex)
• Wound-Wait Mutexes — ww_mutex in full: the ticket/stamp protocol, the -EDEADLK retry idiom, and the DRM/dma_resv users
• Lock Statistics and Contention Analysis — /proc/lock_stat, perf lock contention -Y mutex, and how to read the numbers
• Priority Inversion and Priority Inheritance — why owner tracking is what makes priority inheritance possible, and why semaphores cannot have it
• Queued Spinlocks and qspinlock — the MCS queue the mutex’s osq spinner queue is derived from
• Cache Coherence and the Store Buffer — why an unmanaged spinner stampede on lock->owner would be expensive, and hence why osq exists
• Read-Copy-Update Fundamentals — the preempt-disabled-is-an-RCU-read-side argument that makes the midpath’s speculative task_struct dereference safe
• Linux Kernel Synchronization MOC — parent map (section C, Sleeping Locks)