A read-write semaphore (struct rw_semaphore, universally abbreviated rwsem) is the kernel’s reader/writer sleeping lock: it permits many concurrent readers XOR exactly one writer, and a contending task that cannot acquire it blocks (sleeps) rather than spins. It is the read-mostly counterpart to the plain semaphore and the sleeping counterpart to the non-sleeping Reader-Writer Spinlocks. The entire lock state — reader count, writer-held bit, waiters-present bit, and a hand-off bit — is packed into a single atomic_long_t count, manipulated with lock-free atomics on the fast path; only on contention does a task take the internal wait_lock and join wait_list. Writers get optimistic spinning (they spin while the owning writer is running on another CPU, reusing the same MCS queue as the mutex), and a HANDOFF mechanism guarantees that a waiter starved past a deadline is handed the lock directly. The headline kernel user is mmap_lock — the per-mm_struct rwsem (formerly mmap_sem) that serialises a process’s address-space changes.
The One Idea
A reader/writer lock has a design decision at its heart that a mutex does not: when a writer is waiting and a new reader arrives, who wins?
Both answers are wrong.
Let the reader in (“reader-preferring”). Throughput is excellent — readers never block each other — but a steady stream of readers means the writer’s turn never comes. Writer starvation.
Make the reader wait (“writer-preferring”). The writer is safe, but now a single slow writer blocks every reader behind it, and the reader parallelism you bought the rwsem for evaporates on every write.
Linux refuses to pick. Its rwsem is throughput-preferring up to a bounded latency, then fairness-preferring by fiat. Readers are allowed to steal the lock past queued writers, and writers are allowed to spin optimistically and steal it past queued readers — but every waiter stamps a deadline when it joins the queue, and once the first waiter blows through that deadline it sets RWSEM_FLAG_HANDOFF (bit 2 of count). From that instant, all stealing and all optimistic acquisition are forbidden: the lock must be handed directly to that waiter when it is released.
Understanding that trade-off is the whole point of this note. Everything else — the bit packing, the MCS spin queue, the reader-owned spin threshold — is machinery in service of making the fast case fast without letting the slow case become unbounded.
And the trade-off was measured, not guessed. Waiman Long’s commit that introduced HANDOFF (dated 2019-05-20) reports a microbenchmark on a 2-socket, 40-core, 80-thread Skylake with 240 writer threads and a 5 µs critical section, run for 5 seconds:
min
mean
max
Before HANDOFF
1
7,792
173,696
After HANDOFF
5,842
6,542
7,458
Lock acquisitions per thread over 5 s, from the commit message. What it shows: before the patch, the unluckiest of 240 writer threads acquired the lock once in five seconds while the luckiest acquired it 173,696 times — a 173,000× spread. After, the spread is 1.28×. The insight to take: the mean fell by about 16%, and the commit says so plainly: “there was a drop of about 16% in the mean locking operations done which was a tradeoff of having better fairness.” That is the price of the guarantee, stated by the author, in the commit that made the guarantee. A min of 1 is not “unfair”; it is a hung thread.
All code claims are pinned to v6.12 (released 2024-11-17; still maintained as the longterm branch 6.12.109 as of 2026-09-12 per kernel.org/releases.json), read from raw.githubusercontent.com/torvalds/linux/v6.12/ with line numbers from that tag. It describes the default (non-PREEMPT_RT) implementation; the PREEMPT_RT variant rebuilds rwsem on the RT-mutex and is covered at the end.
Dating the Implementation
The rwsem you read in kernel/locking/rwsem.c today is not the one that existed before 2019, and mixing the two is the most common way to be confidently wrong about this lock. The rewrite is datable precisely from two independent primary sources.
Existence check. Fetching the same paths at successive tags and recording the HTTP status:
Tag
kernel/locking/rwsem-xadd.c
kernel/locking/rwsem.c size
v5.1
HTTP 200
—
v5.2
HTTP 200
4,085 bytes
v5.3
HTTP 404
47,154 bytes
v5.4
HTTP 404
—
v6.12
HTTP 404
47,210 bytes
rwsem-xadd.c — the old implementation — disappears between v5.2 and v5.3, and in the same step rwsem.c grows from a 4 KB wrapper to a 47 KB full implementation. The rewrite landed in v5.3. Note also how little the file has changed since: 47,154 → 47,210 bytes across nine years and twelve releases. This code is stable.
locking/rwsem: Add reader-owned state to the owner field
Waiman Long
+6/−2
2017-10-10
locking/rwsem: Add down_read_killable()
Kirill Tkhai
+16
2019-06-17
locking/rwsem: Merge rwsem.h and rwsem-xadd.c into rwsem.c
Waiman Long
+884
2019-06-17
locking/rwsem: Make rwsem_spin_on_owner() return owner state
Waiman Long
+47/−18
2019-06-17
locking/rwsem: Implement lock handoff to prevent lock starvation
Waiman Long
+171/−54
2019-06-17
locking/rwsem: Always release wait_lock before waking up tasks
Waiman Long
+15/−16
2019-06-17
locking/rwsem: Wake up almost all readers in wait queue
Waiman Long
+26/−5
2019-06-17
locking/rwsem: Enable readers spinning on writer
Waiman Long
+74/−12
2019-06-17
locking/rwsem: Make rwsem->owner an atomic_long_t
Waiman Long
+80/−45
2019-06-17
locking/rwsem: Enable time-based spinning on reader-owned rwsem
Waiman Long
+143/−30
2019-06-17
locking/rwsem: Adaptive disabling of reader optimistic spinning
Waiman Long
+129/−4
2019-06-17
locking/rwsem: Guard against making count negative
Waiman Long
+41/−12
2019-07-25
locking/rwsem: Add ACQUIRE comments + two ACQUIRE fixes
Peter Zijlstra, Jan Stancek
+18/−6
The rwsem timeline. What it shows: fifteen commits by Waiman Long landed on a single day in June 2019, replacing the implementation wholesale. The LWN posting of the series is “locking/rwsem: Rwsem rearchitecture part 2”. The insight to take: anything you remember about rwsem internals from before v5.3 — a separate rwsem-xadd.c, RWSEM_ACTIVE_WRITE_BIAS, a long count with a different encoding — describes code that no longer exists. Optimistic spinning is older (Davidlohr Bueso, 2014, LWN 598577) but was reworked in the same 2019 series. HANDOFF is the newest of the core mechanisms and exists for exactly one reason: the starvation numbers above.
Mental Model
An rwsem is a turnstile with two modes. In read mode any number of tasks can be inside at once; in write mode exactly one task is inside and everyone else — readers and writers alike — waits outside. The classic justification is read-mostly data: a structure read far more often than it is written can let all the readers proceed in parallel, paying for exclusion only on the rare write.
The clever part is that “how many readers, plus is a writer in, plus is anyone waiting, plus is a hand-off pending” is encoded as a single machine word, so the common uncontended acquire is one atomic instruction on that word. You only fall back to the heavyweight wait-list machinery when the fast atomic fails.
The bit layout of count and the two fast paths. What it shows: readers acquire by addingRWSEM_READER_BIAS (1 << 8) so many readers accumulate in the high bits; a writer acquires only by cmpxchg-ing the whole word from exactly 0 to WRITER_LOCKED. The insight to take: a reader fast-path succeeds only if none of the low “fail” bits (writer-locked, waiters-present, hand-off) are set — the bias add is unconditional but the result is inspected, and a reader that lands on a set fail bit backs its bias out and takes the slow path. A writer can fast-acquire only a completely free lock.
/* include/linux/rwsem.h v6.12, L38-67 — comment and ifdefs verbatim *//* * For an uncontended rwsem, count and owner are the only fields a task * needs to touch when acquiring the rwsem. So they are put next to each * other to increase the chance that they will share the same cacheline. * * In a contended rwsem, the owner is likely the most frequently accessed * field in the structure as the optimistic waiter that holds the osq lock * will spin on owner. For an embedded rwsem, other hot fields in the * containing structure should be moved further away from the rwsem to * reduce the chance that they will share the same cacheline causing * cacheline bouncing problem. */struct rw_semaphore { atomic_long_t count; /* * Write owner or one of the read owners as well flags regarding * the current state of the rwsem. Can be used as a speculative * check to see if the write owner is running on the cpu. */ atomic_long_t owner;#ifdef CONFIG_RWSEM_SPIN_ON_OWNER struct optimistic_spin_queue osq; /* spinner MCS lock */#endif raw_spinlock_t wait_lock; struct list_head wait_list;#ifdef CONFIG_DEBUG_RWSEMS void *magic;#endif#ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map;#endif};
Note the three conditional fields. osq exists only with CONFIG_RWSEM_SPIN_ON_OWNER (=y on this host, checked in /boot/config-7.1.13-200.fc44.x86_64), so optimistic spinning is live here. magic and dep_map are debug-only and are absent on this host — CONFIG_DEBUG_RWSEMS and CONFIG_DEBUG_LOCK_ALLOC are both is not set, which matters below when we ask what can be instrumented.
Field by field. count is the heart: a single atomic_long_t (a long-sized atomic) holding the entire lock state. owner holds either the writer’s task_struct pointer (when write-locked) or the last reader’s pointer with a flag bit set — used for optimistic spinning and debugging, “take it with a grain of salt” per the source because it may be stale for reader-owned locks. osq is the optimistic spin queue, an MCS lock (the same queue type used by qspinlock and the mutex) on which spinning writers line up. wait_lock is a raw spinlock guarding the wait_list of blocked tasks. The header deliberately places count and owner adjacent so they share a cache line, since both are touched on every acquire — and the comment goes further, advising driver authors that for an embedded rwsem (one sitting inside a larger struct, as mmap_lock sits inside mm_struct) other hot fields of the container should be moved away from the rwsem, because a spinning writer hammering owner will otherwise bounce that cache line against unrelated accesses.
The count bit layout, copied verbatim from the rwsem.c comment for 64-bit (rwsem.c v6.12):
Bit 0 - writer locked bit (RWSEM_WRITER_LOCKED)
Bit 1 - waiters present bit (RWSEM_FLAG_WAITERS)
Bit 2 - lock handoff bit (RWSEM_FLAG_HANDOFF)
Bits 3-7 - reserved
Bits 8-62 - 55-bit reader count
Bit 63 - read fail bit (RWSEM_FLAG_READFAIL)
The count word at bit accuracy. What it shows:WL = RWSEM_WRITER_LOCKED, WT = RWSEM_FLAG_WAITERS, HO = RWSEM_FLAG_HANDOFF, RF = RWSEM_FLAG_READFAIL. The insight to take: the geometry is the algorithm. Putting the reader count above bit 7 means adding RWSEM_READER_BIAS never disturbs a flag, so atomic_long_add_return(RWSEM_READER_BIAS, &count) both takes the reader’s slot and reports every flag in a single instruction. Putting WRITER_LOCKED at bit 0 means a totally free lock is numerically 0, so a writer’s acquire is cmpxchg(&count, 0, 1) — the cheapest possible test for “nobody at all is here.”
Field by field, with the decisive consequence of each:
Bits
Constant
Meaning
Why it is where it is
0
RWSEM_WRITER_LOCKED (1UL << 0)
a writer holds the lock
Being bit 0 makes “completely free” equal to the integer 0, so try_cmpxchg(&count, &zero, RWSEM_WRITER_LOCKED) is the entire writer fast path.
1
RWSEM_FLAG_WAITERS (1UL << 1)
wait_list is non-empty
Included in RWSEM_READ_FAILED_MASK, so a fast-path reader that sees it must take the slow path. This single bit is what stops readers from casually jumping a queued writer.
2
RWSEM_FLAG_HANDOFF (1UL << 2)
the head waiter has timed out; no stealing allowed
Set and cleared only under wait_lock, and only by the first waiter. The fairness override.
3–7
reserved
—
Five spare bits, kept free so the reader count can start at a byte boundary (RWSEM_READER_SHIFT = 8), making RWSEM_READER_BIAS = 0x100 and reader-count extraction a plain count >> 8.
8–62
reader count (55 bits)
number of concurrent readers
RWSEM_READER_MASK = ~(RWSEM_READER_BIAS - 1). Each down_read() adds 0x100; each up_read() subtracts it.
63
RWSEM_FLAG_READFAIL (1UL << (BITS_PER_LONG - 1))
reader-count overflow guard
Being the sign bit means overflow makes count negative, which the fast path already tests for (WARN_ON_ONCE(*cntp < 0)). It costs nothing to check.
On 32-bit architectures the layout is narrower
The same source comment gives the 32-bit variant: bits 0–2 and 3–7 are identical, but the reader count is bits 8–30 (23 bits) and the read-fail bit is bit 31. The constants are written in terms of BITS_PER_LONG, so one set of code serves both. A 23-bit count still allows ~8.4 million concurrent readers, which is why the source calls overflow unlikely: “It is not likely that the most significant bit (read fail bit) will ever be set. This guard bit is still checked anyway in the down_read() fastpath just in case we need to use up more of the reader bits for other purpose in the future.”
Two derived masks matter later and are worth pinning down now, because their exact membership is what encodes the fairness policy:
RWSEM_LOCK_MASK means “somebody holds this lock, as a reader or a writer” — it covers bit 0 and the whole reader count, and it is what release paths test to decide whether they were the last holder. RWSEM_READ_FAILED_MASK means “a reader must not take the fast path” — and note what is in it that is not in LOCK_MASK: FLAG_WAITERS and FLAG_HANDOFF. Those two additions are the entirety of rwsem’s fairness policy, expressed as two bits in a bitmask. Remove FLAG_WAITERS from that mask and you have a reader-preferring lock with unbounded writer starvation.
Reading this carefully is the whole design:
Reader count lives in the high bits. Because RWSEM_READER_BIAS is 1 << 8, each reader contributes 0x100 to count. Adding the bias is the reader’s acquire; subtracting it is the release. Putting the count in the high bits means the low bits are free for flags, and an atomic_long_add_return of the bias simultaneously increments the reader count and lets the caller read back all the flag bits in one operation.
Bit 0 WRITER_LOCKED marks the word as held by a writer. A writer acquires by cmpxchg-ing the entire word from 0 to RWSEM_WRITER_LOCKED — which can only succeed if there are no readers and no flags, i.e. the lock is completely free.
Bit 1 FLAG_WAITERS means the wait_list is non-empty. A fast-path reader that sees this bit set must take the slow path, because granting it immediately could starve a queued writer.
Bit 2 FLAG_HANDOFF is the anti-starvation hand-off bit, explained in its own section below.
Bit 63 FLAG_READFAIL is a guard against reader-count overflow. With 55 reader bits the count would need ~2^55 concurrent readers to overflow into bit 63; the bit “is still checked anyway in the down_read() fastpath just in case,” per the source. If the bias add ever pushed the count negative (overflow), the read fast path fails safely.
The reader fast path tests count & RWSEM_READ_FAILED_MASK: a reader succeeds only if, after adding its bias, none of {writer-locked, waiters-present, hand-off, read-fail} is set. The presence of FLAG_WAITERS in that mask is what enforces fairness — a reader will not jump ahead of an already-queued writer on the fast path.
static inline bool rwsem_read_trylock(struct rw_semaphore *sem, long *cntp){ *cntp = atomic_long_add_return_acquire(RWSEM_READER_BIAS, &sem->count); if (WARN_ON_ONCE(*cntp < 0)) rwsem_set_nonspinnable(sem); if (!(*cntp & RWSEM_READ_FAILED_MASK)) { rwsem_set_reader_owned(sem); return true; } return false;}
The reader unconditionally adds RWSEM_READER_BIAS with acquire ordering (atomic_long_add_return_acquire) — acquire semantics ensure that reads inside the critical section cannot be hoisted above the lock acquisition. It then inspects the returned value. If none of the fail bits are set, the reader is in: it records itself as a reader-owner and returns. If a fail bit is set — a writer holds the lock, or there are waiters, or a hand-off is pending — the bias is left added for the moment and the caller drops into rwsem_down_read_slowpath(), which will back the bias out (adjustment = -RWSEM_READER_BIAS) and decide whether to queue.
The slow path has a notable optimisation: reader optimistic lock stealing. If the lock is not writer-held and no hand-off is pending, a reader can grab the lock immediately even with the slow path entered, unless the lock is “very likely owned by readers” with more than one reader already present and no writer — in which case it queues, “to prevent a constant stream of readers from starving a sleeping writer.” This is a deliberate fairness compromise: readers normally pile in freely, but once a writer is waiting, new readers stop stealing.
That prose describes the fast path. The slow path is where the fairness policy actually lives, and it is worth drawing in full, because “reader takes the slow path” does not mean “reader sleeps” — there are three separate ways a reader can still win without ever joining the queue:
flowchart TD
A["down_read()"] --> B["atomic_long_add_return_acquire<br/>(RWSEM_READER_BIAS, &count)<br/>rwsem.c L250"]
B --> C{"count < 0?<br/>(reader-count overflow)"}
C -->|yes| C2["WARN_ON_ONCE +<br/>rwsem_set_nonspinnable()"]
C -->|no| D{"count &<br/>RWSEM_READ_FAILED_MASK<br/>== 0 ?"}
C2 --> D
D -->|"clear — nobody in the way"| E["FAST PATH: rwsem_set_reader_owned()<br/>return. Total cost: one atomic add."]
D -->|"a fail bit is set"| F["rwsem_down_read_slowpath()<br/>adjustment = -RWSEM_READER_BIAS"]
F --> G{"owner is READER_OWNED<br/>AND rcnt > 1<br/>AND not WRITER_LOCKED?"}
G -->|yes| Q["queue — do NOT steal.<br/>'prevent a constant stream of readers<br/>from starving a sleeping writer'"]
G -->|no| H{"count &<br/>(WRITER_LOCKED | HANDOFF)<br/>== 0 ?"}
H -->|yes| I["READER LOCK STEALING:<br/>keep the bias, set reader-owned,<br/>lockevent rwsem_rlock_steal.<br/>Never slept."]
I --> I2{"first reader<br/>and WAITERS set?"}
I2 -->|yes| I3["rwsem_mark_wake(RWSEM_WAKE_READ_OWNED)<br/>— drag the other queued readers in too"]
I2 -->|no| Z["return, lock held for read"]
I3 --> Z
H -->|no| Q
Q --> R["waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT<br/>take wait_lock"]
R --> S{"wait_list empty<br/>AND not writer-held?"}
S -->|yes| T["FAST EXIT: bias is already in count,<br/>lockevent rwsem_rlock_fast. Never slept."]
S -->|no| U["adjustment += RWSEM_FLAG_WAITERS<br/>rwsem_add_waiter(); sleep on the wait queue"]
U --> V["woken by rwsem_mark_wake()<br/>or granted by HANDOFF"]
The complete reader acquire path. What it shows: four distinct outcomes, only one of which involves sleeping. The insight to take: the single most important node is the diamond marked “owner is READER_OWNED and rcnt > 1”. That test is the anti-starvation valve on the reader side. When the lock is already held by more than one reader and no writer holds it, a newly arriving reader is sent to the queue even though it could have taken the lock immediately — because a lock that is already reader-saturated is exactly the situation in which a queued writer would never get a turn. The kernel gives up a little reader throughput at precisely the moment reader throughput is already high.
The relevant source, which says it in one sentence:
/* kernel/locking/rwsem.c v6.12, L1001-1013 */ /* * To prevent a constant stream of readers from starving a sleeping * writer, don't attempt optimistic lock stealing if the lock is * very likely owned by readers. */ if ((atomic_long_read(&sem->owner) & RWSEM_READER_OWNED) && (rcnt > 1) && !(count & RWSEM_WRITER_LOCKED)) goto queue; /* * Reader optimistic lock stealing. */ if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF))) { rwsem_set_reader_owned(sem); lockevent_inc(rwsem_rlock_steal);
Note the asymmetry in that second test: a stealing reader checks WRITER_LOCKED | HANDOFF but notFLAG_WAITERS. So a reader in the slow path is allowed to jump a queued writer — right up until that writer’s deadline expires and it sets HANDOFF. That is the bounded-unfairness bargain in two lines of C.
A writer can only fast-acquire a completely free lock: try_cmpxchg succeeds only if count was exactly 0, swapping it to RWSEM_WRITER_LOCKED. Any reader present, any flag set, and the cmpxchg fails, sending the writer to rwsem_down_write_slowpath(). On success the writer stamps its own task_struct into owner, enabling other contenders to see who holds it and whether that holder is running.
Release
up_read() subtracts the bias with release ordering and, if it was the last reader and waiters are present, wakes them:
The condition (tmp & (LOCK_MASK | WAITERS)) == WAITERS is precise: it fires only when, after this reader leaves, there are no remaining lock holders (LOCK_MASK covers writer-locked and the reader-count bits) and the waiters bit is set — i.e. “I was the last reader and someone is queued.” Only then is a wake needed. up_write() is the mirror: it subtracts RWSEM_WRITER_LOCKED with release ordering and wakes if FLAG_WAITERS is set. Release ordering on both ensures that writes done inside the critical section are visible to the next acquirer before the lock appears free.
downgrade_write() atomically converts a held write lock into a read lock — subtracting WRITER_LOCKED and adding READER_BIAS in one fetch_add_release — which is safe and starvation-free because no other writer can sneak in during the conversion, and it lets queued readers proceed without the writer having to fully release and re-contend.
Writer Optimistic Spinning
Sleeping is expensive: two context switches plus scheduler work. If the lock is write-held but the owning writer is currently running on another CPU, it will probably release very soon, so a contending writer does better to spin than to sleep. This is the same bet the mutex’s adaptive spinning makes, and rwsem reuses the same machinery — it is gated on CONFIG_RWSEM_SPIN_ON_OWNER and lines spinners up on the osq MCS queue, the identical queue type the mutex uses.
rwsem_down_write_slowpath() opens with:
if (rwsem_can_spin_on_owner(sem) && rwsem_optimistic_spin(sem)) return sem; /* got it by spinning, never slept */
rwsem_optimistic_spin() takes the osq lock (so only one optimistic spinner contends per CPU queue position, avoiding a cache-line storm) and then loops calling rwsem_spin_on_owner(), which watches the owner field:
for (;;) { new = rwsem_owner_flags(sem, &new_flags); if ((new != owner) || (new_flags != flags)) { ...break; } /* owner changed */ barrier(); if (need_resched() || !owner_on_cpu(owner)) { /* stop spinning */ state = OWNER_NONSPINNABLE; break; } cpu_relax();}
The spinner stops the moment the owning writer is no longer running (!owner_on_cpu(owner)) or the spinner itself needs to reschedule (need_resched()) — spinning is only worth it while the holder is actively making progress on another CPU. Each loop it also retries rwsem_try_write_lock_unqueued() to grab the lock if it just became free.
A subtlety: spinning on a reader-owned lock cannot be bounded by an owner-running flag, because there is no single reader to watch — the owner field holds some past reader, possibly one that has already released. So reader-owned spinning is bounded by wall-clock time instead:
Symbol by symbol: count >> RWSEM_READER_SHIFT (i.e. >> 8) extracts the current reader count from the high bits. That count is clamped to 30. delta = (20 + readers) * NSEC_PER_USEC / 2 is (20 + readers) / 2 microseconds expressed in nanoseconds — so the budget is 10 µs with no readers, rising by 0.5 µs per reader, saturating at 25 µs from 30 readers upward. The function returns an absolute sched_clock() deadline, not a duration, and the spin loop compares against it once every 16 iterations, because the source notes that calling sched_clock() more often would itself add latency between the lock becoming free and the spinner noticing.
The reasoning is that the more readers own the rwsem, the longer it will take for them all to wind down, so a longer spin is justified — but only up to a point, since past ~25 µs sleeping is cheaper than burning a CPU. If a writer spins past the threshold on a reader-owned lock it sets RWSEM_NONSPINNABLE (bit 1 of the owner field, not of count) to disable further futile spinning on this lock, and goes to sleep. Note that the two spin-flag bits live in owner, entirely separately from the count flags: RWSEM_READER_OWNED is 1UL << 0 and RWSEM_NONSPINNABLE is 1UL << 1 of owner (rwsem.c L63-65). Conflating the owner flags with the count flags is an easy mistake — they are two different words with two different bit-0s.
Fairness and the HANDOFF Anti-Starvation Mechanism
Optimistic spinning and reader lock-stealing are great for throughput but dangerous for fairness: a steady stream of readers could starve a queued writer forever, or aggressive writer spinning could starve queued readers. The kernel resolves this with the HANDOFF protocol (RWSEM_FLAG_HANDOFF, bit 2).
The idea: every waiter records a deadline when it joins the queue — waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT (rwsem.c L1035 for readers, L1122 for writers). If the first waiter has been queued past its deadline, it sets FLAG_HANDOFF, and from that point lock-stealing and optimistic acquisition are forbidden — the lock must be handed directly to that first waiter when it is released.
The deadline constant is defined in terms of the tick rate, not in milliseconds:
Walk the arithmetic. HZ is the timer-tick frequency and jiffies counts ticks, so DIV_ROUND_UP(HZ, 250) is “however many jiffies make up 1/250 of a second” — that is, 4 ms, rounded up to a whole jiffy. The value is therefore configuration-dependent:
CONFIG_HZ
DIV_ROUND_UP(HZ, 250)
Real deadline
Jiffy resolution
1000
4 jiffies
exactly 4 ms
1 ms — the deadline is precise
300
2 jiffies
~6.67 ms
3.33 ms
250
1 jiffy
exactly 4 ms
4 ms — deadline is one whole tick
100
1 jiffy
10 ms
10 ms — the “4 ms” is 10 ms in practice
The HANDOFF deadline across tick rates. What it shows: the commonly repeated figure “4 ms” is only correct at CONFIG_HZ of 250 or above. The insight to take: on a HZ=100 kernel the starvation bound is 10 ms, two and a half times the documented intent, purely because a jiffy is the finest granularity available. This host runs CONFIG_HZ_1000=y / CONFIG_HZ=1000 (verified in /boot/config-7.1.13-200.fc44.x86_64), so here the bound really is 4 ms.
That number is also the worst case before the bit is set, not the worst case overall. Once HANDOFF is set, the waiter still has to wait for the current holders to release — which, for a reader-held lock with many readers, means waiting for the last of them. The guarantee HANDOFF provides is “no new acquirer will overtake you after 4 ms,” not “you will have the lock within 4 ms.”
In rwsem_try_write_lock() (the queued writer’s acquire attempt):
if (count & RWSEM_LOCK_MASK) { if (has_handoff || (!rt_or_dl_task(waiter->task) && !time_after(jiffies, waiter->timeout))) return false; new |= RWSEM_FLAG_HANDOFF; /* set the bit, force the issue */}
A waiting writer that has timed out (or is a realtime/deadline task, which gets it immediately) sets FLAG_HANDOFF. Once set, the comment in rwsem.c guarantees ordering: “A writer must also be the first one in the wait_list to be eligible for setting the handoff bit. So concurrent setting/clearing of handoff bit is not possible” — the wait_lock is held for every set/clear, and only the head waiter may set it. With HANDOFF set, rwsem_try_write_lock() for any non-first waiter returns false (“Honor handoff bit and yield only when the first waiter is the one that set it”), so the queued head waiter is guaranteed the next grant.
Drawn as a state machine over the lock word, the fairness regime has two modes and one trapdoor between them:
stateDiagram-v2
direction TB
[*] --> Free: init_rwsem — count = 0
state "THROUGHPUT MODE (HANDOFF clear)" as TP {
Free --> WriteHeld: try_cmpxchg 0 to WRITER_LOCKED
Free --> ReadHeld: add READER_BIAS, no fail bit
ReadHeld --> ReadHeld: another reader adds READER_BIAS
ReadHeld --> Free: last up_read, count back to 0
WriteHeld --> Free: up_write
WriteHeld --> ReadHeld: downgrade_write — atomic swap of WRITER_LOCKED for READER_BIAS
WriteHeld --> Spinning: contending writer, owner is on_cpu
Spinning --> WriteHeld: rwsem_try_write_lock_unqueued won the race
Spinning --> Queued: owner slept, need_resched, or rspin threshold blown
ReadHeld --> Stealing: slow-path reader, rcnt is 1, no WRITER_LOCKED
Stealing --> ReadHeld: rwsem_rlock_steal — jumped a queued writer
Free --> Queued: lost the race, joined wait_list
ReadHeld --> Queued: reader-saturated, must not steal
}
Queued --> Queued: waiting, jiffies < waiter.timeout
Queued --> HandoffSet: head waiter past waiter.timeout<br/>OR is an RT or DL task
state "FAIRNESS MODE (HANDOFF set)" as FM {
HandoffSet --> HandoffSet: holders drain — no new acquirer admitted
HandoffSet --> Granted: last holder releases, lock goes to the head waiter
}
Granted --> Free: HANDOFF cleared in rwsem_del_waiter<br/>or rwsem_try_write_lock
Granted --> [*]
The rwsem fairness state machine. What it shows: the large box is the ordinary, fast, slightly-unfair regime — stealing and spinning are both permitted and both are transitions into a held state. The trapdoor out of it is one edge: the head waiter’s deadline expiring. The insight to take:HANDOFF is not a queue discipline, it is a mode switch. In throughput mode the lock behaves like an unfair, high-throughput lock; in fairness mode it behaves like a strict FIFO queue with exactly one eligible acquirer. Every rwsem in the kernel spends almost all of its life in the left box and only visits the right one under sustained contention — which is precisely the design goal, since the fairness mode costs ~16% throughput.
Two edges in that diagram deserve a second look because they are easy to misread:
Queued --> HandoffSet fires immediately for RT and DL tasks. The condition in rwsem_try_write_lock() is has_handoff || (!rt_or_dl_task(waiter->task) && !time_after(jiffies, waiter->timeout)). A SCHED_FIFO, SCHED_RR or SCHED_DEADLINE waiter does not serve the 4 ms sentence — it sets HANDOFF on its first attempt. The kernel treats “this waiter has a latency contract” as equivalent to “this waiter has already waited too long.” (Those policies are covered in Real-Time Scheduling SCHED_FIFO and SCHED_RR.)
Granted --> Free clears the bit in three different places. The source enumerates them verbatim (rwsem.c v6.12, L108-111):
* There are three places where the lock handoff bit may be set or cleared.
* 1) rwsem_mark_wake() for readers -- set, clear
* 2) rwsem_try_write_lock() for writers -- set, clear
* 3) rwsem_del_waiter() -- clear
The third case is the one people forget: a waiter that gives up — killed, or interrupted via down_read_killable() — must clear the bit it set, or the lock would stay in fairness mode waiting for a waiter that has already left.
Uncertain
Verify: the practical frequency with which real workloads actually enter fairness mode — i.e. what rwsem_wlock_handoff looks like on a busy production box. Reason: the counter requires CONFIG_LOCK_EVENT_COUNTS plus readable /sys/kernel/debug, and neither is available on this host (mode 0700, no sudo, perf not installed). Waiman Long’s 2019 numbers were taken on a deliberately pathological microbenchmark (240 writer threads, 5 µs critical sections) chosen to provoke starvation, so they establish that the mechanism works but say nothing about how often it fires in ordinary use. To resolve: read /sys/kernel/debug/lock_event_ctrs/rwsem_wlock_handoff on a CONFIG_LOCK_EVENT_COUNTS=y kernel under a real workload, or perf lock contention -b as root. uncertain
The serialisation argument for all of this is one sentence in the same comment, and it is the reason no cmpxchg loop is needed around the bit: “For all the above cases, wait_lock will be held. A writer must also be the first one in the wait_list to be eligible for setting the handoff bit. So concurrent setting/clearing of handoff bit is not possible.” One lock, one eligible setter, no race.
On the reader side, rwsem_mark_wake() similarly checks time_after(jiffies, waiter->timeout) and requests a hand-off when waking readers has been blocked too long by writers. The hand-off bit is cleared once the lock is granted to the intended waiter (in rwsem_del_waiter(), rwsem_try_write_lock(), or when readers are woken). The net effect: throughput-friendly stealing and spinning are allowed up to a bounded latency, after which fairness is enforced by fiat. The source describes the reader-wakeup policy as “an adaptation of the phase-fair R/W locks where at the reader phase… all readers are eligible to acquire the lock at the same time.”
The Headline User: mmap_lock
The single most important rwsem in the kernel is mmap_lock, the per-mm_struct lock that protects a process’s virtual address space — its tree of Virtual Memory Areas (VMAs). Every mmap(), munmap(), mprotect(), brk(), and page-fault-driven VMA modification serialises through it. The kernel docs state it plainly: “Each MM has a read/write semaphore mmap_lock which locks at a process address space granularity which can be acquired via mmap_read_lock(), mmap_write_lock() and variants” (process_addrs). Readers (e.g. page-fault handlers that only look up a VMA) take it for read; mutators (mmap/munmap) take it for write. “The first thing any of these locks achieve is to stabilise the VMA within the MM tree” — i.e. guarantee the VMA will not be freed or modified underneath you.
It was historically named mmap_sem. Michel Lespinasse’s work to wrap all direct mmap_sem accesses behind an mmap_*_lock() API renamed the field to mmap_lock; the wrapping was the necessary prerequisite for later scalability work. Jonathan Corbet’s report from the 2019 Linux Storage, Filesystem and Memory-Management summit opens by calling mmap_sem “a known scalability problem for years, but it has proved difficult to remove” — difficult enough that “no less than three sessions were held during the memory-management track” on it alone. The diagnosis quoted there is Matthew Wilcox’s: mmap_sem “is highly contended”, and “part of the problem… is that mmap_sem ‘covers many sins’ beyond the protection of VMAs” (LWN 787629). That is the essential point — the lock was not slow, it was overloaded: one rwsem standing in for several unrelated invariants, so that contention on any one of them serialised all the others.
The v5.8 attribution is not a recollection — it is an existence check against include/linux/mm_types.h and the new wrapper header at successive tags:
Tag
struct mm_struct field
include/linux/mmap_lock.h
v5.7
struct rw_semaphore mmap_sem;
HTTP 404 — does not exist
v5.8
struct rw_semaphore mmap_lock;
HTTP 200 — introduced
v5.9
struct rw_semaphore mmap_lock;
HTTP 200
Dating the rename. What it shows: the field rename and the arrival of the mmap_*_lock() wrapper header happen in the same release. The insight to take: the rename was never cosmetic. Wrapping every access behind an API was the prerequisite for changing what the lock is — you cannot replace a lock that 900 call sites dereference directly. The renaming series bought the mm developers the freedom that per-VMA locking later spent.
Why it mattered so much is set out in LWN’s coverage of the effort (LWN 787629, “How to get rid of mmap_sem”) and its follow-ups on scalability (LWN 893906, LWN 906852). The structural complaint is that a single address-space-wide rwsem makes unrelated work contend: two threads faulting pages in completely different VMAs of the same process still serialise against each other, because the lock’s granularity is the whole mm_struct. LWN’s per-VMA-locking coverage frames the general problem the same way when discussing the alternative of range locking: rather than “locking the entire address space to make a change to a small part of it,” you want exclusive access only to the range of interest, “while allowing accesses to” the rest (LWN 906852). Per-VMA locking is the variant of that idea which actually got merged.
The contention pressure on mmap_lock drove the per-VMA locking work: page-fault handlers can now take a lightweight per-VMA read lock under RCU (lock_vma_under_rcu()) instead of the whole-address-space mmap_lock, so faults to different VMAs run concurrently. The docs confirm “VMA read locking is entirely optimistic — if the lock is contended or a competing write has started, then we do not obtain a read lock,” and that a VMA read lock “can be obtained without any other lock,” with the maple-tree VMA lookup being “RCU safe.” A VMA write lock still requires the mmap_lock write lock to be held. This is the classic rwsem story: a single coarse rwsem becomes a scalability bottleneck, and the fix is to push toward finer-grained or RCU-based reader paths rather than to make the rwsem itself faster.
Configuration and Usage Examples
/* Static declaration */DECLARE_RWSEM(my_rwsem);/* Dynamic initialisation */struct rw_semaphore lock;init_rwsem(&lock);/* Reader critical section */down_read(&lock);... /* read shared data; many readers run here at once */up_read(&lock);/* Writer critical section */down_write(&lock);... /* mutate shared data; exclusive */up_write(&lock);
The complete API surface
Every public entry point declared in include/linux/rwsem.h at v6.12, with its exact semantics. Read the Returns column carefully: the return conventions are not uniform, and mixing them up is a real source of bugs.
Function
Header line
Sleeps?
Returns
Semantics
down_read(sem)
L211
yes
void
Acquire for read. Blocks uninterruptibly.
down_read_interruptible(sem)
L212
yes
0 or -EINTR, __must_check
Acquire for read; abandons on any signal.
down_read_killable(sem)
L213
yes
0 or -EINTR, __must_check
Acquire for read; abandons only on a fatal signal. The right choice in syscall paths that must stay killable.
down_read_trylock(sem)
L218
no
1 on success, 0 on failure
Non-blocking read acquire. Note the convention.
down_write(sem)
L223
yes
void
Acquire for write. Blocks uninterruptibly.
down_write_killable(sem)
L224
yes
0 or -EINTR, __must_check
Killable write acquire. Added 2016-04-22 by Michal Hocko, specifically so OOM-killed tasks stuck on mmap_sem could die.
down_write_trylock(sem)
L229
no
1 on success, 0 on failure
Non-blocking write acquire.
up_read(sem)
L234
no
void
Release a read acquire.
up_write(sem)
L239
no
void
Release a write acquire.
downgrade_write(sem)
L251
no
void
Atomically convert write-held to read-held. Cannot fail and cannot be overtaken.
rwsem_is_locked(sem)
L73 / L165
no
int
count != 0. Advisory only — a debug/assertion aid, never a correctness test.
rwsem_is_contended(sem)
L130 / L180
no
int
!list_empty(&sem->wait_list). Used by long-running readers (notably in mm) to voluntarily drop and re-take the lock when somebody is waiting.
down_read_nested(sem, subclass) etc.
L267-271
yes
varies
lockdep annotation variants; compile to the plain versions without CONFIG_DEBUG_LOCK_ALLOC.
down_read_non_owner(sem) / up_read_non_owner(sem)
L285-286
yes / no
void
Acquire in one context, release in another. Defeats the owner tracking; use is strongly discouraged.
The trylock return-convention trap
down_read_trylock() and down_write_trylock() return 1 on success, matching spin_trylock() and mutex_trylock(). The plain counting semaphore’s down_trylock() returns 0 on success, matching the down_* family’s errno convention. Both are “trylock”, both take a lock-shaped argument, and they are inverted with respect to each other. if (down_read_trylock(&sem)) is correct; if (!down_trylock(&sem)) is correct; swapping the ! silently produces a lock that is taken exactly when it should not be.
Scope-based acquisition with guard()
Since the cleanup.h infrastructure landed, rwsems can be acquired with automatic scope-exit release, which eliminates the classic “goto out forgot the up_read()” leak:
void example(struct my_obj *obj){ guard(rwsem_read)(&obj->lock); /* up_read() runs on any exit from this scope */ if (obj->broken) return; /* released automatically — no goto needed */ do_work(obj);} /* released automatically here too *//* Conditional variants bind to a scope only if acquisition succeeded: */scoped_guard (rwsem_write_try, &obj->lock) { mutate(obj); /* body runs only if the trylock won */}
Note the three conditional variants and the one that is missing: there is rwsem_read_intr but no rwsem_write_intr, mirroring the fact that down_write_interruptible() does not exist. Writers may be killable but not interruptible — a design choice that keeps a half-completed write-side critical section from being abandoned on an ordinary signal.
Diagnosing contention — what is and is not possible here
The kernel exposes rwsem internals through two mechanisms, and it is worth being precise about what each requires, because on most machines neither is available.
Lock event counters.rwsem.c calls lockevent_inc() at every interesting decision point. The full set at v6.12 (lock_events_list.h L53-69):
Counter
Incremented when
What a high value tells you
rwsem_rlock_fast
reader exited the slow path early (empty wait list, no writer)
contention is shallow — readers are being turned away by a transient, not a queue
rwsem_rlock_steal
a slow-path reader stole the lock past a queued writer
the throughput/fairness bargain is being exercised; pair with wlock_handoff
rwsem_rlock / rwsem_wlock
read / write lock acquired via the slow path
baseline slow-path volume
rwsem_rlock_fail / rwsem_wlock_fail
acquisition abandoned (signal, kill)
killable/interruptible waiters are actually being aborted
rwsem_rlock_handoff / rwsem_wlock_handoff
a hand-off was performed
the starvation guard is firing — waiters are blowing the 4 ms deadline
rwsem_opt_lock
write lock acquired by optimistic spinning
spinning is paying off
rwsem_opt_fail
optimistic spin gave up and the task slept
spinning is wasted CPU
rwsem_opt_nospin
spinning was skipped because RWSEM_NONSPINNABLE was set
the adaptive disable has kicked in on a reader-owned lock
rwsem_sleep_reader / rwsem_sleep_writer
a task actually went to sleep
the real “this lock is contended” signal
rwsem_wake_reader / rwsem_wake_writer
a waiter was woken
pairs with the sleep counters
The diagnostic story these tell together: a healthy read-mostly rwsem shows large rlock_fast, near-zero sleep_*, and zero *_handoff. A rwsem in trouble shows wlock_handoff climbing, which means writers are routinely waiting past 4 ms and the lock has switched into fairness mode — the signature of the exact starvation scenario HANDOFF was written to bound.
These counters require CONFIG_LOCK_EVENT_COUNTS, are exported through /sys/kernel/debug/lock_event_ctrs/, and are unavailable on this host. Verified directly:
$ ls /sys/kernel/debug/lock_event_ctrsls: cannot access '/sys/kernel/debug/lock_event_ctrs': Permission denied$ ls -ld /sys/kernel/debugdrwx------. 56 root root 0 Sep 10 16:46 /sys/kernel/debug/$ sudo -n truesudo: a password is required
Per-lock debugging.CONFIG_DEBUG_RWSEMS adds the magic field and owner-mismatch assertions; CONFIG_LOCK_STAT adds per-call-site hold-time and wait-time statistics. Both are off in this kernel:
$ grep -E '^(# )?CONFIG_(LOCK_STAT|DEBUG_RWSEMS|DEBUG_LOCK_ALLOC|PROVE_LOCKING|RWSEM_SPIN_ON_OWNER|PER_VMA_LOCK) ' /boot/config-7.1.13-200.fc44.x86_64# CONFIG_LOCK_STAT is not set# CONFIG_DEBUG_RWSEMS is not set# CONFIG_DEBUG_LOCK_ALLOC is not set# CONFIG_PROVE_LOCKING is not setCONFIG_RWSEM_SPIN_ON_OWNER=yCONFIG_PER_VMA_LOCK=y
This is the normal state of affairs on a distribution kernel — all four debug options carry a real runtime cost, so Fedora ships them off. Optimistic spinning and per-VMA locking, by contrast, are both compiled in.
What this note cannot show you
There are no measured rwsem contention numbers in this note, because none could be produced on this host, and inventing them would be worse than omitting them. The command that would work is:
# perf lock contention -b -- <workload> # BPF mode, aggregates by lock and caller# perf lock contention -a -l sleep 10 # system-wide, report per-lock-address
perf lock contention needs (a) the perf binary — not installed here, command -v perf returns nothing; (b) root, or kernel.perf_event_paranoid lowered from its current value of 2; and for the -b BPF mode, CONFIG_BPF_SYSCALL plus CAP_BPF. bpftrace, bpftool and trace-cmd are also absent, and /sys/kernel/tracing is 0700, so the tracepoint route is closed too. The structural facts in this note are all read from source at a pinned tag; the performance facts are quoted from Waiman Long’s commit messages, where they were measured by the author on named hardware. Nothing here is a number I produced and could not defend.
Failure Modes and Common Misunderstandings
down_read_trylock() vs down_trylock() return inversion. rwsem trylocks return 1 on success (like spin_trylock/mutex_trylock), but the plain semaphore’s down_trylock() returns 0 on success. Mixing the conventions silently inverts the success test.
Reader recursion deadlock. rwsems are not recursive. A task that holds the read lock and then a writer queues behind it must not try to take the read lock again — if a writer is waiting, the second down_read() will block behind that writer (fairness forbids jumping the queue), and since the task is also the one holding the first read lock, it deadlocks against itself. The header explicitly warns “rwsems are not allowed to recurse.”
Writer starvation under reader floods (mitigated, not eliminated). Before HANDOFF, a steady reader stream could starve a writer indefinitely. HANDOFF bounds the starvation to ~4ms+ of waiting, but a latency-sensitive writer can still see multi-millisecond stalls under heavy reader load — mmap_lock write contention is a real source of tail latency in memory-heavy workloads.
Holding it too long. Because it sleeps, an rwsem may only be taken in process context that can block — never in atomic/interrupt context (use Reader-Writer Spinlocks there) and never while holding a spinlock. might_sleep() in the acquire paths will warn if violated.
Stale owner for reader locks. The owner field for a reader-held lock points at some past reader and may be wrong — it is for spinning heuristics and debugging only, never for correctness. The source warns: “it may not be the real owner… when that field is examined.”
Alternatives and When to Choose Them
Plain mutex — if reads and writes are roughly balanced, the reader/writer bookkeeping of an rwsem (the atomic add/sub, the spinning thresholds) often costs more than it saves; a simple mutex wins. Reach for rwsem only when reads genuinely dominate.
Reader-Writer Spinlocks — the non-sleeping reader/writer lock for atomic/interrupt context or very short critical sections. Cannot sleep inside.
RCU (Read-Copy-Update Fundamentals) — when readers must be truly free (no atomic on the fast path, no cache-line bouncing) and the data structure can be updated copy-then-publish. RCU readers never block writers at all, which an rwsem cannot match — but RCU constrains how you mutate the data. This is the direction mmap_lock evolution took with RCU-based per-VMA lookup.
Per-CPU Variables — if the state can be made per-CPU, share nothing and avoid the lock entirely.
Side by side, across the dimensions that actually decide the choice:
The sleeping/non-sleeping reader-writer landscape. What it shows: the two rows that usually settle the decision are “may the holder sleep” and “reader concurrency”. If your critical section can block — it allocates with GFP_KERNEL, does I/O, or takes another sleeping lock — every non-sleeping column is disqualified before you get to performance. The insight to take: look at the “writer starvation” row. rwsem is the only reader/writer primitive here with an explicit, bounded anti-starvation mechanism. rwlock_t genuinely can starve a writer forever, which is a large part of why the kernel has been migrating away from it. That bound is rwsem’s distinguishing feature, not its reader concurrency — plenty of things offer reader concurrency.
A practical decision procedure, in the order the questions actually arise:
Can the critical section sleep? If no, you are choosing among spinlock / rwlock / seqlock / RCU and rwsem is out. If yes, continue.
Are reads genuinely dominant? Not “mostly reads” in the hand-wave sense — measurably, by an order of magnitude. If not, use a mutex: the reader/writer bookkeeping (a read-modify-write on a shared word for every reader, which bounces the cache line exactly as hard as a mutex would) costs more than it saves. A rwsem’s readers are concurrent but not cheap; they all hammer the same count.
Can the data be updated copy-then-publish? If yes, RCU gives you readers that cost literally nothing and never block a writer. This is strictly better than rwsem where it applies — it just does not apply to most mutable structures.
Can the state be made per-CPU? If yes, Per-CPU Variables removes the question entirely.
Otherwise, rwsem. And if write latency then turns out to matter, your fix is almost never “tune the rwsem” — it is to take the lock for write less often, or to shard it.
That last point is the mmap_lock story in miniature, and it is worth stating as a general lesson rather than an mm-specific anecdote: when a single coarse rwsem becomes a bottleneck, the kernel’s answer has consistently been to reduce the scope of what the lock protects, not to make the lock faster. Per-VMA locking, the maple tree, and RCU-based lookup are all instances of that move.
PREEMPT_RT and Production Notes
On a PREEMPT_RT kernel, struct rw_semaphore is not the count-and-atomics implementation above. Everything in this note — the 55-bit reader count, RWSEM_FLAG_HANDOFF, the osq spin queue, the 4 ms deadline — is compiled out and replaced. (This host is notPREEMPT_RT: # CONFIG_PREEMPT_RT is not set, with CONFIG_PREEMPT_DYNAMIC=y. The section below is read from source, not observed here.)
The RT variant is built on struct rwbase_rt, which is two fields:
An embedded RT-mutex does all the blocking, and readers is a counter initialised to READER_BIAS (1U << 31) so that readers are negative while the fast path is open. rw_base_is_write_locked() is literally atomic_read(&rwb->readers) == WRITER_BIAS. The protocol is spelled out at the top of rwbase_rt.c:
* down_write/write_lock()
* 1) Lock rtmutex
* 2) Remove the reader BIAS to force readers into the slow path
* 3) Wait until all readers have left the critical section
* 4) Mark it write locked
*
* down_read/read_lock()
* 1) Try fast path acquisition (reader BIAS is set)
* 2) Take rtmutex::wait_lock, which protects the writelocked flag
* 3) If !writelocked, acquire it for read
* 4) If writelocked, block on rtmutex
* 5) unlock rtmutex, goto 1)
Step 2 of the write path is the whole trick: a writer removes the bias, which does not block anybody immediately but causes every subsequent reader to fall out of the fast path and into the rtmutex — a one-instruction way to close the door behind the readers already inside, then wait for them to drain.
The fairness properties invert. The source is blunt about it:
down_read/read_lock()#3 has the consequence, that rw semaphores and rw locks on RT are not writer fair, but writers, which should be avoided in RT tasks (think mmap_sem), are subject to the rtmutex priority/DL inheritance mechanism. […] The risk of writer starvation is there, but the pathological use cases which trigger it are not necessarily the typical RT workloads.
So PREEMPT_RTgives up the HANDOFF guarantee entirely and accepts possible writer starvation, in exchange for priority inheritance. The comment even explains why the obvious fix was rejected: making the primitive writer-fair would require “keeping a list of active readers” and proxy-locking the rtmutex “for one reader after the other,” because “we can’t use multi-reader inheritance… there is no way to support that with SCHED_DEADLINE.” The verdict is “a major surgery for a very dubious value.”
The locktypes documentation states the same asymmetry from the user’s side. First the baseline, which is the sentence most worth memorising: “On non-PREEMPT_RT kernels the implementation is fair, thus preventing writer starvation.” Then the change:
PREEMPT_RT kernels map rw_semaphore to a separate rt_mutex-based implementation, thus changing the fairness:
Because an rw_semaphore writer cannot grant its priority to multiple readers, a preempted low-priority reader will continue holding its lock, thus starving even high-priority writers. In contrast, because readers can grant their priority to a writer, a preempted low-priority writer will have its priority boosted until it releases the lock, thus preventing that writer from starving readers.
Read that carefully, because the asymmetry is exactly backwards from what most people expect. Priority inheritance works reader → writer but not writer → readers. A blocked writer can boost a single reader? No — it cannot, because there may be many readers and PI has no mechanism to boost a set. So on PREEMPT_RT a low-priority reader that gets preempted keeps the lock and starves a high-priority writer. Going the other way is fine: readers blocked behind a writer boost that one writer, which has a single identity to boost. The direction PI can flow is determined by which side of the lock has exactly one holder.
mainline rwsem (this note)
PREEMPT_RT rwsem
Backing structure
atomic_long_t count + osq + wait_list
atomic_t readers + embedded rt_mutex_base
Reader fast path
atomic_long_add_return_acquire(READER_BIAS)
atomic increment while READER_BIAS is set
Writer optimistic spinning
yes, CONFIG_RWSEM_SPIN_ON_OWNER
no — rtmutex blocking only
Anti-starvation
RWSEM_FLAG_HANDOFF, bounded ~4 ms
none — writer starvation is possible by design
Priority inheritance
no (writer owner is tracked, but no PI)
yes, readers boost a blocked writer
Readers preemptible while holding
yes
yes, and this is the point
The two implementations compared. What it shows: these are not variants of one algorithm; they are different algorithms behind one API. The insight to take: the trade-off is swapped wholesale. Mainline buys bounded writer latency with a fairness mode and no PI; RT buys priority correctness with PI and accepts unbounded writer latency. If you are reasoning about worst-case write latency, the first question is which kernel you are on, and the answer changes the analysis completely. See The PREEMPT_RT Real-Time Kernel.
In a production 6.12 kernel the most-watched rwsem by far is mmap_lock; perf lock and the lock_event_counts (the rwsem_* lockevents the source increments, like rwsem_rlock_steal, rwsem_wlock_handoff, rwsem_opt_lock) are the tools for diagnosing rwsem contention. Heavy mmap_lock write contention shows up as time spent in rwsem_down_write_slowpath; the modern remedy is rarely “tune the rwsem” and usually “stop taking mmap_lock for write so often” — hence the years of per-VMA-lock and maple-tree work in the memory-management subsystem.
See Also
Kernel Semaphores — the non-reader/writer sibling (struct semaphore): a counting permit pool / signal, no owner, the simpler sleeping lock
Kernel Mutexes — the binary sleeping lock; prefer it when reads and writes are balanced
Migration the Stop Class and the Migration Thread — the far end of the exclusion spectrum: where an rwsem read-acquire excludes almost nobody for one atomic instruction, stop_machine() excludes every thread on every CPU