vault

Sleepable RCU and SRCU

14 min read

Sleepable RCU and SRCU

Sleepable Read-Copy-Update (SRCU) is the variant (“flavor”) of RCU whose read-side critical sections are permitted to block and sleep — to allocate memory that may reclaim, wait on I/O, or take a mutex — which classic RCU strictly forbids. SRCU pays for this freedom by abandoning RCU’s near-free reader: where rcu_read_lock() compiles down to little more than disabling preemption, the SRCU reader srcu_read_lock() performs an explicit per-CPU counter increment fenced by a full memory barrier, returns an integer index that the matching srcu_read_unlock() must consume, and is scoped to a caller-allocated domain called a struct srcu_struct. The domain is the second defining feature: grace periods are tracked per srcu_struct, so a reader that sleeps forever inside one domain stalls only updaters of that domain and cannot wedge the global RCU machinery — the property that makes SRCU safe to hand to subsystems whose readers genuinely block (KVM’s memslot lookups, SRCU notifier chains, tracepoints). This note describes SRCU as implemented by CONFIG_TREE_SRCU in Linux v6.12 LTS (kernel/rcu/srcutree.c).

For the RCU concepts SRCU builds on — publish/subscribe, quiescent states, the meaning of a grace period — read Read-Copy-Update Fundamentals and RCU Grace Periods first; this note assumes them and focuses on what is different about the sleepable flavor.

Mental Model

The cleanest way to think about SRCU is: classic RCU, but with the “you may not sleep in a reader” rule lifted, in exchange for (a) you must declare your own bounded scope and (b) your readers are no longer free.

Classic RCU detects the end of a grace period by observing quiescent states — context switches, idle, user-mode — because in a non-preemptible reader those events prove no reader is active. SRCU cannot use that trick: a sleepable reader voluntarily schedules away while still inside its critical section, so a context switch proves nothing. SRCU therefore tracks readers explicitly with per-CPU counters and waits for those counters to drain. That explicit accounting is the source of both SRCU’s extra read-side cost and its per-domain isolation.

flowchart TB
  subgraph DomA["srcu_struct A (e.g. kvm->srcu)"]
    direction TB
    AL["srcu_read_lock(&A)<br/>idx = READ_ONCE(A.srcu_idx) &amp; 1<br/>this_cpu_inc(lock_count[idx])<br/>smp_mb()  →  returns idx"]
    AR["... reader may SLEEP here ..."]
    AU["srcu_read_unlock(&A, idx)<br/>smp_mb()<br/>this_cpu_inc(unlock_count[idx])"]
    AL --> AR --> AU
  end
  subgraph DomB["srcu_struct B (e.g. tracepoint_srcu)"]
    BSYNC["synchronize_srcu(&B)<br/>drains B's counters only —<br/>UNAFFECTED by a stuck reader in A"]
  end
  AGP["synchronize_srcu(&A):<br/>scan unlock sum, flip A.srcu_idx,<br/>wait for old-index lock sum == unlock sum"]
  AU -.->|"counts toward"| AGP
  AR -.->|"stuck/sleeping reader<br/>delays only A's GP"| AGP

The two-domain picture. What it shows: every SRCU operation is scoped to one srcu_struct. A reader enters by reading the domain’s current srcu_idx (0 or 1), bumping that CPU’s lock_count bank, and returning the index; it may then sleep arbitrarily; unlock bumps the matching unlock_count bank. A grace period for domain A waits only on A’s counters. The insight: a pathologically slow or sleeping reader in domain A delays only synchronize_srcu(&A) — domain B’s synchronize_srcu(&B) and the entire global RCU subsystem are completely insulated. This isolation is exactly why SRCU is the flavor you hand to code whose readers may block.

The “two banks” in the diagram are the heart of the algorithm. Each srcu_struct keeps two per-CPU lock counters and two per-CPU unlock counters, indexed [0] and [1]. A single bit, srcu_idx & 1, selects which bank new readers use. To run a grace period the updater (1) waits for the inactive bank to read as “all locks accounted for by unlocks,” (2) flips srcu_idx so new readers go to the other bank, then (3) waits for the now-old bank to drain. After the flip, any reader that started before the flip is counted in the old bank and must finish before the grace period ends; any reader that starts after lands in the new bank and is irrelevant to this grace period. This is the same “wait for pre-existing readers, ignore new ones” guarantee as ordinary RCU, achieved by counting instead of by observing quiescent states.

Mechanical Walk-through

The read side: __srcu_read_lock / __srcu_read_unlock

The entire fast path is two short functions in srcutree.c:

int __srcu_read_lock(struct srcu_struct *ssp)
{
	int idx;
 
	idx = READ_ONCE(ssp->srcu_idx) & 0x1;
	this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter);
	smp_mb(); /* B */  /* Avoid leaking the critical section. */
	return idx;
}
 
void __srcu_read_unlock(struct srcu_struct *ssp, int idx)
{
	smp_mb(); /* C */  /* Avoid leaking the critical section. */
	this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter);
}

Line by line. READ_ONCE(ssp->srcu_idx) & 0x1 reads the domain’s current bank selector and masks it to 0 or 1 — this is the idx that flows back to the caller and must be passed, unchanged, to srcu_read_unlock. this_cpu_inc(...srcu_lock_count[idx]...) bumps this CPU’s lock counter for the selected bank; because it is a per-CPU variable there is no cache-line contention between CPUs on the common path. smp_mb() (barrier “B”) is a full memory barrier ensuring that the counter increment is globally visible before any memory access inside the critical section — without it, the updater could observe a drained counter while the reader’s dereferences are still in flight, freeing data out from under the reader. The unlock is the mirror image: barrier “C” orders the critical section’s accesses before the unlock-counter increment, then bumps the matching unlock_count[idx] bank.

Two consequences fall straight out of this code and are routinely missed:

  • The index is load-bearing. srcu_read_lock() returns an int and srcu_read_unlock() consumes it; passing the wrong index corrupts the counters and can hang grace periods. This is unlike rcu_read_lock()/rcu_read_unlock(), which take and return nothing. The public wrapper even asserts WARN_ON_ONCE(idx & ~0x1) in include/linux/srcu.h.
  • Unlock may run on a different CPU. The comment on __srcu_read_unlock notes “this may well be a different CPU than that which was incremented by the corresponding srcu_read_lock()” — legal precisely because the grace period sums the counters across all CPUs, so it does not matter which CPU each half landed on. (This is what srcu_down_read()/srcu_up_read(), the semaphore-style variant, exploit to hand a critical section between contexts.)

The cost difference versus classic RCU is now concrete. A classic rcu_read_lock() in a CONFIG_PREEMPT kernel increments a task-local nesting counter and does no barrier; an SRCU reader does a per-CPU RMW increment plus a full smp_mb() on both entry and exit. The smp_mb() is the expensive part — a full barrier serializes the store buffer and is far costlier than the compiler barrier classic RCU readers get away with. That is the price of being able to sleep.

The update side: synchronize_srcu, the flip, and counter draining

synchronize_srcu(&ssp) waits until every read-side critical section that began before the call has finished. Its core is srcu_readers_active_idx_check(), which decides whether a given bank has drained:

static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx)
{
	unsigned long unlocks;
 
	unlocks = srcu_readers_unlock_idx(ssp, idx);  /* sum unlock_count[idx] over all CPUs */
	smp_mb(); /* A */
	return srcu_readers_lock_idx(ssp, idx) == unlocks; /* sum lock_count[idx], compare */
}

It first sums the unlock counters for the bank across all CPUs, executes a full barrier “A”, then sums the lock counters and compares. If the two sums are equal, then at some instant during the function there were no active readers on that bank: every lock had a matching unlock. The ordering matters and is subtle — summing unlocks first (before the lock sum) means a reader that increments its lock counter after we read it but unlocks after we read the unlock sum cannot make the sums spuriously match. The long comment in srcu_readers_active_idx_check() proves that the worst-case over-count is bounded by the number of tasks plus CPUs, which the task_struct size makes finite. Barrier “A” pairs with reader barriers “B” and “C” to form a store-buffering pattern that makes this counting sound.

The grace period as a whole, driven through the state machine in srcu_gp_seq (states SRCU_STATE_IDLE, SRCU_STATE_SCAN1, SRCU_STATE_SCAN2), runs roughly:

  1. Wait for the inactive bank (srcu_idx ^ 1) to read as drained (try_check_zero loops on srcu_readers_active_idx_check).
  2. srcu_flip(ssp) does WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1) — new readers now use the other bank. From this moment, the bank we are about to wait on receives no new readers.
  3. Wait for the now-old bank to drain the same way.

srcu_flip carries barrier “E” (smp_mb()) so that the index change is ordered against the counter scans — though the code comments it is technically redundant given the control dependency from the equality check, and kept only until compilers model that dependency.

The waiting itself is adaptive, not a busy-spin: srcu_get_delay() and try_check_zero() spin for a short fixed window (srcu_retry_check_delay, default 5 µs per srcutree.c), then fall back to sleeping for one-jiffy intervals that grow as the grace period ages, capped at SRCU_MAX_INTERVAL (10 jiffies). This is what lets synchronize_srcu() itself block politely rather than burn a CPU while a sleepable reader takes its time.

call_srcu and asynchronous reclamation

call_srcu(ssp, rhp, func) is the deferred-callback form, mirroring [[call_rcu and Deferred Reclamation|call_rcu]] but scoped to a domain. It queues the callback into the per-CPU srcu_cblist (an rcu_segcblist) and kicks the grace-period machinery via srcu_gp_start_if_needed(). When the grace period for that callback’s “needed” sequence number completes, srcu_gp_end() schedules the callbacks to run from a workqueue kthread, in process context. The docstring is explicit that the callback “must nevertheless be fast and must not block” — the sleepability applies to readers, not to callbacks. synchronize_srcu() is implemented on top of call_srcu(): it queues a wakeme_after_rcu callback and does wait_for_completion().

A heuristic worth knowing: synchronize_srcu() checks srcu_might_be_idle(ssp) and, if the domain looks idle (or rcu_gp_is_expedited() is set), routes through synchronize_srcu_expedited() to cut first-request latency. See Expedited Grace Periods for what “expedited” does to SRCU and to RCU generally.

The srcu_struct Domain and Its Data Structures

A domain is declared with one of:

DEFINE_STATIC_SRCU(my_srcu);            /* file-scope static, no init call */
DEFINE_SRCU(my_srcu);                   /* file-scope, exported */
/* or, dynamically: */
struct srcu_struct my_srcu;
init_srcu_struct(&my_srcu);             /* ... and cleanup_srcu_struct() later */

The visible struct srcu_struct (from srcutree.h) is deliberately tiny — srcu_idx, a pointer to the per-CPU srcu_data array (sda), a lockdep map, and a pointer to a separately allocated struct srcu_usage that holds all the heavy update-side state (the grace-period sequence counters, the combining-tree srcu_nodes, mutexes, the barrier machinery). Splitting srcu_usage out keeps statically defined srcu_structs small and lets the same struct be embedded cheaply in things like srcu_notifier_head.

Each per-CPU srcu_data holds the read-side state — atomic_long_t srcu_lock_count[2] and srcu_unlock_count[2], the two banks — plus update-side per-CPU bits (the callback list, a leaf-srcu_node back-pointer). On large machines SRCU lazily grows an srcu_node combining tree (states SRCU_SIZE_SMALL → SRCU_SIZE_BIG) so that summing and callback-funneling scale, exactly as Tree RCU does for the global flavor; small or low-contention domains stay in SRCU_SIZE_SMALL with no tree at all.

The lockdep contract

srcu_read_lock() annotates a dep_map so lockdep can catch the cardinal SRCU sin: calling synchronize_srcu(&ssp) (directly or indirectly) from inside a read-side critical section of that same ssp, which self-deadlocks because the grace period waits for a reader that is itself the waiter. The header comment warns that “one way to indirectly wait on an SRCU grace period is to acquire a mutex that is held elsewhere while calling synchronize_srcu().” Crucially, it is legal to call synchronize_srcu() on one domain from inside a read-side section of a different domain, as long as the graph of domains is acyclic — this is the formal statement of the per-domain isolation.

Failure Modes and Common Misunderstandings

  • Mismatched or lost index. Because the index returned by srcu_read_lock() must be fed back to srcu_read_unlock(), code that stashes it in the wrong variable, or returns early without unlocking, corrupts the per-CPU counters. A leaked lock leaves the old bank permanently non-empty and synchronize_srcu() hangs forever on that domain. KVM guards this with an explicit srcu_depth debug counter (kvm_vcpu_srcu_read_lock WARN_ONCEs on illegal nesting) — a good pattern to copy.
  • Self-deadlock via synchronize_srcu() under the read lock. As above; lockdep with CONFIG_PROVE_RCU catches it, but only if you actually run that config. Indirect cycles through a mutex are the insidious version.
  • Assuming SRCU readers are cheap. They are not. The double smp_mb() makes the SRCU read side meaningfully more expensive than classic RCU’s. If your readers do not need to sleep, using SRCU is a pure pessimization — use plain RCU. SRCU is justified only by the sleeping requirement (or by the need for a private, isolatable grace-period domain).
  • Expecting callbacks to be able to sleep. They cannot. Sleepability is a read-side property only; call_srcu() callbacks run from workqueue context and “must not block.”
  • Forgetting srcu_barrier() before cleanup_srcu_struct(). cleanup_srcu_struct() will WARN and leak the structure if there are still in-flight callbacks or active readers — it deliberately refuses to free memory it cannot prove is quiescent.
  • NMI context. Plain srcu_read_lock() is not NMI-safe (the this_cpu_inc is not an atomic RMW). Code that must read inside an NMI uses srcu_read_lock_nmisafe(), which uses atomic_long_inc and requires the domain be built with CONFIG_NEED_SRCU_NMI_SAFE; mixing NMI-safe and NMI-unsafe readers on one domain is a bug that CONFIG_PROVE_RCU flags.

Alternatives and When to Choose Them

  • Classic RCU (rcu_read_lock). The default. Near-zero read cost, but readers must not sleep. Choose this whenever readers can stay non-blocking — the overwhelming majority of cases (dcache, routing tables, the process list).
  • SRCU. Choose only when a reader genuinely must sleep (allocate-with-reclaim, take a mutex, wait on I/O) or when you need a private grace-period domain that cannot be stalled by, and cannot stall, the rest of the kernel. Accept the heavier read side.
  • RCU Tasks Trace (rcu_read_lock_trace). A separate sleepable-capable flavor built for BPF sleepable programs; different trade-offs (no per-CPU index, tuned for tracing). Mentioned here only so it is not confused with SRCU — tracepoint.h historically used SRCU (tracepoint_srcu) for the rcuidle path.
  • [[Read-Write Semaphores|rw_semaphore]]. If you actually want reader/writer mutual exclusion with sleeping readers — and writers genuinely block readers — an rw_semaphore is simpler. SRCU is not a lock: writers never block readers, they publish and defer. Reach for SRCU only when the RCU publish/defer model fits and the lock model does not.

Production Notes

The flagship SRCU user is KVM. struct kvm embeds struct srcu_struct srcu and irq_srcu (include/linux/kvm_host.h), and vCPU code enters srcu_read_lock(&vcpu->kvm->srcu) (wrapped by kvm_vcpu_srcu_read_lock) to read the memslots and I/O bus arrays — kvm_memslots() is literally srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu, ...). SRCU is required here because the read-side may take paths that sleep (fault-in, allocation), and because memslot updates use synchronize_srcu() to publish a new memslot generation without blocking vCPUs. The header even documents a deadlock hazard between a kvm->srcu read-side section and synchronize_srcu if slots_lock is acquired in the wrong order — a real-world instance of the indirect-cycle failure mode above.

The SRCU notifier chain (struct srcu_notifier_head in include/linux/notifier.h) uses SRCU so that srcu_notifier_call_chain() runs callbacks “with no cache bounces and no memory barriers” on the call path (relative to a rwsem-based chain), accepting that unregister (synchronize_srcu) is “rather expensive.” This is the canonical “many fast call-chain traversals, rare registration changes” shape. Tracepoints declare extern struct srcu_struct tracepoint_srcu and use srcu_read_lock_notrace() for the rcuidle probe path (where sched-RCU cannot be used because the CPU is RCU-idle), with synchronize_srcu(&tracepoint_srcu) on probe teardown.

Uncertain

Verify: the precise read-side cost comparison (“classic RCU reader does no barrier; SRCU does a full smp_mb() on entry and exit”). Reason: this is accurate for CONFIG_PREEMPT/preemptible-RCU read sides per the v6.12 source, but the exact cost of a classic rcu_read_lock() depends on the RCU configuration (preemptible vs. not) and the comparison’s magnitude is qualitative here, not benchmarked against a specific microbenchmark. To resolve: benchmark srcu_read_lock/unlock against rcu_read_lock/unlock on the target microarchitecture, or cite a maintainer-published number. uncertain

Uncertain

Verify: that tracepoint_srcu / the rcuidle SRCU path is still the live mechanism in v6.12 rather than a vestige, given ongoing churn in how tracepoints handle RCU-idle CPUs (context-tracking rework). Reason: the grep confirms the symbol and synchronize_srcu(&tracepoint_srcu) exist in v6.12’s tracepoint.h, but the surrounding __DO_TRACE/rcuidle machinery has changed across releases. To resolve: trace the v6.12 __DO_TRACE_CALL/tracepoint.c path end-to-end. uncertain

See Also

Graph View