vault

RCU Grace Periods

20 min read

RCU Grace Periods

A grace period (GP) is the central temporal abstraction of Read-Copy-Update: it is the interval after which every RCU read-side critical section that was already in progress when the grace period began is guaranteed to have completed. The kernel’s own definition is terse — synchronize_rcu() is documented to “wait until a grace period has elapsed,” meaning “after all currently executing RCU read-side critical sections have completed” (kernel/rcu/tree.c v6.12). Crucially, a grace period makes no promise about readers that start after it begins — those may run concurrently with, and outlive, the grace period. That asymmetry is the whole trick: an updater who unlinks an object, waits one grace period, then frees it, knows that no surviving reader can still hold a pointer to the freed object, because any reader that could have grabbed the pointer was a pre-existing reader and has by definition finished. RCU detects a grace period not by tracking individual readers (which would defeat the zero-cost read side — see RCU Read-Side Critical Sections) but by waiting for every CPU to pass through a quiescent state: a point at which that CPU provably holds no RCU read lock. This note explains what a grace period guarantees, what a quiescent state is, how a grace period is detected across all CPUs, and why “normal” grace periods deliberately take milliseconds. Pinned to Linux 6.12 LTS (released 2024-11-17).

This note builds on Read-Copy-Update Fundamentals (the publish/subscribe + defer model) and is the prerequisite for call_rcu and Deferred Reclamation, Tree RCU, and Expedited Grace Periods.

Mental Model — the grace period is a fence in time

Think of a grace period as a fence dropped at time T. Every reader is a horizontal line segment on a timeline: it starts at rcu_read_lock() and ends at rcu_read_unlock(). When an updater drops the fence at T (by calling synchronize_rcu() or call_rcu()), RCU’s only job is to wait until every segment that crosses the fence from the left has ended. Segments entirely to the right of the fence (readers that start after T) are irrelevant — RCU is “under no obligation to wait for these new readers” (Requirements.rst v6.12).

sequenceDiagram
    participant U as Updater
    participant R1 as Reader A (pre-existing)
    participant R2 as Reader B (pre-existing)
    participant R3 as Reader C (new)
    Note over U,R3: GP begins at T (synchronize_rcu called)
    R1->>R1: rcu_read_lock (before T)
    R2->>R2: rcu_read_lock (before T)
    U->>U: list_del + synchronize_rcu()  [fence at T]
    R3->>R3: rcu_read_lock (after T) — NOT waited for
    R1-->>R1: rcu_read_unlock  (GP waits for this)
    R2-->>R2: rcu_read_unlock  (GP waits for this)
    Note over U: GP ends: A and B done -> safe to free old object
    R3-->>R3: rcu_read_unlock (may outlive GP — fine, never saw old object)

The grace-period fence. What it shows: the updater unlinks an object and starts a grace period at T; readers A and B began before T so the grace period blocks until both finish; reader C began after T and is never waited for. The insight: RCU only ever waits for pre-existing readers. Reader C is safe to ignore because, having started after the unlink, it can never obtain a pointer to the now-unreachable old object — the publish/subscribe rule of The publish-subscribe Pattern in RCU guarantees C sees the new structure. Once A and B end, no live reader can reference the old object, so the updater frees it.

The deep reason this is correct — not just convenient — is stated by Paul McKenney: “The key observation here is that subsequent RCU read-side critical sections have no way to gain a reference to the newly removed element” (What is RCU, Fundamentally?, LWN 2007). A reader fetches its pointer inside its critical section; once the object is unlinked, only readers that already dereferenced the old pointer can hold it, and all of those are pre-existing. So “wait for pre-existing readers” is exactly “wait until no one can reference the old version.”

Quiescent States — how RCU knows a reader has finished without watching readers

RCU’s read side is deliberately invisible: in a non-preemptible build, rcu_read_lock() and rcu_read_unlock() expand to nothing more than compiler barriers (see RCU Read-Side Critical Sections). There is no per-reader counter to inspect. So how can the kernel possibly know when “all pre-existing readers have completed”? The answer is the quiescent state (QS): a per-CPU event that proves the CPU is not currently inside any RCU read-side critical section.

The defining rule is that it is illegal to block inside an RCU read-side critical section (in classic, non-preemptible RCU). From this, the toy implementation in whatisRCU.rst draws the foundational inference:

“Remember that it is illegal to block while in an RCU read-side critical section. Therefore, if a given CPU executes a context switch, we know that it must have completed all preceding RCU read-side critical sections. Once all CPUs have executed a context switch, then all preceding RCU read-side critical sections will have completed.” (whatisRCU.rst v6.12)

That single observation is the engine of grace-period detection. A context switch is one quiescent state; in practice classic RCU recognizes several. Per the requirements document, in non-preemptible kernels the quiescent states include scheduling-clock interrupts, context switches, the idle loop, and transitions to user-mode code (Requirements.rst v6.12). The intuition: if a CPU is running the idle task, executing in user mode, or taking a context switch, it cannot possibly be holding an RCU read lock (read locks may only be held in kernel mode and never across a sleep), so that CPU has “checked in.” A grace period completes once every CPU has checked in at least once since the grace period began.

Because read locks compile to nothing, RCU does not detect quiescent states at the lock itself — it “infers quiescent states only at special locations, for example, within the scheduler” (Requirements.rst v6.12). The cost of detection is paid by code that already runs (the scheduler, the timer tick, the idle loop), not by readers. This is why RCU read-side overhead is genuinely zero: the entire grace-period machinery piggybacks on events the kernel performs anyway.

Classic vs preemptible RCU

The picture above assumes a reader cannot be preempted mid-critical-section. On a CONFIG_PREEMPT_RCU kernel (the default for preemptible and PREEMPT_RT configurations), a reader can be preempted while holding rcu_read_lock() — and then a context switch on that CPU does not prove the reader finished, because the reader will be resumed later, still inside its critical section. Preemptible RCU therefore tracks read-lock nesting explicitly in the task.

__rcu_read_lock() in preemptible RCU does almost nothing visible: it just increments a per-task counter. The code comment is blunt — “Just increment →rcu_read_lock_nesting, shared state will be updated if we block” — and the implementation is one line:

static void rcu_preempt_read_enter(void)
{
	WRITE_ONCE(current->rcu_read_lock_nesting,
	           READ_ONCE(current->rcu_read_lock_nesting) + 1);
}

(kernel/rcu/tree_plugin.h v6.12.) The work happens only if the reader is actually preempted. At that context switch, rcu_preempt_note_context_switch() notices the nonzero nesting count and enqueues the task on the rcu_node’s blocked-tasks list. The comment states the mechanism precisely:

“…we will no longer be able to rely on the CPU to record that fact, so we enqueue the task on the blkd_tasks list. The task will dequeue itself when it exits the outermost enclosing RCU read-side critical section. Therefore, the current grace period cannot be permitted to complete until the blkd_tasks list entries predating the current grace period drain, in other words, until rnp→gp_tasks becomes NULL.” (tree_plugin.h v6.12)

So the cardinal rule survives preemption: a reader that started before the grace period keeps its old view and keeps the grace period open until it finishes — even if it is preempted and migrated across CPUs in between. The difference is only in bookkeeping: classic RCU infers completion from CPU events; preemptible RCU additionally tracks the specific blocked tasks via ->blkd_tasks and the ->gp_tasks pointer on each rcu_node. When rcu_read_unlock() brings the nesting count to zero and there is deferred work flagged in rcu_read_unlock_special, the reader removes itself from the list, and once the list of pre-existing blocked tasks drains, that branch of the tree reports its quiescent state.

Uncertain

Verify: the exact set of quiescent states recognized for expedited preemptible-RCU grace periods (rcu_exp_handler / IPI-driven reporting) versus normal grace periods. Reason: tree_plugin.h notes “quiescent state reports for expedited grace periods are handled separately via deferred quiescent states and context switch events,” but the precise expedited path is not fully traced here. To resolve: read synchronize_rcu_expedited() and rcu_exp_handler() in kernel/rcu/tree_exp.h v6.12. Covered in Expedited Grace Periods. uncertain

Mechanical Walk-through — detecting a grace period across all CPUs

A grace period is driven by a dedicated kernel thread, the grace-period kthread (rcu_gp_kthread, named rcu_preempt or rcu_sched in ps). It runs a state machine; the states are enumerated in kernel/rcu/tree.h v6.12:

#define RCU_GP_IDLE      0  /* Initial state and no GP in progress. */
#define RCU_GP_WAIT_GPS  1  /* Wait for grace-period start. */
#define RCU_GP_DONE_GPS  2  /* Wait done for grace-period start. */
#define RCU_GP_ONOFF     3  /* Grace-period initialization hotplug. */
#define RCU_GP_INIT      4  /* Grace-period initialization. */
#define RCU_GP_WAIT_FQS  5  /* Wait for force-quiescent-state time. */
#define RCU_GP_DOING_FQS 6  /* Wait done for force-quiescent-state time. */
#define RCU_GP_CLEANUP   7  /* Grace-period cleanup started. */
#define RCU_GP_CLEANED   8  /* Grace-period cleanup complete. */

The thread loops through three phases. The mechanism rests on the combining tree — the hierarchy of rcu_node structures that gives Tree RCU its name. Each rcu_node carries a qsmask bitmask of CPUs (or child nodes) that have not yet reported a quiescent state for the current grace period.

  1. Initialization (rcu_gp_init). The kthread waits for the RCU_GP_FLAG_INIT flag (set when callbacks need a new grace period), then advances the global gp_seq sequence number and sets every qsmask bit in the tree, declaring “every CPU still owes a quiescent state for this grace period.” It also processes any pending CPU hotplug (online/offline) so that an offline CPU — which can never report a QS — does not stall the grace period forever.

  2. Waiting for / forcing quiescent states (rcu_gp_fqs_loop). Now the kthread sleeps, waking either when the grace period is reported complete or after a timeout to force quiescent states (rcu_gp_fqs → force_qs_rnp). On the normal path, CPUs report themselves: a context switch calls rcu_note_context_switch(), which (via rcu_qs() and rcu_report_qs_rdp()) grabs its leaf rcu_node lock and clears its bit in that node’s qsmask. When a leaf node’s mask reaches zero, rcu_report_qs_rnp() walks up the tree, clearing the parent’s bit, and recurses until it reaches the root — at which point the grace period is complete. The combining tree is what makes this scale: on a 256-CPU box, a flat bitmask would be a brutally contended cache line, but the tree lets, say, 16 CPUs contend on each leaf node and only the rare “last CPU of this leaf” touches the parent. This is the scalability core of Tree RCU.

    The forcing path handles CPUs that are not taking context switches — a CPU spinning in a long kernel loop, or one that is dyntick-idle. force_qs_rnp() first snapshots each CPU’s dyntick-idle counter (rcu_watching_snap_save); on a later pass it rechecks (rcu_watching_snap_recheck) and, if the CPU has been idle the whole time, reports the QS on that CPU’s behalf — an idle CPU is trivially quiescent. A CPU stuck in the kernel gets nudged toward cond_resched()-style quiescent points.

  3. Cleanup (rcu_gp_cleanup). Once the root reports complete, the kthread records the grace-period duration, advances gp_seq to mark the end (rcu_seq_end) through the whole tree, wakes every blocked synchronize_rcu() caller (rcu_sr_normal_gp_cleanup), and — if more callbacks are already queued — immediately kicks off the next grace period.

The gp_seq sequence number is the linchpin of the whole protocol. It packs both a generation counter and a low-bit state field; each CPU keeps its own snapshot in its rcu_data. A CPU can therefore tell lockless-ly (via note_gp_changes) “has a new grace period started since I last looked?” without taking any lock — which is how a quiescent state gets correctly attributed to the grace period that was open when the reader actually ran, not a later one.

What synchronize_rcu() actually does

synchronize_rcu() does not itself drive the state machine — it enqueues a request and sleeps. The v6.12 body shows two paths:

void synchronize_rcu(void)
{
	...
	RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
	                 lock_is_held(&rcu_lock_map) ||
	                 lock_is_held(&rcu_sched_lock_map),
	    "Illegal synchronize_rcu() in RCU read-side critical section");
	if (!rcu_blocking_is_gp()) {
		if (rcu_gp_is_expedited())
			synchronize_rcu_expedited();
		else
			synchronize_rcu_normal();
		return;
	}
	// Context allows vacuous grace periods.  ... runs with !PREEMPT && !SMP.
	...
}

(kernel/rcu/tree.c v6.12.) Reading top to bottom:

  • The RCU_LOCKDEP_WARN is a built-in safety net: calling synchronize_rcu() inside an RCU read-side critical section is a self-deadlock — you would be waiting for yourself to finish. lockdep Runtime Lock Validator catches it. It is also illegal to call from atomic context (it sleeps).
  • rcu_blocking_is_gp() is the vacuous grace-period fast path: very early in boot, with only one online CPU and no preemption, a grace period is trivially over (the single CPU calling synchronize_rcu() is itself a quiescent state), so the function just bumps the counters and returns without blocking.
  • Otherwise it dispatches to synchronize_rcu_expedited() (Expedited Grace Periods — microseconds, but sends IPIs) or synchronize_rcu_normal() (the throughput-optimized path). The normal path calls wait_rcu_gp(), which enqueues an rcu_synchronize completion structure via call_rcu_hurry() and then sleeps on a completion until the grace-period kthread, in its cleanup phase, signals it.

This is the key structural insight: synchronize_rcu() is call_rcu() plus a wait. It registers a callback whose only job is to wake the sleeper, then blocks on that callback. Everything about how the grace period is detected is identical for both APIs; they differ only in whether the updater blocks (synchronize_rcu) or carries on and lets the callback do the reclamation asynchronously (call_rcu and Deferred Reclamation).

Normal-GP latency is a feature, not a bug

A normal grace period commonly takes milliseconds. The requirements document is explicit that synchronize_rcu() is “optimized for throughput” and “may therefore incur several milliseconds of latency in addition to the duration of the longest RCU read-side critical section” (Requirements.rst v6.12). New users almost always misread this as a defect. It is the opposite — it is the entire point of RCU’s update side.

The latency comes from the force-quiescent-state cadence. The kthread waits roughly RCU_JIFFIES_TILL_FORCE_QS jiffies between checks, defined as (1 + (HZ > 250) + (HZ > 500)) jiffies (tree.h v6.12) — so on a HZ=1000 kernel that is 3 jiffies = 3 milliseconds between FQS scans, scaled up by nr_cpu_ids / 256 on very large systems. RCU deliberately does not poll aggressively; it lets quiescent states accumulate naturally from context switches and ticks.

Why is slowness good? Because it enables batching. Per the requirements document, “multiple concurrent invocations of synchronize_rcu() are required to use batching optimizations so that they can be satisfied by a single underlying grace-period-wait operation,” with a single operation sometimes serving “more than 1,000 separate invocations,” thereby “amortizing the per-invocation overhead down to nearly zero” (Requirements.rst v6.12). A grace period is a shared event: while one is in flight, every other updater that calls synchronize_rcu() or call_rcu() simply piggybacks on the same grace period. So the system-wide cost of N concurrent updaters is one grace period’s worth of overhead, not N. Speeding up individual grace periods would reduce the batching window and raise aggregate overhead — which is exactly why the low-latency Expedited Grace Periods path is reserved for cases that truly need it and pays for its speed with a storm of inter-processor interrupts (IPIs).

A second consequence: synchronize_rcu() “does not necessarily return immediately after the last pre-existing RCU read-side critical section completes” because “there might well be scheduling delays” (whatisRCU.rst v6.12) — the grace period is over as soon as the last reader checks in, but the sleeping updater is only woken on the next pass of the kthread. The guarantee is one-directional: when synchronize_rcu() returns, the grace period has definitely elapsed; it may have elapsed somewhat earlier.

Code — measuring grace-period latency in practice

You can observe grace-period duration directly. The cleanest tool is the rcu_torture test module’s stats, but for ad-hoc measurement use tracepoints or the rcu:rcu_grace_period trace event:

# Watch grace-period start/end events live (root, on a kernel with CONFIG_RCU_TRACE)
cd /sys/kernel/tracing
echo 1 > events/rcu/rcu_grace_period/enable
cat trace_pipe
# Sample lines: "rcu_preempt 8881 start" ... "rcu_preempt 8881 end"
#  -> the gp_seq (8881) brackets one grace period; the time delta is its duration.

Line by line: enabling the rcu_grace_period event makes the kthread emit a record each time it starts (start) or ends (end) a grace period, tagged by the gp_seq value. Matching a start to its end and subtracting timestamps gives the real latency on your hardware — typically single-digit milliseconds idle, longer under load.

To force expedited grace periods globally (trading IPI overhead for latency — useful for latency-sensitive boot or shutdown):

echo 1 > /sys/kernel/rcu_expedited    # all synchronize_rcu() take the expedited path
# or boot with rcupdate.rcu_expedited=1

This flips rcu_gp_is_expedited() so the dispatch in synchronize_rcu() chooses synchronize_rcu_expedited(). Do not leave this on under steady-state load: the IPIs perturb every CPU and can hurt throughput far more than the millisecond GP latency ever did. See Expedited Grace Periods.

A typical correct use of a grace period in update code:

/* Remove 'old' from an RCU-protected list and free it safely. */
spin_lock(&list_lock);
list_del_rcu(&old->node);          /* unlink; readers may still hold 'old' */
spin_unlock(&list_lock);
synchronize_rcu();                 /* block until every pre-existing reader finishes */
kfree(old);                        /* now provably unreachable — safe to free */

The list_del_rcu() makes old unreachable to new readers (publish side); synchronize_rcu() waits out the pre-existing readers; only then is kfree() safe. Replacing synchronize_rcu(); kfree(old); with kfree_rcu(old, node); or call_rcu(&old->rcu, free_cb); makes it asynchronous — see call_rcu and Deferred Reclamation and RCU-Protected Linked Lists.

Failure Modes and Common Misunderstandings

“synchronize_rcu() waits for my reader, the one I care about.” No — it waits for all pre-existing readers on all CPUs. A single long-running reader anywhere in the system extends the grace period for everyone. A reader that holds rcu_read_lock() across a schedule() in a non-preemptible kernel is a bug that will eventually trigger an RCU CPU stall warning (“rcu_preempt detected stalls on CPUs/tasks”) because that CPU never reports a quiescent state.

Calling synchronize_rcu() in atomic context or inside a read-side section. It sleeps, so calling it with a spinlock held, in an interrupt handler, or inside rcu_read_lock() is illegal; the RCU_LOCKDEP_WARN and the may-sleep checks fire. The inside-a-reader case is a literal self-deadlock.

Assuming new readers are blocked during a grace period. They are not. The grace period does not pause the system; readers keep entering and exiting freely. The whole design depends on new readers being cheap and unimpeded — they simply never see the old, already-unlinked object.

Expecting microsecond latency from the normal path. As above, normal grace periods are milliseconds by design. Code that calls synchronize_rcu() in a hot path will serialize on grace-period latency and crater throughput; the fix is call_rcu() (don’t block) or batching, not expedited grace periods.

Forgetting that “since v5.0” preempt/irq/bh-disabled regions are also read-side sections. The synchronize_rcu() and call_rcu() kerneldoc both state that “in v5.0 and later, regions of code across which interrupts, preemption, or softirqs have been disabled also serve as RCU read-side critical sections. This includes hardware interrupt handlers, softirq handlers, and NMI handlers” (tree.c v6.12). This is the RCU flavor consolidation: the old separate rcu_bh and rcu_sched flavors were folded into one vanilla RCU, so a single synchronize_rcu() now also waits out any CPU that merely has preemption or bottom-halves disabled — even without an explicit rcu_read_lock(). Code written against the old multi-flavor assumptions can wait longer than expected.

Alternatives and When to Choose Them

  • synchronize_rcu() (block) vs call_rcu() (defer). Use synchronize_rcu() in process context when blocking for a few milliseconds is acceptable and the code is simpler if reclamation is synchronous. Use call_rcu() when you must not block (atomic context, latency-sensitive paths, or freeing inside the update fast path) — it registers a callback and returns immediately. Both wait exactly one grace period; see call_rcu and Deferred Reclamation.
  • Normal vs Expedited Grace Periods. Normal: throughput-optimized, milliseconds, no IPIs. Expedited (synchronize_rcu_expedited): “a few tens of microseconds on small systems” (Requirements.rst v6.12) but disrupts every CPU with IPIs. Choose expedited only when grace-period latency is genuinely on a critical path (e.g., some teardown/hotplug scenarios), never as a blanket setting under load.
  • Sleepable RCU and SRCU. Plain RCU readers may not sleep (the foundation of the quiescent-state inference). When a reader must block — sleep, wait on I/O, take a mutex — use SRCU, which uses explicit per-domain counters and so tolerates sleeping readers, at the cost of slightly heavier read-side primitives.
  • Sequence Locks and seqlock. When updates are rare but readers must see a fully consistent snapshot and can cheaply retry, a seqlock (the timekeeping workhorse) is the alternative read-mostly tool — it makes readers retry on a concurrent write rather than deferring reclamation.

Production Notes

Grace periods are the heartbeat of the kernel’s read-mostly data structures — the dentry cache, network routing tables, the process list, SELinux policy. Under steady-state load on a busy server, RCU is constantly running back-to-back grace periods to service the firehose of call_rcu() callbacks; the kthread immediately starts a new grace period if callbacks remain at cleanup time. On NOHZ_FULL / housekeeping setups, grace-period work and callback invocation can be offloaded off the isolated CPUs onto dedicated kthreads so that latency-sensitive userspace on those CPUs is never perturbed — see RCU and NOCB Offloaded Callbacks.

The most common operational signal involving grace periods is the RCU CPU stall warning: if a grace period runs longer than rcu_cpu_stall_timeout (default 21 seconds), RCU dumps the offending CPUs/tasks. In production this almost always means a CPU is stuck in a long non-preemptible kernel loop (failing to reach a quiescent state), an interrupt storm, or — on preemptible kernels — a reader blocked on the ->blkd_tasks list that never gets to run. The fix is in the stalling code path (insert cond_resched(), shorten the section), not in RCU.

rcutorture, the in-tree stress tester, deliberately races readers, updaters, and grace periods (including forcing expedited and stall conditions) and is the canonical way RCU correctness is validated across releases; the grace-period guarantee it checks is precisely the one above — no reader ever observes freed memory.

Uncertain

Verify: the precise default value of rcu_cpu_stall_timeout in v6.12 (stated as 21 s from prior knowledge of CONFIG_RCU_CPU_STALL_TIMEOUT, not re-confirmed against the v6.12 Kconfig in this task). Reason: not directly fetched here. To resolve: check kernel/rcu/Kconfig.debug or update.c rcu_cpu_stall_timeout default in v6.12. uncertain

See Also

Graph View

Backlinks