call_rcu and Deferred Reclamation

call_rcu(&obj->rcu_head, callback) is the non-blocking writer path of Read-Copy-Update (RCU). Rather than the updater stopping to wait for a grace period — the interval after which every reader that could still hold a reference to the old object is guaranteed to be gone — it registers a callback and returns immediately. The kernel records the request in a per-CPU list and, once a grace period has elapsed, invokes callback(rcu_head) to do the actual reclamation (typically kfree). This is the deferred, asynchronous twin of the blocking synchronize_rcu(): same correctness guarantee, but the writer never sleeps, which is exactly what you need in atomic or interrupt context where blocking is forbidden (whatisRCU; call_rcu() in tree.c). The cost of this convenience is that the kernel now owns a queue of pending callbacks per CPU, and if a workload issues call_rcu() faster than grace periods can retire them, that queue can grow without bound — the callback-flooding hazard this note treats in depth.

This note covers the writer-side machinery: the struct rcu_head embedded in the protected object, how callbacks are batched into per-CPU segmented lists, how RCU_SOFTIRQ and the RCU kthreads invoke them, the optimized kfree_rcu()/kvfree_rcu() family, the rcu_barrier() synchronization needed before module unload, and the throttling the kernel applies under flood. Version facts are pinned to Linux 6.12 LTS (released 2024-11-17; a maintained long-term-support branch — mainline had moved into the 7.x series by the time of writing, 2026-09-04). Every code excerpt below was read from the v6.12 tree with curl, not from memory.

---

Scope: This Note Owns the Callback Machinery

RCU is covered by a cluster of notes in this vault and the split matters, because the obvious way to deepen any one of them is to re-derive the others. This note’s territory is the callback: the object that carries a deferred free, the list it waits on, the engine that runs it, and what happens when there are too many of them.

flowchart TB
  FUND["<b>[[Read-Copy-Update Fundamentals]]</b><br/>WHAT RCU IS — the read/copy/update/reclaim<br/>triad, flavour consolidation, the honest costs,<br/>and <i>which</i> deferral primitive to pick"]
  GP["<b>[[RCU Grace Periods]]</b> + <b>[[Tree RCU]]</b><br/>HOW 'all readers are done' is DETECTED —<br/>quiescent states, the combining tree, gp_seq"]
  ME["<b>THIS NOTE</b> — WHAT HAPPENS TO THE CALLBACK<br/>struct rcu_head · the segmented list ·<br/>rcu_do_batch and its throttles · the kvfree bulk path ·<br/>rcu_barrier and kvfree_rcu_barrier ·<br/>callback flooding and the memory-footprint bill"]
  NOCB["<b>[[RCU and NOCB Offloaded Callbacks]]</b><br/>WHERE the callback runs when you move it —<br/>rcu_nocbs, rcuog/rcuop/rcuoc kthreads, CPU isolation"]
  EXP["<b>[[Expedited Grace Periods]]</b><br/>the IPI-driven short cut and its cost"]
  FUND -->|"defers callback<br/>machinery to"| ME
  GP -->|"supplies the<br/>gp_seq that promotes<br/>segments"| ME
  ME -->|"the flooding failure mode<br/>whose production answer is"| NOCB
  ME -.->|"back-pressure option"| EXP

Where this note sits in the RCU cluster. What it shows: the four neighbours each own a different question, and this note owns the middle of the pipeline — everything between “the updater has decided to defer” and “the memory is actually free”. The insight to take: the boundary is stated by the neighbours themselves, not invented here. Read-Copy-Update Fundamentals describes its own scope as staying “at the level of what RCU is and when to reach for it” and names “the asynchronous-callback machinery in call_rcu and Deferred Reclamation” as deferred to this note; its rcu_barrier() table row likewise says the callbacks-versus-grace-period distinction is “covered in call_rcu and Deferred Reclamation”. So the choice between synchronize_rcu(), call_rcu() and kfree_rcu() belongs there and is not re-derived here; what each one then does to your memory belongs here.

Concretely: if your question is “should I use call_rcu() or synchronize_rcu() here?”, read Read-Copy-Update Fundamentals. If it is “why is my machine sitting on 400 MB of freed-but-not-freed objects?”, or “why did my module unload crash three seconds after it succeeded?”, or “what actually happens between call_rcu() returning and my callback running?”, you are in the right note. If it is “why did the grace period take 40 ms?”, read RCU Grace Periods.

---

Mental Model

Think of call_rcu() as dropping a note into a mailbox marked “deliver after the next safe point.” You do not wait by the mailbox; you walk away. A background postal service (RCU’s grace-period machinery) periodically determines that a safe point has passed — every reader who could have been looking at your old data has since let go — and only then delivers all the accumulated notes by running their callbacks. The writer’s latency is decoupled entirely from the reader’s; the writer pays microseconds to enqueue, and the system pays the millisecond-scale grace-period wait in the background, amortized across every callback queued during that window.

flowchart TB
  W["Writer:<br/>unpublish old object<br/>(rcu_assign_pointer / list_del_rcu)"] --> CR["call_rcu(&obj->rcu_head, cb)"]
  CR --> ENQ["Enqueue rcu_head into<br/>this CPU's segmented<br/>callback list (RCU_NEXT_TAIL)"]
  ENQ --> RET["Writer returns immediately<br/>(no blocking)"]
  GP["Grace period elapses<br/>(all pre-existing readers done)"] -.->|"advances segments"| READY["Callbacks reach<br/>RCU_DONE_TAIL"]
  READY --> SI{"use_softirq?"}
  SI -->|"yes (default)"| SOFT["RCU_SOFTIRQ -> rcu_core()<br/>-> rcu_do_batch()"]
  SI -->|"no (PREEMPT_RT / NOCB)"| KT["rcuc / rcuo kthread<br/>-> rcu_do_batch()"]
  SOFT --> INV["Invoke cb(rcu_head)<br/>e.g. kfree(obj)"]
  KT --> INV

The deferred-reclamation pipeline. What it shows: the writer’s path (top) ends the instant the rcu_head is enqueued; the invocation path (bottom) is driven asynchronously by the grace-period machinery, which promotes callbacks through the segmented list until they are “done” and then runs them either from the RCU_SOFTIRQ softirq or from a dedicated kthread. The insight to take: there is a deliberate, queue-mediated decoupling between when you ask for reclamation and when reclamation happens — and that queue is the thing that can overflow under flooding.

The delay is unbounded, and that is the whole story

The single most important property of call_rcu() is negative: it makes no promise whatsoever about when your callback runs. Not a bound in milliseconds, not a bound in callbacks, not a bound at all. Every failure mode in this note is a consequence of that one fact, so it is worth drawing the timeline explicitly, with the stalls marked.

sequenceDiagram
    autonumber
    participant U as Updater<br/>(any context)
    participant L as This CPU's<br/>rcu_segcblist
    participant GP as Grace-period<br/>kthread (rcu_preempt)
    participant IV as Invoker<br/>(RCU_SOFTIRQ / rcuc / rcuoc)
    participant M as Allocator

    U->>L: call_rcu(&obj->rcu_head, cb)
    Note over U,L: ~tens of nanoseconds.<br/>No allocation. No blocking.<br/>Lands in RCU_NEXT_TAIL.
    U-->>U: returns immediately
    rect rgb(250, 235, 215)
    Note over L,GP: STALL 1 — the callback has no grace-period<br/>number yet. If no GP is in progress one must be<br/>STARTED first. With CONFIG_RCU_LAZY it may be<br/>deliberately held back for SECONDS.
    end
    GP->>L: rcu_segcblist_accelerate(): tag segments with gp_seq
    rect rgb(250, 235, 215)
    Note over GP: STALL 2 — wait out the grace period.<br/>Requirements.rst: "several milliseconds… in<br/>addition to the duration of the longest RCU<br/>read-side critical section". A looping reader,<br/>a preempted reader, or a stuck GP kthread<br/>extends this without limit.
    end
    GP->>L: rcu_segcblist_advance(): segments reach RCU_DONE_TAIL
    GP->>IV: invoke_rcu_core() / wake the offload kthread
    rect rgb(250, 235, 215)
    Note over IV: STALL 3 — throttled invocation.<br/>rcu_do_batch() runs at most<br/>bl = max(blimit, pending >> 7) per pass,<br/>and bails out after ~3 ms or on need_resched().<br/>The rest waits for the next pass.
    end
    IV->>M: cb(rcu_head) → kfree(obj)
    Note over U,M: Only NOW is the memory actually free.<br/>Between step 1 and step 9 the object is<br/>allocated, unreachable, and unusable.

The full call_rcu() timeline with its three independent stalls. What it shows: the writer’s fast return buys latency for the updater and nothing else; the object’s memory is held from step 1 to step 9, and each of the three shaded regions can be extended independently. The insight to take: these are three different problems with three different cures, and confusing them is why “my RCU memory usage is too high” is so often misdiagnosed. Stall 1 is a scheduling problem, cured by call_rcu_hurry() or by not enabling CONFIG_RCU_LAZY. Stall 2 is a grace-period problem, cured (at a price) by synchronize_rcu_expedited() or diagnosed as an RCU stall. Stall 3 is an invocation-throughput problem, cured by raising rcutree.blimit, lowering rcutree.rcu_divisor, or offloading to rcuo kthreads. The kernel’s own Requirements.rst frames the whole thing as a forward-progress obligation rather than a performance nicety: “not only are memory sizes finite but also callbacks sometimes do wakeups, and sufficiently deferred wakeups can be difficult to distinguish from system hangs.”

The contrast with synchronize_rcu() is exactly the absence of that first arrow’s “returns immediately”. A blocking updater cannot get ahead of the grace-period machinery, so its memory footprint is bounded by construction. checklist.rst (v6.12) states the trade in one sentence: synchronize_rcu() “automatically self-limits: if grace periods are delayed for whatever reason, then the synchronize_rcu() primitive will correspondingly delay updates. In contrast, code using call_rcu() should explicitly limit update rate in cases where grace periods are delayed, as failing to do so can result in excessive realtime latencies or even OOM conditions.”

---

The struct rcu_head: the callback’s anchor

For the kernel to track a deferred free without allocating memory at the worst possible moment (you are, after all, in the middle of freeing things), the bookkeeping must live inside the object being freed. That is the struct rcu_head — a tiny two-word structure you embed in any RCU-protected object:

struct callback_head {
        struct callback_head *next;
        void (*func)(struct callback_head *head);
} __attribute__((aligned(sizeof(void *))));
#define rcu_head callback_head

(include/linux/types.h)

Two design points are worth dwelling on. First, rcu_head is literally a #define for callback_head, a type shared with the task_work subsystem — RCU and deferred task work both reuse the same “link + function pointer” primitive, which is why the kernel-doc comment above it reads “callback structure for use with RCU and task_work”. Second, the structure is explicitly aligned to the size of a pointer, and the header spells out both consequences of that (v6.12):

“The alignment is required to guarantee that bit 0 of @next will be [zero]… This guarantee is important for few reasons: future call_rcu_lazy() will make use of lower bits in the pointer; the structure shares storage space in struct page with @compound_head, which encode PageTail() in bit 0. The guarantee is needed to avoid false-positive PageTail().”

That second reason is the concrete one and it is easy to miss: struct page is a union-packed 64-byte object, and the rcu_head a page’s deferred free uses overlaps the compound_head field whose bit 0 is the “this is a tail page” flag. A misaligned rcu_head would therefore not merely be untidy — it would make the memory-management code mistake an ordinary page for the tail of a compound page. The comment also notes that the explicit __attribute__((aligned(sizeof(void *)))) is belt-and-braces on most targets: “on most architectures it happens naturally due ABI requirements, but some architectures (like CRIS) have weird ABI and we need to ask it explicitly.”

packet-beta
0-0: "bit 0 of ->next — GUARANTEED ZERO by the alignment attribute"
1-63: "->next : the remaining 63 bits of struct callback_head *"
64-127: "->func : void (*)(struct callback_head *head)"

The 16-byte struct rcu_head on a 64-bit kernel, drawn at bit accuracy (bit 0 of ->next shown first, as the reserved field rather than in memory order). What it shows: two words, of which 127 bits carry a pointer and a function pointer and one bit is deliberately reserved. The insight to take: that single reserved bit is why the __attribute__((aligned(sizeof(void *)))) is not cosmetic. When an rcu_head is unioned into struct page, it overlaps compound_head, whose bit 0 encodes PageTail(); a misaligned head would make the memory-management code believe an ordinary page is the tail of a compound page. The header also earmarks the same bit for a future call_rcu_lazy()’s tagging. Sixteen bytes, and one of the 128 bits is load-bearing for a completely different subsystem.

The ->next pointer threads the object onto a singly linked callback list; ->func is the reclamation function the kernel will call. Note the resulting cost, which Read-Copy-Update Fundamentals lists in its bill of RCU’s overheads: every object you intend to reclaim with call_rcu() must carry two extra pointers, permanently. For a task_struct that is noise; for a struct page or a networking sk_buff it is enough to force the union tricks just described.

Because the rcu_head lives in the object, no allocation is needed to queue a callback — the writer just fills in ->func, links the object onto its CPU’s list, and leaves. The callback receives a pointer to the embedded rcu_head, from which it recovers the enclosing object via container_of(). A canonical pattern, from the kernel’s own process-teardown code:

void release_task(struct task_struct *p)
{
        write_lock(&tasklist_lock);
        list_del_rcu(&p->tasks);          // unpublish from the task list
        write_unlock(&tasklist_lock);
        call_rcu(&p->rcu, delayed_put_task_struct);  // free after a GP
}

(listRCU) — the task_struct is removed from the globally visible task list, but the actual put is deferred until any reader walking for_each_process() without the lock has finished, which a grace period guarantees.

---

Mechanical Walk-through: from call_rcu() to invocation

Enqueue

Every call_rcu() and its lazy/hurry variants funnel into one internal routine, __call_rcu_common() (tree.c). The essential sequence:

static void
__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
{
        ...
        WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); // misaligned rcu_head!
        if (debug_rcu_head_queue(head)) { ... WRITE_ONCE(head->func, rcu_leak_callback); return; }
        head->func = func;
        head->next = NULL;
        local_irq_save(flags);
        rdp = this_cpu_ptr(&rcu_data);
        lazy = lazy_in && !rcu_async_should_hurry();
        ...
        check_cb_ovld(rdp);
        if (unlikely(rcu_rdp_is_offloaded(rdp)))
                call_rcu_nocb(rdp, head, func, flags, lazy);
        else
                call_rcu_core(rdp, head, func, flags);
        local_irq_restore(flags);
}

Reading this line by line teaches most of the writer path. The WARN_ON_ONCE on alignment is the runtime check backing the rcu_head alignment requirement above — a misaligned head is a bug. debug_rcu_head_queue() (active under CONFIG_DEBUG_OBJECTS_RCU_HEAD) detects a double call_rcu() on the same head: if the head is already queued, the kernel leaks the callback deliberately (rcu_leak_callback) rather than corrupting the list, and prints the offending function. Interrupts are disabled (local_irq_save) because the per-CPU callback list can be touched from interrupt context, and this_cpu_ptr(&rcu_data) selects this CPU’s rcu_data — the per-CPU RCU state that owns the callback list. check_cb_ovld(rdp) is the flood detector, discussed below. Finally, the callback is queued either to the NOCB (“no-callback”) offload path — see RCU and NOCB Offloaded Callbacks — or, in the common case, to the local core path. The whole thing runs in tens of nanoseconds: no allocation, no blocking, no grace-period wait.

A 6.12-era subtlety: when CONFIG_RCU_LAZY is built in — which it is not by default, see the resolved callout below — call_rcu() is lazy by default. The public call_rcu() passes enable_rcu_lazy as the lazy_in argument; lazy callbacks are held back rather than immediately driving a new grace period, which saves power on idle systems by letting callbacks accumulate. Code that needs prompt reclamation uses the explicit call_rcu_hurry() variant, which forces lazy = false (tree.c lazy section).

Batching: the segmented callback list

Callbacks are not held in one undifferentiated pile. Each CPU’s rcu_data owns a struct rcu_segcblist — a segmented list split into four regions by grace-period readiness (rcu_segcblist.h):

Segment	Meaning
RCU_DONE_TAIL	Callbacks whose grace period has elapsed — ready to invoke now.
RCU_WAIT_TAIL	Callbacks waiting on the current grace period.
RCU_NEXT_READY_TAIL	Callbacks that arrived before the next GP started; the next GP will retire them.
RCU_NEXT_TAIL	Newly arrived callbacks not yet assigned a grace-period number.

A fresh call_rcu() lands at RCU_NEXT_TAIL. As grace periods begin and end, the RCU core advances callbacks down the segments — RCU_NEXT_TAIL → RCU_NEXT_READY_TAIL → RCU_WAIT_TAIL → RCU_DONE_TAIL — by relabeling segment boundaries rather than moving list nodes (the header keeps a per-segment ->gp_seq[] recording which grace period each segment is waiting for). The payoff of this design is batching: a single grace period retires all the callbacks that were waiting on it, so the per-callback cost of grace-period detection is amortized across however many call_rcu()s accumulated in that window. Under load this can be thousands of callbacks per grace period; that amortization is what makes deferred reclamation cheap in aggregate.

stateDiagram-v2
    direction LR
    [*] --> NEXT: call_rcu()<br/>rcu_segcblist_enqueue()
    state "RCU_NEXT_TAIL" as NEXT
    state "RCU_NEXT_READY_TAIL" as READY
    state "RCU_WAIT_TAIL" as WAIT
    state "RCU_DONE_TAIL" as DONE
    state "invoked, ->func poisoned" as GONE

    NEXT: No grace-period number yet.<br/>"Might have arrived after the<br/>next GP started" — the CPU<br/>cannot know, so it is conservative.
    READY: gp_seq[] = the NEXT grace period.<br/>Arrived before that GP began.
    WAIT: gp_seq[] = the CURRENT grace period.<br/>Cannot be empty unless<br/>RCU_NEXT_READY_TAIL is also empty.
    DONE: Grace period elapsed.<br/>Safe to invoke NOW.
    GONE: rcu_do_batch() ran f(rhp)<br/>and wrote ->func = 0.

    NEXT --> READY: rcu_segcblist_accelerate(seq)<br/>relabel a tail pointer, assign gp_seq
    READY --> WAIT: a new grace period starts<br/>(__note_gp_changes)
    WAIT --> DONE: rcu_segcblist_advance(seq)<br/>this GP completed
    NEXT --> DONE: accelerate + advance in<br/>one rcu_core() pass
    DONE --> GONE: rcu_do_batch(), up to<br/>bl callbacks per pass
    GONE --> [*]
    DONE --> DONE: batch limit / 3 ms<br/>time limit hit — remainder REQUEUED

A callback’s journey through the four segments of rcu_segcblist. What it shows: the four segments are not four lists — they are four tail pointers into one singly-linked list, and a “promotion” is an assignment to rsclp->tails[i], not a walk. rcu_segcblist_advance() in kernel/rcu/rcu_segcblist.c (v6.12) does exactly WRITE_ONCE(rsclp->tails[RCU_DONE_TAIL], rsclp->tails[i]) in a loop over at most three indices. The insight to take: promoting a hundred thousand callbacks costs the same three pointer stores as promoting one. That is the mechanism behind the amortisation claim — grace-period detection is expensive, so the design makes the number of callbacks waiting on a grace period irrelevant to the cost of detecting it. The self-loop on RCU_DONE_TAIL is the one that matters for the failure modes below: reaching “done” is not the same as being freed, and a large DONE segment drains at a throttled rate.

The NEXT → READY transition is the interesting one, because it is where the conservatism gets undone. When a callback is enqueued, the local CPU genuinely does not know where the global grace period stands — include/linux/rcu_segcblist.h puts it as “there is some uncertainty as to when a given GP starts and ends, but a CPU knows the exact times if it is the one starting or ending the GP.” So new callbacks are parked in RCU_NEXT_TAIL with no grace-period number, which is the safe assumption. Later, when better information arrives, rcu_segcblist_accelerate(seq) re-labels them. Its comment states the design directly:

“RCU does not synchronize the beginnings and ends of grace periods, and… callbacks are posted locally. This in turn means that the callbacks must be labelled conservatively early on, as getting exact information would degrade both performance and scalability. When more accurate grace-period information becomes available, previously posted callbacks can be ‘accelerated’, marking them to complete at the end of the earlier grace period.”

Acceleration is therefore a pure optimisation: it never makes a callback wait longer, only shorter, and skipping it entirely would still be correct. That is why rcu_accelerate_cbs() is called opportunistically from several places — from note_gp_changes(), from the grace-period cleanup path, and from call_rcu_core() when a CPU notices it is accumulating callbacks.

Two structural details round out the picture. The struct rcu_segcblist carries long seglen[RCU_CBLIST_NSEGS] — a per-segment count — as well as a total len, and on CONFIG_RCU_NOCB_CPU kernels that total is an atomic_long_t rather than a plain long, because an offload kthread on another CPU may be draining the list concurrently. And the whole structure lives in a header the comment calls “seemingly RCU-private” because SRCU embeds one too: “the size of the TREE SRCU srcu_struct structure depends on these definitions.” The segmented list is shared machinery — see Sleepable RCU and SRCU and RCU Grace Periods for how segment advancement is tied to the grace-period counter.

Uncertain

Verify: the large ASCII “NOCB Offloading state machine” diagram at the top of include/linux/rcu_segcblist.h (v6.12). Reason: it is stale in-tree documentation. The diagram’s states are named in terms of SEGCBLIST_RCU_CORE, SEGCBLIST_LOCKING and SEGCBLIST_KTHREAD_GP, but the #define block immediately below it defines only two flags — SEGCBLIST_ENABLED BIT(0) and SEGCBLIST_OFFLOADED BIT(1) — and grepping the whole of kernel/rcu/ at v6.12 (tree.c, tree.h, tree_nocb.h, tree_plugin.h, rcu_segcblist.c, rcu_segcblist.h) finds zero uses of the three missing names outside comments. The offload state machine was simplified and the header comment was not updated. To resolve: git log -p include/linux/rcu_segcblist.h to find the commit that removed the flags, and file a documentation fix. Do not cite that diagram for how offloading works in 6.12; RCU and NOCB Offloaded Callbacks describes the current mechanism. This is the general lesson restated: in-tree comments go stale; verify every named identifier against the code at the tag you are reading. uncertain

Invocation: RCU_SOFTIRQ and the RCU kthreads

Once callbacks reach RCU_DONE_TAIL, something must run them. In the default configuration that “something” is the RCU_SOFTIRQ softirq. RCU registers a handler at init:

open_softirq(RCU_SOFTIRQ, rcu_core_si);

(tree.c init), and RCU_SOFTIRQ is deliberately the last entry in the softirq enum (include/linux/interrupt.h comments “Preferable RCU should always be the last softirq”), giving RCU callback processing the lowest softirq priority so it does not starve timer or network softirqs. The scheduler-clock tick, or the completion of a grace period, calls invoke_rcu_core():

static void invoke_rcu_core(void)
{
        if (!cpu_online(smp_processor_id())) return;
        if (use_softirq)
                raise_softirq(RCU_SOFTIRQ);
        else
                invoke_rcu_core_kthread();
}

use_softirq defaults to !IS_ENABLED(CONFIG_PREEMPT_RT) — true on a normal kernel, false on a real-time kernel (tree.c). On PREEMPT_RT, running callbacks in softirq context would inject unbounded latency into the softirq path, so RCU instead wakes a per-CPU rcuc kthread that can be scheduled and preempted like any task. Either route lands in rcu_core() → rcu_do_batch(), which extracts the done segment and runs each callback:

f = rhp->func;
WRITE_ONCE(rhp->func, (rcu_callback_t)0L);
f(rhp);          // the actual reclamation, e.g. kfree(container_of(rhp, ...))

rcu_do_batch() clears ->func before calling it (poisoning the head to catch reuse) and runs callbacks until a batch limit is hit. There are, in v6.12, four engines that can end up executing that loop, and knowing which one your system uses is the difference between “callbacks are slow” and “callbacks are slow for this reason”:

flowchart TB
  START["A callback has reached RCU_DONE_TAIL.<br/>Who runs it?"]
  Q1{"Was it queued with<br/>kfree_rcu / kvfree_rcu<br/>on the array fast path?"}
  KRC["<b>Engine 4 — the kvfree workqueue</b><br/>kfree_rcu_monitor() on a delayed_work,<br/>draining every KFREE_DRAIN_JIFFIES = 5*HZ.<br/>Runs kfree_bulk() on a page of pointers.<br/>Never touches rcu_do_batch() at all."]
  Q2{"Is this CPU offloaded?<br/>(rcu_rdp_is_offloaded)"}
  NOCB["<b>Engine 3 — rcuoc kthread</b><br/>A schedulable, pinnable kthread.<br/>Chosen by rcu_nocbs= at boot.<br/>See [[RCU and NOCB Offloaded Callbacks]]."]
  Q3{"use_softirq?<br/>(defaults to<br/>!CONFIG_PREEMPT_RT)"}
  SI["<b>Engine 1 — RCU_SOFTIRQ</b><br/>raise_softirq(RCU_SOFTIRQ) → rcu_core_si()<br/>→ rcu_core() → rcu_do_batch().<br/>The default on an ordinary kernel."]
  KT["<b>Engine 2 — per-CPU rcuc kthread</b><br/>invoke_rcu_core_kthread().<br/>The PREEMPT_RT default, so callback<br/>work is schedulable and preemptible."]
  START --> Q1
  Q1 -->|yes| KRC
  Q1 -->|"no — plain call_rcu()"| Q2
  Q2 -->|yes| NOCB
  Q2 -->|no| Q3
  Q3 -->|"true"| SI
  Q3 -->|"false"| KT
  NOTE["All four are throttled, but differently.<br/>Engines 1–3 share rcu_do_batch()'s blimit and<br/>3 ms time budget; engine 4 is paced by a<br/>5-second workqueue timer instead."]
  SI -.-> NOTE
  KT -.-> NOTE
  NOCB -.-> NOTE
  KRC -.-> NOTE

The four callback-invocation engines and how one is selected. What it shows: the choice is made by three independent axes — whether the object went down the kvfree_rcu() array path, whether the CPU is offloaded, and whether the kernel is PREEMPT_RT — and the four destinations have genuinely different latency characteristics. The insight to take: engine 4 is the one people forget. A kfree_rcu() object does not ride the rcu_segcblist; it is batched into a page of pointers and drained by a workqueue on a five-second timer, so its worst-case reclamation delay is bounded below by that timer and not by the grace period. That is why rcu_barrier() — which entrains a callback on the segmented list — does not wait for kfree_rcu() objects, and why v6.12 had to add a separate kvfree_rcu_barrier(). If you are hunting deferred memory, check which engine owns it before tuning anything.

The throttles inside rcu_do_batch() deserve reading literally, because “only 10 at a time” is a considerable simplification of what v6.12 actually does:

	pending = rcu_segcblist_get_seglen(&rdp->cblist, RCU_DONE_TAIL);
	div = READ_ONCE(rcu_divisor);
	div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div;
	bl = max(rdp->blimit, pending >> div);
	if ((in_serving_softirq() || rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING) &&
	    (IS_ENABLED(CONFIG_RCU_DOUBLE_CHECK_CB_TIME) || unlikely(bl > 100))) {
		long rrn = READ_ONCE(rcu_resched_ns);
		rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn;
		tlimit = local_clock() + rrn;
		...
	}

Line by line. pending is how many callbacks are sitting in the done segment right now. rcu_divisor defaults to 7, so pending >> div is pending / 128: the batch limit scales with the backlog, and a CPU sitting on 128,000 ready callbacks will run 1,000 of them in one pass rather than 10. bl = max(rdp->blimit, pending >> div) means the fixed blimit is a floor, not a ceiling — the “10” figure is only the steady-state minimum. The second clause installs a wall-clock budget: rcu_resched_ns defaults to 3 * NSEC_PER_MSEC, so once the batch limit exceeds 100 callbacks, the loop also bails out after roughly 3 ms to avoid starving other softirq vectors. rcu_do_batch_check_time() only reads local_clock() “once per 32 consecutive callbacks” to keep the check itself cheap. And in the loop, the softirq path stops early only if it has work to do — if (count >= bl && (need_resched() || !is_idle_task(current))) break; — so an idle CPU will happily drain the whole backlog.

All of that is throttling by design: it caps the damage a callback flood does to interrupt latency, at the cost of making the flood drain more slowly. The remainder is put back with rcu_segcblist_insert_done_cbs() and picked up on the next pass. The orthogonal NOCB configuration moves invocation off the softirq/rcuc path entirely onto rcuo kthreads that can be pinned away from latency-sensitive CPUs; that is its own topic in RCU and NOCB Offloaded Callbacks.

One ordering guarantee is worth extracting from rcu_do_batch()’s comment, because it is the reason a callback may safely dereference anything the updater wrote before calling call_rcu(): “Callbacks execution is fully ordered against preceding grace period completion (materialized by rnp->gp_seq update) thanks to the smp_mb__after_unlock_lock() upon node locking required for callbacks advancing.” The call_rcu() kernel-doc states the same guarantee from the caller’s side, and it is stronger than most people assume: “if CPU A invoked call_rcu() and CPU B invoked the resulting RCU callback function func(), then both CPU A and CPU B are guaranteed to execute a full memory barrier during the time interval between the call to call_rcu() and the invocation of func() — even if CPU A and CPU B are the same CPU (but again only if the system has more than one CPU).” See Memory Barriers in the Linux Kernel for what smp_mb__after_unlock_lock() costs (it is a real sync on PowerPC and free everywhere else).

---

kfree_rcu() and kvfree_rcu(): the no-callback fast path

The overwhelmingly common RCU callback is just kfree(obj). Writing a one-line wrapper function for every such type is tedious, bloats the kernel with near-identical functions, and — crucially — forces any module that uses them to call the expensive rcu_barrier() at unload time (see below). kfree_rcu() eliminates all three problems:

#define kfree_rcu(ptr, rhf)  kvfree_rcu_arg_2(ptr, rhf)
#define kvfree_rcu(ptr, rhf) kvfree_rcu_arg_2(ptr, rhf)

(rcupdate.h) — in 6.12 kfree_rcu and kvfree_rcu are the same macro; the latter handles objects allocated with either kmalloc or vmalloc. You pass the object pointer and the name of the rcu_head field within it. There is no callback function at all. The macro encodes the offset of the rcu_head inside the object and verifies it is small enough:

#define kvfree_rcu_arg_2(ptr, rhf)                                          \
do {                                                                        \
        typeof (ptr) ___p = (ptr);                                          \
        if (___p) {                                                         \
                BUILD_BUG_ON(!__is_kvfree_rcu_offset(offsetof(typeof(*(ptr)), rhf))); \
                kvfree_call_rcu(&((___p)->rhf), (void *) (___p));           \
        }                                                                   \
} while (0)

__is_kvfree_rcu_offset(offset) is ((offset) < 4096). The reason for the 4096-byte ceiling is a clever overloading: the RCU core stores either a function pointer or an offset in the same field, and since the kernel disallows function addresses in the low 4096 bytes of virtual memory, any value under 4096 is unambiguously an offset rather than a function (rcupdate.h kfree_rcu doc). If your rcu_head is more than 4095 bytes into the structure, the BUILD_BUG_ON fails at compile time and you must either move the field or fall back to call_rcu().

There is also a headless single-argument form, kfree_rcu_mightsleep(ptr) (and its alias kvfree_rcu_mightsleep), for objects that have no embedded rcu_head at all. As the name warns, it may sleep, so it is only legal from might_sleep() context. (In older kernels this was spelled kfree_rcu(ptr) with one argument; it was renamed to make the sleeping behavior explicit.)

Beyond ergonomics, kvfree_rcu() has a genuinely different implementation under the hood. kvfree_call_rcu() does not simply enqueue an rcu_head per object. It maintains per-CPU kfree_rcu_cpu (krcp) structures holding arrays of pointers (bulk blocks), so many freed objects share one block and one grace-period snapshot. The batch is drained every KFREE_DRAIN_JIFFIES = 5 * HZ (five seconds) from workqueue context, where it is safe to call the page allocator (kvfree_call_rcu doc, tree.c). The header comment states the goal directly: “batch requests together to reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.”

The bulk path, and why it is roughly 500× better

The saving is worth quantifying, because it explains why checklist.rst pushes so hard for kfree_rcu() over call_rcu().

flowchart TB
  subgraph CR["call_rcu(&obj->rcu_head, kfree_cb) — one head per object"]
    direction TB
    O1["obj 1<br/>+16 B rcu_head"] --> O2["obj 2<br/>+16 B rcu_head"] --> O3["obj 3<br/>+16 B rcu_head"] --> ON["… obj N"]
    CRN["N linked-list nodes on the segcblist.<br/>N indirect calls in rcu_do_batch().<br/>N × 16 B of per-object overhead.<br/>Throttled by blimit."]
    ON --> CRN
  end
  subgraph KV["kfree_rcu(ptr, rhf) — one page per ~507 objects"]
    direction TB
    PG["struct kvfree_rcu_bulk_data — exactly one page<br/>list (16 B) + gp_snap (16 B) + nr_records (8 B)<br/>then records[] : an array of void *"]
    PTRS["records[0..506] → the objects<br/>ONE grace-period snapshot for all of them"]
    FREE["kfree_bulk(nr_records, records)<br/>— ONE slab call frees the lot"]
    PG --> PTRS --> FREE
  end
  CH["Two channels, indexed by is_vmalloc_addr(ptr):<br/>channel 0 → kfree_bulk()<br/>channel 1 → a vfree() loop"]
  KV --> CH
  FB["Fallback (channel 3): if no page can be allocated,<br/>thread the object's own rcu_head onto krcp->head.<br/>Headless kvfree_rcu(one_arg) instead does<br/>synchronize_rcu() + kvfree() inline — and may sleep."]
  KV --> FB

call_rcu()’s per-object callback versus kvfree_rcu()’s bulk block. What it shows: the bulk path replaces N rcu_head list nodes and N indirect calls with one page of pointers, one grace-period snapshot, and one kfree_bulk(). The insight to take: the numbers are concrete. KVFREE_BULK_MAX_ENTR is defined as (PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *), and the header comment says the sizing is chosen “based on the fact that the size of kvfree_rcu_bulk_data structure becomes exactly one page” — so on a 64-bit kernel with 4 KiB pages the header is 40 bytes and the array holds 507 pointers. One page, one grace period, one bulk free, five hundred objects. The records[] array also uses __counted_by(nr_records), so the bounds are visible to the compiler’s array-bounds sanitiser.

Uncertain

Verify: the specific figure of 507 entries per bulk block. Reason: it is arithmetic on verified inputs, not a value read from a running kernel — KVFREE_BULK_MAX_ENTR is (PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *), and struct kvfree_rcu_bulk_data is struct list_head list (16 B) + struct rcu_gp_oldstate gp_snap (two unsigned long, 16 B, read from include/linux/rcutree.h v6.12) + unsigned long nr_records (8 B) = 40 B, giving (4096 - 40) / 8 = 507. The figure is therefore correct only for a 64-bit kernel with 4 KiB pages; on arm64 with 64 KiB pages it is 8,187, and on 32-bit it differs again. To resolve: compile with a BUILD_BUG_ON or read /sys/kernel/debug/ tracing output for rcu_invoke_kfree_bulk_callback, whose nr_records argument is the live value. uncertain

The three paths in kvfree_call_rcu() are worth naming because they degrade in a specific order, and the degradation is what you see under memory pressure. The fast path is add_ptr_to_bulk_krc_lock(), which appends the pointer to the current bulk block, or allocates a fresh page if the block is full. That allocation is deliberately timid: __get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN), with in-line comments explaining each flag — __GFP_NORETRY “allows a light-weight direct reclaim… apart of that it forbids any OOM invoking what is also beneficial since we are about to release memory soon”; __GFP_NOMEMALLOC “prevents from consuming of all the memory reserves”; __GFP_NOWARN because “an allocation can be failed under low memory or high memory pressure scenarios.” A per-CPU cache of spare pages (krcp->bkvcache, refilled to rcu_min_cached_objs = 5 pages by a background worker) usually avoids the allocator entirely.

The first fallback is used when no page can be had: the object’s own embedded rcu_head is threaded onto a plain krcp->head list with a single grace-period cookie taken via get_state_synchronize_rcu(). Slower, but no allocation. The second fallback applies only to the headless single-argument form, which has no rcu_head to fall back on; there, kvfree_call_rcu() does the thing it exists to avoid — synchronize_rcu(); kvfree(ptr); inline. That is why the API is spelled kfree_rcu_mightsleep(): the sleep is not hypothetical, it is the documented degraded path.

Draining is done by kfree_rcu_monitor(), a delayed_work rearmed every KFREE_DRAIN_JIFFIES. It first drains anything already past its grace period (kvfree_rcu_drain_ready(), gated on poll_state_synchronize_rcu_full(&bnode->gp_snap) — a poll, not a wait), then queues a batch via queue_rcu_work(), which is a workqueue item that runs after a grace period. KFREE_N_BATCHES = 2 batches are available per CPU, so a new batch can accumulate while the previous one is still waiting out its grace period. The final free is kfree_bulk(bnode->nr_records, bnode->records) for channel 0 and a vfree() loop for channel 1, after which the now-empty page is returned to krcp->bkvcache for reuse rather than to the allocator.

kvfree_rcu_barrier() — new in v6.12

Because kfree_rcu() objects never enter the rcu_segcblist, rcu_barrier() cannot see them, and for years that was simply a gap: kfree_rcu() needed no barrier because its callback lives in the core kernel, so a module could unload safely. But kmem_cache_destroy() has the same lifetime problem for a different reason — destroying a slab cache while objects allocated from it are still queued for a deferred free is a use-after-free of the cache. v6.12 adds the missing primitive:

/**
 * kvfree_rcu_barrier - Wait until all in-flight kvfree_rcu() complete.
 *
 * Note that a single argument of kvfree_rcu() call has a slow path that
 * triggers synchronize_rcu() following by freeing a pointer. It is done
 * before the return from the function. Therefore for any single-argument
 * call that will result in a kfree() to a cache that is to be destroyed
 * during module exit, it is developer's responsibility to ensure that all
 * such calls have returned before the call to kmem_cache_destroy().
 */
void kvfree_rcu_barrier(void)

Its implementation is a two-pass sweep of every possible CPU: first force each krcp with pending objects to queue a batch (looping and calling flush_rcu_work() if a previous batch is still in flight), then, in a second pass, cancel_delayed_work_sync(&krcp->monitor_work) and flush_rcu_work() on both KFREE_N_BATCHES slots. Dating it by reading the same file at several tags: kernel/rcu/tree.c contains zero occurrences of kvfree_rcu_barrier at v6.10 and v6.11, and three at v6.12 and v6.13. So this is genuinely a 6.12 addition — any text describing “kfree_rcu() needs no barrier” without qualification predates it, and is still true for module text but not for slab caches.

---

rcu_barrier(): waiting for callbacks to drain

call_rcu() is fire-and-forget, which creates a lifetime problem: if the code that registered a callback can disappear, you must ensure no callback is still pending against it. The flagship case is module unload. If a module calls call_rcu(&obj->rcu_head, my_module_func) and is then unloaded, a callback referencing my_module_func may still be sitting in some CPU’s list; invoking it after the module’s text is freed jumps into freed memory. rcu_barrier() is the fix — it blocks until every callback registered before the barrier has been invoked:

/**
 * rcu_barrier - Wait until all in-flight call_rcu() callbacks complete.
 * Note that this primitive does not necessarily wait for an RCU grace period
 * to complete... if there are no RCU callbacks queued anywhere in the system,
 * then rcu_barrier() is within its rights to return immediately.
 */
void rcu_barrier(void)

(tree.c)

The mechanism is elegant. rcu_barrier() takes barrier_mutex (serializing concurrent callers), then for each CPU that has callbacks, it enqueues its own callback (rcu_barrier_handler via smp_call_function_single) at the tail of that CPU’s list and bumps an atomic counter barrier_cpu_count. Because callbacks on a CPU’s list run in FIFO order, the barrier’s callback can only run after every callback ahead of it has run. Each barrier callback decrements the counter; when it reaches zero, a completion is signaled and rcu_barrier() wakes.

sequenceDiagram
    autonumber
    participant T as Task calling rcu_barrier()
    participant S as rcu_state (global)
    participant C0 as CPU 0 cblist
    participant C1 as CPU 1 cblist
    participant C2 as CPU 2 cblist (empty)

    T->>S: mutex_lock(barrier_mutex)
    T->>S: rcu_seq_snap / rcu_seq_start(barrier_sequence)
    T->>S: atomic_set(barrier_cpu_count, 2)
    Note over S: Initialised to 2, not 0 — otherwise an<br/>immediately-invoked first callback could<br/>drive the count to zero before the loop ends.
    T->>C0: smp_call_function_single(0, rcu_barrier_handler)
    C0->>C0: rcu_segcblist_entrain(barrier_head)<br/>append at the last NON-EMPTY segment
    C0-->>S: atomic_inc(barrier_cpu_count)
    T->>C1: smp_call_function_single(1, rcu_barrier_handler)
    C1->>C1: rcu_segcblist_entrain(barrier_head)
    C1-->>S: atomic_inc(barrier_cpu_count)
    T->>C2: no callbacks queued (n_cbs == 0)
    Note over C2: SKIPPED. Nothing is entrained here —<br/>this is why rcu_barrier() can return<br/>immediately on an otherwise idle system.
    T->>T: atomic_dec(barrier_cpu_count) x2, then wait_for_completion()
    C0->>S: barrier callback runs → atomic_dec_and_test
    C1->>S: barrier callback runs → atomic_dec_and_test → 0
    S-->>T: complete(barrier_completion)
    T->>S: mutex_unlock(barrier_mutex)

How rcu_barrier() actually waits. What it shows: it does not wait for a grace period; it plants a sentinel callback behind every existing callback on every CPU that has any, and waits for those sentinels. The FIFO discipline of the per-CPU list does the rest. The insight to take: the CPU-2 arrow is the whole caveat in the kernel-doc — “if there are no RCU callbacks queued anywhere in the system, then rcu_barrier() is within its rights to return immediately, without waiting for anything, much less an RCU grace period.” So rcu_barrier() is not a substitute for synchronize_rcu() and vice versa. checklist.rst (v6.12) is explicit: “if you need to wait for both a grace period and for all pre-existing callbacks, you will need to invoke both functions.”

Two implementation details repay attention. First, the sentinel is planted with rcu_segcblist_entrain(), not rcu_segcblist_enqueue(), and the difference is the point: entrain() appends “at the end of the last non-empty segment”, walking down from RCU_NEXT_TAIL to find it, rather than always landing in RCU_NEXT_TAIL. Its comment carries the warning that makes the semantics precise: “This is intended for use by rcu_barrier()-like primitives, -not- for normal grace-period use. IMPORTANT: The callback you enqueue will wait for all prior callbacks, NOT necessarily for a grace period. You have been warned.” Second, rcu_barrier() is idempotent under concurrency via a sequence counter: a second caller that arrives while a barrier is in flight sees rcu_seq_done(&rcu_state.barrier_sequence, s) and returns after an smp_mb(), having piggybacked on the first.

checklist.rst also lists the barrier for each asynchronous flavour, which is the table to have to hand at module-exit time:

Registration primitive	Matching barrier
call_rcu()	rcu_barrier()
call_srcu()	srcu_barrier()
call_rcu_tasks()	rcu_barrier_tasks()
call_rcu_tasks_trace()	rcu_barrier_tasks_trace()
kfree_rcu() / kvfree_rcu() (for kmem_cache_destroy())	kvfree_rcu_barrier() — new in v6.12

The registration-to-barrier correspondence. What it shows: each asynchronous API has its own callback queue and therefore its own barrier; none of them covers another. The insight to take: checklist.rst is blunt about why waiting for a grace period is not enough — “it is absolutely not sufficient to wait for a grace period! For example, synchronize_rcu() implementation is not guaranteed to wait for callbacks registered on other CPUs via call_rcu(). Or even on the current CPU if that CPU recently went offline and came back online.”

The standard module-exit idiom is therefore: stop generating new callbacks, then call rcu_barrier(), then free module-private data — exactly mirroring how kfree_rcu() was designed to avoid needing this call for module text, and exactly why kvfree_rcu_barrier() had to be added for slab caches.

---

Callback Flooding and the Memory-Footprint Bill

This is the section the rest of the note exists to support. call_rcu() converts a latency cost into a memory cost, and the conversion rate is entirely under the caller’s control — which means it is entirely the caller’s responsibility.

The arithmetic

Do the sum once and the hazard stops being abstract. Suppose a workload unlinks and call_rcu()s objects of 512 bytes at a rate of 200,000 per second across the machine, and grace periods on this system take 25 ms end to end (a plausible figure for a loaded server; Requirements.rst characterises the normal grace period as “several milliseconds… in addition to the duration of the longest RCU read-side critical section”). One grace period’s worth of in-flight objects is 200,000 × 0.025 = 5,000 objects, or about 2.6 MB — trivial. Now suppose the grace period stretches to 2 seconds because one CPU is running a nohz_full tight loop, or a preempted reader is being starved by a real-time task. The same workload now holds 400,000 objects, or roughly 205 MB, none of which is reachable, none of which is free, and none of which the allocator can reclaim. Nothing in the code changed. The footprint is a product of your update rate and someone else’s grace-period latency, and you control only one factor.

That is why the design puts a queue between the two, and why the queue has an ever-escalating series of throttles bolted to it:

flowchart TB
  Q["Per-CPU rcu_segcblist backlog<br/>rcu_segcblist_n_cbs(&rdp->cblist)"]
  T0["<b>0 – 9,999 callbacks: steady state</b><br/>rcu_do_batch() invokes up to<br/>bl = max(blimit, pending &gt;&gt; 7) per pass.<br/>blimit = DEFAULT_RCU_BLIMIT = 10."]
  T1["<b>≥ qhimark (10,000): work harder</b><br/>call_rcu_core() notices<br/>n_cbs &gt; qlen_last_fqs_check + qhimark and:<br/>1. starts a GP if none is in progress<br/>2. sets blimit = DEFAULT_MAX_RCU_BLIMIT<br/>3. calls rcu_force_quiescent_state()"]
  T2["<b>≥ qovld (20,000 = 2 × qhimark): hammer QS</b><br/>check_cb_ovld() sets this CPU's bit in the leaf<br/>rcu_node's cbovldmask, telling the whole<br/>grace-period machinery that this CPU is drowning."]
  T3["<b>Back under qlowmark (100): stand down</b><br/>rcu_do_batch() restores<br/>rdp->blimit = blimit."]
  OFF["<b>The structural answer: rcu_nocbs</b><br/>Move invocation to rcuoc kthreads that can be<br/>scheduled, prioritised and pinned — so a flood<br/>costs scheduler time on chosen CPUs instead of<br/>softirq latency on every CPU.<br/>See [[RCU and NOCB Offloaded Callbacks]]."]
  DANGER["<b>…and the failure it does NOT prevent.</b><br/>Requirements.rst: a 64-CPU box with<br/>rcu_nocbs=1-63 and CPUs 1–63 spinning on<br/>call_rcu() means 'CPU 0 simply will not be able<br/>to invoke callbacks as fast as the other 63 CPUs<br/>can register them, at least not until the system<br/>runs out of memory.'"]
  Q --> T0 --> T1 --> T2
  T2 --> T3
  T2 --> OFF
  OFF --> DANGER

RCU’s escalating response to a growing callback backlog, and where it stops working. What it shows: three thresholds — qlowmark = 100, qhimark = 10000, qovld = 20000 — each triggering a different, stronger reaction, all read from kernel/rcu/tree.c v6.12. The insight to take: every one of these throttles makes callbacks drain faster; none of them makes the updater go slower. There is no back-pressure anywhere in this picture. That asymmetry is the entire hazard, and it is why checklist.rst puts the responsibility on the caller and why the concentration warning at the bottom is a real production configuration mistake — “a determined user or administrator can still exhaust memory. This is especially the case if a system with a large number of CPUs has been configured to offload all of its RCU callbacks onto a single CPU.”

Requirements.rst (v6.12) enumerates the qhimark reaction as four numbered steps, and reading them against the code above is a good check that the documentation and the implementation still agree. RCU, when a non-rcu_nocbs CPU “has 10,000 callbacks, or has 10,000 more callbacks than it had the last time encouragement was provided”, will:

1. “Start a grace period, if one is not already in progress” — rcu_accelerate_cbs_unlocked() in call_rcu_core().
2. “Force immediate checking for quiescent states, rather than waiting for three milliseconds to have elapsed since the beginning of the grace period” — rcu_force_quiescent_state().
3. “Immediately tag the CPU’s callbacks with their grace period completion numbers, rather than waiting for the RCU_SOFTIRQ handler to get around to it” — the note_gp_changes() call.
4. “Lift callback-execution batch limits, which speeds up callback invocation at the expense of degrading realtime response” — rdp->blimit = DEFAULT_MAX_RCU_BLIMIT.

The same document is candid that this is best-effort and that the rcu_nocbs case is weaker: “callback-invocation forward progress for rcu_nocbs CPUs is much less well-developed, in part because workloads benefiting from rcu_nocbs CPUs tend to invoke call_rcu() relatively infrequently. If workloads emerge that need both rcu_nocbs CPUs and high call_rcu() invocation rates, then additional forward-progress work will be required.”

Restoring back-pressure yourself

Since RCU supplies none, checklist.rst enumerates four ways to add it, and they are worth knowing as a menu rather than as a list, because they suit different situations:

Technique	How it works	When it fits
Cap the in-flight count	Keep a counter of elements awaiting reclamation; when it hits a ceiling, stall the updater. checklist.rst: “One way to stall the updates is to acquire the update-side mutex. (Don’t try this with a spinlock — other CPUs spinning on the lock could prevent the grace period from ever ending.)”	Any data structure with a natural element count. The standard answer
Simulate OOM in an allocation wrapper	Wrap the allocator so it fails “when there is too much memory awaiting an RCU grace period”, pushing back through the existing error path	Codebases that already handle allocation failure gracefully
Limit the update rate structurally	If updates happen once an hour, no explicit limiting is needed. “Older versions of the dcache subsystem take this approach, guarding updates with a global lock, limiting their rate”	Configuration-driven or administrative update paths
Trusted updater only	If only root can drive updates, “superuser already has lots of ways to crash the machine”	Debugfs knobs, module parameters
Periodic rcu_barrier()	“Permitting a limited number of updates per grace period” — the barrier blocks until the current backlog has drained	Bulk operations that can tolerate a stall every N items

The five documented ways to put back-pressure on a call_rcu() producer. What it shows: four of the five work by making the updater wait; only the allocation-wrapper trick converts the problem into an ordinary error path. The insight to take: the parenthetical about spinlocks is the sharpest thing in the list and is a genuine deadlock, not a performance note. Stalling on a spinlock while waiting for a grace period can prevent that grace period from ever ending, because a CPU spinning with preemption disabled never reaches a quiescent state. The rule generalises: any back-pressure mechanism that blocks with preemption disabled can deadlock against the thing it is waiting for.

There is a sixth option the checklist does not list because it is newer, and it is the cheapest of them all when it applies: the polled grace-period API. Snapshot the grace-period counter at removal time with get_state_synchronize_rcu(), do other work, and then either poll with poll_state_synchronize_rcu(cookie) — which returns true at no cost if a grace period has already elapsed — or block only if needed with cond_synchronize_rcu(cookie). Requirements.rst frames this as the middle ground: “line 16 returns immediately if a grace period has elapsed in the meantime, but otherwise waits as required. RCU thus provides a range of tools to allow updaters to strike the required tradeoff between latency, flexibility and CPU overhead.” This is exactly the mechanism the kvfree_rcu() bulk path uses internally — each bulk block carries a gp_snap taken with get_state_synchronize_rcu_full() and is drained when poll_state_synchronize_rcu_full() says so — so the technique is proven at scale inside RCU itself.

SLAB_TYPESAFE_BY_RCU: the alternative discipline

There is one way out of the memory-footprint problem that does not throttle anything: stop deferring the object’s free at all, and defer only the slab page. That is SLAB_TYPESAFE_BY_RCU, a kmem_cache_create() flag whose kernel-doc in include/linux/slab.h (v6.12) opens with the words “WARNING READ THIS!” — which is a fair summary of the trade.

“This delays freeing the SLAB page by a grace period, it does NOT delay object freeing. This means that if you do kmem_cache_free() that memory location is free to be reused at any time. Thus it may be possible to see another object there in the same RCU grace period. This feature only ensures the memory location backing the object stays valid, the trick to using this is relying on an independent object validation pass.”

The guarantee weakens from “this object stays alive” to “this memory stays this type”, and the reader must compensate:

begin:
 rcu_read_lock();
 obj = lockless_lookup(key);
 if (obj) {
   if (!try_get_ref(obj))            // might fail for free objects
     rcu_read_unlock();
     goto begin;
 
   if (obj->key != key) {            // not the object we expected
     put_ref(obj);
     rcu_read_unlock();
     goto begin;
   }
 }
 rcu_read_unlock();

Reading it line by line: the lookup may return a pointer to memory that has already been recycled into a different object of the same cache, so the reader must first take a reference (which fails on a free object) and then re-validate the key, restarting if either check fails. The payoff is that kmem_cache_free() is immediate — there is no per-object grace-period delay, no rcu_head, and no queue to overflow. The price is a retry loop and one further, easily-missed constraint that the header spells out: “SLAB_TYPESAFE_BY_RCU pages are not zeroed before being given to the slab, which means that any locks must be initialized after each and every kmem_struct_alloc().” Or, alternatively, initialise them in the cache’s constructor — the header names three in-tree examples that do exactly that: __i915_request_ctor(), sighand_ctor() and anon_vma_ctor(). The flag was called SLAB_DESTROY_BY_RCU before it was renamed, which is worth knowing when reading older material.

This is the right tool for update-heavy RCU-protected caches — precisely the workloads that checklist.rst’s “more than about 10 % of the time” gate says should not be using ordinary RCU. Documentation/RCU/rculist_nulls.rst develops the same idea for hash tables, where the recycling hazard has an extra wrinkle: an object moved to a different hash bucket can send a lockless reader walking off into the wrong chain, which the “nulls” marker at the end of each bucket detects.

---

Failure Modes

Callback flooding (the headline hazard). Because call_rcu() never blocks, a writer can issue callbacks far faster than grace periods retire them. The whatisRCU documentation states the trade-off plainly: synchronize_rcu() “has the nice property of automatically limiting update rate,” whereas call_rcu() “requires manual rate-limiting for denial-of-service resilience” (whatisRCU). The mechanism, the arithmetic, the three throttle thresholds and the five ways to restore back-pressure are the subject of the preceding section; what belongs here are the tunables you would reach for at 3 a.m., all module parameters under rcutree.:

Parameter	Default (v6.12)	Effect of raising it	Effect of lowering it
rcutree.blimit	10 (but 1000 if CONFIG_RCU_STRICT_GRACE_PERIOD=y)	Drains the backlog faster in steady state	Gentler on softirq latency
rcutree.qhimark	10000	RCU tolerates a bigger backlog before forcing quiescent states	Reacts to a flood sooner
rcutree.qlowmark	100	Stays in “drain hard” mode longer	Returns to steady state sooner
rcutree.qovld	20000 (= 2 × qhimark)	Later cbovldmask escalation	Earlier system-wide “this CPU is drowning” signal
rcutree.rcu_divisor	7 (so bl floor is pending / 128)	Lower it to drain a proportionally larger slice per pass	Higher = smaller slices
rcutree.rcu_resched_ns	3000000 (3 ms), clamped to [1 ms, 1 s]	Longer uninterrupted callback batches	Shorter batches, better latency
rcutree.rcu_min_cached_objs	5	More per-CPU spare pages for the kvfree_rcu() bulk path	Fewer, so more allocator traffic under load

The callback-throughput tunables and which direction to move each. What it shows: four of the seven are read-only after boot (0444 permissions in the module_param() calls) — blimit, qhimark, qlowmark, qovld and rcu_min_cached_objs — while rcu_divisor and rcu_resched_ns are 0644 and therefore writable at runtime through /sys/module/rcutree/parameters/. The insight to take: that permission split is itself the advice. The two knobs the kernel lets you turn on a live system are the two that trade callback throughput against interrupt latency, which is the trade you are actually making during an incident; the thresholds that change RCU’s escalation policy are boot-time decisions. Note also that DEFAULT_RCU_BLIMIT is (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10), so a debug kernel reports a completely different default — check CONFIG_RCU_STRICT_GRACE_PERIOD before comparing two machines.

The structural fix, when a CPU legitimately produces floods of callbacks, is to offload them to rcuo kthreads — see RCU and NOCB Offloaded Callbacks — or to throttle the producer using one of the five techniques above.

Double call_rcu() / double kfree_rcu(). Queuing the same rcu_head twice corrupts the callback list, and the corruption surfaces far from its cause. RCU’s defences here are worth knowing in detail because they are unusually good.

The first is unconditional. __call_rcu_common() opens with WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); — the runtime check backing the rcu_head alignment requirement — and then, under CONFIG_DEBUG_OBJECTS_RCU_HEAD, calls debug_rcu_head_queue(head). If the head is already queued, the kernel deliberately leaks the callback rather than corrupting the list:

	if (debug_rcu_head_queue(head)) {
		/*
		 * Probable double call_rcu(), so leak the callback.
		 * Use rcu:rcu_callback trace event to find the previous
		 * time callback was passed to call_rcu().
		 */
		if (atomic_inc_return(&doublefrees) < 4) {
			pr_err("%s(): Double-freed CB %p->%pS()!!!  ", __func__, head, head->func);
			mem_dump_obj(head);
		}
		WRITE_ONCE(head->func, rcu_leak_callback);
		return;
	}

Three details make this a good bug report rather than a splat. %pS prints the symbolic name of the previously-registered callback function, which usually identifies the offending subsystem outright. mem_dump_obj(head) dumps what the memory allocator knows about that address — which slab cache it came from, or that it is on the stack or in vmalloc space. And atomic_inc_return(&doublefrees) < 4 caps the noise at three reports, because a double-free bug is usually a loop. The comment’s advice — use the rcu:rcu_callback trace event to find the first registration — is the natural next step.

The second defence is opt-in and precise. If you want to ask whether a given rcu_head has already been handed to call_rcu(), include/linux/rcupdate.h (v6.12) provides a pair: call rcu_head_init(rhp) just after allocating the structure (it stores the sentinel (rcu_callback_t)~0L in ->func), and thereafter rcu_head_after_call_rcu(rhp, f) returns true if and only if that head was passed to call_rcu() with f, and warns “in any other case, including the case where @rhp has already been invoked after a grace period.” Both have a strict anti-race requirement stated in their kernel-doc: calls “must not race with calls to call_rcu(), rcu_head_after_call_rcu(), or callback invocation”, and the suggested discipline is to enclose the test “in an RCU read-side critical section that includes a read-side fetch of the pointer to the structure containing @rhp.” Paul McKenney’s The RCU API, 2019 edition introduced this pair for exactly this class of bug (LWN 777036).

The usual root cause is freeing an object via two code paths, or a use-after-free that lets a recycled head be re-queued — which is also why SLAB_TYPESAFE_BY_RCU caches, where recycling is the normal case, need the validation loop described above.

Calling a sleeping function from a callback. Callbacks run in softirq (or kthread) context. The whatisRCU doc is explicit: “This invocation might happen from either softirq or process context, so the function is not permitted to block” (whatisRCU). A callback that calls a blocking allocator or takes a mutex is a bug — use a workqueue if you need to sleep during reclamation.

Forgetting rcu_barrier() at module unload. The classic crash: a module is removed while one of its call_rcu() callbacks is still pending, and the callback later jumps into freed module text. rcu_barrier() in the module’s exit path prevents it. kfree_rcu() sidesteps the issue because its callback (a plain free) lives in the core kernel, not the module.

---

Alternatives and When to Choose Them

• synchronize_rcu() — the blocking counterpart. The writer sleeps until a grace period completes, then frees inline. Simpler code, automatic rate-limiting, no rcu_head needed, no rcu_barrier() worry. Choose it in process context where blocking is fine and update rate is modest. Choose call_rcu() when you cannot block (atomic/interrupt context) or when the update path is hot enough that a synchronous wait per update would be unacceptable. See RCU Grace Periods.
• kfree_rcu() / kvfree_rcu() — choose over raw call_rcu() whenever the callback is “just free it.” Less code, batched implementation, no rcu_barrier() requirement. The only reasons to fall back to call_rcu() are a non-trivial callback or an rcu_head offset ≥ 4096.
• call_rcu_hurry() — choose over call_rcu() on CONFIG_RCU_LAZY kernels when you specifically need prompt reclamation and cannot tolerate the power-saving lazy delay.
• SLAB_TYPESAFE_BY_RCU — choose it when the update rate is high enough that per-object deferral is the problem. It defers the slab page, not the object, so kmem_cache_free() is immediate and there is no queue to overflow — at the price of a reader-side validate-and-retry loop and constructor-initialised locks. Detailed above.
• The polled grace-period API — get_state_synchronize_rcu() plus cond_synchronize_rcu() or poll_state_synchronize_rcu(). Choose it when you can do useful work between the removal and the free: you pay a grace-period wait only if one has not already elapsed, and you need no rcu_head and no callback. This is the mechanism the kvfree_rcu() bulk path uses internally.
• Reference counting / hazard pointers — alternatives to RCU entirely when readers are not overwhelmingly common or when bounded memory footprint matters more than reader speed. RCU trades unbounded (but throttled) reclamation latency for zero-cost readers.

	Deferral unit	Extra per-object storage	Bounded footprint?	Reader cost
synchronize_rcu()	the updater itself	none	yes — self-limiting by construction	zero
call_rcu()	one object per callback	rcu_head, 2 pointers	no — you must add back-pressure	zero
kfree_rcu() / kvfree_rcu()	~507 objects per page block	rcu_head at offset < 4096	no, but far denser and drained on a 5 s timer	zero
kfree_rcu_mightsleep()	one object; falls back to synchronize_rcu()	none	yes on the fallback path	zero
polled API (cond_synchronize_rcu)	the updater, conditionally	one unsigned long cookie	yes	zero
SLAB_TYPESAFE_BY_RCU	the slab page, not the object	none	yes — objects free immediately	validate-and-retry loop

The deferral strategies compared on the axis this note cares about: memory. What it shows: only three of the six bound the footprint, and two of those do it by making the updater wait. The insight to take: the row that stands out is the last one. SLAB_TYPESAFE_BY_RCU is the only option that gives you both an unbounded-rate non-blocking updater and a bounded footprint, and it pays for that by moving the cost onto the reader — which is the exact opposite of RCU’s usual bargain. That is why it is the recommended answer for the update-heavy workloads checklist.rst says should not be using ordinary RCU at all.

---

Production Notes

The deferred-reclamation pattern is everywhere in the kernel: release_task() defers task_struct freeing via call_rcu() (listRCU); the dentry cache, the networking routing tables, and Open vSwitch flow tables all use kfree_rcu() to retire RCU-protected objects after readers drain. Paul McKenney’s The RCU API, 2019 edition states the recommendation this note has been building toward in two sentences: “The kfree_rcu() primitive serves as a shortcut for an RCU callback that does nothing but free the structure passed to it. Use of kfree_rcu() can both simplify code and reduce the need for rcu_barrier()” (LWN 777036, verified by fetch 2026-09-04).

Debugging a deferred-reclamation problem

Three tools, in the order you would reach for them.

CONFIG_DEBUG_OBJECTS_RCU_HEAD is the first and the most valuable. checklist.rst describes what it buys: it will “check that you don’t pass the same object to call_rcu() (or friends) before an RCU grace period has elapsed since the last time that you passed that same object to call_rcu() (or friends).” The 2019 API survey adds the mechanism — the debug-objects subsystem already “checks for memory-allocation usage bugs, for example, double kfree()”, and this Kconfig option extends the same machinery to call_rcu(). There is one setup wrinkle worth knowing, because the failure is a silent no-check rather than a warning: debug-objects “automatically sets up its state for global variables and heap memory”, but an rcu_head on the stack needs explicit init_rcu_head_on_stack() and a matching destroy_rcu_head_on_stack() before the containing function returns. And there is a re-entrancy hazard: “if that call_rcu() occurs in the memory allocator or in some other function used by debug-objects, this implicit call_rcu()-time invocation can result in deadlock. Functions called by debug-objects that also use call_rcu() should therefore manually invoke init_rcu_head() during initialization in order to break such deadlocks.”

Trace events. rcu_callback records each registration (with the function symbol and the resulting queue length), rcu_invoke_callback each invocation, rcu_batch_start/rcu_batch_end each rcu_do_batch() pass with the batch limit and the reason it stopped, rcu_segcb_stats the per-segment lengths, and rcu_invoke_kfree_bulk_callback the bulk-free path with its live nr_records. Between rcu_callback and rcu_invoke_callback you can measure the actual registration-to-invocation latency of the pipeline drawn at the top of this note, per callback, rather than reasoning about it.

rcutree.do_rcu_barrier. v6.12 exposes rcu_barrier() itself as a writable module parameter (module_param_cb(do_rcu_barrier, &do_rcu_barrier_ops, &do_rcu_barrier, 0644)), so an operator can force a full callback drain from /sys/module/rcutree/parameters/do_rcu_barrier and observe how long it takes — a direct measurement of how deep the backlog really is.

The last word belongs to Requirements.rst, which is unusually honest that none of the machinery in this note is a guarantee. Its worked counter-example is a 64-CPU system built CONFIG_RCU_NOCB_CPU=y and booted rcu_nocbs=1-63, with CPUs 1 through 63 spinning in tight loops that call call_rcu(): “even if these tight loops also contain calls to cond_resched() (thus allowing grace periods to complete), CPU 0 simply will not be able to invoke callbacks as fast as the other 63 CPUs can register them, at least not until the system runs out of memory. In both of these examples, the Spiderman principle applies: With great power comes great responsibility.” Deferred reclamation gives an updater the power to never block; the responsibility that comes with it is to bound the queue yourself.

Resolved

An earlier revision flagged the throttle defaults as unverified. They are now read directly from kernel/rcu/tree.c at v6.12 (fetched 2026-09-04): #define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10), #define DEFAULT_RCU_QHIMARK 10000, #define DEFAULT_RCU_QLOMARK 100, #define DEFAULT_RCU_QOVLD_MULT 2 with DEFAULT_RCU_QOVLD (DEFAULT_RCU_QOVLD_MULT * DEFAULT_RCU_QHIMARK) = 20,000, plus static int rcu_divisor = 7 and static long rcu_resched_ns = 3 * NSEC_PER_MSEC. The only wrinkle is the one the macro itself states: blimit is 1,000, not 10, on a CONFIG_RCU_STRICT_GRACE_PERIOD debug build, so the “10” figure is the production default only.

Resolved

An earlier revision asked whether call_rcu() is lazy by default on a shipped 6.12 kernel. Reading kernel/rcu/Kconfig at v6.12 settles it: CONFIG_RCU_LAZY is default n, and it depends on RCU_NOCB_CPU with the help text adding “Requires rcu_nocbs=all to be set.” So on a kernel built without that option, tree.c’s #else branch defines enable_rcu_lazy as the literal false and call_rcu() is never lazy. Where CONFIG_RCU_LAZY=y is selected, a second option CONFIG_RCU_LAZY_DEFAULT_OFF (also default n) exists to “build the kernel with CONFIG_RCU_LAZY=y yet keep it default off”, and the runtime knob is rcutree.enable_rcu_lazy. The practical summary: laziness is opt-in twice over — a Kconfig option that is off by default and a boot parameter — so on a stock distribution kernel call_rcu() is not lazy, and call_rcu_hurry() is a no-op difference there. Verify a specific machine with grep RCU_LAZY /boot/config-$(uname -r).

---

See Also

• Read-Copy-Update Fundamentals — the hub: the publish/grace-period/reclaim model this note implements the reclaim half of, and the note that owns the choice between synchronize_rcu(), call_rcu() and kfree_rcu()
• RCU Grace Periods — what a grace period is and how synchronize_rcu() (the blocking sibling) waits for one; the source of the gp_seq values that promote segments here
• Tree RCU — the scalable grace-period-detection engine that decides when queued callbacks become ready, and the rcu_node tree whose cbovldmask this note’s overload detector sets
• RCU and NOCB Offloaded Callbacks — moving callback invocation off the softirq/rcuc path onto rcuo kthreads, the production answer to callback flooding on latency-sensitive CPUs
• Expedited Grace Periods — the IPI-driven short cut that shortens stall 2 in the timeline above, and what it costs every other CPU
• Sleepable RCU and SRCU — reuses this note’s rcu_segcblist verbatim, and has its own call_srcu()/srcu_barrier() pair
• RCU Read-Side Critical Sections — the readers whose completion the grace period is waiting for, and whose duration extends stall 2
• The publish-subscribe Pattern in RCU — rcu_assign_pointer()/rcu_dereference(), the removal step that must precede every call_rcu()
• RCU-Protected Linked Lists — list_del_rcu() and friends, the usual thing that happens on the line above a call_rcu()
• Memory Barriers in the Linux Kernel — smp_mb__after_unlock_lock(), the barrier that orders callback invocation against grace-period completion
• The Slab Allocator and SLUB and Kmem Caches and Object Slabs — the cache SLAB_TYPESAFE_BY_RCU is a flag on, and where kfree_bulk() does the bulk path’s actual freeing
• Softirqs and the Softirq Vector — RCU_SOFTIRQ, deliberately the last entry in the softirq enum so callback processing has the lowest priority
• The PREEMPT_RT Real-Time Kernel — why use_softirq defaults to false there, moving invocation to the rcuc kthreads
• Linux Kernel Synchronization MOC — parent map (section E, Read-Copy-Update)