vault

Kernel Preemption Models

15 min read

Kernel Preemption Models

A preemption model answers one question: while code is running inside the kernel (in a system call, a fault handler, a kernel thread), may the scheduler forcibly take the CPU away to run something more deserving, and if so, where? Userspace is always preemptible — the timer tick can yank a CPU-bound user program at any instruction. The hard choice is about kernel code, which may hold locks or be mid-operation. Linux historically shipped this as a compile-time choice among three models — PREEMPT_NONE (server: kernel code runs to a natural yield point), PREEMPT_VOLUNTARY (desktop: extra explicit yield points), and PREEMPT (low-latency: kernel code is preemptible anywhere it does not hold a lock) — selected in kernel/Kconfig.preempt. Since v5.12 CONFIG_PREEMPT_DYNAMIC lets a single kernel binary carry all three and pick one at boot via preempt= or switch at runtime, implemented with static calls / static keys that patch the preemption points to live code or no-ops. The model is, mechanically, a choice of which preemption points are armed (verified against kernel/Kconfig.preempt and kernel/sched/core.c at v6.12).

The single most important idea: the models do not differ in how a reschedule is requested — they all set TIF_NEED_RESCHED the same way — they differ in which checkpoints honour it. PREEMPT_NONE arms only the return-to-userspace and explicit-cond_resched() points; PREEMPT additionally arms the interrupt-return and preempt_enable() points so kernel code can be preempted involuntarily. That single difference is the entire trade-off between throughput (fewer, cheaper context switches; better cache locality) and latency (faster reaction to a woken high-priority task).

Uncertain

Verify: the claim that PREEMPT_LAZY is absent in 6.12 and present in 6.18 is confirmed by direct diff of kernel/Kconfig.preempt between the two tags — v6.12 has no PREEMPT_LAZY config stanza at all; v6.18 adds config PREEMPT_LAZY and config ARCH_HAS_PREEMPT_LAZY. However, the default model at both versions is still default PREEMPT_NONE in the Kconfig choice — distributions override this. Reason: “what a distro ships” (e.g. Fedora/Ubuntu choosing PREEMPT_DYNAMIC defaulting to voluntary/full) is a config fact, not a mainline-source fact, and varies. To resolve for a given system: read /boot/config-$(uname -r) for CONFIG_PREEMPT* and check dmesg | grep -i "Dynamic Preempt" for the runtime mode. uncertain

Mental Model

Think of the kernel as a building with several exit turnstiles where a guard (the scheduler) can ask you to step aside for someone more important. The turnstiles are the preemption points: the door back to userspace, the door at the end of every interrupt, the door at each preempt_enable(), and a few voluntary “rest stops” (cond_resched()). A preemption model decides which turnstiles have a guard standing at them.

  • PREEMPT_NONE staffs only the userspace door and the voluntary rest stops. Once you are deep inside the kernel and not returning to userspace, you run uninterrupted until you reach a rest stop or leave — maximally efficient, occasionally laggy.
  • PREEMPT_VOLUNTARY adds more rest stops by turning might_resched() (inside might_sleep() debug points) into a real yield, so long kernel operations volunteer to pause more often.
  • PREEMPT (full) staffs every door, including the interrupt-return door and every preempt_enable(). Now a guard can stop you the instant a higher-priority task wakes, as long as you are not holding a lock.
flowchart TB
  subgraph POINTS["The four preemption points (same flag, TIF_NEED_RESCHED)"]
    P1["return to userspace<br/>(exit_to_user_mode_loop)"]
    P4["explicit cond_resched()<br/>/ might_resched()"]
    P2["return from interrupt<br/>(irqentry_exit_cond_resched)"]
    P3["preempt_enable()<br/>(count hits 0)"]
  end
  NONE["PREEMPT_NONE<br/>(server, throughput)"] --> P1
  NONE --> CR1["cond_resched: ON<br/>might_resched: OFF"]
  VOL["PREEMPT_VOLUNTARY<br/>(desktop)"] --> P1
  VOL --> CR2["cond_resched: ON<br/>might_resched: ON"]
  FULL["PREEMPT (FULL)<br/>(low-latency)"] --> P1
  FULL --> P2
  FULL --> P3
  FULL --> CR3["cond_resched: OFF<br/>(no longer needed)"]

The three classic models as a choice of armed preemption points. What it shows: all models arm the universal return-to-userspace point (P1). PREEMPT_NONE adds only cond_resched(); PREEMPT_VOLUNTARY additionally arms might_resched(); PREEMPT (full) arms the interrupt-return (P2) and preempt_enable() (P3) points and turns cond_resched() off because it is redundant once kernel code is preemptible everywhere. The insight to take: going down the list trades throughput for latency by arming progressively more (and more involuntary) checkpoints. Under CONFIG_PREEMPT_DYNAMIC, these on/off switches are runtime-patchable static calls, not compile-time decisions.

The Three Classic Models

PREEMPT_NONE — No Forced Preemption (Server)

This is the original Linux behaviour, geared for throughput. Kernel code is not preemptible: once a task enters the kernel, it keeps the CPU until it (a) returns to userspace, (b) blocks/sleeps, or (c) hits an explicit cond_resched(). The Kconfig help text states it plainly: “geared towards throughput… there are no guarantees and occasional longer delays are possible… maximize the raw processing power of the kernel, irrespective of scheduling latencies” (Kconfig.preempt @ v6.12). The win is fewer context switches and better cache/TLB locality; the cost is that a high-priority task waking up may wait until the running task next yields. This is the right choice for batch/HPC and many database/storage servers where aggregate work-per-second beats tail latency.

PREEMPT_VOLUNTARY — Voluntary Kernel Preemption (Desktop)

A middle ground. It does not make kernel code involuntarily preemptible; instead it adds explicit preemption points. Concretely, it turns might_resched() — which is invoked by the might_sleep() macro scattered through the kernel at points where blocking would be legal — into a real reschedule check rather than a no-op. The Kconfig text: “adds more explicit preemption points… selected to reduce the maximum latency of rescheduling… at the cost of slightly lower throughput” (Kconfig.preempt @ v6.12). Because might_sleep() annotations already mark “I could block here” spots, reusing them as yield points is nearly free and dramatically shortens worst-case in-kernel runs without the locking overhead of full preemption. This was the long-standing default for desktop distributions.

PREEMPT — Preemptible Kernel (Low-Latency / “Full”)

Here all kernel code that is not inside a critical section is preemptible (Kconfig.preempt @ v6.12). This is achieved by selecting PREEMPT_BUILD → PREEMPTION → PREEMPT_COUNT, which makes preempt_enable() perform the decrement-and-test check (see The need_resched Flag and Preemption Points) and arms irqentry_exit_cond_resched() so an interrupt returning into kernel mode can preempt. A task spinning in a long kernel computation is now preempted at the very next timer tick, giving milliseconds-range worst-case latency. The cost is “slightly lower throughput and a slight runtime overhead to kernel code” from the extra preempt_count maintenance and more frequent switches. This is the basis for low-latency desktop, audio, and many container hosts. (Beyond PREEMPT lies PREEMPT_RT, the fully real-time variant that also makes spinlocks sleepable and threads interrupts — that is The PREEMPT_RT Real-Time Kernel, a separate note; it mainlined in 6.12.)

The Kconfig choice and PREEMPTION

These three (plus PREEMPT_RT) form a Kconfig choice — mutually exclusive at build time — with default PREEMPT_NONE (Kconfig.preempt @ v6.12 and v6.18). The build-time symbol that distinguishes “kernel code is preemptible” from “it is not” is CONFIG_PREEMPTION, selected by PREEMPT, PREEMPT_RT, and (importantly) PREEMPT_DYNAMIC. CONFIG_PREEMPTION in turn selects CONFIG_PREEMPT_COUNT, which is what makes preempt_disable()/preempt_enable() actually maintain the counter rather than compile to barriers. So “is this a preemptible kernel?” reduces to “is CONFIG_PREEMPTION set?”

Dynamic Preemption — CONFIG_PREEMPT_DYNAMIC

The classic model was frustrating for distributions: shipping a server-tuned and a desktop-tuned kernel doubles the build/test/maintenance matrix. CONFIG_PREEMPT_DYNAMIC (first appears in kernel/Kconfig.preempt at v5.12; absent at v5.11) fixes this by compiling the kernel as PREEMPT-capable (it selects PREEMPT_BUILD) but making the individual preemption points switchable at boot and runtime. The Kconfig text: “allows to define the preemption model on the kernel command line… override the default preemption model defined during compile time… primarily interesting for Linux distributions which provide a pre-built kernel binary to reduce the number of kernel flavors” (Kconfig.preempt @ v6.12). It defaults to y when the architecture supports static-call-based dispatch (default y if HAVE_PREEMPT_DYNAMIC_CALL).

Mechanism: static calls and static keys

The preemption points are not ordinary function calls; they are static calls (DEFINE_STATIC_CALL) — indirect call sites whose target the kernel can rewrite at runtime by patching the call instruction, with essentially zero steady-state overhead (unlike a function pointer, there is no indirect-branch misprediction). On architectures without static-call inline patching, PREEMPT_DYNAMIC falls back to static keys (HAVE_PREEMPT_DYNAMIC_KEY), the jump-label mechanism that patches a nop/jmp to enable or disable a branch. Either way the toggled functions are (core.c @ v6.12):

  • cond_resched — the explicit voluntary yield
  • might_resched — the might_sleep()-driven yield
  • preempt_schedule — the preempt_enable() reschedule path
  • preempt_schedule_notrace — the tracer-safe variant
  • irqentry_exit_cond_resched — the interrupt-return reschedule path

Each is defined with a “dynamic_enabled” target (the real implementation) and a “dynamic_disabled” target (NULL for the schedule paths, or __static_call_return0 — a stub returning 0 — for cond_resched/might_resched). For example (asm/preempt.h @ v6.12, core.c @ v6.12 line ~7243):

DEFINE_STATIC_CALL(preempt_schedule, preempt_schedule_dynamic_enabled);
DEFINE_STATIC_CALL_RET0(cond_resched, __cond_resched);     /* disabled => returns 0 */
DEFINE_STATIC_CALL_RET0(might_resched, __cond_resched);

How a mode is applied

__sched_dynamic_update(mode) is the switchboard. It first enables everything, then, per mode, disables the points that mode does not want (core.c @ v6.12 line ~7406). Reading the actual switch makes the model-as-armed-points idea concrete:

switch (mode) {
case preempt_dynamic_none:
	preempt_dynamic_enable(cond_resched);              /* keep voluntary yield */
	preempt_dynamic_disable(might_resched);            /* OFF */
	preempt_dynamic_disable(preempt_schedule);         /* OFF: no preempt_enable resched */
	preempt_dynamic_disable(preempt_schedule_notrace); /* OFF */
	preempt_dynamic_disable(irqentry_exit_cond_resched);/* OFF: no irq-return preempt */
	break;
case preempt_dynamic_voluntary:
	preempt_dynamic_enable(cond_resched);
	preempt_dynamic_enable(might_resched);             /* ON: extra yield points */
	preempt_dynamic_disable(preempt_schedule);
	preempt_dynamic_disable(preempt_schedule_notrace);
	preempt_dynamic_disable(irqentry_exit_cond_resched);
	break;
case preempt_dynamic_full:
	preempt_dynamic_disable(cond_resched);             /* OFF: redundant now */
	preempt_dynamic_disable(might_resched);
	preempt_dynamic_enable(preempt_schedule);          /* ON */
	preempt_dynamic_enable(preempt_schedule_notrace);
	preempt_dynamic_enable(irqentry_exit_cond_resched);/* ON: irq-return preempts */
	break;
}
preempt_dynamic_mode = mode;

This is the canonical mapping of model → armed points the mental-model diagram depicts: none keeps cond_resched only; voluntary adds might_resched; full swaps to the involuntary preempt_schedule + irqentry_exit_cond_resched points and turns cond_resched off because once the kernel is fully preemptible those manual yields are redundant. (The comment at the top of __sched_dynamic_update — “Avoid {NONE,VOLUNTARY} → FULL transitions from ever ending up in the ZERO state” — explains why it enables-then-disables rather than disabling first: a momentary all-off state would be a window with no preemption at all.)

Selecting the mode

At boot, the preempt= kernel command-line parameter is parsed by setup_preempt_mode() → sched_dynamic_mode(), which accepts "none", "voluntary", "full" (and "lazy" at v6.18) (core.c @ v6.12 line ~7494):

static int __init setup_preempt_mode(char *str)
{
	int mode = sched_dynamic_mode(str);
	if (mode < 0) { pr_warn("Dynamic Preempt: unsupported mode: %s\n", str); return 0; }
	sched_dynamic_update(mode);
	return 1;
}
__setup("preempt=", setup_preempt_mode);

If preempt= is absent, preempt_dynamic_init() falls back to whatever the compiled-in choice selected (CONFIG_PREEMPT_NONE/_VOLUNTARY/_PREEMPT) (core.c @ v6.12 line ~7507). At runtime, writing to /sys/kernel/debug/sched/preempt calls sched_dynamic_update() and re-patches the static calls live — no reboot, no recompile. The boot log prints e.g. Dynamic Preempt: full.

The preempt_model_none()/voluntary()/full() accessors (generated by PREEMPT_MODEL_ACCESSOR) let other kernel code query the active mode at runtime (core.c @ v6.12 line ~7524). Note PREEMPT_DYNAMIC depends on !PREEMPT_RT at v6.12 — a real-time kernel is always fully preemptible and not a runtime choice — though at v6.18 that dependency was relaxed (depends on HAVE_PREEMPT_DYNAMIC) as the dynamic machinery grew to cover lazy mode (Kconfig diff v6.12→v6.18).

DEBUG_PREEMPT

Orthogonal to the model, CONFIG_DEBUG_PREEMPT instruments preempt_count operations to catch imbalances — a preempt_disable() without a matching preempt_enable(), or using a per-CPU value or smp_processor_id() in a preemptible region. It does not change which points are armed; it validates that code respects the preempt_count invariants. Pair it with CONFIG_DEBUG_ATOMIC_SLEEP when chasing “scheduling while atomic” bugs.

PREEMPT_LAZY — the newer model (6.18, not 6.12)

A fourth model, PREEMPT_LAZY, was added to mainline in 6.13: config PREEMPT_LAZY and config ARCH_HAS_PREEMPT_LAZY first appear in kernel/Kconfig.preempt at v6.13 and are absent at v6.12 (corroborated by Phoronix on the 6.13 lazy-preempt merge). It is therefore present in 6.18 but absent in 6.12 — a direct diff of kernel/Kconfig.preempt confirms v6.12 has no PREEMPT_LAZY/ARCH_HAS_PREEMPT_LAZY stanzas, while v6.18 adds both, plus a preempt_dynamic_lazy mode and a "lazy" value for preempt=. Its Kconfig description: “a scheduler driven preemption model that is fundamentally similar to full preemption, but is less eager to preempt SCHED_NORMAL tasks in an attempt to reduce lock holder preemption and recover some of the performance gains seen from using Voluntary preemption” (Kconfig.preempt @ v6.18).

The idea is to get full-preemption latency for real-time/high-priority tasks while avoiding the throughput cost of preempting an ordinary fair task the instant another ordinary task becomes slightly more eligible. It does this with a second flag bit, TIF_NEED_RESCHED_LAZY, set by a new resched_curr_lazy(): lazy requests do not force an immediate kernel preemption and are instead resolved at the next tick or return to userspace, whereas urgent requests still use resched_curr()/TIF_NEED_RESCHED for immediate preemption. The full mechanism — the two-bit scheme, how the fair class decides eager vs lazy, and why this is poised to replace the scattered cond_resched() calls — is its own note: Lazy Preemption and PREEMPT_LAZY. The takeaway here is only that the model landscape grew from three to four between 6.12 and 6.18, and PREEMPT_DYNAMIC can now select lazy alongside none/voluntary/full.

What Each Model Guarantees: Latency vs Throughput

ModelKernel code preemptible?Armed pointsLatencyThroughputTypical use
PREEMPT_NONENouserspace return, cond_reschedWorst (ms–10s of ms tails)BestHPC, batch, throughput servers
PREEMPT_VOLUNTARYNo (more yield points)+ might_reschedBetterSlightly below NONEGeneral desktop/server
PREEMPT (full)Yes (outside locks)+ irq-return, preempt_enableGood (ms-range)Slightly below VOLUNTARYLow-latency desktop, audio, containers
PREEMPT_LAZY (6.18)Yes, but lazy for fair tasksfull points + lazy bitGood for RT, near-VOLUNTARY throughput for fairNear VOLUNTARYThe intended future default
PREEMPT_RTYes, + sleepable locks, threaded IRQseverythingBounded worst-caseLowestHard real-time

The guarantees are qualitative, not hard bounds — only PREEMPT_RT aims at a bounded worst-case latency. The non-RT models reduce expected and typical-tail latency by arming more checkpoints, at the cost of more context switches (cache/TLB churn) and the per-operation preempt_count overhead. There is no free lunch: every involuntary preemption is a context switch that the throughput-oriented PREEMPT_NONE avoids.

Failure Modes and Common Misunderstandings

  • “PREEMPT_NONE means userspace can’t be preempted.” Wrong — userspace is always preemptible (the tick yanks it). The model only governs kernel code. A PREEMPT_NONE kernel still time-slices user programs perfectly.
  • Soft lockups from missing cond_resched() (NONE/VOLUNTARY only). A kernel loop that never yields, never sleeps, and never returns to userspace will hog the CPU until the watchdog fires. This class of bug disappears under full/lazy preemption (the irq-return point preempts the loop) — and eliminating the need for scattered cond_resched() calls is an explicit motivation for lazy preemption (Phoronix on the 6.13 lazy-preempt merge). The source confirms the mechanics: full/lazy mode disables cond_resched in __sched_dynamic_update() precisely because the involuntary points make it redundant.
  • Lock-holder preemption hurting throughput (full). Full preemption can preempt a task while it holds a contended lock, stalling every waiter — exactly the pathology PREEMPT_LAZY targets by not eagerly preempting fair tasks.
  • Assuming the compiled default is what’s running. On a CONFIG_PREEMPT_DYNAMIC distro kernel (most modern ones), the active model is whatever preempt= or the runtime sysfs knob set — not the choice default. Always check dmesg | grep "Dynamic Preempt" and /sys/kernel/debug/sched/preempt.

Alternatives and When to Choose Them

  • Throughput-bound, latency-tolerant (HPC, encoding farms, batch ETL) → PREEMPT_NONE (or preempt=none on a dynamic kernel). Maximize work-per-second; tail latency does not matter.
  • General-purpose / mixed → PREEMPT_VOLUNTARY historically; on 6.13+ the trajectory is PREEMPT_LAZY as the new general default once mature.
  • Interactive / soft-real-time (audio, desktop, latency-sensitive services, container hosts) → PREEMPT (full).
  • Hard real-time (industrial control, robotics) → PREEMPT_RT — but accept the lowest throughput and the sleepable-spinlock semantics.
  • Don’t want to pick / ship one binary → CONFIG_PREEMPT_DYNAMIC and decide per-deployment with preempt=.

Production Notes

The push behind PREEMPT_DYNAMIC and then PREEMPT_LAZY is consolidation: rather than a menu of compile-time models plus hand-placed cond_resched() annotations, the direction is a single dynamically-tunable scheme where the scheduler — not scattered annotations — decides when to preempt. That PREEMPT_LAZY is described in its own Kconfig as “a scheduler driven preemption model” meant to recover “the performance gains seen from using Voluntary preemption” while keeping full-preemption responsiveness (Kconfig.preempt @ v6.18) is the in-tree statement of this goal.

Uncertain

Verify: the broader claim that lazy preemption is intended to eventually replace the scattered cond_resched() calls and become the general default. Reason: this is a roadmap/intent claim about kernel-community direction, supported by the Kconfig wording and the Phoronix 6.13 write-up but not by a single authoritative primary source read during this note. To resolve: read the lazy-preemption merge cover letter / the relevant LWN deep-dive (e.g. Steven Rostedt / Thomas Gleixner threads on lazy preemption) and pin the intent to it. uncertain

To inspect a live system: cat /sys/kernel/debug/sched/preempt shows the active mode (the current one in brackets); grep CONFIG_PREEMPT /boot/config-$(uname -r) shows what was compiled in; dmesg | grep "Dynamic Preempt" shows the boot-time resolution. To measure the cost of your choice, the preemptirqsoff ftrace tracer reports the longest stretch the kernel ran with preemption disabled — directly the worst-case added latency — and perf sched latency shows per-task wakeup-to-run delays under load.

See Also

Graph View

Backlinks