vault

The PREEMPT_RT Real-Time Kernel

13 min read

The PREEMPT_RT Real-Time Kernel

PREEMPT_RT is the Linux configuration that turns the general-purpose kernel into a real-time kernel — one with bounded, predictable worst-case scheduling latency rather than merely good average throughput. It does this by making nearly all kernel code preemptible: it converts the kernel’s ordinary spinning locks (spinlock_t, rwlock_t, local_lock) into sleeping, priority-inheriting locks built on rt_mutex, forces almost all interrupt handlers to run in schedulable kernel threads, and shrinks the handful of truly non-preemptible regions down to the lowest-level entry, scheduler, and raw-spinlock code. The most important fact about it in 2026: PREEMPT_RT is now mainline. After roughly 20 years out of tree (begun ~2004), the final missing piece — a reworked, threaded printk() — was merged for Linux 6.12 (the LTS released 2024-11-17), making CONFIG_PREEMPT_RT a selectable build option in the upstream kernel on x86-64, arm64, and RISC-V (Corbet, “The realtime preemption end game — for real this time,” LWN, 2024-09-06; the 6.12 release, LWN). Do not describe PREEMPT_RT as “an out-of-tree patchset” — that framing is stale as of 6.12.

Why a real-time kernel is a different goal

A stock Linux kernel optimizes for throughput and good average latency. A real-time kernel optimizes for bounded worst-case latency — the guarantee that a high-priority task, once it becomes runnable, will run within some provable upper bound regardless of what else the system is doing. The two goals conflict: the tricks that maximize throughput (batching work, holding locks across long sections, deferring to softirqs, running interrupt handlers to completion with preemption disabled) all add worst-case latency.

The enemy of bounded latency is any stretch of code during which the scheduler cannot run. In a normal kernel those stretches are everywhere: a held spinlock_t disables preemption for its whole critical section; a hardware interrupt handler runs to completion with that CPU’s preemption (and often other interrupts) disabled; softirqs and tasklets run in a non-preemptible context; long-running kernel loops may not check in for a reschedule. Each is a window in which a just-woken real-time task must wait. PREEMPT_RT’s entire program is to eliminate or bound every one of these windows.

This is a strictly stronger goal than the PREEMPT_FULL model (see Kernel Preemption Models). PREEMPT_FULL already makes most kernel code preemptible, but it leaves spinlock sections, IRQ handlers, and softirqs non-preemptible. PREEMPT_RT attacks exactly those.

Mental model: make almost everything schedulable

flowchart TB
  subgraph STOCK["Stock kernel: non-preemptible islands"]
    S1["spin_lock(): preempt off<br/>for whole critical section"]
    S2["hardirq handler runs<br/>to completion, preempt off"]
    S3["softirq/tasklet:<br/>non-preemptible context"]
  end
  subgraph RT["PREEMPT_RT: islands made schedulable"]
    R1["spinlock_t -> rt_mutex<br/>sleeping + priority inheritance<br/>(preemption stays ON)"]
    R2["hardirq -> threaded IRQ<br/>(kthread, schedulable)"]
    R3["softirq runs in task ctx,<br/>preemptible"]
  end
  S1 -->|"convert"| R1
  S2 -->|"force threading"| R2
  S3 -->|"per-CPU local_lock"| R3
  RAW["raw_spinlock_t<br/>(stays a true spinlock,<br/>preempt off — used only in<br/>scheduler/entry/low-level IRQ)"]
  R1 -. "still spins in tiny<br/>core paths" .- RAW

The core transformation. What it shows: the three big sources of non-preemptible time in a stock kernel — spinlocks, hardware interrupt handlers, and softirqs — are each converted into something the scheduler can preempt. spinlock_t becomes a sleeping rt_mutex (so a task waiting on a contended lock blocks instead of spinning with preemption off), hardware IRQ handlers become threaded (a schedulable kthread does the real work), and softirqs run in preemptible task context. The insight to take: the only thing left that truly disables preemption is raw_spinlock_t, reserved for the irreducible core — the scheduler itself, entry/exit code, and the lowest-level interrupt plumbing. Bounded latency is achieved by shrinking the non-preemptible set to that irreducible core.

The mechanism, piece by piece

1. Sleeping spinlocks: spinlock_t becomes an rt_mutex

This is the heart of PREEMPT_RT, and the in-tree Documentation/locking/locktypes.rst (verified at tag v6.12) spells it out precisely. The kernel deliberately distinguishes two spinning-lock types:

  • raw_spinlock_t — “a strict spinning lock implementation in all kernels, including PREEMPT_RT kernels.” It busy-waits with preemption disabled. It is reserved for “real critical core code, low-level interrupt handling and places where disabling preemption or interrupts is required, for example, to safely access hardware state.” This is the irreducible non-preemptible primitive.
  • spinlock_t — on a non-RT kernel this is mapped to raw_spinlock_t and behaves identically (spin, preempt off). On a PREEMPT_RT kernel, spinlock_t is “mapped to a separate implementation based on rt_mutex,” which changes the semantics dramatically:
    • Preemption is not disabled while the lock is held.
    • The hard-IRQ suffixes (spin_lock_irq, spin_lock_irqsave) do not actually disable interrupts — interrupts stay on.
    • A task that contends for the lock sleeps (blocks and is taken off the CPU) rather than spinning.

The same treatment applies to rwlock_t, local_lock (mapped to a per-CPU spinlock_t), and rw_semaphore (mapped to a separate rt_mutex-based implementation). The consequence: holding a spinlock_t no longer creates a non-preemptible, interrupt-disabled latency window. A higher-priority task can preempt a lock holder; correctness is preserved by priority inheritance (next section). The trade-off is that lock acquisition/release is heavier (it can sleep and must do PI bookkeeping), which is why genuinely tiny, hardware-touching critical sections are left as raw_spinlock_t.

Uncertain

Verify: that printk()/console output specifically uses raw_spinlock_t-style primitives such that it remains usable from the most restricted RT contexts. Reason: the reason printk was the last RT blocker (it must work everywhere, including NMI and the scheduler core) is well attested in the LWN coverage I read, but I did not trace the exact 6.12 console-lock implementation in source. To resolve: read kernel/printk/ at tag v6.12. uncertain

2. Priority inheritance via rt_mutex

Once a lock holder can be preempted, the classic real-time hazard appears: priority inversion. A low-priority task holds a lock; a high-priority task blocks on it; a medium-priority task, runnable and unrelated to the lock, preempts the low-priority holder — so the high-priority task is indirectly blocked by a medium-priority task it has nothing to do with. This is the failure that nearly killed the Mars Pathfinder mission in 1997, and it is unacceptable in a real-time system.

rt_mutex — the priority-inheriting mutex that all RT-converted locks are built on — solves it: when a high-priority task blocks on a lock held by a lower-priority task, the holder temporarily inherits the waiter’s priority, so it cannot be preempted by anything below the waiter’s level. As soon as it releases the lock, its priority drops back. This bounds the inversion to the length of the actual critical section. (See Real-Time Priorities and Priority Inversion for the general mechanism; PREEMPT_RT is what makes PI pervasive throughout the kernel rather than only in user-space PI_FUTEX/pthread mutexes.) The Documentation/locking/rt-mutex.html page is the primary reference.

3. Threaded interrupt handlers

In a stock kernel a hardware interrupt handler runs in hard-IRQ context — preemption disabled, often with other interrupts masked, the handler running to completion before anything else on that CPU. That is a latency window proportional to the slowest driver’s handler. PREEMPT_RT forces interrupt handlers to run in dedicated kernel threads (irq/NN-name kthreads): the tiny hard-IRQ “primary” handler does the bare minimum (acknowledge the device, mask the line) and wakes a schedulable thread that runs the real work. Because that thread is an ordinary schedulable entity, a higher-priority real-time task can preempt it, and the system administrator can even prioritize specific IRQ threads. (The infrastructure — request_threaded_irq(), IRQF_NO_THREAD for the handlers that must stay in hard-IRQ context — predates RT being mainlined and is reused by the stock kernel too; RT simply makes threading the default for almost all IRQs. See Linux Interrupts and Deferred Work MOC for the general threaded-IRQ story.)

4. Preemptible softirqs and the shrinking preempt_disable set

Softirqs (the deferred-work mechanism behind networking, timers, block I/O completion) run in a non-preemptible context on a stock kernel. Under PREEMPT_RT they run in preemptible task context, serialized where necessary by per-CPU local_locks (which, recall, are themselves sleeping locks under RT) rather than by blanket preemption disabling. More broadly, the RT effort spent years auditing and removing or shrinking preempt_disable() / local_irq_disable() regions throughout the kernel so that the non-preemptible set is as small as possible.

5. What stays non-preemptible

The kernel cannot be fully preemptible — something has to schedule, and that something cannot itself be preempted mid-decision. The PREEMPT_RT Kconfig help text (identical in v6.12 and v6.18) is explicit: the kernel becomes fully preemptible “except for very low level and critical code paths (entry code, scheduler, low level interrupt handling).” Those paths use raw_spinlock_t and explicit preemption disabling. The art of PREEMPT_RT is keeping that residue tiny and bounded in duration.

Configuration

PREEMPT_RT is the fourth member of the preemption-model choice in kernel/Kconfig.preempt. As of 6.12 it is a normal upstream option:

config PREEMPT_RT
	bool "Fully Preemptible Kernel (Real-Time)"
	depends on EXPERT && ARCH_SUPPORTS_RT
	select PREEMPTION
	help
	  This option turns the kernel into a real-time kernel by replacing
	  various locking primitives (spinlocks, rwlocks, etc.) with
	  preemptible priority-inheritance aware variants, enforcing
	  interrupt threading and introducing mechanisms to break up long
	  non-preemptible sections. ...

Reading the dependencies: EXPERT hides it behind the “expert options” menu, and ARCH_SUPPORTS_RT is the gate that limits it to architectures that have done the porting work — x86-64, arm64, and RISC-V at the 6.12 merge. select PREEMPTION means an RT kernel is always at least fully preemptible. In the 6.18 Kconfig.preempt, the option additionally gains depends on ... && !COMPILE_TEST and a sibling PREEMPT_RT_NEEDS_BH_LOCK, and — importantly — PREEMPT_NONE and PREEMPT_VOLUNTARY now carry depends on !PREEMPT_RT, so on an RT kernel you cannot select a non-preemptible model. The sched_dynamic_mode() parser in kernel/sched/core.c likewise refuses preempt=none / preempt=voluntary under CONFIG_PREEMPT_RT. The relationship to Lazy Preemption and PREEMPT_LAZY: an RT kernel still gives ordinary fair tasks lazy treatment, but real-time tasks always get the eager TIF_NEED_RESCHED flag for near-immediate preemption.

Building and selecting:

# In menuconfig: General setup -> (expert) -> Preemption Model
#   -> Fully Preemptible Kernel (Real-Time)
# Or in the .config:
CONFIG_PREEMPT_RT=y
# Verify at runtime:
uname -v          # the version string carries "PREEMPT_RT" on an RT build
cat /sys/kernel/realtime   # prints 1 on an RT kernel (when present)

Uncertain

Verify: that /sys/kernel/realtime exists and reads 1 specifically on a mainline 6.12/6.18 CONFIG_PREEMPT_RT build (this sysfs file came from the out-of-tree patch series; confirm it survived mainlining). Reason: I did not locate its definition in the 6.12/6.18 source during this task. The uname -v “PREEMPT_RT” marker is reliable. To resolve: grep the 6.12 tree for kernel/realtime / the sysfs attribute. uncertain

The 20-year road to mainline

PREEMPT_RT (originally the “-rt patch set,” driven by Ingo Molnar, Thomas Gleixner, Steven Rostedt, Sebastian Andrzej Siewior and others) began around 2004. Corbet’s end-game article notes the effort “began almost exactly 20 years ago” (LWN 989212, 2024-09-06). Rather than landing as one giant merge, it was upstreamed incrementally over two decades: high-resolution timers (hrtimers), generic interrupt handling and threaded IRQs, the rt_mutex priority-inheritance infrastructure, the raw_spinlock_t / spinlock_t split, lockdep, the futex PI work, and much else were all merged piecemeal — each useful to the mainline kernel in its own right. The realtime preemption locking core landed years earlier (LWN 867919). The Linux Foundation’s Real-Time Linux project (formerly the RTL Collaborative Project, established 2015) coordinated and funded the long tail.

The final blocker was printk(). The console-output path must work from every context — including NMI, the scheduler core, and inside the most restricted locks — yet the legacy printk implementation took locks and ran console drivers in ways incompatible with full preemptibility. Reworking it into an asynchronous, threaded console design (a multi-year subproject of its own) was the last piece; once it merged for 6.12, CONFIG_PREEMPT_RT could finally be enabled in mainline (LWN 989212; 6.12 merge window, LWN 990750).

Uncertain

Verify: the exact start year (2004 vs 2005). Reason: Corbet’s 2024 article says “almost exactly 20 years ago” (→ ~2004); Wikipedia’s PREEMPT_RT page gives 2005. Both agree on “roughly two decades” and on the 6.12 completion. To resolve: the discrepancy is one year and does not affect any substantive claim. uncertain

Where PREEMPT_RT is used

Real-time Linux is deployed where a missed deadline is a failure, not just a hiccup:

  • Professional and pro-sumer audio — low-latency audio (JACK, Ardour) needs bounded buffer-fill latency to avoid xruns (audible glitches). RT kernels are a staple of audio-focused distributions.
  • Industrial control and robotics — motion control, PLCs, and robot arms run control loops at fixed periods (e.g. 1 kHz); jitter translates to mechanical error. Frameworks like ROS 2’s real-time work and the OSADL (Open Source Automation Development Lab) QA farm target RT Linux. OSADL publishes continuous worst-case-latency measurements across many boards.
  • Telecom and high-frequency / low-latency systems — packet-processing and trading systems that need bounded tail latency.
  • Measurement and test equipment, aerospace ground systems — anywhere deterministic response is contractually required.

RT is usually combined with the isolation techniques in this MOC — CPU Isolation isolcpus and nohz_full, CPU Affinity and sched_setaffinity, and Housekeeping CPUs and Tickless Isolation — to clear a CPU of all other work and remove even the timer tick, plus a real-time scheduling policy (Real-Time Scheduling SCHED_FIFO and SCHED_RR or SCHED_DEADLINE and Earliest Deadline First) for the critical task. PREEMPT_RT bounds kernel latency; the RT scheduling class bounds which task runs; isolation removes interference. All three are usually needed together for the lowest jitter.

Common misunderstandings

  • “PREEMPT_RT is still an out-of-tree patch.” Stale. The last pieces merged for 6.12 (2024); CONFIG_PREEMPT_RT is a mainline option on x86-64/arm64/riscv. Distributions still ship dedicated RT kernel flavours, but the code is upstream.
  • “RT makes Linux faster.” No — it usually lowers throughput. Sleeping locks, threaded IRQs, and pervasive PI add overhead. RT trades average performance for bounded worst-case latency. If you do not need determinism, an RT kernel is a net loss.
  • “spinlock_t always spins.” Not under RT, where it is an rt_mutex that sleeps. Only raw_spinlock_t always spins. Code that requires true spinning / preempt-off (touching hardware registers, the scheduler) must use raw_spinlock_t explicitly.
  • “RT eliminates all non-preemptible code.” No — entry code, the scheduler, and low-level IRQ handling stay non-preemptible (via raw_spinlock_t). RT minimizes and bounds the non-preemptible set; it does not erase it.
  • “PREEMPT_RT is the same as SCHED_FIFO.” Different layers. SCHED_FIFO/SCHED_RR are scheduling policies (which task runs); PREEMPT_RT is a kernel build that makes the kernel preemptible enough for those policies to actually meet deadlines. You typically use both.

See Also

Graph View

Backlinks