The PREEMPT_RT Real-Time Kernel

PREEMPT_RT is the Linux configuration that turns the general-purpose kernel into a real-time kernel — one with bounded, predictable worst-case scheduling latency rather than merely good average throughput. It does this by making nearly all kernel code preemptible: it converts the kernel’s ordinary spinning locks (spinlock_t, rwlock_t, local_lock) into sleeping, priority-inheriting locks built on rt_mutex, forces almost all interrupt handlers to run in schedulable kernel threads, and shrinks the handful of truly non-preemptible regions down to the lowest-level entry, scheduler, and raw-spinlock code. The most important fact about it in 2026: PREEMPT_RT is now mainline. After roughly 20 years out of tree (begun 2004), the final missing piece — a reworked, threaded printk() — was merged for Linux 6.12 (released 2024-11-17), making CONFIG_PREEMPT_RT a selectable build option in the upstream kernel on x86-64, arm64, and RISC-V (Corbet, “The realtime preemption end game — for real this time,” LWN, 2024-09-06; the 6.12 release, LWN). The precise wording matters and is verified below by existence-checking the source tree tag by tag: the symbol CONFIG_PREEMPT_RT has been in kernel/Kconfig.preempt since v5.3 (2019), but it was un-selectable dead config for five years because no architecture set its ARCH_SUPPORTS_RT gate until v6.12. Do not describe PREEMPT_RT as “an out-of-tree patchset” — that framing is stale as of 6.12. This note pins Linux v6.12 as its reference tree (the vault’s pin, and still a longterm release as of 2026-09-11 at 6.12.109 per kernel.org/releases.json) and calls out where v6.18 and later differ.

Why a real-time kernel is a different goal

A stock Linux kernel optimizes for throughput and good average latency. A real-time kernel optimizes for bounded worst-case latency — the guarantee that a high-priority task, once it becomes runnable, will run within some provable upper bound regardless of what else the system is doing. The two goals conflict: the tricks that maximize throughput (batching work, holding locks across long sections, deferring to softirqs, running interrupt handlers to completion with preemption disabled) all add worst-case latency.

The enemy of bounded latency is any stretch of code during which the scheduler cannot run. In a normal kernel those stretches are everywhere: a held spinlock_t disables preemption for its whole critical section; a hardware interrupt handler runs to completion with that CPU’s preemption (and often other interrupts) disabled; softirqs and tasklets run in a non-preemptible context; long-running kernel loops may not check in for a reschedule. Each is a window in which a just-woken real-time task must wait. PREEMPT_RT’s entire program is to eliminate or bound every one of these windows.

This is a strictly stronger goal than the PREEMPT_FULL model (see Kernel Preemption Models). PREEMPT_FULL already makes most kernel code preemptible, but it leaves spinlock sections, IRQ handlers, and softirqs non-preemptible. PREEMPT_RT attacks exactly those.

Mental model: make almost everything schedulable

flowchart TB
  subgraph STOCK["Stock kernel: non-preemptible islands"]
    S1["spin_lock(): preempt off<br/>for whole critical section"]
    S2["hardirq handler runs<br/>to completion, preempt off"]
    S3["softirq/tasklet:<br/>non-preemptible context"]
  end
  subgraph RT["PREEMPT_RT: islands made schedulable"]
    R1["spinlock_t -> rt_mutex<br/>sleeping + priority inheritance<br/>(preemption stays ON)"]
    R2["hardirq -> threaded IRQ<br/>(kthread, schedulable)"]
    R3["softirq runs in task ctx,<br/>preemptible"]
  end
  S1 -->|"convert"| R1
  S2 -->|"force threading"| R2
  S3 -->|"per-CPU local_lock"| R3
  RAW["raw_spinlock_t<br/>(stays a true spinlock,<br/>preempt off — used only in<br/>scheduler/entry/low-level IRQ)"]
  R1 -. "still spins in tiny<br/>core paths" .- RAW

The core transformation. What it shows: the three big sources of non-preemptible time in a stock kernel — spinlocks, hardware interrupt handlers, and softirqs — are each converted into something the scheduler can preempt. spinlock_t becomes a sleeping rt_mutex (so a task waiting on a contended lock blocks instead of spinning with preemption off), hardware IRQ handlers become threaded (a schedulable kthread does the real work), and softirqs run in preemptible task context. The insight to take: the only thing left that truly disables preemption is raw_spinlock_t, reserved for the irreducible core — the scheduler itself, entry/exit code, and the lowest-level interrupt plumbing. Bounded latency is achieved by shrinking the non-preemptible set to that irreducible core.

The four preemption models, side by side

PREEMPT_RT only makes sense against its siblings. At tag v6.12 the choice block in kernel/Kconfig.preempt has exactly four members, in this order: PREEMPT_NONE, PREEMPT_VOLUNTARY, PREEMPT, PREEMPT_RT (verified by reading the file; endchoice sits immediately after the PREEMPT_RT block at line 87). Each model is defined by which of the three preemption opportunities the kernel honours while running kernel-mode code:

Model (Kconfig symbol)	Prompt string in menuconfig	Preempts on return to user space	cond_resched() is a real preemption point	might_sleep() is a preemption point	Involuntary preemption anywhere preempt_count == 0	spinlock_t sections preemptible	IRQ handlers threaded by default	Worst-case kernel latency	Throughput cost	Intended use
PREEMPT_NONE	“No Forced Preemption (Server)”	yes	yes	no	no	no	no	“no guarantees and occasional longer delays are possible” (Kconfig)	none (baseline)	servers, batch, HPC
PREEMPT_VOLUNTARY	“Voluntary Kernel Preemption (Desktop)”	yes	yes	yes	no	no	no	reduced but still unbounded	“slightly lower throughput”	general desktop
PREEMPT (a.k.a. PREEMPT_FULL)	“Preemptible Kernel (Low-Latency Desktop)”	yes	no-op (redundant)	no-op (redundant)	yes	no	no	“latency requirements in the milliseconds range” (Kconfig)	“slightly lower throughput and a slight runtime overhead”	low-latency desktop, embedded
PREEMPT_RT	“Fully Preemptible Kernel (Real-Time)”	yes	no-op (redundant)	no-op (redundant)	yes	yes (spinlock_t → rt_mutex)	yes (forced)	bounded by design	largest — sleeping locks + PI + IRQ threads	hard real-time

Table 1 — the preemption choice at v6.12. Prompt strings and the quoted latency/throughput phrasings come from the Kconfig help text itself, not from folklore; the cond_resched() / might_sleep() columns were read out of the preprocessor in include/linux/sched.h and include/linux/kernel.h at the same tag. What surprises most readers: cond_resched() is a working preemption point under PREEMPT_NONE — the guard is #if !defined(CONFIG_PREEMPTION) || defined(CONFIG_PREEMPT_DYNAMIC), so both non-preemptible models honour it and both fully preemptible models compile it to return 0, because involuntary preemption already covers those points. What actually separates NONE from VOLUNTARY is might_resched() (invoked by every might_sleep() annotation), which is __cond_resched() only under CONFIG_PREEMPT_VOLUNTARY_BUILD and do { } while (0) otherwise. The insight: the first three models differ only in where the kernel checks whether to reschedule; only PREEMPT_RT changes what the kernel does with its locks and interrupts, which is why it alone can claim a bound rather than merely “usually good.”

Two further symbols are not members of the choice but change how the choice behaves:

Symbol	What it is	Status at v6.12	Status at v6.18
PREEMPT_DYNAMIC	Lets the model be picked at boot with preempt=none|voluntary|full instead of at compile time, by patching static calls/keys	depends on HAVE_PREEMPT_DYNAMIC && !PREEMPT_RT — mutually exclusive with RT	depends on HAVE_PREEMPT_DYNAMIC only — the !PREEMPT_RT exclusion has been dropped
PREEMPT_LAZY	A fifth model, “Scheduler controlled preemption model”: full preemption for real-time tasks, deferred (lazy) preemption for SCHED_NORMAL to recover throughput	absent — does not exist in kernel/Kconfig.preempt at v6.12	present, a fifth member of the choice, gated on ARCH_HAS_PREEMPT_LAZY

Table 2 — the two modifiers, and a version-pinning trap. What it shows: both rows are moving targets. PREEMPT_LAZY is frequently described as “part of the 6.12 preemption work”; it is not — existence-checking kernel/Kconfig.preempt shows zero occurrences of config PREEMPT_LAZY at v6.12 and one at v6.13, so v6.13 (2025-01-19) is where lazy preemption became selectable. The insight: any claim about which preemption models exist must name a kernel version, because the set changed in three consecutive releases (6.12 RT, 6.13 lazy, and by 6.18 PREEMPT_RT has moved outside the choice block entirely and PREEMPT_NONE/PREEMPT_VOLUNTARY have gained depends on !PREEMPT_RT).

See Kernel Preemption Models for the first three models in depth and Lazy Preemption and PREEMPT_LAZY for the fifth.

The mechanism, piece by piece

1. Sleeping spinlocks: spinlock_t becomes an rt_mutex

This is the heart of PREEMPT_RT, and the in-tree Documentation/locking/locktypes.rst (verified at tag v6.12) spells it out precisely. The kernel deliberately distinguishes two spinning-lock types:

• raw_spinlock_t — “a strict spinning lock implementation in all kernels, including PREEMPT_RT kernels.” It busy-waits with preemption disabled. It is reserved for “real critical core code, low-level interrupt handling and places where disabling preemption or interrupts is required, for example, to safely access hardware state.” This is the irreducible non-preemptible primitive.
• spinlock_t — on a non-RT kernel this is mapped to raw_spinlock_t and behaves identically (spin, preempt off). On a PREEMPT_RT kernel, spinlock_t is “mapped to a separate implementation based on rt_mutex,” which changes the semantics dramatically:
  • Preemption is not disabled while the lock is held.
  • The hard-IRQ suffixes (spin_lock_irq, spin_lock_irqsave) do not actually disable interrupts — interrupts stay on.
  • A task that contends for the lock sleeps (blocks and is taken off the CPU) rather than spinning.

The same treatment applies to rwlock_t, local_lock (mapped to a per-CPU spinlock_t), and rw_semaphore (mapped to a separate rt_mutex-based implementation). The consequence: holding a spinlock_t no longer creates a non-preemptible, interrupt-disabled latency window. A higher-priority task can preempt a lock holder; correctness is preserved by priority inheritance (next section). The trade-off is that lock acquisition/release is heavier (it can sleep and must do PI bookkeeping), which is why genuinely tiny, hardware-touching critical sections are left as raw_spinlock_t.

What spin_lock() actually expands to on an RT kernel

Prose descriptions of this conversion routinely stop at “it sleeps instead of spinning,” which is both incomplete and, in the common case, wrong. The real substitution lives in kernel/locking/spinlock_rt.c at v6.12, and it is thirteen lines long:

static __always_inline void __rt_spin_lock(spinlock_t *lock)
{
	rtlock_might_resched();      /* debug: assert we are in a context that may block */
	rtlock_lock(&lock->lock);    /* the rt_mutex acquisition proper */
	rcu_read_lock();             /* stand in for the RCU quiescence that preempt_disable() gave */
	migrate_disable();           /* stand in for the CPU-pinning that preempt_disable() gave */
}

Read it as a replacement of side effects. On a non-RT kernel spin_lock() calls preempt_disable(), which incidentally does three things at once: it serialises against other tasks on this CPU, it pins the task to this CPU (so this_cpu_ptr() stays valid), and it suppresses RCU quiescent states (so an RCU read-side critical section cannot end mid-lock). RT cannot keep the first effect — that is the latency window — so it re-creates the other two explicitly: migrate_disable() for the pinning and rcu_read_lock() for the RCU property, while preemption stays on. The file’s own header comment says so: “Non RT spin/rwlocks disable preemption and eventually interrupts. Disabling preemption has the side effect of disabling migration and preventing RCU grace periods. The RT substitutions explicitly disable migration and take rcu_read_lock() across the lock held section.”

rtlock_lock() itself takes the uncontended fast path with a single rt_mutex_cmpxchg_acquire(); only on contention does it fall into rtlock_slowlock().

The contended path does not immediately sleep — it spins adaptively

Inside rtlock_slowlock_locked() (kernel/locking/rtmutex.c, v6.12) the waiter loop is:

if (!owner || !rtmutex_spin_on_owner(lock, &waiter, owner))
	schedule_rtlock();

rtmutex_spin_on_owner() — the “adaptive spinlocks” work whose copyright header credits Gregory Haskins, Sven Dietrich and Peter Morreale (Novell, 2008) and a simplification by Steven Rostedt (Red Hat, 2008) — busy-waits with cpu_relax() for as long as all four of these hold:

1. the lock owner has not changed,
2. the owner is currently executing on some CPU (owner_on_cpu(owner)),
3. this task is still the top waiter, and
4. need_resched() is false.

Only when one of those fails does the waiter call schedule_rtlock() and actually block. This matters for the mental model: under RT a contended spinlock_t behaves like a spinlock when spinning is the cheap answer (owner running on another CPU, short section) and like a mutex when it is not (owner descheduled). The claim “spinlock_t always sleeps under PREEMPT_RT” is a common oversimplification.

Task state is preserved, via a dedicated task state

A plain mutex sets the blocking task to TASK_UNINTERRUPTIBLE, clobbering whatever state it had. That would be a correctness disaster for a lock substitution, because kernel code takes spinlock_t while in TASK_INTERRUPTIBLE all the time (classic prepare_to_wait() patterns). RT therefore saves and restores it: current_save_and_set_rtlock_wait_state() before blocking, current_restore_rtlock_saved_state() after acquisition, with the intermediate state being the RT-specific TASK_RTLOCK_WAIT. Documentation/locking/locktypes.rst walks the interaction with a concurrent non-lock wakeup — the wakeup writes TASK_RUNNING into task->saved_state rather than task->state, so “the real wakeup cannot be lost” — and kernel/locking/spinlock_rt.c states the invariant flatly: “spinlocks and rwlocks are state preserving.”

The suffix table: what _irq, _irqsave and _bh mean under RT

Primitive	Non-RT kernel	PREEMPT_RT kernel	Why
raw_spin_lock()	spin, preempt_disable()	identical — spin, preempt_disable()	the one primitive RT does not touch
spin_lock()	spin, preempt_disable()	rt_mutex acquire + rcu_read_lock() + migrate_disable(); preemption stays on	the whole point
spin_lock_irq() / spin_lock_irqsave()	spin, preempt_disable(), interrupts off	interrupts stay on; otherwise as spin_lock()	an rt_mutex “requires a fully preemptible context”; you cannot block with IRQs off
spin_lock_bh()	spin, preempt_disable(), softirqs off	softirqs are still disabled, via a per-CPU local_lock (softirq_ctrl.lock), preemption stays on	BH exclusion is a real semantic other code relies on, so it is preserved by a different mechanism
rwlock_t (read side)	fair spinning reader/writer lock	rt_mutex-based; readers boost a blocked writer, writers cannot boost multiple readers	a preempted low-priority reader can therefore starve a high-priority writer — an RT-specific hazard called out explicitly in locktypes.rst
local_lock	wrapper around preempt_disable() / local_irq_save()	per-CPU spinlock_t (so: sleeping)	it becomes a real lock with a real, named protection scope
mutex, rw_semaphore, ww_mutex	sleeping locks	sleeping locks, re-implemented on rt_mutex so they carry PI	PI has to be pervasive or the chain breaks

Table 3 — the conversion matrix, from Documentation/locking/locktypes.rst and kernel/locking/spinlock_rt.c at v6.12. The insight to take: the _irq* suffixes are the surprising row — code that says spin_lock_irqsave() and then relies on interrupts actually being off (to serialise against its own hardware interrupt handler, say) is silently broken on an RT kernel. The _bh row is the opposite surprise: it keeps working, but through a lock rather than through a counter.

printk() — the last thing to be converted, and the reason the merge took until 6.12 — is a good illustration of the raw/non-raw split in practice. Console output must be callable from NMI, from inside the scheduler, and from inside raw_spinlock_t sections, so it cannot itself take a sleeping lock. The resolution was not “make printk take raw locks” but a redesign: a lockless ring buffer plus a threaded console printer, so the emitting side stays usable from any context while the expensive device I/O happens in a schedulable kthread (LWN 989212; Thomas Gleixner’s 2021 remaining-work list on LWN 867919 names printk as one of three blockers left at that time).

Uncertain

Verify: the internal structure of the 6.12 nbcon/threaded-console implementation — specifically which locks the emit path takes and how the atomic (panic/NMI) console handover works. Reason: I verified that printk was the final blocker and why (attested by both LWN articles read and by Gleixner’s own comment listing “memory management, printk, fs/namespace”), and I verified the locking primitives in kernel/locking/ directly, but I did not read kernel/printk/nbcon.c in this pass. To resolve: read kernel/printk/printk.c and kernel/printk/nbcon.c at tag v6.12. uncertain

2. Priority inheritance via rt_mutex

Once a lock holder can be preempted, the classic real-time hazard appears: priority inversion. A low-priority task holds a lock; a high-priority task blocks on it; a medium-priority task, runnable and unrelated to the lock, preempts the low-priority holder — so the high-priority task is indirectly blocked by a medium-priority task it has nothing to do with. This is the failure that nearly killed the Mars Pathfinder mission in 1997, and it is unacceptable in a real-time system.

Documentation/locking/rt-mutex-design.rst (Steven Rostedt, 2006, still shipped at v6.12) draws the distinction the name hides: a priority inversion per se is unavoidable and harmless — “Anytime a high priority process wants to use a resource that a lower priority process has… the high priority process must wait.” What must be prevented is unbounded priority inversion, “when the high priority process is prevented from running by a lower priority process for an undetermined amount of time.” The middle-priority task B is what makes the wait undetermined: nothing about the length of A’s wait is now a function of C’s critical section.

sequenceDiagram
    autonumber
    participant A as Task A<br/>(SCHED_FIFO prio 90)
    participant B as Task B<br/>(SCHED_FIFO prio 50)
    participant C as Task C<br/>(SCHED_FIFO prio 10)
    participant L as rt_mutex L

    Note over C,L: C is the only runnable task
    C->>L: rt_spin_lock(L) — uncontended, cmpxchg succeeds
    Note over C: C owns L, prio 10

    A->>L: rt_spin_lock(L) — contended
    L-->>A: blocked: enqueue A on L->waiters (rbtree, keyed by prio)
    Note over L: WITHOUT inheritance
    B->>B: becomes runnable at prio 50
    B-->>C: preempts C (50 > 10)
    Note over A,C: A (prio 90) now waits on B (prio 50)<br/>for an UNBOUNDED time — B never touched L

    Note over L: WITH inheritance (what rt_mutex does)
    L->>C: rt_mutex_adjust_prio(): C inherits A's prio 90
    B->>B: becomes runnable at prio 50
    Note over B,C: B cannot preempt C (50 < 90)
    C->>L: rt_spin_unlock(L)
    L->>C: deboost C back to prio 10
    L->>A: mark_wakeup_next_waiter(): hand L to A
    Note over A: A runs. Its wait was bounded by<br/>the length of C's critical section

Priority inversion and the inheritance fix, as a message sequence. What it shows: the same three tasks and one lock, run twice — the upper half without priority inheritance, the lower half with it. In the upper half an unrelated medium-priority task B indirectly blocks the highest-priority task A for an arbitrarily long time. In the lower half the lock handoff boosts the holder C to the waiter’s priority the instant A blocks, so B is locked out until C releases. The insight to take: inheritance does not make A’s wait short — A still waits for C’s whole critical section. It makes A’s wait bounded by something the system designer controls (the length of that critical section) instead of by an unrelated task’s runtime. This is the single concept PREEMPT_RT exists to make pervasive.

rt_mutex — the priority-inheriting mutex that all RT-converted locks are built on — implements this: when a high-priority task blocks on a lock held by a lower-priority task, the holder temporarily inherits the waiter’s priority, so it cannot be preempted by anything below the waiter’s level. As soon as it releases the lock, its priority drops back.

The mechanics in kernel/locking/rtmutex.c at v6.12 are worth naming precisely, because “the holder inherits the priority” hides a graph walk:

• Two red-black trees, not one. struct rt_mutex_base holds waiters, an rb_root_cached of the tasks blocked on this lock, keyed by priority so the top waiter is the cached leftmost node (rt_mutex_enqueue()). Separately, each task_struct carries pi_waiters, an rbtree of the top waiters of every lock this task currently owns (rt_mutex_enqueue_pi()). A task’s effective priority is then simply max(own priority, priority of its top pi_waiter) — which is exactly what rt_mutex_adjust_prio() computes before calling rt_mutex_setprio().
• Boosting is a chain walk, not a single step. If C is itself blocked on a lock held by D, boosting C is useless unless D is boosted too. rt_mutex_adjust_prio_chain() walks that chain, and its header comment is an unusually explicit 13-step protocol with the lock held at each step annotated ([P1] = task->pi_lock, [L] = rtmutex->wait_lock). It deliberately holds “a maximum of two locks per step” so that the boosting walk is itself preemptible — a PI implementation that walked the chain with preemption off would reintroduce the latency it exists to remove.
• The chain is bounded by a sysctl. int max_lock_depth = 1024; in kernel/locking/rtmutex_api.c caps the walk; it is exposed as /proc/sys/kernel/max_lock_depth (registered in kernel/sysctl.c under #ifdef CONFIG_RT_MUTEXES). Exceeding it aborts the walk and is how deadlock/livelock in the chain is contained. Do not confuse this with lockdep’s unrelated MAX_LOCK_DEPTH (48UL in include/linux/sched.h), which bounds how many locks lockdep will track per task — same-ish name, different subsystem, different value.
• rt_mutex internals are built on raw_spinlock_t. struct rt_mutex_base embeds raw_spinlock_t wait_lock (include/linux/rtmutex.h). The sleeping lock is implemented using the spinning lock — which is the neat structural reason raw_spinlock_t can never itself be converted.

(See Real-Time Priorities and Priority Inversion for the general mechanism and the Mars Pathfinder story; Kernel Mutexes for the plain mutex this replaces; PREEMPT_RT is what makes PI pervasive throughout the kernel rather than only in user-space PI_FUTEX/pthread mutexes.)

3. Threaded interrupt handlers

In a stock kernel a hardware interrupt handler runs in hard-IRQ context — preemption disabled, often with other interrupts masked, the handler running to completion before anything else on that CPU. That is a latency window proportional to the slowest driver’s handler. PREEMPT_RT forces interrupt handlers to run in dedicated kernel threads (irq/NN-name kthreads): the tiny hard-IRQ “primary” handler does the bare minimum (acknowledge the device, mask the line) and wakes a schedulable thread that runs the real work. Because that thread is an ordinary schedulable entity, a higher-priority real-time task can preempt it, and the administrator can prioritise specific IRQ threads with chrt.

The forcing is not a policy written in prose; it is three lines of preprocessor in include/linux/interrupt.h at v6.12:

#ifdef CONFIG_IRQ_FORCED_THREADING
# ifdef CONFIG_PREEMPT_RT
#  define force_irqthreads()	(true)                              /* RT: always on */
# else
DECLARE_STATIC_KEY_FALSE(force_irqthreads_key);
#  define force_irqthreads()	(static_branch_unlikely(&force_irqthreads_key))
# endif
#else
#define force_irqthreads()	(false)
#endif

On a non-RT kernel force_irqthreads() is a static branch that is off until the threadirqs boot parameter flips it (early_param("threadirqs", setup_forced_irqthreads) in kernel/irq/manage.c; note that the static key is compiled only on non-RT — #if defined(CONFIG_IRQ_FORCED_THREADING) && !defined(CONFIG_PREEMPT_RT)). On an RT kernel the compiler folds it to a constant true and every eligible IRQ is threaded with no boot-time choice.

“Eligible” is the interesting part. irq_setup_forced_threading() bails out early for three flags:

if (!force_irqthreads())
	return 0;
if (new->flags & (IRQF_NO_THREAD | IRQF_PERCPU | IRQF_ONESHOT))
	return 0;

IRQF_NO_THREAD is the driver’s explicit “this must run in hard-IRQ context” opt-out (the timer tick and IPIs are the archetypes); IRQF_PERCPU handlers are per-CPU and already constrained. Everything else gets IRQF_ONESHOT forced on and, if the driver registered both a primary handler and a thread function, a secondary irqaction is allocated so that both halves end up threaded.

flowchart TB
  DEV["Device asserts IRQ line"] --> ENTRY["CPU takes the interrupt<br/>low-level entry code<br/>(raw_spinlock_t territory,<br/>non-preemptible either way)"]
  ENTRY --> Q{"force_irqthreads()<br/>AND not IRQF_NO_THREAD<br/>/ IRQF_PERCPU / IRQF_ONESHOT?"}
  Q -- "no (stock kernel default,<br/>or opted out)" --> HARD["Full driver handler runs in<br/>hard-IRQ context<br/>preempt OFF, line masked<br/><b>latency = this handler's runtime</b>"]
  HARD --> SOFT1["raise_softirq() -> softirq runs<br/>on IRQ exit, non-preemptible"]
  Q -- "yes (always, on PREEMPT_RT)" --> PRIM["irq_default_primary_handler():<br/>mask the line, return<br/>IRQ_WAKE_THREAD<br/><b>runtime: a few hundred ns</b>"]
  PRIM --> WAKE["wake_up_process(irq thread)"]
  WAKE --> THR["irq/NN-name kthread<br/>SCHED_FIFO, preemptible,<br/>runs thread_fn(), unmasks line"]
  THR --> SOFT2["softirq work runs in<br/>preemptible task context,<br/>serialised by per-CPU local_lock"]
  RT["High-priority RT task<br/>becomes runnable"] -. "can preempt" .-> THR
  RT -. "CANNOT preempt" .-> HARD

The two interrupt paths, stock versus forced-threaded. What it shows: the same hardware event handled two ways. On the left the entire driver handler executes with preemption off, so a real-time task that wakes during it must wait for the slowest driver on the system. On the right only a few hundred nanoseconds of ack-and-mask run in hard-IRQ context; the real work is a SCHED_FIFO kernel thread that the real-time task can simply preempt. The insight to take: threading does not make interrupt handling faster — it makes it schedulable, which converts “however long the worst driver takes” into “a scheduling decision you control by setting priorities.” The dashed arrows are the whole argument.

The infrastructure itself — request_threaded_irq(), IRQF_NO_THREAD, the irq/NN-name kthreads — predates RT being mainlined and is used by stock kernels too; RT only changes the default. See Threaded Interrupt Handlers for the general mechanism in depth and Linux Interrupts and Deferred Work MOC for the surrounding map.

4. Preemptible softirqs and the shrinking preempt_disable set

Softirqs (the deferred-work mechanism behind networking, timers, block I/O completion) run in a non-preemptible context on a stock kernel. Under PREEMPT_RT they run in preemptible task context, serialized where necessary by per-CPU local_locks (which, recall, are themselves sleeping locks under RT) rather than by blanket preemption disabling. More broadly, the RT effort spent years auditing and removing or shrinking preempt_disable() / local_irq_disable() regions throughout the kernel so that the non-preemptible set is as small as possible.

The concrete substitution is in kernel/softirq.c at v6.12, inside a #ifdef CONFIG_PREEMPT_RT block that replaces the whole bottom-half accounting scheme:

struct softirq_ctrl {
	local_lock_t	lock;
	int		cnt;
};
static DEFINE_PER_CPU(struct softirq_ctrl, softirq_ctrl) = {
	.lock	= INIT_LOCAL_LOCK(softirq_ctrl.lock),
};

On a stock kernel local_bh_disable() is a counter bump inside preempt_count, and a task in a BH-disabled section therefore cannot be preempted. On RT the same call takes the per-CPU softirq_ctrl.lock (a sleeping spinlock_t, recall) and rcu_read_lock(), then bumps two counters: the per-CPU softirq_ctrl.cnt and a new per-task current->softirq_disable_cnt. The comment explains exactly why two are needed: “RT accounts for BH disabled sections in task::softirqs_disabled_cnt and also in per CPU softirq_ctrl::cnt. This is necessary to allow tasks in a softirq disabled section to be preempted. The per task counter is used for softirq_count(), in_softirq() and in_serving_softirqs() because these counts are only valid when the task holding softirq_ctrl::lock is running.”

That sentence is the mechanism: once a BH-disabled section can be preempted and even migrated, a per-CPU counter no longer answers “am I in a softirq?” for the current task, because some other task may now be running on this CPU. So the state moves onto the task, and the per-CPU counter is kept only to suppress pointless ksoftirqd wakeups. A companion helper, local_bh_blocked(), exists purely so the idle task can tell “softirqs are pending but blocked by a preempted lock holder” apart from a genuine stuck-softirq bug.

At v6.18 this area gained a debugging escape hatch, CONFIG_PREEMPT_RT_NEEDS_BH_LOCK (“Enforce softirq synchronisation on PREEMPT_RT”), whose help text says it “provides the old behaviour for testing reasons” — evidence that preemptible softirqs are still an area where RT-specific bugs surface.

5. What stays non-preemptible

The kernel cannot be fully preemptible — something has to schedule, and that something cannot itself be preempted mid-decision. The PREEMPT_RT Kconfig help text (identical in v6.12 and v6.18) is explicit: the kernel becomes fully preemptible “except for very low level and critical code paths (entry code, scheduler, low level interrupt handling).” Those paths use raw_spinlock_t and explicit preemption disabling. The art of PREEMPT_RT is keeping that residue tiny and bounded in duration.

Two consequences follow, both counter-intuitive and both documented in Documentation/locking/locktypes.rst.

raw_spinlock_t sections get more restrictive under RT, not less. On a stock kernel this is fine:

raw_spin_lock(&lock);
p = kmalloc(sizeof(*p), GFP_ATOMIC);   /* OK on non-RT */

Under RT it breaks, “because the memory allocator is fully preemptible and therefore cannot be invoked from truly atomic contexts.” And the converse is the part that catches people out: the same allocation is fine under a plain spin_lock() on an RT kernel, “because they do not disable preemption on PREEMPT_RT kernels.” RT inverts which of the two locks is the dangerous one to hold.

Bit spinlocks cannot be converted at all. “PREEMPT_RT cannot substitute bit spinlocks because a single bit is too small to accommodate an RT-mutex.” A bit spinlock therefore keeps hard spinning semantics on an RT kernel and inherits all the raw_spinlock_t caveats. Where a bit spinlock had to become sleepable, the conversion was done by hand with #ifdefs at the call site — unlike spinlock_t, whose substitution is entirely transparent in headers.

The result is a strict three-level nesting order that lockdep enforces on every kernel, RT or not:

1. sleeping locks (mutex, rw_semaphore, rt_mutex, …)
2. spinlock_t, rwlock_t, local_lock
3. raw_spinlock_t and bit spinlocks

You may nest downward but never upward. On a stock kernel levels 2 and 3 are the same category, so the rule is nearly vacuous; under RT level 2 becomes sleeping, and every place the kernel took a spinlock_t inside a raw_spinlock_t became a real bug that had to be found and fixed. That audit — thousands of call sites across two decades — is most of what the twenty years went into.

Configuration

PREEMPT_RT is the fourth member of the preemption-model choice in kernel/Kconfig.preempt. As of 6.12 it is a normal upstream option:

config PREEMPT_RT
	bool "Fully Preemptible Kernel (Real-Time)"
	depends on EXPERT && ARCH_SUPPORTS_RT
	select PREEMPTION
	help
	  This option turns the kernel into a real-time kernel by replacing
	  various locking primitives (spinlocks, rwlocks, etc.) with
	  preemptible priority-inheritance aware variants, enforcing
	  interrupt threading and introducing mechanisms to break up long
	  non-preemptible sections. ...

Reading the dependencies line by line: EXPERT hides the option behind the “expert options” menu, so it does not appear in a default menuconfig at all. ARCH_SUPPORTS_RT is the real gate — a bare bool with no prompt that an architecture must select to advertise “I have done the porting work.” select PREEMPTION means an RT kernel is always at least fully preemptible.

ARCH_SUPPORTS_RT is where the twenty-year story actually ends, and it is directly checkable. Fetching arch/<arch>/Kconfig at successive tags and counting occurrences of the symbol gives:

Architecture	arch/*/Kconfig at v6.11	at v6.12	at v6.13	at v7.1
x86	absent	present	present	present
arm64	absent	present	present	present
riscv	absent	present	present	present
arm (32-bit)	absent	absent	absent	absent

Table 4 — select ARCH_SUPPORTS_RT by architecture and tag, measured by fetching each arch/<arch>/Kconfig from the mainline tree and grepping. The insight: the transition is a clean step function at v6.12 on exactly the three architectures LWN named, and 32-bit ARM never joined — so “PREEMPT_RT is mainline” has always meant “on x86-64, arm64 and RISC-V”, never “everywhere.”

At v6.18 the same area has moved on in four ways, all verified by diffing kernel/Kconfig.preempt between the two tags:

1. PREEMPT_RT gains && !COMPILE_TEST in its depends on line, keeping it out of automated compile-test configs.
2. A sibling option PREEMPT_RT_NEEDS_BH_LOCK appears (see the softirq section above).
3. PREEMPT_NONE and PREEMPT_VOLUNTARY each gain depends on !PREEMPT_RT, so on an RT kernel you cannot select a non-preemptible model. sched_dynamic_mode() in kernel/sched/core.c enforces the same thing at boot — the "none" and "voluntary" string comparisons are inside # ifndef CONFIG_PREEMPT_RT, so preempt=none on an RT kernel produces Dynamic Preempt: unsupported mode: none and is ignored.
4. PREEMPT_DYNAMIC is no longer mutually exclusive with RT. At v6.12 it reads depends on HAVE_PREEMPT_DYNAMIC && !PREEMPT_RT; at v6.18 the !PREEMPT_RT clause is gone. An RT kernel can now be built with dynamic preemption, where the reachable modes are full and (on architectures with ARCH_HAS_PREEMPT_LAZY) lazy.

The relationship to Lazy Preemption and PREEMPT_LAZY: lazy preemption did not exist at v6.12 at all, and where it does exist it defers preemption only for SCHED_NORMAL tasks — real-time tasks always get the eager TIF_NEED_RESCHED flag for near-immediate preemption.

Building and selecting:

# In menuconfig: General setup -> (expert) -> Preemption Model
#   -> Fully Preemptible Kernel (Real-Time)
# Or in the .config:
CONFIG_PREEMPT_RT=y
# Verify at runtime — the version string carries "PREEMPT_RT" on an RT build:
uname -v

Resolved: there is no /sys/kernel/realtime in mainline

A previous revision of this note suggested cat /sys/kernel/realtime as a runtime check, flagged as unverified. It is not a mainline interface. All /sys/kernel/* attributes are registered in kernel/ksysfs.c; grepping that file at tags v6.11, v6.12 and v6.18, and the live torvalds/linux tree on git.kernel.org, returns zero matches for “realtime”, and the kernel_attrs[] array contains no such entry. The attribute came from the out-of-tree -rt patch series and did not come with it. uname -v (or uname -a) containing the literal string PREEMPT_RT is the reliable check; distributions that still carry the leftover patch may also expose the sysfs file, so its presence proves RT but its absence proves nothing.

The 20-year road to mainline

PREEMPT_RT (originally the “-rt patch set,” driven by Ingo Molnar, Thomas Gleixner, Steven Rostedt, Sebastian Andrzej Siewior and others) began in 2004. Three independent primary sources agree, which is enough to close the “2004 or 2005?” question a previous revision of this note left open:

• Corbet’s end-game brief says the work “got its start almost exactly 20 years ago” — dated 2024-09-06, so ~2004 (LWN 989212).
• Corbet’s 2021 brief says flatly “This work began in 2004” (LWN 867919).
• The source itself: kernel/locking/rtmutex.c at v6.12 still carries Copyright (C) 2004-2006 Red Hat, Inc., Ingo Molnar in its header, alongside Copyright (C) 2005-2006 Timesys Corp., Thomas Gleixner.

Rather than landing as one giant merge, the work was upstreamed incrementally over two decades: high-resolution timers (hrtimers), generic interrupt handling and threaded IRQs, the rt_mutex priority-inheritance infrastructure, the raw_spinlock_t / spinlock_t split, lockdep, and the futex PI work were all merged piecemeal — each useful to the mainline kernel in its own right, which is precisely why a twenty-year out-of-tree patch set did not bit-rot. The Linux Foundation’s Real-Time Linux project (formerly the RTL Collaborative Project) coordinated and funded the long tail.

Dating the endgame by existence-checking the tree

The popular summary is “PREEMPT_RT merged in 6.12,” which is true but hides a five-year interval that matters. Fetching kernel/Kconfig.preempt at successive tags and grepping for ^config PREEMPT_RT$ gives a hard boundary:

Tag	config PREEMPT_RT in kernel/Kconfig.preempt	File size	Any arch selects ARCH_SUPPORTS_RT
v4.19	absent	2,208 B	no
v5.0 – v5.2	absent	2,209–2,249 B	no
v5.3	present	3,033 B	no
v5.4, v5.10	present	~3,032 B	no
v5.15	present	4,517 B	no
v6.1 – v6.11	present	4,885 B	no
v6.12	present	5,989 B	yes — x86, arm64, riscv
v6.18	present (moved outside the choice)	7,127 B	yes

Table 5 — the two-step merge, measured rather than remembered. Sizes are the byte counts returned by raw.githubusercontent.com for each tag; every row was fetched in this research pass. The insight: the option symbol appeared at v5.3 (tarball published 2019-09-16) and sat there un-selectable for thirteen releases, because ARCH_SUPPORTS_RT had no select anywhere in the tree. It became genuinely buildable at v6.12 (tarball published 2024-11-18). Both halves of that sentence are needed: “PREEMPT_RT has been in Kconfig since 5.3” and “you could not build an RT kernel until 6.12” are simultaneously true.

The interval is independently corroborated by a reader comment on the 5.15-era LWN brief, posted 2021-08-31: “This preemption model is not available on any of the archs yet, right? I wanted to try it, but it’s not shown in the menuconfig and it seems no arch sets ARCH_SUPPORTS_RT=y” (LWN 867919) — matching the table’s v5.15 row exactly.

timeline
    title PREEMPT_RT from patch set to selectable mainline option
    2004 : "-rt patch set begins (Molnar, Gleixner)" : "rtmutex.c copyright starts 2004"
    2005-2018 : "Piecemeal upstreaming" : "hrtimers, genirq + threaded IRQs, rt_mutex PI, raw_spinlock_t/spinlock_t split, futex PI, lockdep"
    2019 (v5.3) : "config PREEMPT_RT appears in Kconfig.preempt" : "un-selectable: no arch sets ARCH_SUPPORTS_RT"
    2021 (v5.15) : "Realtime preemption LOCKING CORE merged" : "sleeping spinlocks land in mainline" : "still not selectable"
    2021-2024 : "Gleixner's remaining-blockers list" : "memory management, printk, fs/namespace trylock loop"
    2024 (v6.12) : "x86 / arm64 / riscv select ARCH_SUPPORTS_RT" : "threaded printk merged - the last blocker" : "CONFIG_PREEMPT_RT is now buildable"
    2025 (v6.13) : "PREEMPT_LAZY added as a fifth model" : "a separate line of work, not part of RT"

Twenty years in six steps. What it shows: the two milestones people conflate — the locking core merging (5.15, 2021) and RT becoming selectable (6.12, 2024) — are three years apart, with a documented list of blockers in between. The insight to take: “merged” is not one event for a change this large; ask which piece and which release, because the answer is different for the locking core, the config symbol, and the arch gate.

The final blockers

Asked in August 2021 what was left, Thomas Gleixner answered on the LWN comment thread with a three-item list: “memory management: The bulk of those changes sits in the akpm tree… printk: Another batch is destined for 5.15-rc1, but we still have a few steps to overcome there… fs/namespace: The last trylock loop which causes headache.” (LWN 867919)

The one that took three more years was printk(). The console-output path must work from every context — including NMI, the scheduler core, and inside raw_spinlock_t sections — yet the legacy implementation took locks and drove console hardware in ways incompatible with full preemptibility. Reworking it into a lockless-ringbuffer plus threaded-console design was a multi-year subproject of its own. Once it landed, Sebastian Andrzej Siewior posted the patch series that flipped the arch gates, quoted in full by Corbet: “With the printk bits merged, PREEMPT_RT could be enabled on X86, ARM64 and Risc-V. These three architectures merged required changes over the years leaving me in a position where I have no essential changes in the queue that would affect them.” (LWN 989212, 2024-09-06 — one merge window ahead of 6.12; 6.12 merge window, LWN 990750; 6.12 released, LWN 997958).

What a stock kernel actually reports — a worked example

Everything above is easier to hold onto after looking at a real, non-RT machine and naming exactly which of these mechanisms are and are not in play. The following was measured on 2026-09-11 on a stock Fedora 44 desktop, 32 logical CPUs, running distribution kernel 7.1.13-200.fc44.x86_64.

Step 1 — what does the version string say?

$ uname -v
#1 SMP PREEMPT_DYNAMIC Wed Sep  2 13:58:38 UTC 2026

PREEMPT_DYNAMIC, not PREEMPT_RT. This is not a real-time kernel, and nothing below should be read as an RT measurement. That is the point of showing it: a distribution kernel is the baseline RT is measured against.

Step 2 — what is in the build config?

$ grep -E '^(CONFIG_PREEMPT|# CONFIG_PREEMPT)' /boot/config-7.1.13-200.fc44.x86_64
CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT_LAZY=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_RT is not set

Read against Table 1 and Table 2: the compile-time member of the choice selected here is PREEMPT_LAZY — the fifth model that did not exist at v6.12 — with PREEMPT_DYNAMIC=y layered on top so the effective model can still be overridden at boot with preempt=. CONFIG_PREEMPT_RT is not set, so spinlock_t on this machine is a plain spinning lock that disables preemption, exactly as Table 3’s non-RT column says. CONFIG_RT_MUTEXES=y is set, but that only means rt_mutex is compiled in for the futex PI and PI_FUTEX paths — it is not the pervasive lock substitution.

Uncertain

Verify: the runtime preemption mode this machine is actually in. Reason: with CONFIG_PREEMPT_DYNAMIC=y the live mode is readable at /sys/kernel/debug/sched/preempt, but /sys/kernel/debug is mode 0700 root-owned and this session has no passwordless sudo, so I could not read it (ls: cannot open directory '/sys/kernel/debug/': Permission denied). The kernel command line (cat /proc/cmdline) carries no preempt= parameter, so by preempt_dynamic_init() in kernel/sched/core.c the mode should fall through to the compile-time default, lazy. To resolve: sudo cat /sys/kernel/debug/sched/preempt — the active mode is shown in brackets. uncertain

Step 3 — how many interrupts are threaded?

$ grep -cE '^\s*[0-9]+:' /proc/interrupts     # numbered IRQ lines
66
$ ps -eo pid,class,rtprio,comm | grep 'irq/'
    227 FF      50 irq/9-acpi
    239 FF      50 irq/26-AMD-Vi
    246 FF      50 irq/33-pciehp
    247 FF      50 irq/34-pciehp
    382 FF      50 irq/27-ACPI:Event
    ... (10 threads total)

Ten threaded IRQs out of sixty-six lines — and note class=FF, i.e. SCHED_FIFO, at real-time priority 50. This is the detail that a purely theoretical treatment gets wrong in both directions. A stock kernel is not “zero threaded IRQs”: any driver that calls request_threaded_irq() with a real thread_fn gets a thread regardless of force_irqthreads(), and ACPI, the AMD IOMMU and PCIe hotplug all do. What PREEMPT_RT changes is that force_irqthreads() becomes a compile-time true, so every handler that has not set IRQF_NO_THREAD / IRQF_PERCPU / IRQF_ONESHOT is threaded too — that ten would become most of the sixty-six.

Step 4 — where do softirqs run?

$ ps -eo pid,class,rtprio,comm | grep ksoftirq | head -3
     15 TS       - ksoftirqd/0
     26 TS       - ksoftirqd/1
     32 TS       - ksoftirqd/2

class=TS — SCHED_OTHER, no real-time priority. On this kernel ksoftirqd is only the overflow path: softirqs normally run inline on interrupt exit, in a non-preemptible context, and are pushed to ksoftirqd only when they overrun. Under PREEMPT_RT the accounting described in §4 above moves that work into preemptible task context as a matter of course.

What this does and does not tell you. It tells you the shape of the baseline: preemption is enabled, most interrupt work is unthreaded, softirqs are non-preemptible, and spinlock_t sections are preempt-off windows. It tells you nothing about numbers — measuring worst-case latency requires cyclictest from the rt-tests package (absent on this box) under a load generator, on a tuned, isolated machine, for hours.

Uncertain

Verify: concrete worst-case latency figures for RT versus non-RT kernels. Reason: the authoritative public dataset is the OSADL QA Farm, which runs cyclictest continuously across a rack of boards and publishes per-system plots; I fetched osadl.org/Latency-plots.latency-plots.0.html (HTTP 200, 68 KB) and an individual system page (HTTP 200, 63 KB) and confirmed the methodology statement — “Data to construct the above plot have been generated using the RT test utility cyclictest” — but the numeric series are rendered client-side by JavaScript with no plain-text or JSON endpoint I could find, so I have not extracted specific microsecond figures and deliberately quote none. To resolve: open an OSADL per-system latency plot in a browser, or run cyclictest -m -S -p90 -i200 -h400 -D 4h locally on both kernel flavours. uncertain

RT-specific failure modes: code that is correct on a stock kernel and broken under RT

Because PREEMPT_RT changes what existing primitives mean rather than adding new ones, the characteristic bug is old code that compiles fine and is silently wrong. Documentation/locking/locktypes.rst at v6.12 enumerates the patterns; these are the four worth memorising.

1. Disabling interrupts and then taking a spinlock_t. On a stock kernel these two are interchangeable:

local_irq_disable();
spin_lock(&lock);          /* equivalent to spin_lock_irq(&lock) on non-RT */

Under RT they are not, because “RT-mutex requires a fully preemptible context” — you cannot block with interrupts off. The documented fix is to use spin_lock_irq() / spin_lock_irqsave() (which, recall from Table 3, do not actually disable interrupts under RT) or, where the interrupt-disabling genuinely must be separable from the locking, a local_lock.

2. Taking a spinlock_t inside get_cpu_ptr(). This is the per-CPU idiom:

struct foo *p = get_cpu_ptr(&var1);   /* implicitly preempt_disable() */
spin_lock(&p->lock);                  /* BREAKS on RT: sleeping lock in atomic context */

The fix in the documentation is to pin explicitly instead of implicitly:

migrate_disable();
p = this_cpu_ptr(&var1);
spin_lock(&p->lock);

migrate_disable() keeps the per-CPU pointer valid while leaving the task preemptible — which is the same trick __rt_spin_lock() itself uses.

3. Assuming migrate_disable() gives you mutual exclusion. It does not. The documentation is explicit that migrate_disable() “does not protect against reentrancy from a preempting task” — on a stock kernel preempt_disable() happened to give both pinning and exclusion, and code that conflated them breaks. The correct substitution when exclusion is needed is a real local_lock.

4. Using two different local_locks to protect one thing. On a stock kernel local_lock_irqsave(&lock_a) and local_lock_irqsave(&lock_b) both compile down to local_irq_save(), so two call sites with different locks still serialise against each other. Under RT they are two distinct per-CPU spinlock_ts and serialise nothing, and the lockdep_assert_irqs_disabled() in the shared callee will fire. Each local_lock has “a specific protection scope”; sharing data means sharing the lock.

5. rwlock_t reader starvation. A genuinely RT-only hazard rather than a porting bug: “Because an rwlock_t writer cannot grant its priority to multiple readers, a preempted low-priority reader will continue holding its lock, thus starving even high-priority writers.” PI is asymmetric here — readers boost a blocked writer, but a writer cannot boost an unbounded set of readers. Read-heavy rwlock_t use is therefore a latency risk specific to RT kernels.

The general lesson: an RT kernel does not just run existing code more preemptibly, it re-types the locking vocabulary, and drivers written against the non-RT meanings need auditing. This is a large part of why the merge took twenty years and why ARCH_SUPPORTS_RT is a per-architecture opt-in rather than a global default.

Where PREEMPT_RT is used

Real-time Linux is deployed where a missed deadline is a failure, not just a hiccup:

• Professional and pro-sumer audio — low-latency audio (JACK, Ardour) needs bounded buffer-fill latency to avoid xruns (audible glitches). RT kernels are a staple of audio-focused distributions.
• Industrial control and robotics — motion control, PLCs, and robot arms run control loops at fixed periods (e.g. 1 kHz); jitter translates to mechanical error. Frameworks like ROS 2’s real-time work and the OSADL (Open Source Automation Development Lab) QA farm target RT Linux. OSADL publishes continuous worst-case-latency measurements across many boards.
• Telecom and high-frequency / low-latency systems — packet-processing and trading systems that need bounded tail latency.
• Measurement and test equipment, aerospace ground systems — anywhere deterministic response is contractually required.

RT is usually combined with the isolation techniques in this MOC — CPU Isolation isolcpus and nohz_full, CPU Affinity and sched_setaffinity, and Housekeeping CPUs and Tickless Isolation — to clear a CPU of all other work and remove even the timer tick, plus a real-time scheduling policy (Real-Time Scheduling SCHED_FIFO and SCHED_RR or SCHED_DEADLINE and Earliest Deadline First) for the critical task. PREEMPT_RT bounds kernel latency; the RT scheduling class bounds which task runs; isolation removes interference. All three are usually needed together for the lowest jitter.

Common misunderstandings

• “PREEMPT_RT is still an out-of-tree patch.” Stale. The last pieces merged for 6.12 (2024); CONFIG_PREEMPT_RT is a mainline option on x86-64/arm64/riscv. Distributions still ship dedicated RT kernel flavours, but the code is upstream.
• “RT makes Linux faster.” No — it usually lowers throughput. Sleeping locks, threaded IRQs, and pervasive PI add overhead. RT trades average performance for bounded worst-case latency. If you do not need determinism, an RT kernel is a net loss.
• “spinlock_t always spins.” Not under RT, where it is an rt_mutex that sleeps. Only raw_spinlock_t always spins. Code that requires true spinning / preempt-off (touching hardware registers, the scheduler) must use raw_spinlock_t explicitly.
• “RT eliminates all non-preemptible code.” No — entry code, the scheduler, and low-level IRQ handling stay non-preemptible (via raw_spinlock_t). RT minimizes and bounds the non-preemptible set; it does not erase it.
• “PREEMPT_RT is the same as SCHED_FIFO.” Different layers. SCHED_FIFO/SCHED_RR are scheduling policies (which task runs); PREEMPT_RT is a kernel build that makes the kernel preemptible enough for those policies to actually meet deadlines. You typically use both.
• “spinlock_t under RT always sleeps on contention.” It spins first. rtmutex_spin_on_owner() busy-waits with cpu_relax() while the owner is running on another CPU and this task is the top waiter and need_resched() is false; only when one of those stops holding does it call schedule_rtlock(). The 2008 “adaptive spinlocks” work exists precisely so short critical sections do not pay a context-switch pair.
• “PREEMPT_RT was merged in 6.12, so CONFIG_PREEMPT_RT is new.” The symbol has been in kernel/Kconfig.preempt since v5.3 (2019) and the locking core merged in v5.15 (2021). What changed at 6.12 is that x86, arm64 and RISC-V began to select ARCH_SUPPORTS_RT, which is what made the option selectable. See Table 5.
• “PREEMPT_LAZY is part of the RT work.” It is a separate line of work on the non-RT models, and it did not exist at v6.12 — config PREEMPT_LAZY first appears in kernel/Kconfig.preempt at v6.13. Under RT, real-time tasks are never treated lazily.
• “spin_lock_bh() stops working under RT.” It does not. The _bh suffix still disables softirq handlers — through a per-CPU local_lock (softirq_ctrl.lock) instead of a preempt_count bump. It is the _irq / _irqsave suffixes that change meaning, becoming no-ops with respect to the hardware interrupt-disabled state.
• “An RT kernel means every interrupt is threaded.” Handlers marked IRQF_NO_THREAD, IRQF_PERCPU or IRQF_ONESHOT are exempt — irq_setup_forced_threading() returns early for all three. The timer tick and inter-processor interrupts stay in hard-IRQ context on an RT kernel, as they must.
• “A stock kernel has no IRQ threads.” Also false, in the other direction: any driver that registers a thread_fn via request_threaded_irq() gets one regardless of the preemption model. The stock Fedora box measured above has ten, all SCHED_FIFO priority 50, with force_irqthreads() off.

See Also

• Kernel Preemption Models — PREEMPT_RT is the fourth member of the preemption choice at v6.12, alongside none/voluntary/full
• Lazy Preemption and PREEMPT_LAZY — the fifth model, added at v6.13 (verified absent at v6.12); a separate line of work from RT
• Kernel Spinlocks — the spinlock_t / raw_spinlock_t primitives this note re-types
• Kernel Mutexes — the plain mutex, re-implemented on rt_mutex under RT so it carries priority inheritance
• Threaded Interrupt Handlers — request_threaded_irq(), IRQF_NO_THREAD and the irq/NN kthreads, in depth
• Context Switch Mechanics switch_to and switch_mm — what actually happens at each of the extra preemption points RT creates
• The EEVDF Scheduler — the fair-class scheduler that runs alongside the RT classes on the same kernel
• Real-Time Scheduling SCHED_FIFO and SCHED_RR — the scheduling policies an RT kernel is built to serve
• Real-Time Priorities and Priority Inversion — the priority-inheritance problem that rt_mutex solves pervasively under RT
• SCHED_DEADLINE and Earliest Deadline First — EDF/CBS deadline scheduling, often paired with an RT kernel
• The need_resched Flag and Preemption Points — the flag and preemption points that RT makes reachable from almost everywhere
• CPU Isolation isolcpus and nohz_full, Housekeeping CPUs and Tickless Isolation — the isolation usually paired with RT for lowest jitter
• Linux Kernel Synchronization MOC — the sleeping-spinlock / rt_mutex / raw_spinlock_t machinery in depth
• Linux Interrupts and Deferred Work MOC — threaded IRQs and softirqs, which RT makes the default
• Linux Process Scheduling MOC — parent map