Real-Time Scheduling SCHED_FIFO and SCHED_RR
Linux implements the two POSIX real-time scheduling policies —
SCHED_FIFOandSCHED_RR— as a single scheduling class,rt_sched_class, that sits above the fair (SCHED_OTHER) class in the kernel’s strict class hierarchy. A runnable real-time task therefore always preempts any ordinary task, no matter how “important” the fair task believes it is. Real-time tasks carry a static priority from 1 (low) to 99 (high) (sched(7)); the scheduler always runs the highest-priority runnable real-time task, and only chooses among tasks of equal priority using the policy:SCHED_FIFOlets a task run until it blocks, yields, or is preempted by something of higher priority (no time slice at all), whereasSCHED_RRrotates among equal-priority tasks everysched_rr_timeslice(100 ms by default on a 1000 Hz kernel). Because nothing below the real-time class runs while an RT task is runnable, a runaway, non-blocking RT task can monopolise a CPU forever — the hazard that Linux has spent seventeen years building safety valves against. The classic valve, RT throttling, capped real-time tasks atsched_rt_runtime_us(950,000 µs) out of everysched_rt_period_us(1,000,000 µs); as of v6.12 that valve is gone unlessCONFIG_RT_GROUP_SCHED=y, replaced by a deferred deadline server that hands the fair class a 50 ms reservation only when fair tasks are actually being starved (commit5f6bd380c7bd, “sched/rt: Remove default bandwidth control”). Both mechanisms are traced below. Verified against Linux 6.12 LTS source (kernel/sched/rt.c,kernel/sched/deadline.c,include/linux/sched/prio.h) and measured on a stock Fedora 44 box (kernel 7.1.13,CONFIG_RT_GROUP_SCHEDunset) on 2026-09-12.
These are fixed-priority policies, the classical real-time model: the programmer assigns priorities, and the scheduler mechanically honours them. There is no notion of fairness, lag, or virtual runtime here — those belong to EEVDF. For deadline-driven work where you can state a (runtime, period, deadline) triple and want the kernel to compute schedulability, see SCHED_DEADLINE, which sits above the real-time class.
What this note owns, and what it does not. This note owns the two POSIX fixed-priority policies as policies: the selection rule, the four priority number spaces and their conversions, the rt_prio_array data structure, the SCHED_FIFO/SCHED_RR queue disciplines, and the default-bandwidth safety valve (RT throttling and its v6.12 deadline-server replacement) as it applies to an ordinary system with no real-time cgroups. It deliberately does not own: the PREEMPT_RT patch set, rtmutex conversion, or the ARCH_SUPPORTS_RT story — that is The PREEMPT_RT Real-Time Kernel; fair-class policy and EEVDF’s lag arithmetic — The EEVDF Scheduler; the mechanics of the switch itself — Context Switch Mechanics switch_to and switch_mm; the stop class and migration threads — Migration the Stop Class and the Migration Thread; the per-cgroup cpu.rt_runtime_us interface — The cgroup v2 CPU Controller and Real-Time Throttling and the RT Bandwidth Limit; why a SCHED_FIFO handler ends up in a kthread at all — Threaded Interrupt Handlers; and the struct rq envelope that holds the rt_rq — Per-CPU Run Queues and struct rq.
Mental Model
Think of the real-time class as 99 numbered shelves stacked from 1 at the bottom to 99 at the top. Each shelf holds a FIFO queue of runnable tasks at that priority. To pick the next task, the scheduler scans from the top shelf downward and takes the first task on the first non-empty shelf. A task only ever runs if every shelf above it is empty. SCHED_FIFO and SCHED_RR tasks share the same shelves and the same selection rule — they differ only in when a running task gives up its shelf position: a SCHED_FIFO task holds its spot until it voluntarily blocks or is preempted; a SCHED_RR task is sent to the back of its own shelf when its time quantum expires, letting an equal-priority sibling run.
flowchart TD PICK["pick_next_task_rt()<br/>on this CPU's rt_rq"] --> SCAN["sched_find_first_bit(array.bitmap)<br/>find highest set priority"] SCAN --> SHELF99["prio 99 queue<br/>(FIFO list)"] SCAN --> SHELF50["prio 50 queue"] SCAN --> SHELF1["prio 1 queue"] SHELF99 -->|non-empty wins| RUN["run head of highest<br/>non-empty queue"] SHELF50 -.->|only if 99 empty| RUN SHELF1 -.->|only if all above empty| RUN RUN --> FIFO{"policy?"} FIFO -->|SCHED_FIFO| HOLD["runs until block / yield /<br/>higher-prio preempt"] FIFO -->|SCHED_RR| TICK["on tick: decrement time_slice;<br/>at 0 → requeue at tail,<br/>reset to sched_rr_timeslice"]
The real-time run queue and its selection rule. What it shows: a per-priority array of FIFO queues with a companion bitmap; sched_find_first_bit reads the bitmap to locate the highest populated priority in O(1), and the head of that queue runs. The insight to take: priority is absolute — priority 50 never runs while priority 51 has a runnable task — and the only difference between the two policies is the bottom of the diagram: SCHED_FIFO keeps the shelf position indefinitely, SCHED_RR rotates within a shelf on a tick-driven quantum.
The Class Cascade — Where rt_sched_class Actually Sits
“RT beats fair” is usually stated as a rule. It is not a rule; it is a linker script. Each scheduling class is emitted into its own named ELF section by the DEFINE_SCHED_CLASS(name) macro (kernel/sched/sched.h:2499), and include/asm-generic/vmlinux.lds.h:129–138 concatenates those sections into one contiguous array in a hard-coded order, bracketed by the symbols __sched_class_highest and __sched_class_lowest:
#define SCHED_DATA \
STRUCT_ALIGN(); \
__sched_class_highest = .; \
*(__stop_sched_class) \
*(__dl_sched_class) \
*(__rt_sched_class) \
*(__fair_sched_class) \
*(__ext_sched_class) \
*(__idle_sched_class) \
__sched_class_lowest = .;for_each_class(class) (sched.h:2546) is therefore a plain pointer walk over an array, not a linked-list traversal — which is why the class order is a build-time constant with zero runtime cost, and why sched_class_above(a, b) is just a pointer comparison. Note where __ext_sched_class landed: sched_ext sits below fair, not above it.
| # | Class (ELF section) | Policies it serves | Per-CPU sub-runqueue | Internal prio of its tasks | Ordering rule inside the class |
|---|---|---|---|---|---|
| 1 | stop_sched_class (__stop_sched_class) | none — kernel-internal only; rq->stop, i.e. migration/N and kstopmachine work | none (a single task pointer) | 0 — set by sched_set_stop_task() via a cosmetic SCHED_FIFO/99 sched_setscheduler_nocheck() | one task per CPU; no queue |
| 2 | dl_sched_class (__dl_sched_class) | SCHED_DEADLINE (6), plus the servers rq->fair_server and rq->dl_server | struct dl_rq dl — rb_root_cached root keyed by absolute deadline | MAX_DL_PRIO - 1 = −1 | earliest absolute deadline first (EDF), with admission control |
| 3 | rt_sched_class (__rt_sched_class) | SCHED_FIFO (1), SCHED_RR (2) | struct rt_rq rt — rt_prio_array of 100 list heads + bitmap | MAX_RT_PRIO - 1 - rt_priority = 0…98 | strict priority, then FIFO order (RR rotates on quantum) |
| 4 | fair_sched_class (__fair_sched_class) | SCHED_NORMAL/SCHED_OTHER (0), SCHED_BATCH (3), SCHED_IDLE (5) | struct cfs_rq cfs — rb_root_cached tasks_timeline keyed by virtual deadline | NICE_TO_PRIO(nice) = 100…139 | EEVDF — earliest eligible virtual deadline first |
| 5 | ext_sched_class (__ext_sched_class) | SCHED_EXT (7), under CONFIG_SCHED_CLASS_EXT | struct scx_rq scx — BPF-managed dispatch queues | shares the fair range (prio 120 by default) | whatever the loaded BPF scheduler decides |
| 6 | idle_sched_class (__idle_sched_class) | none — rq->idle, the per-CPU swapper task | none (a single task pointer) | MAX_PRIO = 140 conceptually; never compared | one task per CPU; runs only when everything else is empty |
Policy numbers come from include/uapi/linux/sched.h:114–121. Note the hole at 4: the header still carries the comment /* SCHED_ISO: reserved but not implemented yet */ — an “isochronous” policy proposed years ago and never merged, so sched_setscheduler(2) with policy 4 returns EINVAL.
flowchart TD SCHED["__schedule()<br/>picks via __pick_next_task()"] --> FAST{"fast path?<br/>prev class not above fair<br/>AND rq->nr_running == rq->cfs.h_nr_running"} FAST -->|yes| FAIRONLY["pick_next_task_fair()<br/>skip the whole walk"] FAST -->|no| WALK["for_each_active_class(class)<br/>walk the linker-ordered array"] WALK --> STOP["__stop_sched_class<br/>migration/N, kstopmachine"] STOP -->|"pick_task returns NULL"| DL["__dl_sched_class<br/>SCHED_DEADLINE + rq->fair_server"] DL -->|NULL| RT["__rt_sched_class<br/>SCHED_FIFO / SCHED_RR"] RT -->|NULL| FAIR["__fair_sched_class<br/>SCHED_NORMAL / BATCH / IDLE (EEVDF)"] FAIR -->|NULL| EXT["__ext_sched_class<br/>SCHED_EXT (BPF), if enabled"] EXT -->|NULL| IDLE["__idle_sched_class<br/>the per-CPU swapper task"] IDLE -->|NULL| BUG["BUG()<br/>idle must always be runnable"] STOP -.->|"returns a task"| RUN[["run it: put_prev_set_next_task()"]] DL -.-> RUN RT -.-> RUN FAIR -.-> RUN EXT -.-> RUN IDLE -.-> RUN
The class cascade as __pick_next_task() walks it (kernel/sched/core.c:5952). What it shows: the real code has two paths. A fast path short-circuits the entire walk when prev was not in a class above fair and every runnable task on this CPU is a fair task (rq->nr_running == rq->cfs.h_nr_running) — the overwhelmingly common case, which is why the class hierarchy costs almost nothing on an ordinary desktop. The slow path walks the array top-down and returns the first task any class offers. The insight to take: rt_sched_class is third, not first. A SCHED_FIFO task at priority 99 is still preempted by any SCHED_DEADLINE task, by rq->fair_server when it wakes, and by the stop class — which is exactly how the kernel retains the ability to migrate and to throttle a runaway RT task.
Uncertain
Verify: whether
for_each_active_class()(as opposed tofor_each_class()) skips theextandfairclasses wholesale when a BPF scheduler has taken over all fair tasks. Reason: thefor_active_class_rangemacro atsched.h:2553was read, but thescx_switched_all()predicate it depends on was not traced throughkernel/sched/ext.c. To resolve: readscx_switched_all()and thefor_active_class_rangedefinition at v6.12 in full. This does not affect any claim above about the order of the classes. uncertain
How the Real-Time Class Works
The per-CPU rt_rq and the priority array
Each CPU’s run queue (struct rq) embeds an rt_rq. The heart of the rt_rq is a struct rt_prio_array named active, defined in kernel/sched/sched.h (Linux 6.12) as:
struct rt_prio_array {
DECLARE_BITMAP(bitmap, MAX_RT_PRIO+1); /* include 1 bit for delimiter */
struct list_head queue[MAX_RT_PRIO];
};MAX_RT_PRIO is 100 (include/linux/sched/prio.h, Linux 6.12), so there are 100 FIFO queues — one per internal real-time priority level 0…99 — and a bitmap with one bit per queue plus a trailing “delimiter” bit. When a real-time task becomes runnable it is appended to queue[prio] and the corresponding bit is set; when the queue empties the bit is cleared. Selecting the next task is then just sched_find_first_bit(array->bitmap) — an O(1) scan for the lowest set bit (which, because of the priority inversion in the internal numbering described below, is the highest user-visible priority) — followed by taking the head of that queue (rt.c, Linux 6.12). This is what makes fixed-priority scheduling constant-time regardless of how many real-time tasks are runnable, a property classical Unix schedulers lacked.
init_rt_rq() initialises every queue empty, clears all bits, and then sets the delimiter bit at index MAX_RT_PRIO so that sched_find_first_bit always terminates even when no real-time task is runnable; it also seeds highest_prio.curr = MAX_RT_PRIO-1 (i.e. 99, the lowest-priority sentinel) (rt.c init_rt_rq, Linux 6.12).
Internal vs. user-visible priority numbering — a deliberate inversion
There are two priority number spaces and they run in opposite directions, which is a perennial source of confusion. The userspace API (sched_priority in struct sched_param) uses 1 = lowest, 99 = highest, as documented in sched(7): “Processes scheduled under one of the real-time policies … have a sched_priority value in the range 1 (low) to 99 (high)” (sched(7)). Internally the kernel stores a task_struct.prio where smaller numbers are more urgent so that the same comparison logic works across all classes. The conversion lives in __normal_prio() (kernel/sched/syscalls.c, Linux 6.12):
prio = MAX_RT_PRIO - 1 - rt_prio; /* 100 - 1 - rt_priority */So a user priority of 99 becomes internal prio = 0 (most urgent), and user priority 1 becomes internal prio = 98. The fair (SCHED_OTHER) tasks occupy internal priorities MAX_RT_PRIO…MAX_PRIO-1, i.e. 100…139 — strictly numerically larger, hence strictly less urgent, than every real-time priority. That is the entire mechanism by which “real-time always beats fair”: it falls straight out of the number line (prio.h, Linux 6.12). The sched_get_priority_min(2)/sched_get_priority_max(2) calls return 1 and 99 for both SCHED_FIFO and SCHED_RR (sched_get_priority_max(2)).
There are in fact four numbering spaces in play at once, and confusing any two of them is the single most common source of misreadings of Linux scheduling. They are: the nice value (fair tasks only), rt_priority (RT tasks only), the internal task_struct.prio, and the number /proc/<pid>/stat prints in field 18. The last one is task_prio() (kernel/sched/syscalls.c:191–194), which is simply
int task_prio(const struct task_struct *p)
{
return p->prio - MAX_RT_PRIO; /* prio - 100 */
}so field 18 is the internal priority shifted so that nice 0 reads as 20. Laying all four side by side:
| Task | nice (−20…19) | rt_priority (0…99) | internal prio (−1…139) | /proc/<pid>/stat field 18 (prio − 100) | Which class picks it |
|---|---|---|---|---|---|
SCHED_DEADLINE task | ignored | 0 | −1 (MAX_DL_PRIO - 1) | −101 | dl_sched_class |
stop-class kthread (migration/N) | ignored | 99 (cosmetic) | 0 | −100 | stop_sched_class |
SCHED_FIFO/RR at user prio 99 | ignored | 99 | 0 | −100 | rt_sched_class |
SCHED_FIFO/RR at user prio 50 | ignored | 50 | 49 | −51 | rt_sched_class |
SCHED_FIFO/RR at user prio 1 | ignored | 1 | 98 | −2 | rt_sched_class |
SCHED_OTHER at nice −20 | −20 | 0 | 100 (MAX_RT_PRIO) | 0 | fair_sched_class |
SCHED_OTHER at nice 0 | 0 | 0 | 120 (DEFAULT_PRIO) | 20 | fair_sched_class |
SCHED_OTHER at nice +19 | +19 | 0 | 139 (MAX_PRIO - 1) | 39 | fair_sched_class |
SCHED_BATCH / SCHED_IDLE | as set | 0 | 100…139 (same range) | 0…39 | fair_sched_class |
Read three things off that table. First, prio runs backwards relative to rt_priority: the most urgent task has the smallest number, because __normal_prio() computes MAX_RT_PRIO - 1 - rt_prio (kernel/sched/syscalls.c:19–31). Second, a stop-class kthread and an rt_priority 99 task share internal prio 0 — the numbers do not separate them; the class array order does. Third, the constants are all derived from two numbers, both in include/linux/sched/prio.h: MAX_RT_PRIO is 100 and NICE_WIDTH is 40, whence MAX_PRIO = 140 and DEFAULT_PRIO = 120. The nice range is not mapped onto RT priorities at all — the two spaces are disjoint halves of one number line.
flowchart LR subgraph DLL["SCHED_DEADLINE"] D1["no static priority<br/>internal prio = MAX_DL_PRIO - 1 = -1"] end subgraph STOPL["stop class (kernel only)"] S1["sched_priority 99 cosmetically<br/>internal prio = 0"] end subgraph RTL["SCHED_FIFO / SCHED_RR"] R99["rt_priority 99 --> prio 0"] R50["rt_priority 50 --> prio 49"] R1["rt_priority 1 --> prio 98"] end subgraph FL["SCHED_NORMAL / BATCH / IDLE"] F20["nice -20 --> prio 100"] F0["nice 0 --> prio 120 (DEFAULT_PRIO)"] F19["nice +19 --> prio 139 (MAX_PRIO-1)"] end DLL --> STOPL --> RTL --> FL
The four priority number spaces, aligned left (most urgent) to right (least urgent). What it shows: one internal prio number line running from −1 to 139, cut into three disjoint regions — deadline at −1, real-time at 0…98, fair at 100…139 — with the two userspace-facing scales (rt_priority counting up toward urgency, nice counting up toward un-urgency) mapped onto it. The insight to take: the arrows inside the RT box point the opposite way to the arrows inside the fair box. That single inversion is why renice cannot touch an RT task (it writes static_prio, which __normal_prio() ignores for RT policies — see set_user_nice() at kernel/sched/syscalls.c:65–90, which explicitly stores the nice value and returns without recomputing p->prio) and why internal prio 49 means “quite urgent” while internal prio 100 means “least urgent fair task there is”.
Priority inheritance is the one thing that can move a task across the RT/fair boundary without a syscall. effective_prio() (syscalls.c:52–62) returns p->normal_prio for ordinary tasks but leaves p->prio untouched if the task is currently RT-boosted, and rt_mutex_setprio() / __rt_effective_prio() (kernel/sched/core.c:7095) is what pushes a nice-0 lock holder from prio 120 down into the RT range so it can release the lock. rt_task(p) in include/linux/sched/rt.h:23 therefore returns true for a PI-boosted fair task — the header says so explicitly — and code that wants “is this really an RT-policy task” must use rt_or_dl_task_policy() instead. See Real-Time Priorities and Priority Inversion.
SCHED_FIFO: run until you stop yourself
SCHED_FIFO is, in the words of sched(7), “a simple scheduling algorithm without time slicing.” A SCHED_FIFO thread “runs until either it is blocked by an I/O request, it is preempted by a higher priority thread, or it calls sched_yield(2)” (sched(7)). There is no quantum and no preemption by equal-priority peers — a SCHED_FIFO task at priority 50 will run to the exclusion of every other priority-50 task until it blocks. The list discipline is precisely specified:
- When a
SCHED_FIFOtask is preempted by a higher-priority task, “it will stay at the head of the list for its priority and will resume execution as soon as all threads of higher priority are blocked again” (sched(7)). It does not lose its place. - When a blocked
SCHED_FIFOtask becomes runnable again, “it will be inserted at the end of the list for its priority” (sched(7)). - When a running task’s priority is changed via
sched_setscheduler(2)/sched_setparam(2): raising it places the task “at the end of the list for its new priority”; an unchanged value leaves “its position in the run list … unchanged”; lowering it places it “at the front of the list for its new priority” (sched(7)). The asymmetry (front on lower, back on higher) exists so a task that raises its own priority cannot use the change to leapfrog peers it just joined.
sched_yield(2) under SCHED_FIFO moves the caller to the tail of its priority list, giving equal-priority siblings a turn — the one cooperative knob a SCHED_FIFO programmer has.
SCHED_RR: round-robin within a priority
SCHED_RR is “a simple enhancement of SCHED_FIFO” — identical in every respect except that “each thread is allowed to run only for a maximum time quantum” (sched(7)). When a SCHED_RR task exhausts its quantum it is moved to the tail of its priority’s queue and the next equal-priority task runs; tasks of different priority are still strictly ordered. The default quantum is the kernel variable sched_rr_timeslice, initialised from the macro RR_TIMESLICE in include/linux/sched/rt.h (Linux 6.12):
#define RR_TIMESLICE (100 * HZ / 1000) /* 100ms */With the common server tick rate HZ = 1000 this evaluates to 100 jiffies, i.e. 100 ms; with HZ = 250 it is 25 jiffies, still 100 ms of wall-clock. The value is tunable at runtime through /proc/sys/kernel/sched_rr_timeslice_ms (exposed via the sysctl_sched_rr_timeslice table in rt.c). Userspace can query the quantum for a specific task with sched_rr_get_interval(2), which “writes into the timespec structure … the round-robin time quantum for the process” (sched_rr_get_interval(2)).
Mechanically, the per-tick accounting happens in task_tick_rt() (rt.c): for a SCHED_RR task it decrements p->rt.time_slice; while the slice remains positive nothing happens, but on reaching zero it resets time_slice back to sched_rr_timeslice and, only if the task has a queue-mate at the same priority, requeues it at the tail and sets the reschedule flag. A lone SCHED_RR task at its priority therefore behaves exactly like SCHED_FIFO — there is no one to round-robin with, so the quantum expiry is a no-op beyond the reset.
The guard is literally rt_se->run_list.prev != rt_se->run_list.next (kernel/sched/rt.c:2595–2601) — a one-element list_head has both pointers aimed at itself, so the test is “do I have a queue-mate?”. On the test box HZ = 1000 (CONFIG_HZ=1000 in /boot/config-7.1.13-200.fc44.x86_64) and /proc/sys/kernel/sched_rr_timeslice_ms reads 100, so a quantum is 100 jiffies of 1 ms each. Querying the quantum for a SCHED_FIFO task returns zero, not 100 ms: get_rr_interval_rt() (rt.c:2604–2611) returns sched_rr_timeslice only if (task->policy == SCHED_RR).
sequenceDiagram autonumber participant T as scheduler_tick participant A as Task A (prio 50) participant B as Task B (prio 50) Note over A,B: Case 1 - both SCHED_FIFO T->>A: task_tick_rt() sees policy != SCHED_RR, returns T->>A: task_tick_rt() returns T->>A: task_tick_rt() returns Note over A,B: A keeps the CPU indefinitely / B never runs until A blocks Note over A,B: Case 2 - both SCHED_RR, sched_rr_timeslice = 100 jiffies T->>A: --time_slice (99 left) - still positive, return T->>A: ... 98 more ticks ... T->>A: --time_slice hits 0 A->>A: time_slice = sched_rr_timeslice (refill) A->>B: run_list.prev != run_list.next, so requeue_task_rt(tail) + resched_curr() T->>B: B is now head of the prio-50 queue and runs its own 100 ms
SCHED_FIFO versus SCHED_RR at the same priority, driven by task_tick_rt(). What it shows: the two policies enter the same function on every tick and differ by one early return. FIFO’s tick does nothing at all; RR’s decrements a counter, and only when the counter reaches zero and a same-priority sibling exists does it requeue and set need_resched. The insight to take: the quantum is enforced by the periodic tick, not by a high-resolution timer — so RR’s granularity is one jiffy (1 ms at HZ=1000, 4 ms at HZ=250), and on a nohz_full CPU running a single RR task the tick may be stopped entirely, at which point the quantum stops being enforced because there is nothing to round-robin with anyway.
Choosing among the policies
SCHED_FIFO (1) | SCHED_RR (2) | SCHED_DEADLINE (6) | SCHED_OTHER/NORMAL (0) | SCHED_BATCH (3) | SCHED_IDLE (5) | |
|---|---|---|---|---|---|---|
| Class | rt_sched_class | rt_sched_class | dl_sched_class | fair_sched_class | fair_sched_class | fair_sched_class |
| Priority knob | sched_priority 1…99 | sched_priority 1…99 | none — (runtime, deadline, period) triple | nice −20…19 | nice −20…19 | nice ignored; weight 3 |
sched_priority must be | 1…99 | 1…99 | 0 | 0 | 0 | 0 |
| Preemption rule | strictly higher rt_priority only | strictly higher rt_priority, or own quantum expiry | earlier absolute deadline | EEVDF eligibility + virtual deadline | same as NORMAL but never wakeup-preempts | same, lowest weight |
| Timeslice | none — runs until block/yield/preempt | sched_rr_timeslice (100 ms default) | dl_runtime per dl_period | sysctl_sched_base_slice, scaled by weight | as NORMAL | as NORMAL |
sched_rr_get_interval(2) returns | 0 | sched_rr_timeslice | 0 | base slice | base slice | base slice |
| Admission control | none — over-subscribe freely | none | yes — sched_dl_overflow() returns -EBUSY if the utilisation sum would exceed the cap | none | none | none |
| Unprivileged use | needs RLIMIT_RTPRIO > 0 or CAP_SYS_NICE | same | never — dl_policy(policy) unconditionally jumps to req_priv (syscalls.c:497) | yes | yes | yes |
| Starvation risk it creates | highest — a spinner starves every lower class | high — starves lower classes, shares within its own priority | bounded by construction, but DL outranks RT | none | none | none |
| Starvation risk it suffers | from DL, stop class, and the fair server | same | from the stop class only | from any RT/DL task | same | worst — yields to everything |
| Typical user | audio graph thread, motion control loop, irq/* kthreads | several peers that must share one priority band (PipeWire’s data-loop) | video decode, periodic control loops with provable schedulability | everything else | compilers, batch analytics | screensavers, idle scrubbers |
Two entries there are worth dwelling on. SCHED_FIFO has no admission control at all — the kernel will happily let you create a hundred SCHED_FIFO threads at priority 99 and will not warn you; SCHED_DEADLINE refuses over-subscription with EBUSY. That asymmetry is the whole argument for preferring SCHED_DEADLINE where the workload can be characterised (see The Constant Bandwidth Server and Admission Control). And SCHED_DEADLINE is unavailable to unprivileged users regardless of any rlimit: user_check_sched_setscheduler() has a bare if (dl_policy(policy)) goto req_priv; with the comment “Can’t set/change SCHED_DEADLINE policy at all for now (safest behavior)”. There is no RLIMIT_DLBW.
Configuring Real-Time Tasks
From C: sched_setscheduler(2)
#include <sched.h>
#include <string.h>
struct sched_param param;
memset(¶m, 0, sizeof(param));
param.sched_priority = 80; /* user priority 1..99 */
/* Switch the calling thread (pid 0 == self) to SCHED_FIFO at priority 80. */
if (sched_setscheduler(0, SCHED_FIFO, ¶m) == -1)
perror("sched_setscheduler"); /* EPERM if unprivileged */Line by line: sched_priority is the only meaningful field of struct sched_param, and for the RT policies it must be in 1…99 (the kernel rejects out-of-range values with EINVAL; in __sched_setscheduler the check is attr->sched_priority > MAX_RT_PRIO-1, and it also enforces that a non-zero priority is given iff the policy is a real-time policy — rt_policy(policy) != (attr->sched_priority != 0) (syscalls.c, Linux 6.12)). Passing pid == 0 targets the caller. The call fails with EPERM unless the caller has CAP_SYS_NICE or the requested priority is within the thread’s RLIMIT_RTPRIO soft limit. To make the policy change not survive fork() or execve()-style resets in newer code, the SCHED_RESET_ON_FORK flag can be OR-ed into the policy argument.
From the shell: chrt(1)
# Run a command as SCHED_FIFO priority 80
$ chrt --fifo 80 ./latency_sensitive_app
# Run as SCHED_RR priority 10
$ chrt --rr 10 ./worker
# Inspect an existing thread's policy and priority
$ chrt -p 1234
pid 1234's current scheduling policy: SCHED_FIFO
pid 1234's current scheduling priority: 80
# Change a running thread back to normal (SCHED_OTHER, priority must be 0)
$ chrt --other -p 0 1234chrt(1) is the util-linux front-end to sched_setscheduler(2) (chrt(1)). chrt -m prints the valid priority range for each policy on the running kernel (1–99 for SCHED_FIFO/SCHED_RR, 0 for SCHED_OTHER/SCHED_BATCH/SCHED_IDLE). Note that SCHED_OTHER must be given priority 0 — supplying any other value is the classic chrt EINVAL.
Privilege and the RLIMIT_RTPRIO ceiling
Since Linux 2.6.12, an unprivileged thread can set real-time policies only up to its RLIMIT_RTPRIO soft limit: “the RLIMIT_RTPRIO resource limit defines a ceiling on an unprivileged thread’s static priority for the SCHED_RR and SCHED_FIFO policies” (sched(7)). A thread with CAP_SYS_NICE bypasses the limit entirely. This is why audio daemons (PipeWire, JACK) ship /etc/security/limits.d rules granting members of an audio/realtime group a non-zero rtprio — without it, the daemon’s request for SCHED_FIFO priority returns EPERM.
RT Throttling, and the v6.12 Rug-Pull
This is the part of real-time scheduling that surprises people in production, and it is also the part that most documentation — including the kernel’s own and sched(7) — currently describes wrongly for v6.12 and later. Read this section before trusting any number you find elsewhere.
The original mechanism (Linux 2.6.25 → 6.11)
Real-time group scheduling landed in 2.6.25 and brought with it the notion of real-time bandwidth: a (period, runtime) pair per group. Peter Zijlstra and Ingo Molnar then applied it to the default group, so that even a system with no cgroups got a cap. Jonathan Corbet’s write-up of the ensuing argument is the best surviving statement of the rationale (LWN, September 2008). Molnar’s case was empirical:
The thing is, i got far more bugreports about locked up RT tasks where the lockup was unintentional, than real bugreports about anyone intending for the whole box to come to a grinding halt because a high-prio RT tasks is monopolizing the CPU.
The counter-argument, from Nick Piggin, was that this is a userspace API change that breaks POSIX semantics: “It would be perfectly valid to engineer a realtime process that uses a peak of say 90% of the CPU with a 10% margin for safety and other services. Now they only have 5%.” Corbet’s closing line — “the 0.95-second limit — the one option that nobody seems to like — remains unchanged” — turned out to be prophetic: it stayed for sixteen years.
Mechanically, at v6.11 and earlier the accounting ran on every tick and every update_curr call, with no CONFIG_RT_GROUP_SCHED guard:
/* kernel/sched/rt.c, v6.11, update_curr_rt() */
if (!rt_bandwidth_enabled())
return;
for_each_sched_rt_entity(rt_se) {
struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
if (sched_rt_runtime(rt_rq) != RUNTIME_INF) {
raw_spin_lock(&rt_rq->rt_runtime_lock);
rt_rq->rt_time += delta_exec; /* charge it */
exceeded = sched_rt_runtime_exceeded(rt_rq); /* over budget? */
if (exceeded)
resched_curr(rq);
raw_spin_unlock(&rt_rq->rt_runtime_lock);
...
}
}Three consequences fall out of those few lines, and all three matter operationally.
The cap is per-CPU, not system-wide. rt_time and rt_runtime live in the per-CPU struct rt_rq, so each CPU independently gets 950 ms of real-time execution per second. There is a mechanism to rebalance unused budget between CPUs — balance_runtime() — but it is gated on the RT_RUNTIME_SHARE scheduler feature, which is false by default (kernel/sched/features.h, verified identical at v6.11 and v6.12). A 32-CPU box therefore has 32 independent 95% budgets, not one 3040% budget.
Throttling dequeues the entire class, not the offending task. sched_rt_runtime_exceeded() sets rt_rq->rt_throttled = 1, emits exactly one printk_deferred_once("sched: RT throttling activated\n") line to the kernel log for the lifetime of the boot, and calls sched_rt_rq_dequeue(), which subtracts rt_rq->rt_nr_running from rq->nr_running and clears rt_rq->rt_queued. From the core scheduler’s point of view the real-time class on that CPU has simply become empty. Every RT task on the CPU stops, not merely the one that burned the budget — so a well-behaved 1%-duty-cycle audio thread sharing a CPU with a runaway spinner is collateral damage.
Unthrottling is driven by an hrtimer, not by the tick. sched_rt_period_timer() fires every rt_period; do_sched_rt_period_timer() walks every CPU in the span, subtracts overrun * runtime from each rt_time, and where the result falls below runtime clears rt_throttled and calls sched_rt_rq_enqueue() to put the class back. Because the subtraction is rt_time -= min(rt_time, overrun*runtime) rather than a reset to zero, a task that overruns badly can stay throttled across more than one period while the debt drains.
stateDiagram-v2 [*] --> Running: RT task enqueued on rt_rq Running --> Running: task_tick_rt() / update_curr_rt()<br/>rt_rq->rt_time += delta_exec Running --> Throttled: rt_time > rt_runtime<br/>rt_throttled = 1<br/>printk_deferred_once("sched RT throttling activated")<br/>sched_rt_rq_dequeue() + resched_curr() Throttled --> Throttled: rt_rq is off rq->nr_running<br/>fair tasks run - or the CPU goes IDLE Throttled --> Running: sched_rt_period_timer fires<br/>rt_time -= overrun * runtime<br/>rt_time < runtime, so rt_throttled = 0<br/>sched_rt_rq_enqueue() Running --> [*]: last RT task dequeued note right of Throttled The defect: this state is entered on budget exhaustion alone, with no check for whether any lower-priority task actually wants the CPU. end note
The pre-v6.12 rt_rq throttle state machine. What it shows: budget is charged on the running path, the transition into Throttled yanks the whole real-time class off the run queue, and only a periodic hrtimer can bring it back. The insight to take: the note on the right is the reason the mechanism was eventually removed. Nothing in the transition consults rq->cfs.nr_running. If a SCHED_FIFO task genuinely needs 100% of a CPU and no fair task is waiting, the kernel throttles it anyway and the CPU sits idle for 50 ms out of every second — a 5% throughput loss bought for no benefit at all. Corbet: “The idle time is an unwanted artifact of how the throttling is implemented rather than a desired feature in its own right” (LWN 2023).
What changed at v6.12
Commit 5f6bd380c7bd, “sched/rt: Remove default bandwidth control” (Peter Zijlstra, authored 2024-05-27, from Daniel Bristot de Oliveira’s deadline-server series), states the change in two sentences: “Now that fair_server exists, we no longer need RT bandwidth control unless RT_GROUP_SCHED. Enable fair_server with parameters equivalent to RT throttling.”
Established by fetching kernel/sched/rt.c at successive tags and checking whether the rt_time += delta_exec accounting inside update_curr_rt() sits under a #ifdef CONFIG_RT_GROUP_SCHED:
| Tag | update_curr_rt() bandwidth accounting | rt_rq has rt_time/rt_runtime/rt_throttled |
|---|---|---|
| v6.1, v6.5, v6.6, v6.9, v6.10, v6.11 | unconditional | unconditionally (only rt_nr_boosted, rq, tg were group-gated) |
| v6.12 | #ifdef CONFIG_RT_GROUP_SCHED | only under CONFIG_RT_GROUP_SCHED |
| v6.13, v6.16, v6.18, v7.0 | #ifdef CONFIG_RT_GROUP_SCHED | only under CONFIG_RT_GROUP_SCHED |
So v6.12 is exactly the release where the boundary moved — and it is the vault’s pinned LTS, which makes this the single most consequential version fact in the note. On a kernel built with CONFIG_RT_GROUP_SCHED unset (Fedora’s setting, measured below), the fields that would hold the throttle state do not exist in the struct. There is nothing to throttle with.
Uncertain
Verify: whether the major enterprise distributions other than Fedora ship
CONFIG_RT_GROUP_SCHED=yon their v6.12-or-later kernels, which would keep classic throttling alive for their users. Reason: only the local Fedora 44 kernel config was read (/boot/config-7.1.13-200.fc44.x86_64,CONFIG_RT_GROUP_SCHEDabsent); no other distribution’s config was fetched, andlore.kernel.org— where the distro-kernel discussion lives — returns HTTP 403 to this host. To resolve: read/boot/config-*on a RHEL 9/10, Debian 13 and Ubuntu 24.04-HWE box, or fetch those distributions’ published kernel configs. Operational consequence:grep RT_GROUP_SCHED /boot/config-$(uname -r)is the first thing to run before reasoning about throttling on any given machine. uncertain
The replacement: rq->fair_server, a deferred deadline server
The new safety valve is not in the real-time class at all. It is a struct sched_dl_entity fair_server embedded directly in struct rq (kernel/sched/sched.h:1135) — a deadline-class entity whose job, when it runs, is to pick fair tasks. Because the deadline class sits above the real-time class in the linker-ordered array, the server can preempt a SCHED_FIFO task at priority 99 without any special-case code.
Its parameters are set in dl_server_start() (kernel/sched/deadline.c:1636–1645) on first use:
if (!dl_server(dl_se)) {
u64 runtime = 50 * NSEC_PER_MSEC; /* 50 ms */
u64 period = 1000 * NSEC_PER_MSEC; /* 1 s */
dl_server_apply_params(dl_se, runtime, period, 1);
dl_se->dl_server = 1;
dl_se->dl_defer = 1; /* deferred, not eager */
setup_new_dl_entity(dl_se);
}50 ms out of 1000 ms is numerically identical to the 5% the old throttle reserved — the commit message’s “parameters equivalent to RT throttling” is literal. What differs is the dl_defer = 1 flag. A deferred server does not run at the start of its period; start_dl_timer() arms its hrtimer at deadline - runtime (deadline.c:1138), the zero-laxity instant — the last possible moment at which it can still receive its full 50 ms before its deadline. With the defaults that instant is 950 ms into each 1-second period. The brief’s “950 ms out of every 1000 ms” figure therefore survives the rewrite, but it now means something completely different: not “RT is cut off at 950 ms” but “the fair class’s rescue timer is armed for the 950 ms mark, and fires only if it is still needed”.
“Still needed” is the other half. If fair tasks have been getting CPU anyway — because the RT task blocked, or the CPU went idle — dl_server_update_idle_time() (deadline.c:1593–1618) charges that progress against the server’s own runtime, and the timer handler pushes the activation forward by deadline - now - runtime instead of firing (deadline.c:1220–1233). A healthy system never sees the server run at all, and therefore never loses the 5%.
sequenceDiagram autonumber participant C as rq clock / hrtimer participant FS as rq->fair_server (dl_se, 50ms/1000ms, dl_defer=1) participant RT as SCHED_FIFO spinner (prio 80) participant F as fair task (nice 0) Note over FS: period starts - replenish_dl_entity() sets deadline = now + 1s,<br/>runtime = 50 ms, then arms the defer timer FS->>C: start_dl_timer() - act = deadline - runtime = t0 + 950 ms loop t0 .. t0+950 ms RT->>RT: rt_sched_class wins every pick / F gets nothing end C->>FS: defer timer fires at t0+950 ms FS->>FS: dl_defer_running = 1 / enqueue_dl_entity(ENQUEUE_REPLENISH) FS->>RT: dl class is ABOVE rt, so resched_curr() preempts the spinner FS->>F: pick_task_fair() runs under the server's 50 ms budget Note over F: F gets up to 50 ms, then the server throttles<br/>and RT resumes for the next period Note over C,FS: If F had been getting CPU anyway, dl_server_update_idle_time()<br/>burns the budget in background and the timer is pushed forward - no preemption
The deferred fair server rescuing starved fair tasks from a SCHED_FIFO spinner, v6.12. What it shows: the timeline of one 1-second server period. The server is armed but throttled for the first 950 ms; the RT spinner owns the CPU. At the zero-laxity instant the timer fires, the deadline class outranks the real-time class, and 50 ms of fair execution is forced through. The insight to take: compare this with the previous diagram. The old mechanism reserved the 5% unconditionally and burned it as idle time when nobody wanted it; the new one reserves the same 5% but only spends it on demand — which is why a SCHED_FIFO task that legitimately needs an entire CPU now gets it, while the administrator still keeps a guaranteed 50 ms/s window to run kill.
The knobs, and what each one actually does now
| Knob | Default (measured on Fedora 44 / 7.1.13) | Range | What it controls at v6.12+ with RT_GROUP_SCHED=n | What it controlled at ≤ v6.11 |
|---|---|---|---|---|
/proc/sys/kernel/sched_rt_period_us | 1000000 | 1 … INT_MAX | only the denominator of the SCHED_DEADLINE admission-control ratio dl_b->bw = to_ratio(period, runtime) (deadline.c:527) | the RT bandwidth accounting period |
/proc/sys/kernel/sched_rt_runtime_us | 950000 | −1 … sched_rt_period_us; −1 means ”= period”, i.e. no cap | only the numerator of that same DL ratio — so it caps total SCHED_DEADLINE utilisation at 95% per root domain, and no longer affects SCHED_FIFO/SCHED_RR at all | RT bandwidth per period; exceeding it throttled the whole rt_rq |
/proc/sys/kernel/sched_rr_timeslice_ms | 100 | > 0 (converted to jiffies by sched_rr_handler) | the SCHED_RR quantum — unchanged by any of this | same |
/sys/kernel/debug/sched/fair_server/cpu<N>/runtime | not readable here — /sys/kernel/debug is mode 0700 and there is no passwordless sudo on this host | ns | per-CPU fair-server runtime; 0 disables the server on that CPU, restoring pre-2.6.25 “RT can hang the box” semantics | did not exist |
/sys/kernel/debug/sched/fair_server/cpu<N>/period | not readable here (same reason) | ns | per-CPU fair-server period | did not exist |
RLIMIT_RTTIME (per process, setrlimit(2)) | unlimited (measured: /proc/self/limits) | µs | unchanged, and now the most important per-task valve: CPU time an RT task may consume without making a blocking system call | same |
Uncertain
Verify: the
fair_serverruntime/perioddebugfs values as they exist on a running v6.12+ kernel, and whether writing0toruntimedisables the server cleanly. Reason:/sys/kernel/debugis mode0700 root:rooton this host and no privilege escalation is available, so the two debugfs files were confirmed only by reading theirdebugfs_create_file("runtime", 0644, d_cpu, …)registrations inkernel/sched/debug.c:479–491at v6.12 — never by reading the live files. To resolve: runcat /sys/kernel/debug/sched/fair_server/cpu0/{runtime,period}as root on a v6.12+ box. uncertain
What happens to a task that hits the limit
Combine the pieces and the answers are quite specific, and they differ by mechanism:
- Classic RT throttling (≤ v6.11, or v6.12+ with
RT_GROUP_SCHED=y). The task receives no signal and no notification. It is simply not scheduled for the remainder of the period; from inside the task, wall-clock time jumps forward by up to 50 ms with no CPU time consumed — which looks exactly like a scheduling-latency spike, and is the reason this is so often misdiagnosed.dmesgcarries onesched: RT throttling activatedline per boot, and nothing more. Every RT task on the CPU is stopped, not just the greedy one. - The deferred fair server (v6.12+,
RT_GROUP_SCHED=n). Again no signal. The RT task is preempted by the deadline class for up to 50 ms once per second, and only when fair tasks were actually starved. There is noprintkat all, so the mechanism is invisible in the log; the only in-kernel evidence is the fair server’s state in/sys/kernel/debug/sched/debug. RLIMIT_RTTIME. This one does signal, and it is the mechanism to reach for when you want to be told. Per getrlimit(2): on reaching the soft limit the process is sentSIGXCPU; if it catches or ignores the signal and keeps burning CPU,SIGXCPUis regenerated once per second until the hard limit, at which point the kernel sendsSIGKILL. The counter resets to zero every time the task makes a blocking system call, and — critically — is not reset by preemption, quantum expiry, orsched_yield(2). Implementation:watchdog()inkernel/sched/rt.c:2535–2557, called fromtask_tick_rt(), which incrementsp->rt.timeoutonce per jiffy and hands off toposix_cputimers_rt_watchdog(). It is compiled in only underCONFIG_POSIX_TIMERS.SCHED_RESET_ON_FORK.sched(7)records that reset-on-fork exists specifically so that “applications [cannot evade] theRLIMIT_RTTIMEresource limit … by creating multiple child processes” — a child of a reset-on-fork RT thread comes back asSCHED_OTHER. Worth knowing because it shows up in real output: on the test box every user-space RT thread found (rtkit-daemon, PipeWire’sdata-loop.0) reportsSCHED_RR|SCHED_RESET_ON_FORK.
Measured on a stock, unprivileged Fedora 44 box
Everything in this subsection was read from the live system on 2026-09-12 (kernel 7.1.13-200.fc44.x86_64, 32 logical CPUs, no sudo). It is included because the shape of a default system’s RT usage is the context every capacity decision needs.
The account cannot create real-time tasks at all:
$ ulimit -r ; ulimit -Hr
0
0
$ grep -i 'real' /proc/self/limits
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
$ chrt -m
SCHED_OTHER min/max priority : 0/0
SCHED_FIFO min/max priority : 1/99
SCHED_RR min/max priority : 1/99
SCHED_BATCH min/max priority : 0/0
SCHED_IDLE min/max priority : 0/0
SCHED_DEADLINE min/max priority : 0/0RLIMIT_RTPRIO is 0 for both soft and hard limits, so per sched(7) “the only permitted changes are to lower the priority, or to switch to a non-real-time policy” — sched_setscheduler(0, SCHED_FIFO, …) returns EPERM. Note that chrt -m still advertises 1/99: it reports sched_get_priority_min/max(2), which are policy limits and know nothing about the caller’s rlimit. And SCHED_DEADLINE shows 0/0 because deadline tasks have no static priority, not because the policy is unavailable.
The sysctls read exactly the documented defaults — and are now, on this kernel, DL-admission knobs only:
$ cat /proc/sys/kernel/sched_rt_period_us # 1 s
1000000
$ cat /proc/sys/kernel/sched_rt_runtime_us # 0.95 s
950000
$ cat /proc/sys/kernel/sched_rr_timeslice_ms # 100 jiffies at HZ=1000
100
$ grep -E 'CONFIG_(HZ|RT_GROUP_SCHED|PREEMPT_DYNAMIC|SCHED_CORE|SCHED_CLASS_EXT|IRQ_FORCED_THREADING)=' \
/boot/config-$(uname -r)
CONFIG_HZ=1000
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_SCHED_CORE=y
CONFIG_SCHED_CLASS_EXT=y
CONFIG_IRQ_FORCED_THREADING=y
# CONFIG_RT_GROUP_SCHED does not appear at allA census of every thread not in SCHED_OTHER, across 4,509 threads:
| Class / priority | Count | Who |
|---|---|---|
SCHED_FIFO 99 | 32 | migration/0 … migration/31 — one per CPU. ps and chrt both report these as SCHED_FIFO 99, but they are not in the RT class: sched_set_stop_task() (core.c:3553–3570) calls sched_setscheduler_nocheck(stop, SCHED_FIFO, {.sched_priority = MAX_RT_PRIO - 1}) purely so that “it will make it appear like a SCHED_FIFO task, its something userspace knows about and won’t get confused about”, then overwrites stop->sched_class = &stop_sched_class. See Migration the Stop Class and the Migration Thread. |
SCHED_FIFO 50 | 47 | 32 × idle_inject/N, 10 × irq/* threaded interrupt handlers, watchdogd, 4 × card1-crtcN (amdgpu vblank workers) |
SCHED_FIFO 1 | 4 | 3 × psimon (PSI trigger monitors), mt76-tx phy0 (Wi-Fi TX worker) |
SCHED_RR 99 | 1 | rtkit-daemon (`SCHED_RR |
SCHED_RR 20 | 3 | PipeWire data-loop.0 threads (`SCHED_RR |
SCHED_BATCH | 142 | browser/renderer helper threads |
SCHED_IDLE | 1 | alsactl |
The threaded-interrupt count is worth being precise about, because “Linux runs interrupts in threads now” is a common overstatement. /proc/interrupts has 66 numbered IRQ lines on this box (plus 21 named architecture counters such as NMI, LOC, RES, TLB). Exactly 10 of those 66 are threaded, all at SCHED_FIFO 50: irq/9-acpi, irq/26-AMD-Vi (the IOMMU), irq/33-pciehp and irq/34-pciehp (PCIe hotplug), and irq/27…irq/32-ACPI:Event. Everything else — including every NVMe and network queue — still runs its handler in hard-IRQ context. CONFIG_IRQ_FORCED_THREADING=y only makes threadirqs available as a boot parameter; it does not thread anything by itself. For why these specific drivers opt in, see Threaded Interrupt Handlers.
Reading a task’s policy without chrt uses /proc/<pid>/stat. Field 18 is task_prio() (prio − 100), field 19 is nice, field 40 is rt_priority and field 41 is the raw policy number (proc_pid_stat(5)):
$ awk '{printf "prio=%s nice=%s rt_priority=%s policy=%s\n",$18,$19,$40,$41}' /proc/227/stat
prio=-51 nice=0 rt_priority=50 policy=1 # irq/9-acpi: SCHED_FIFO(1), user prio 50
$ awk '{printf "prio=%s nice=%s rt_priority=%s policy=%s\n",$18,$19,$40,$41}' /proc/19/stat
prio=-100 nice=0 rt_priority=99 policy=1 # migration/0: cosmetically FIFO 99
$ awk '{printf "prio=%s nice=%s rt_priority=%s policy=%s\n",$18,$19,$40,$41}' /proc/self/stat
prio=20 nice=0 rt_priority=0 policy=0 # this shell: SCHED_NORMAL(0), nice 0Check those against the priority table above: −51 = 49 − 100 for internal prio 49, −100 = 0 − 100 for internal prio 0, 20 = 120 − 100 for nice 0. Field 18 is a derived number and cannot be compared across classes without the conversion; field 41 is the ground truth.
Finally, two things that are not available here and are worth flagging so nobody wastes time on them. /sys/kernel/debug/sched/ — which at v6.12 holds debug (the per-CPU run-queue dump), preempt (the CONFIG_PREEMPT_DYNAMIC mode), and the fair_server/cpu<N>/ tunables — is mode 0700 root:root and unreadable. And /proc/sched_debug does not exist on any modern kernel: existence-checking proc_create in kernel/sched/debug.c across tags shows it present at v5.11 and v5.12 and gone from v5.13 onward, having moved to debugfs. Documentation and blog posts that tell you to cat /proc/sched_debug predate 2021.
Failure Modes
A runaway RT task hangs the machine
The defining hazard of fixed-priority real-time scheduling is that, because nothing in a lower class runs while an RT task is runnable, a single non-blocking RT task starves everything below it. sched(7) states it plainly: “A nonblocking infinite loop in a thread scheduled under the SCHED_FIFO, SCHED_RR, or SCHED_DEADLINE policy can potentially block all other threads from accessing the CPU forever” (sched(7)). On a single-CPU system this means the shell you would use to kill the offending task never gets to run; on SMP it pins one CPU and, if the task migrates or there are several such tasks, can take the whole machine down. Linux’s defence has two eras, both traced in full in RT Throttling, and the v6.12 Rug-Pull above. Until v6.11 it was RT throttling: sched_rt_runtime_us = 950000 of every sched_rt_period_us = 1000000, reserving 5% of each period for non-RT tasks “so that a run-away real-time tasks will not lock up the machine but leave a little time to recover it” (sched-rt-group, Linux 6.12). From v6.12, unless CONFIG_RT_GROUP_SCHED=y, it is instead rq->fair_server — a deferred deadline server holding a 50 ms/1 s reservation that it spends only when fair tasks are genuinely starved. Either way the guarantee to the administrator is the same 5% window in which to run kill; what changed is that the modern mechanism stops forcing the CPU idle when nobody wants that window. Note also that the defence is only a defence of scheduling: neither mechanism helps if the RT task holds a lock, a mmap_lock or a cpu_hotplug_lock that the rest of the system needs. The deeper history and the group-scheduling interface are in Real-Time Throttling and the RT Bandwidth Limit.
Priority inversion: an RT task blocked by a lower-priority lock holder
A higher-priority RT task that needs a lock held by a lower-priority task is, transitively, at the mercy of whatever can preempt that holder — a medium-priority task can then keep the holder (and thus the high-priority waiter) off the CPU indefinitely. This is the priority inversion problem, the bug that famously reset the Mars Pathfinder lander. Linux’s answer is priority inheritance via rt_mutex and PI-futexes; the full story, including the inheritance algorithm, is in Real-Time Priorities and Priority Inversion.
Mis-set priority and CPU pinning interactions
Two subtler traps: (1) A SCHED_FIFO task spinning at priority 99 can starve per-CPU kernel threads (kworker, RCU callbacks, the migration thread) that themselves run as RT — leading to RCU stalls and watchdog warnings rather than a clean hang. Best practice is to leave headroom (run application RT work in the middle of the range, e.g. 1–50) and to pin RT work to dedicated cores (see CPU Isolation isolcpus and nohz_full). (2) Because real-time tasks ignore nice values entirely, renice has no effect on them — a common surprise when an operator tries to tame a runaway RT process with nice.
Common misunderstandings worth naming
- “
sched_rt_runtime_uslimits mySCHED_FIFOtask.” On v6.12+ withCONFIG_RT_GROUP_SCHED=nit does not, and the file is still there reading950000, which makes the misreading almost inevitable. It now capsSCHED_DEADLINEadmission only. BothDocumentation/scheduler/sched-rt-group.rstat v6.12 and v6.13 (“Even without CONFIG_RT_GROUP_SCHED enabled, this will limit time reserved to real-time processes”) andsched(7)as served today still state the old behaviour; the kernel doc was corrected at v6.14 to “Without CONFIG_RT_GROUP_SCHED enabled, this only serves for admission control of deadline tasks”, the man page has not been. - “The 95% cap is system-wide.” It never was.
rt_time/rt_runtimeare per-CPUrt_rqfields, and cross-CPU budget borrowing (balance_runtime()) is gated on theRT_RUNTIME_SHAREscheduler feature, which defaults tofalse. Likewiserq->fair_serveris per-CPU: each CPU has its own independent 50 ms/1 s reservation. - “
SCHED_FIFO99 is the highest priority on the system.” It is not. The deadline class, the fair server, and the stop class all outrank it, andmigration/N— whichpsshows asSCHED_FIFO99 — is actually in the stop class. Setting an application to 99 buys nothing over 98 except the risk of tying with kernel threads. - “
nice/renicecan tame a runaway RT process.”set_user_nice()stores the newstatic_prioand returns without recomputingp->priofor any task with an RT or DL policy (kernel/sched/syscalls.c:86–89). The call succeeds and changes nothing. Usechrt --other -p 0 <pid>, orRLIMIT_RTTIMEset in advance. - “
sched_yield(2)underSCHED_FIFOgives other work a chance.” It moves the caller to the tail of its own priority list only. If it is the sole task at that priority,sched_yieldis very nearly a no-op, and it never lets a lower-priority task run.
Alternatives and When to Choose Them
SCHED_OTHER(EEVDF) — the default for almost everything. Choose it unless you have a genuine bounded-latency requirement; fixed-priority RT is a sharp tool that trades system robustness for determinism. See The EEVDF Scheduler.SCHED_RRvs.SCHED_FIFO— preferSCHED_RRwhen several tasks legitimately share one priority and you want time-sharing among them; preferSCHED_FIFOwhen a single task per priority should run to completion. If you only ever have one task per priority, the two are indistinguishable.SCHED_DEADLINE— choose when you can characterise work as (runtime every period, by this deadline) and want the kernel to admission-control the set so it provably meets deadlines. UnlikeSCHED_FIFO/RR,SCHED_DEADLINEis not fixed-priority and refuses over-subscription. See SCHED_DEADLINE and Earliest Deadline First and The Constant Bandwidth Server and Admission Control.PREEMPT_RTkernel — orthogonal to policy choice: it makes the kernel itself preemptible so that even a high-priority RT task is not blocked by long non-preemptible kernel sections. You still useSCHED_FIFO/RR;PREEMPT_RTlowers the worst-case latency before your RT task runs. See The PREEMPT_RT Real-Time Kernel.
Production Notes
Real-world RT deployments treat priority as a scarce, carefully budgeted resource. Audio servers (JACK, PipeWire) run their processing graph as SCHED_FIFO at a modest priority and rely on PI-futex-backed pthread_mutex_t with PTHREAD_PRIO_INHERIT to avoid inversion when the real-time thread takes a lock shared with a non-RT setup thread. Industrial and robotics stacks (e.g. ROS 2 real-time configurations) combine SCHED_FIFO with mlockall(2) to lock pages (a page fault in an RT task is an unbounded delay) and CPU isolation to keep housekeeping work off the RT cores. The kernel’s own machinery — the migration/N stop-class thread sits above even RT, and several per-CPU kthreads run as SCHED_FIFO priority 1 — which is why a careless application grabbing priority 99 across all CPUs can deadlock the scheduler’s internals and not merely its own peers. The standard diagnostic when an RT workload “freezes” the box has changed with the mechanism, and getting it wrong wastes hours. On a kernel with classic throttling still active, dmesg | grep 'RT throttling' is definitive — but it prints once per boot (printk_deferred_once), so its absence after the first occurrence proves nothing. On a v6.12+ kernel with CONFIG_RT_GROUP_SCHED=n the fair server logs nothing at all, and the throttle fields it replaced no longer exist, so the only in-kernel evidence is the deadline-server state in /sys/kernel/debug/sched/debug (root-only). Do not reach for /proc/sched_debug — it was removed in v5.13 and moved to debugfs. The portable first steps are therefore: grep RT_GROUP_SCHED /boot/config-$(uname -r) to learn which mechanism you are even looking at; chrt -p <tid> and /proc/<tid>/stat field 41 to confirm the policy is what you believe; ps -eLo cls,rtprio,comm to find the other RT threads competing on the same CPU; and a check that mlockall(2) and core isolation were actually applied, since an unbounded page fault inside an RT task produces the same “frozen” symptom with none of the same cause.
See Also
- Real-Time Priorities and Priority Inversion — the inversion problem and priority inheritance (the necessary companion to using these policies safely)
- Real-Time Throttling and the RT Bandwidth Limit — the safety valve that keeps a runaway RT task from killing the box
- SCHED_DEADLINE and Earliest Deadline First — the deadline-driven RT policy that sits above this class
- Scheduling Classes and the sched_class Interface — the strict-priority class stack that puts
rt_sched_classabove fair - Per-CPU Run Queues and struct rq — the
rqthat embeds thert_rq - Nice Values Weights and Priority Scaling — why
nicedoes nothing to RT tasks - The PREEMPT_RT Real-Time Kernel — making the kernel itself preemptible for bounded latency
- Futex and OS Synchronization Primitives — the userspace locking primitive whose PI variant backs
PTHREAD_PRIO_INHERIT - Scheduler Entry Points and pick_next_task — the class-cascade walk this note’s cascade diagram traces
- Migration the Stop Class and the Migration Thread — the class above
dl, and whymigration/Nreports asSCHED_FIFO99 - Threaded Interrupt Handlers — why 10 of this box’s 66 IRQ lines run at
SCHED_FIFO50 - The cgroup v2 CPU Controller — where
CONFIG_RT_GROUP_SCHED’scpu.rt_runtime_usinterface lives - Context Switch Mechanics switch_to and switch_mm — the cost of every preemption this note describes
- Linux Process Scheduling MOC — parent map