Magic SysRq Key

The Magic System Request key (SysRq) is a set of low-level, kernel-resident emergency commands you can trigger even when a Linux system is so broken that userspace — the shell, the login prompt, the windowing system, sometimes even the scheduler — is no longer responding. Each command is a single keystroke (Alt+SysRq+<key> on a console, or a write to /proc/sysrq-trigger from userspace) that the kernel handles directly inside the keyboard interrupt path, bypassing every layer that might be wedged. The commands run the gamut from benign diagnostics (dump all task states, print memory statistics, backtrace every CPU) to last-resort sledgehammers (sync all disks, remount read-only, kill every process, deliberately panic to capture a crash dump, or reboot immediately without syncing anything). Because the handler executes in interrupt context — “you are also in an interrupt handler, which means don’t sleep!” (sysrq.rst, v6.12) — every command must be minimal and non-blocking, which both constrains what SysRq can do and explains why it works when nothing else will.

Mental Model

The right way to think about SysRq is as a hardware-near “back door” into a sick kernel. Ordinary control flow — type a command, the shell fork/execs a program, the program issues syscalls, the kernel schedules it — depends on a long chain of working subsystems. If any link is broken (the scheduler is deadlocked, the disk subsystem is hung, the GUI has grabbed the keyboard and died), that chain is unusable. SysRq deliberately sits underneath that chain: the keyboard driver recognizes the magic combination at the moment the scan code arrives, in interrupt context, and dispatches straight to a small table of handler functions that touch as little machinery as possible.

flowchart TD
  KBD["Keyboard interrupt<br/>(Alt + SysRq + key held)"] --> HSR["handle_sysrq(key)"]
  PROC["root writes to<br/>/proc/sysrq-trigger"] --> HSR2["__handle_sysrq(key,<br/>check_mask=false)"]
  HSR --> ON{"sysrq_on()?<br/>(kernel.sysrq != 0)"}
  ON -- no --> DROP["ignored"]
  ON -- yes --> HSR3["__handle_sysrq(key,<br/>check_mask=true)"]
  HSR3 --> MASK{"sysrq_on_mask(<br/>op->enable_mask)?"}
  HSR2 --> TABLE["sysrq_key_table[key]<br/>→ op->handler(key)"]
  MASK -- bit set / value==1 --> TABLE
  MASK -- bit clear --> DISABLED["'This sysrq operation<br/>is disabled.'"]
  TABLE --> B["b: emergency_restart()"]
  TABLE --> C["c: panic() → kdump"]
  TABLE --> S["s: emergency_sync()"]
  TABLE --> ETC["e/i/f/t/w/l/m/p/u/…"]

The two SysRq entry points and the gate each passes through. What it shows: the keyboard path (top-left) runs through both the global on/off check (kernel.sysrq != 0) and the per-command bitmask, while the /proc/sysrq-trigger path (top-right, root-only file) calls the dispatcher with check_mask=false and so bypasses the bitmask entirely. The insight to take: the sysctl bitmask restricts the keyboard, not root writing to proc — a hardened box can disable risky key commands at the console yet still let an automation script trigger any of them through the file.

The handler table is keyed by the character of the command, not the physical key, so the same dispatch serves the keyboard, the serial console (BREAK then a key), and the /proc interface. Each entry is a struct sysrq_key_op carrying a .handler function, a human-readable .help_msg, an .action_msg printed before the action runs (positive feedback that the key registered), and an .enable_mask naming which sysctl bit must be set for the keyboard to invoke it.

Triggering SysRq

There are three ways to reach the dispatcher, and they differ in what works and what is checked.

From a physical/console keyboard. On x86 you hold Alt, press SysRq (the key historically labeled Print Screen / SysRq), and press the command key. The official docs give the per-architecture combinations: x86 is ALT-SysRq-<command key>; SPARC is ALT-STOP-<command key>; PowerPC is ALT - Print Screen (or F13) - <command key> (kernel.org sysrq, v6.12). This path runs in the keyboard interrupt handler and is the one that survives a frozen userspace, because no process needs to be scheduled for it to fire.

From a serial console. Send a BREAK, then within 5 seconds the command key (kernel.org sysrq, v6.12). This is invaluable for headless servers and virtual machines reached over a serial line, where there is no physical SysRq key — the out-of-band BREAK signal plays its role.

From userspace via /proc/sysrq-trigger. Writing a single character to this file invokes the corresponding command: echo c > /proc/sysrq-trigger triggers a crash, echo t > /proc/sysrq-trigger dumps task state. The file is mode 0200 (S_IWUSR) — writable only by root. Two subtleties are easy to miss. First, normally only the first character written is processed; if you echo a multi-character string, the rest is ignored — unless the first character is an underscore, in which case the kernel processes the whole string. That is what echo _reisub > /proc/sysrq-trigger does: the leading _ puts the writer into “bulk” mode so all of r, e, i, s, u, b are handled in order. The source makes both behaviors explicit:

/*
 * writing 'C' to /proc/sysrq-trigger is like sysrq-C
 * Normally, only the first character written is processed.
 * However, if the first character is an underscore,
 * all characters are processed.
 */
static ssize_t write_sysrq_trigger(struct file *file, const char __user *buf,
                                   size_t count, loff_t *ppos)
{
        bool bulk = false;
        size_t i;
        for (i = 0; i < count; i++) {
                char c;
                if (get_user(c, buf + i))
                        return -EFAULT;
                if (c == '_')
                        bulk = true;
                else
                        __handle_sysrq(c, false);   /* check_mask == false */
                if (!bulk)
                        break;
        }
        return count;
}

(sysrq.c, v6.12, lines ~1160–1189)

Second — and this is the load-bearing detail — the proc path calls __handle_sysrq(c, false). The false is check_mask, so the proc interface ignores the kernel.sysrq bitmask and will run any command regardless of which keyboard commands the sysctl has disabled. The proc path runs in normal process context (the writing task), not interrupt context, which is why it can never be the recovery path when the scheduler itself is wedged — if you can run echo, your userspace is healthy enough not to need the magic. Use proc for scripting deliberate actions (forcing a crash dump in a test, OOM-stress experiments); use the keyboard/serial path for true emergencies.

The kernel.sysrq Bitmask

Whether the keyboard path does anything at all is governed by the sysctl kernel.sysrq, exposed as /proc/sys/kernel/sysrq. Its value is read by sysrq_on() (is SysRq enabled at all?) and sysrq_on_mask() (is this particular command enabled?):

static bool sysrq_on(void)
{
        return sysrq_enabled || sysrq_always_enabled;
}
 
/* A value of 1 means 'all', other nonzero values are an op mask: */
static bool sysrq_on_mask(int mask)
{
        return sysrq_always_enabled ||
               sysrq_enabled == 1 ||
               (sysrq_enabled & mask);
}

(sysrq.c, v6.12, lines ~62–88)

So 0 disables SysRq entirely; 1 enables everything; and any other value is a bitmask that enables only the commands whose .enable_mask bit it includes. The mask bits, defined in include/linux/sysrq.h, are:

Value (hex)	Constant	Enables
0x0002 (2)	SYSRQ_ENABLE_LOG	console log-level control (0–9, R)
0x0004 (4)	SYSRQ_ENABLE_KEYBOARD	keyboard control — SAK (k), unraw (r)
0x0008 (8)	SYSRQ_ENABLE_DUMP	debugging dumps — task list, registers, memory, backtraces, crash (c)
0x0010 (16)	SYSRQ_ENABLE_SYNC	the s sync command
0x0020 (32)	SYSRQ_ENABLE_REMOUNT	the u remount-read-only command
0x0040 (64)	SYSRQ_ENABLE_SIGNAL	signalling processes — e (SIGTERM), i (SIGKILL), f (OOM kill), j (thaw)
0x0080 (128)	SYSRQ_ENABLE_BOOT	the b reboot / o power-off commands
0x0100 (256)	SYSRQ_ENABLE_RTNICE	nicing real-time tasks (n)

(sysrq.h, v6.12, lines ~23–30; the human-readable descriptions are from kernel.org sysrq, v6.12).

Uncertain

Verify: that the crash command c is gated by SYSRQ_ENABLE_DUMP (0x8) rather than a boot/reboot bit. Reason: the v6.12 source shows sysrq_crash_op.enable_mask = SYSRQ_ENABLE_DUMP, which is the dump group, not the boot group — a reasonable reader might expect a “crash” to live with b/o under SYSRQ_ENABLE_BOOT. The source is authoritative here, but the grouping is non-obvious. To resolve: it is already resolved against sysrq.c line 160 — flagging only because it is a common point of confusion. uncertain

To compute a value, OR the bits you want: echo 176 > /proc/sys/kernel/sysrq (128|32|16 = 0xB0) enables reboot + remount + sync but nothing else — a sensible “I want the safe-shutdown subset and nothing dangerous” choice. Distributions vary in their default: many ship kernel.sysrq=1 (all enabled) on desktops, while hardened/server images often set a restrictive value or 0. The setting is also fixable at boot via the sysctl.conf/sysctl.d machinery, and there is a kernel command-line option sysrq_always_enabled that forces sysrq_always_enabled = true so the box honors SysRq regardless of the runtime sysctl (sysrq.c, v6.12, lines ~90–95).

The Command Keys

Below are the commands that matter most for recovery and diagnosis, each tied to its actual handler in drivers/tty/sysrq.c (v6.12). The grouping mirrors how you would reach for them.

Last-resort shutdown and crash.

• b — immediate reboot, no sync, no unmount. sysrq_handle_reboot() calls lockdep_off(); local_irq_enable(); emergency_restart(); — it does not flush dirty buffers or unmount anything, so any unsynced data is lost. This is the “the box is gone, just bring it back” button.
• o — power off (if the platform supports it), the orderly-poweroff analogue of b.
• c — deliberately crash the kernel. sysrq_handle_crash() releases the RCU read lock it is holding and then calls panic("sysrq triggered crash\n"). If a crash kernel is configured, the panic path hands off to kdump to capture a vmcore. This is the canonical way to force a memory image of a hung-but-not-panicking machine for post-mortem analysis.

Filesystem safety (the “save my data” trio).

• s — sync. sysrq_handle_sync() calls emergency_sync(), which schedules a flush of all mounted filesystems’ dirty pages to disk. Crucially, sync is asynchronous in this context — it kicks off the writeback but does not wait for completion, so you should give it a moment before rebooting.
• u — remount read-only. sysrq_handle_mountro() calls emergency_remount(), forcing every mounted filesystem to read-only. This stops further writes and lets the next boot’s fsck see a cleaner state.
• j — thaw filesystems. emergency_thaw_all() un-freezes filesystems previously frozen by the FIFREEZE ioctl (e.g. by a snapshot tool that died mid-freeze).

Killing processes.

• e — SIGTERM to all but init. send_sig_all(SIGTERM) walks the task list (for_each_process), skipping kernel threads (PF_KTHREAD) and the global init (is_global_init), asking each to terminate gracefully.
• i — SIGKILL to all but init. Same walk, but SIGKILL — unconditional, after e fails to clear a runaway.
• f — invoke the OOM killer. sysrq_handle_moom() does not run the OOM killer inline; it schedule_work(&moom_work), deferring the actual out_of_memory() call to a workqueue. This is a deliberate design choice: the OOM killer takes sleeping locks (mutex_lock(&oom_lock)) and cannot run in interrupt context, so the handler punts it to process context. It manually triggers a kill of the largest memory consumer when the system is thrashing in memory pressure.

Diagnostics (the “tell me what’s wrong” set).

• t — dump all task states. sysrq_handle_showstate() calls show_state() and show_all_workqueues(), printing every task with its state and a kernel stack backtrace — the single most useful command for finding what is stuck.
• w — dump blocked (D-state) tasks. show_state_filter(TASK_UNINTERRUPTIBLE) prints only tasks in uninterruptible sleep, i.e. those wedged in the kernel waiting on I/O or a lock that never came back. This narrows t’s firehose to the tasks that actually matter when diagnosing a hang.
• l — backtrace all active CPUs. sysrq_handle_showallcpus() first tries trigger_all_cpu_backtrace() (an NMI-driven path that even pokes a spinning CPU), and falls back to an IPI (smp_call_function(showacpu, …)) that makes each non-idle CPU print its own stack. This is how you catch a CPU spinning in a tight loop with interrupts disabled.
• m — memory info. show_mem() dumps current memory statistics to the console.
• p — registers and flags. sysrq_handle_showregs() prints the current CPU registers (from get_irq_regs() if in hardirq) and calls perf_event_print_debug().
• d — show held locks (with lockdep), q — pending hrtimers and clock devices, z — dump the ftrace buffer.

Keyboard recovery.

• r — unraw. sysrq_handle_unraw() takes the keyboard out of raw mode and back to XLATE — the fix when a crashed X server or game left the keyboard ungrabbed-but-raw and the console unusable.
• k — Secure Access Key (SAK). Kills all programs on the current virtual console, guaranteeing you a clean, un-trojaned login prompt.

The REISUB Safe-Reboot Sequence

When a box is hung but you want to minimize data loss before forcing a reboot, the kernel docs recommend a specific ordering of commands, memorized via the mnemonic “Raise skinny elephants is utterly boring” — the bold letters spell r e i s u b (sysrq.rst, v6.12):

1. r — unRaw the keyboard (regain console control).
2. e — tErminate all processes (SIGTERM, graceful).
3. i — kIll all processes (SIGKILL, forceful — for those that ignored SIGTERM).
4. s — Sync filesystems (flush dirty data to disk).
5. u — Unmount / remount read-only (stop further writes).
6. b — reBoot.

The logic of the order is data integrity: terminate then kill the processes that might still be writing, then sync the data they wrote, then freeze the filesystems, and only then reboot. Pause a few seconds between s and b to let the asynchronous sync complete. The whole sequence can be fired from userspace as echo _reisub > /proc/sysrq-trigger, though if you can run that command your system probably is not hung enough to need it — REISUB earns its keep at a frozen console, one key at a time.

Why Interrupt Context Constrains Everything

The defining constraint on SysRq is stated plainly in the documentation: when your code runs as part of a SysRq handler, “you are also in an interrupt handler, which means don’t sleep!” (sysrq.rst, v6.12). The keyboard combination is recognized inside the keyboard interrupt path, so the handler runs in atomic context where blocking is forbidden — it may not sleep, may not acquire mutexes, may not allocate memory that could block, and must finish quickly. This is why the commands look the way they do:

• The OOM kill (f) defers to a workqueue rather than running out_of_memory() inline — that function takes a sleeping mutex.
• emergency_sync() (s) schedules writeback rather than waiting for it — waiting would block.
• The crash handler (c) explicitly rcu_read_unlock()s before panic() because the dispatcher took an RCU read-lock (rcu_read_lock() in __handle_sysrq) and a panic must not leave it held.
• The all-CPU backtrace (l) prefers an NMI/IPI mechanism precisely because a target CPU might be spinning with interrupts off and unable to respond to anything gentler.

This constraint is also the source of SysRq’s superpower: because it does the absolute minimum and touches almost no subsystems, it keeps working when the scheduler, the block layer, or the GUI has died. A tool that “did more” would depend on more, and would fail exactly when you need it.

Failure Modes and Gotchas

“Nothing happens when I press Alt+SysRq+key.” Three common causes. (1) kernel.sysrq is 0 or a bitmask that excludes your command — check cat /proc/sys/kernel/sysrq. (2) The physical key combination is wrong for your hardware; some laptops require Fn to access the SysRq/PrtSc key, and the timing of holding Alt+SysRq then tapping the command key matters. (3) SysRq input can get “stuck” — the docs suggest “try tapping shift, alt and control on both sides of the keyboard, and hitting an invalid sysrq sequence again” or switching virtual consoles to reset the input state (sysrq.rst, v6.12).

b ate my data. b reboots with no sync. If you skip s (and u) before b, any dirty page cache is lost and the filesystem may need recovery on next boot. Always prefer the full REISUB ordering unless you have already synced or genuinely do not care about the data.

Output went nowhere. SysRq output goes to the console, via printk into the kernel ring buffer. On a graphical desktop with no visible text console, the output may not be on screen — but it is in the ring buffer, readable afterward via dmesg (if the box survives) or recovered from the crash dump. The handler temporarily raises console_loglevel to default so the header is visible, then restores it.

Security note. SysRq is a privileged, no-authentication channel to dangerous operations. On a machine with physical-console or serial-console exposure, leaving kernel.sysrq=1 means anyone at the keyboard can reboot, crash, or kill the box. Lock it down with a restrictive bitmask (or 0) on exposed systems, and remember the proc path is already root-gated.

Alternatives and When to Choose Them

SysRq is the bottom of the debugging stack — reach for it only when higher-level tools cannot run. If userspace is alive, prefer normal tools: dmesg and journalctl for logs, ps/top for process state, the proc filesystem for live kernel state, and ftrace/perf for tracing. For interactive source-level kernel debugging, kgdb/kdb (reachable via SysRq g) lets you set breakpoints and inspect memory — far more powerful than SysRq’s fixed command set, but requiring a second machine or a serial connection. For post-mortem analysis of a dead kernel, kdump captures a full vmcore (and SysRq c is the standard way to trigger that capture on demand). And for automatically catching hangs, the kernel’s lockup and hang detectors (soft/hard lockup watchdogs, hung-task detector) can be configured to panic — and thus kdump — without a human pressing c. SysRq’s niche is the human-driven emergency: a box you can see but not log into, where you need to either rescue its data or force a dump right now.

Production Notes

In production, the most common deliberate use of SysRq is forcing a crash dump from a hung box: configure kdump, then echo c > /proc/sysrq-trigger (or press Alt+SysRq+c at the console / over IPMI serial-over-LAN) to panic the kernel and capture a vmcore for offline analysis with crash. This converts an opaque “the server froze” into an analyzable memory image. The second common use is the REISUB safe reboot of a box whose userspace has wedged but whose kernel still services interrupts — recovering it without a hard power cycle and the filesystem damage that follows. Operators of remote/headless fleets wire SysRq into their out-of-band management: serial-over-LAN consoles deliver BREAK + key to a kernel that has lost its network stack, which is sometimes the only remaining channel into a machine. The hung-task and lockup detectors are frequently set to panic (via kernel.hung_task_panic, kernel.softlockup_panic, etc.) so that hangs auto-trigger the same kdump path SysRq c does — turning the manual emergency button into an automatic safety net.

See Also

• Kernel Crash Dumps and kdump — c triggers the panic that kdump captures into a vmcore
• Kernel Lockup and Hang Detection — the watchdogs that can auto-panic (and thus auto-kdump) instead of waiting for a human to press c
• The Kernel Ring Buffer and dmesg — where all SysRq diagnostic output lands (t, m, p, l)
• ftrace Framework — the z command dumps its buffer
• Linux Tracing and Observability MOC — parent map (§8, kernel debugging and crash facilities)