vault

Kernel Lockup and Hang Detection

20 min read

Kernel Lockup and Hang Detection

The Linux kernel ships a family of in-kernel watchdogs whose single job is to notice when the kernel itself has stopped making forward progress and to surface that fact as a stack trace in the kernel log before the machine becomes a black box. There are four distinct detectors, each tuned for a different failure shape: the soft-lockup detector catches a CPU spinning in kernel mode with interrupts enabled; the hard-lockup detector catches a CPU stuck with interrupts disabled (where the soft-lockup timer can no longer even fire); the hung-task detector (khungtaskd) catches a task — not a CPU — wedged in uninterruptible sleep for too long; and the RCU CPU-stall detector catches a CPU that never reports a Read-Copy-Update quiescent state. A fifth, the workqueue watchdog, catches a stalled worker pool. All of them share one design idea: a known-good code path periodically bumps a timestamp or counter, and a separate checker fires on a timer or interrupt to verify that bump happened — if it didn’t, something is stuck. Every detector can be configured to turn a silent hang into a clean panic so that kdump can capture a vmcore for postmortem analysis (per Documentation/admin-guide/lockup-watchdogs.rst, Linux v6.12).

This note covers the v6.12 LTS implementation. Soft-lockup, hard-lockup, hung-task, and RCU-stall live in kernel/watchdog.c, kernel/watchdog_perf.c, kernel/watchdog_buddy.c, kernel/hung_task.c, and kernel/rcu/tree_stall.h respectively. The NMI mechanism that the hard-lockup detector rides on is covered separately in Non-Maskable Interrupts and the NMI Watchdog; this note is the detector-family view.

Mental Model — A Liveness Heartbeat, Checked From the Outside

The unifying trick is liveness by heartbeat. A piece of code that is guaranteed to run if the system is healthy writes a fresh timestamp (or increments a counter). A separate, independently-scheduled checker reads that value and asks: “has it advanced since I last looked?” If not, the heartbeat producer never ran, which means the thing it lives inside is wedged. The four detectors differ only in what the heartbeat is and who checks it — and that choice is dictated by the kind of stall each one must survive.

flowchart TB
  subgraph SOFT["Soft lockup — CPU stuck, IRQs ON"]
    S1["hrtimer fires<br/>watchdog_timer_fn()"] --> S2["kicks softlockup_fn<br/>via CPU stopper"]
    S2 --> S3["update_touch_ts()<br/>bumps timestamp"]
    S1 --> S4{"now > touch_ts<br/>+ 2*watchdog_thresh?"}
    S4 -->|yes| S5["BUG: soft lockup<br/>dump stack"]
  end
  subgraph HARD["Hard lockup — CPU stuck, IRQs OFF"]
    H1["PMU cycle counter<br/>overflows -> NMI"] --> H2["watchdog_overflow_callback()"]
    H2 --> H3{"hrtimer_interrupts<br/>advanced?"}
    H3 -->|no| H4["Watchdog detected<br/>hard LOCKUP"]
  end
  subgraph HUNG["Hung task — TASK stuck in D state"]
    T1["khungtaskd wakes<br/>every timeout secs"] --> T2["scan TASK_UNINTERRUPTIBLE"]
    T2 --> T3{"switch_count<br/>unchanged?"}
    T3 -->|yes| T4["INFO: task blocked<br/>for N seconds"]
  end
  subgraph RCU["RCU stall — no quiescent state"]
    R1["GP kthread checks<br/>elapsed time"] --> R2{"> stall timeout<br/>~21s?"}
    R2 -->|yes| R3["INFO: rcu_sched<br/>detected stalls"]
  end

The four lockup detectors and what each one watches. What it shows: each detector pairs a heartbeat producer (left) with a checker (the diamond) and a log splat on failure (right). The insight to take: the choice of checker is forced by the failure being caught — a soft lockup can still take timer interrupts so a normal hrtimer suffices; a hard lockup has disabled interrupts so the checker must be a non-maskable interrupt (NMI) driven by the hardware PMU; a hung task is a scheduling/blocking problem so a kernel thread scans task state; an RCU stall is about grace-period progress so the RCU machinery itself checks. The same heartbeat-and-checker skeleton, four times.

The forcing function is worth dwelling on, because it explains why there isn’t just one watchdog. If a CPU is genuinely wedged with interrupts disabled, no timer callback on that CPU will ever run — the soft-lockup hrtimer is itself a casualty of the lockup. So detecting that case demands a checker that fires even when ordinary interrupts are masked: a non-maskable interrupt, generated either by a hardware performance counter overflowing (the PMU/perf path) or by another CPU acting as a watcher (the buddy path). Conversely, a hung task is not a stuck CPU at all — the CPU is fine, scheduling other work; it’s one specific thread that has gone to sleep on a lock or I/O and never woken. That requires walking the task list, not watching a CPU. Each detector is the minimal mechanism that can observe its specific failure.

Detector 1 — The Soft-Lockup Detector

A soft lockup is a bug in which the kernel loops in kernel mode for an extended period without rescheduling, but with interrupts still enabled (lockup-watchdogs.rst). The classic cause is a kernel code path holding a CPU in a tight loop or under a spinlock for too long. Because interrupts are still on, timer interrupts keep arriving — and that is exactly what makes the soft lockup detectable from the same CPU.

The heartbeat producer is a per-CPU high-resolution timer (hrtimer) that fires watchdog_timer_fn(). On a healthy CPU, that function does not directly stamp the timestamp; in the modern v6.12 implementation it kicks a small helper, softlockup_fn(), onto the CPU via the CPU stopper (stop_one_cpu_nowait(smp_processor_id(), softlockup_fn, NULL, ...), per kernel/watchdog.c). softlockup_fn() runs update_touch_ts() to write the current time into a per-CPU “touch” timestamp, stops the IRQ-counting machinery, and signals a completion. Routing the touch through the stopper means the timestamp is only refreshed if the CPU can actually run a queued stopper task — a stronger liveness signal than just “a timer interrupt arrived.”

The checker is the same watchdog_timer_fn() on its next firing. It calls is_softlockup(), which compares the current time against the stored touch timestamp:

/* from kernel/watchdog.c, v6.12 */
if (time_after(now, period_ts + get_softlockup_thresh()))
        return now - touch_ts;   /* soft lockup! */

get_softlockup_thresh() returns watchdog_thresh * 2. With the default watchdog_thresh of 10 seconds (int __read_mostly watchdog_thresh = 10;), the soft-lockup threshold is 20 seconds — i.e. 2 × watchdog_thresh (kernel.rst: “Softlockup threshold equals 2 * watchdog_thresh”). The doubling is deliberate: the comment in watchdog.c notes that “soft-lockups can have false positives under extreme conditions” so a larger margin is wanted than for hard lockups.

The hrtimer does not wait the full 20 seconds between samples. The sample period is get_softlockup_thresh() * (NSEC_PER_SEC / NUM_SAMPLE_PERIODS) with NUM_SAMPLE_PERIODS = 5, so the timer fires five times within the soft-lockup window — frequent enough to refresh the touch timestamp promptly on a healthy CPU but cheap enough to ignore. The timestamp itself is coarse: get_timestamp() returns running_clock() >> 30LL (a bit-shift that divides nanoseconds by ~2³⁰ ≈ 10⁹, giving approximate seconds), because second-granularity is all a 20-second threshold needs.

When the threshold is exceeded, the kernel prints the signature line and a backtrace of the stuck CPU:

BUG: soft lockup - CPU#3 stuck for 22s! [some_process:12345]

followed by registers and a stack dump showing exactly what code the CPU is spinning in. If kernel.softlockup_panic is set to 1, the detector calls panic("softlockup: hung tasks") instead of merely logging — converting the hang into a kdump-capturable panic.

Detector 2 — The Hard-Lockup Detector

A hard lockup is the more severe sibling: a CPU stuck in kernel mode with interrupts disabled, “without permitting interrupts to run” (lockup-watchdogs.rst). The documented threshold is 10 seconds (one watchdog_thresh, not doubled). The defining problem is that the soft-lockup hrtimer cannot fire on a CPU whose interrupts are masked — so the soft-lockup detector is blind to this case, and a fundamentally different checker is required: one that runs as a non-maskable interrupt (NMI), which by definition cannot be blocked by the normal interrupt-disable.

There are two NMI-capable backends, selected at build/boot time:

The perf/PMU backend (kernel/watchdog_perf.c) programs a hardware Performance Monitoring Unit (PMU) counter to count CPU cycles and overflow at a chosen rate, with the overflow delivered as an NMI:

/* kernel/watchdog_perf.c, v6.12 */
static struct perf_event_attr wd_hw_attr = {
        .type     = PERF_TYPE_HARDWARE,
        .config   = PERF_COUNT_HW_CPU_CYCLES,
        .pinned   = 1,
        .disabled = 1,
};

When the counter overflows, watchdog_overflow_callback() runs in NMI context — even on a CPU whose ordinary interrupts are masked. It first validates timing via watchdog_check_timestamp() to avoid false positives from CPU turbo/frequency scaling (which would otherwise make the NMI arrive too soon), then calls watchdog_hardlockup_check(). The heartbeat it checks is the soft-lockup hrtimer’s interrupt counter. is_hardlockup() asks whether that counter advanced since the last NMI:

/* kernel/watchdog.c, v6.12 */
if (per_cpu(hrtimer_interrupts_saved, cpu) == hrint)
        return true;   /* hardlockup: hrtimer never advanced */

The logic is elegant: the soft-lockup hrtimer increments hrtimer_interrupts every time it fires. If a CPU is hard-locked, its hrtimer stops firing (interrupts are off), so hrtimer_interrupts freezes. The NMI — which still fires — notices the frozen counter and declares a hard lockup, printing:

Watchdog detected hard LOCKUP on cpu 3

with a backtrace of the wedged CPU. Setting kernel.hardlockup_panic to 1 turns this into a panic.

The buddy backend (kernel/watchdog_buddy.c) exists for systems whose PMU cannot deliver an NMI (or where the perf NMI is unavailable). Instead of self-monitoring via NMI, each CPU watches its neighbour — its “buddy”. watchdog_next_cpu() selects the buddy by walking the active watchdog CPUs in a circular pattern (cpumask_next, wrapping to cpumask_first). In watchdog_buddy_check_hardlockup(), a CPU checks its buddy only every third sample — the comment explains: “Test for hardlockups every 3 samples. The sample period is watchdog_thresh * 2 / 5, so 3 samples gets us back to slightly over watchdog_thresh (over by 20%).” After an smp_rmb() memory barrier (to ensure the buddy’s touch is visible), it calls watchdog_hardlockup_check(next_cpu, NULL) against the buddy’s counters. The buddy approach is decentralised — no single monitor CPU — but it depends on at least one other CPU still running, so it cannot detect a whole-machine freeze.

Uncertain

Verify: the precise menu of values accepted by the nmi_watchdog= boot parameter on v6.12 (commonly cited as 0 to disable, nopanic/panic to control panic-on-hard-lockup). Reason: the primary Documentation/admin-guide/kernel-parameters.txt blob could not be fetched in full (the file is very large and the fetch truncated before the nmi_watchdog= entry; a kernel.org fetch returned 503). The lockup-watchdogs.rst doc confirms nmi_watchdog is the kernel-parameter knob for hard-lockup panic behaviour but does not enumerate the exact option strings. To resolve: grep Documentation/admin-guide/kernel-parameters.txt in a v6.12 checkout for nmi_watchdog=. uncertain

Detector 3 — The Hung-Task Detector (khungtaskd)

The hung-task detector watches tasks, not CPUs. Its target is a thread stuck in uninterruptible sleep — the D state — for too long, the canonical symptom of a process blocked indefinitely on I/O or on a lock it will never get. The CPUs are all healthy; one particular task simply never wakes.

The mechanism (kernel/hung_task.c) is a dedicated kernel thread, khungtaskd. It sleeps for hung_task_timeout_secs (computed via hung_timeout_jiffies()) and on each wakeup calls check_hung_uninterruptible_tasks(), which iterates every task with for_each_process_thread() and filters for tasks in TASK_UNINTERRUPTIBLE while excluding TASK_WAKEKILL and TASK_NOLOAD (the latter excludes tasks that deliberately don’t count toward load, like idle kthreads). For each suspect, check_hung_task() compares the task’s current switch_count — the sum of voluntary and involuntary context switches — against the last_switch_count it recorded last scan. If switch_count is unchanged, the task has not been scheduled at all in the interval: it is genuinely wedged, not merely sleeping-then-running.

The default timeout is 120 seconds (sysctl_hung_task_timeout_secs, from CONFIG_DEFAULT_HUNG_TASK_TIMEOUT). When a hang is confirmed, the detector dumps the task’s stack with sched_show_task(t), producing the familiar:

INFO: task some_process:12345 blocked for more than 120 seconds.
      Tainted: G        ...
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<stack trace showing where the task is blocked, e.g. in mutex_lock or a filesystem read>

Several knobs shape its behaviour (all in kernel.rst): hung_task_warnings defaults to 10, capping how many such reports are emitted before the detector goes quiet (set to -1 for unlimited); hung_task_check_count bounds how many tasks are scanned per pass (default PID_MAX_LIMIT, effectively all); hung_task_check_interval_secs lets the scan run more often than the timeout (default 0 means “use the timeout as the interval”); and kernel.hung_task_panic set to 1 makes a confirmed hang call panic("hung_task: blocked tasks"). During the scan, khungtaskd periodically calls rcu_lock_break() so that walking a very long task list does not itself stall an RCU grace period — a nice example of one watchdog being careful not to trip another.

A crucial subtlety: a task legitimately in D state for a long time (a slow but progressing NFS mount, say) is not necessarily a bug — which is why the message is INFO: and the default action is to warn, not panic. The detector measures lack of scheduling, which is a strong but not infallible proxy for “stuck.”

Detector 4 — The RCU CPU-Stall Detector

Read-Copy-Update (RCU) is a synchronisation mechanism in which writers wait for a grace period — a window during which every CPU has passed through a quiescent state (a point where it holds no RCU read-side references) — before reclaiming memory (see RCU Grace Periods for the full mechanism). If one CPU never reports a quiescent state, the grace period never ends, RCU callbacks pile up unprocessed, and memory is never freed. The RCU CPU-stall detector (kernel/rcu/tree_stall.h) exists to surface that.

The detector fires when a grace period has been in progress longer than the stall timeout, which is normally 21 seconds (CONFIG_RCU_CPU_STALL_TIMEOUT, tunable at runtime via the rcupdate.rcu_cpu_stall_timeout module parameter, exposed at /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout, per Documentation/RCU/stallwarn.rst). The documented causes overlap heavily with the other detectors — “a CPU looping in an RCU read-side critical section,” “a CPU looping with interrupts disabled,” “a CPU looping with preemption disabled,” grace-period kthread starvation, and so on — which is why an RCU stall and a soft/hard lockup frequently fire together on the same incident.

The splat names the offending CPU and how far behind it is:

INFO: rcu_sched detected stalls on CPUs/tasks: 2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0

Reading this: CPU 2 is stalling, it is “3 GPs behind” (three grace periods behind where it should be), with its dyntick-idle state, softirq counters, and force-quiescent-state pass count appended for diagnosis. By default the kernel logs and tries to continue; setting kernel.panic_on_rcu_stall to 1 makes it panic() after the stall message, and max_rcu_stall_to_panic lets you require several stalls before panicking (it has no effect when panic_on_rcu_stall is 0).

Detector 5 — The Workqueue Watchdog

A fifth, narrower watchdog targets stalled worker pools rather than CPUs or arbitrary tasks. Workqueues run deferred kernel work on kworker threads; a pool can wedge — for example through a missing WQ_MEM_RECLAIM flag causing a memory-reclaim deadlock, or a concurrency-managed work item that stays RUNNING forever — and, as the detector’s author Tejun Heo noted when introducing it, “these stalls can be extremely difficult to hunt down because the usual warning mechanisms can’t detect workqueue stalls and the internal state is pretty opaque” (LKML, 2015). The watchdog (kernel/workqueue.c) periodically checks each pool’s watchdog_ts; if a pool has pending work but its timestamp has not advanced past the threshold, it emits:

BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 31s!

and dumps workqueue state. The default threshold is 30 seconds (wq_watchdog_thresh, tunable via the workqueue.watchdog_thresh boot/module parameter). As the documentation cautions, a workqueue lockup is usually a side-effect of some other problem (a real deadlock elsewhere starving the pool), so it is a symptom pointer rather than a root cause.

Controls — The Complete Knob Set

The detectors are governed by sysctl (runtime, under /proc/sys/kernel/) and matching boot parameters. The most important, all from kernel.rst:

# Master switch: enable/disable BOTH soft- and hard-lockup detectors at once
sysctl kernel.watchdog=1            # 1 = on (default), 0 = off
 
# The core threshold. hrtimer/NMI frequency and lockup windows derive from this.
sysctl kernel.watchdog_thresh=10    # seconds; soft-lockup window = 2*this = 20s
                                    # 0 disables lockup detection entirely
 
# Individual detector switches
sysctl kernel.soft_watchdog=1       # soft-lockup detector only
sysctl kernel.nmi_watchdog=1        # hard-lockup (NMI) detector only
 
# Which CPUs run the watchdog (cpulist format). Default = all possible CPUs,
# excluding nohz_full cores when NO_HZ_FULL is configured (to keep them tickless).
sysctl kernel.watchdog_cpumask=0-3
 
# Turn each hang into a clean panic -> kdump
sysctl kernel.softlockup_panic=1    # default 0
sysctl kernel.hardlockup_panic=1    # default 0
sysctl kernel.hung_task_panic=1     # default 0
sysctl kernel.panic_on_rcu_stall=1  # default 0
 
# Hung-task tuning
sysctl kernel.hung_task_timeout_secs=120   # D-state threshold; 0 = no checking
sysctl kernel.hung_task_warnings=10        # cap on reports; -1 = unlimited
sysctl kernel.hung_task_check_interval_secs=0  # 0 = use timeout as interval
 
# Capture all-CPU backtraces (via NMI broadcast) at detection time
sysctl kernel.softlockup_all_cpu_backtrace=1   # default 0
sysctl kernel.hardlockup_all_cpu_backtrace=1   # default 0
sysctl kernel.hung_task_all_cpu_backtrace=1    # default 0
 
# Reboot some seconds after panic (combine with the *_panic knobs above)
sysctl kernel.panic=10              # >0: reboot after N s; 0: loop forever; <0: reboot now

Two relationships are worth re-stating because they trip people up. First, watchdog_thresh is the one tunable that scales everything: the soft-lockup window is exactly 2 × watchdog_thresh, the hard-lockup window is watchdog_thresh, and the hrtimer sample period is watchdog_thresh × 2 / 5. Lowering watchdog_thresh makes detection faster but raises false-positive risk. Second, the *_panic knobs are what tie this whole subsystem to crash analysis: with softlockup_panic=1 (or its siblings) plus a configured crash kernel, a hang becomes a panic(), which fires the kdump path and writes a vmcore you can open in crash/gdb — turning an unactionable freeze into a debuggable artefact.

Failure Modes and How They Surface

The detectors are diagnostic aids, and reading their output correctly is a skill. A soft lockup splat (BUG: soft lockup - CPU#N stuck for Xs!) points the backtrace at the exact kernel function the CPU is spinning in — most often a too-long loop, a spinlock held across an expensive operation, or a buggy driver. Because interrupts were on, the trace is reliable. A hard lockup splat (Watchdog detected hard LOCKUP on cpu N) is graver: it means interrupts were off, so the CPU was unreachable by normal means — typical culprits are a deadlock with IRQs disabled, a hardware fault, or a bad spinlock-irqsave path. The trace here is captured from NMI context, which is part of why the NMI/PMU mechanism (see Non-Maskable Interrupts and the NMI Watchdog) is indispensable: nothing else could have interrupted that CPU.

A hung-task report (INFO: task ...:PID blocked for more than 120 seconds) is qualitatively different — the message is INFO, not BUG, because a task in D state is not necessarily broken. The backtrace shows where the task is blocked (e.g. inside mutex_lock, down, or a filesystem read), which usually points at the resource it is waiting on. False positives are real: a genuinely slow but progressing I/O (an overloaded disk, a stalled NFS server) can trip it. The detector measures “did this task get scheduled?” not “is this task making semantic progress,” so the right response is to investigate what it is blocked on, not to assume corruption.

An RCU CPU-stall warning frequently co-occurs with a soft or hard lockup on the same CPU — because the same “CPU looping with preemption/interrupts disabled” that triggers a lockup also prevents that CPU from reporting a quiescent state. When you see all three on one incident, they are three views of one stuck CPU, and the soft/hard-lockup backtrace is usually the more directly actionable. A standalone RCU stall (no accompanying lockup) more often indicates grace-period-kthread starvation or an RCU read-side critical section held too long.

One operational hazard: a false positive can cost you the machine if a *_panic knob is set too aggressively. On heavily loaded virtualized guests, scheduling delays from the hypervisor can make a guest CPU appear soft-locked when it was merely descheduled by the host — which is exactly why Red Hat documents keeping NMI/lockup panic parameters disabled in some virtualized environments to avoid spurious panics (Red Hat solution 1354963). The watchdog_thresh value and the panic knobs together set the trade-off between catching real hangs fast and surviving transient stalls.

Alternatives and When to Choose Them

These software watchdogs detect a kernel that is still partly alive — alive enough to run a timer, an NMI, or a kthread and to format a log message. They cannot help once the machine is completely dead (no NMI, no scheduling, console wedged). For that, you escalate to mechanisms outside the running kernel:

  • A hardware watchdog timer (the /dev/watchdog device, an independent timer chip or BMC feature) reboots the box if userspace stops petting it — it survives a total kernel death that the software detectors cannot.
  • The Magic SysRq Key lets an operator force a backtrace (l/t), trigger a crash (c), or reboot from a still-responsive console even when normal task handling is gone.
  • For interactive postmortem, kgdb and kdb Kernel Debuggers attach a source-level debugger.
  • For automated capture, kdump is the destination the *_panic knobs feed into.

The right tool is a function of how dead the system is: software lockup detectors for a partly-stuck kernel (the common case, and the one that produces the most actionable backtraces), a hardware watchdog as the backstop for total death, and SysRq/kgdb/kdump for capturing and inspecting whatever state remains.

Production Notes

In practice, the most valuable configuration on a fleet is softlockup_panic=1, hardlockup_panic=1, hung_task_panic=1 (where the workload tolerates it), panic_on_rcu_stall=1, plus a reserved crash kernel and kernel.panic=<small N> to auto-reboot after the dump. This converts every class of hang into a captured vmcore and a fast recovery, instead of a node that sits dark until someone notices. The cost is sensitivity to false positives — which is the central tuning tension, especially on virtualized or heavily-overcommitted hosts where host-side scheduling delays masquerade as guest lockups. Conservative operators raise watchdog_thresh (e.g. to 30) on noisy-neighbour guests, or keep the lockup panics off there entirely per the Red Hat guidance above, while keeping the logging (non-panic) detectors on so incidents are still recorded.

When triaging a real incident, the workflow is: read dmesg for the watchdog splat, identify which detector fired (the message prefix tells you: BUG: soft lockup, Watchdog detected hard LOCKUP, INFO: task ... blocked, INFO: rcu_... detected stalls, BUG: workqueue lockup), read the attached backtrace to find the stuck function or blocked resource, and — if a vmcore was captured — open it to inspect the full state. The detector that fired narrows the failure class immediately: soft = CPU spinning with IRQs on; hard = CPU spinning with IRQs off; hung task = a thread blocked on a resource; RCU stall = a CPU not reaching a quiescent state; workqueue = a worker pool starved.

See Also

Graph View