vault

The faulthandler Module

19 min read

The faulthandler Module

Abstract

faulthandler is the standard-library module (docs, Python 3.14) that answers a question ordinary Python error handling cannot: what was my program doing when the interpreter itself died or froze? When CPython takes a fatal C-level fault — a SIGSEGV (segmentation fault), SIGFPE (floating-point exception), SIGABRT (abort), SIGBUS (bus error) or SIGILL (illegal instruction) — the normal exception machinery never runs, because the interpreter loop is no longer in a state where it can execute Python bytecode. You get a bare Segmentation fault (core dumped) from the shell and nothing else. faulthandler installs C signal handlers that, the instant such a fault arrives, walk the interpreter’s frame structures directly and write the Python traceback of every thread to a file descriptor — using only async-signal-safe operations, allocating no memory and running no Python code. It also handles the other undebuggable scenario, a hang or deadlock, via a watchdog thread that dumps all tracebacks after a timeout. Added in Python 3.3; this note tracks the 3.14 implementation in Modules/faulthandler.c and Python/traceback.c at the v3.14.5 tag.

Why You Need It: the Dead-Interpreter Problem

A Python exception — ValueError, KeyError, even KeyboardInterrupt — is a Python-level event. The bytecode evaluation loop catches it, unwinds the frame stack, builds a traceback object, and either runs your except block or prints the familiar Traceback (most recent call first): to stderr. All of that is Python code running on a healthy interpreter.

A fatal C-level fault is not a Python event. When a C extension dereferences a null pointer, or a bug in CPython itself corrupts memory, the operating system delivers a hardware signal like SIGSEGV to the process. There is no Python exception to raise, no except clause that can catch it, and — critically — the interpreter may be in a structurally broken state where running any Python bytecode (including the code that would format a traceback) would crash again or hang forever. The default disposition of these signals is to terminate the process, often dumping a core file. From the operator’s point of view, the program simply vanishes with Segmentation fault, and the single most useful piece of information — which Python line was executing — is gone.

The hang is the mirror image. A deadlock on the Global Interpreter Lock (GIL) or a C call that never returns leaves the process alive but frozen. Ctrl-C may do nothing (the signal can’t be processed if no thread is running the eval loop to check for pending signals). Again you are blind: the process is up, consuming a core, and you have no idea where.

faulthandler closes both gaps. Its premise is that even when the interpreter cannot run, its in-memory data structures — the chain of frames, the code objects, the line tables — are still readable. So instead of asking Python to introspect itself, faulthandler reads those structures from C, in the signal handler, the way an external tool like py-spy reads them from another process (see Sampling Profilers and py-spy).

Mental Model

flowchart TD
    subgraph proc["CPython process"]
        eval["Eval loop running<br/>thread frames live in memory"]
    end
    fault["Fatal fault:<br/>SIGSEGV / SIGFPE / SIGABRT<br/>SIGBUS / SIGILL"] -->|kernel delivers signal| handler
    hang["Hang / deadlock<br/>(no fault, process frozen)"] -->|timeout elapses| watchdog
    user["Operator: kill -USR1 PID"] -->|user signal| usersig
    eval -.->|frames readable| handler
    eval -.->|frames readable| watchdog
    eval -.->|frames readable| usersig
    subgraph fh["faulthandler (C, async-signal-safe)"]
        handler["faulthandler_fatal_error()<br/>walks frames, writes to fd"]
        watchdog["faulthandler_thread()<br/>watchdog, separate thread"]
        usersig["faulthandler_user()<br/>chosen-signal handler"]
    end
    handler -->|write() raw bytes| fd["File descriptor<br/>(default: stderr fd 2)"]
    watchdog -->|write() raw bytes| fd
    usersig -->|write() raw bytes| fd
    handler -->|raise(signum)| dflt["Re-raise to default handler<br/>→ process still dies / cores"]

The diagram shows the three entry points into faulthandler and what they share. The crash handler (left), the hang watchdog (center), and the operator-triggered handler (right) all do the same core job — read the still-readable frame structures and write a traceback to a file descriptor using only signal-safe operations — but they are armed by different triggers. The insight to take away: faulthandler never tries to “recover.” On a fatal fault it dumps and then deliberately re-raises the signal so the process still dies the way it would have; it just dies with a Python traceback attached instead of silently.

Turning It On

There are three ways to enable the crash handler, and they exist precisely because by the time a crash happens it is too late to import anything.

The programmatic call is faulthandler.enable(file=sys.stderr, all_threads=True, c_stack=True). It installs the C signal handlers for the five fatal signals. file is where tracebacks go (it may be a file object or a raw integer file descriptor since 3.5); all_threads=True dumps every thread, not just the faulting one; and c_stack=True — new in 3.14 — additionally prints the native C stack after the Python traceback when the platform supports it (via dump_c_stack(), also new in 3.14, which needs backtrace(3) or dladdr1(3)). You enable it once at process startup:

import faulthandler
faulthandler.enable()           # handlers armed for SIGSEGV/SIGFPE/SIGABRT/SIGBUS/SIGILL
# ... rest of the program; if it segfaults, you get a Python traceback on stderr

Because you cannot rely on a crashing program having gotten as far as your enable() call — the crash might be during import of a misbehaving C extension — CPython offers two pre-main activations that the interpreter wires up at startup:

python -X faulthandler myscript.py     # command-line option
PYTHONFAULTHANDLER=1 python myscript.py # environment variable (any non-empty value)

Both are equivalent to calling faulthandler.enable() with defaults before your code runs, so a crash even during the earliest imports is captured. The Python Development Mode (-X dev) also calls faulthandler.enable() automatically at startup, which is why dev-mode crashes come with tracebacks for free.

faulthandler.disable() uninstalls the handlers and faulthandler.is_enabled() reports the current state.

Mechanical Walk-through: the Crash Path

The actual mechanism is in Modules/faulthandler.c. When enable() runs, faulthandler_enable() first calls faulthandler_allocate_stack(), which allocates a block of memory and registers it as an alternate signal stack with sigaltstack(&stack, &old_stack). This matters for one specific fatal case: a stack overflow. A stack overflow is a SIGSEGV, but the normal stack is exhausted, so a handler that ran on the normal stack would itself overflow and you would learn nothing. By installing SA_ONSTACK in the sigaction flags (action.sa_flags |= SA_ONSTACK), the handler runs on the separate alternate stack and can still dump a traceback even when the program’s real stack is blown.

The handler installed is faulthandler_fatal_error. The flags set in faulthandler_enable() include SA_NODEFER (action.sa_flags = SA_NODEFER). When a fatal signal fires, the kernel invokes faulthandler_fatal_error(signum). A source comment states its discipline plainly: “This function is signal-safe and should only call signal-safe functions.” Here is what it does, in order:

  1. It identifies the signal from the faulthandler_handlers array (which is ordered so that SIGSEGV is “the default choice if searching the handler fails,” per the source comment) and writes a header with the PUTS(fd, str) macro — Fatal Python error: followed by the signal name. PUTS is defined as (void)_Py_write_noraise(fd, str, strlen(str)): a thin wrapper over a raw write() to the file descriptor that never raises a Python exception. No print, no buffering, no allocation.

  2. It calls faulthandler_dump_traceback(fd, deduce_all_threads(), fatal_error.interp), which dispatches to the frame-walking code in Python/traceback.c (detailed in the next section). When all_threads is set it calls _Py_DumpTracebackThreads(fd, NULL, tstate); otherwise just _Py_DumpTraceback(fd, tstate) for the current thread.

  3. With c_stack enabled (3.14 default), it calls faulthandler_dump_c_stack(fd) to print the native backtrace after the Python frames.

  4. Finally — and this is the design decision people miss — it does not swallow the signal. It calls raise(signum) to re-deliver the signal to the previously-installed handler (typically the OS default). The source comment explains the SA_NODEFER choice: “call the previous signal handler: it is called immediately if we use sigaction() thanks to SA_NODEFER flag, otherwise it is deferred.” So the process still terminates and still produces a core dump if one was configured; faulthandler is purely additive. (On Windows, for SIGSEGV, it returns without re-raising, because re-raising would suppress the Windows fault handler.)

A subtle but important detail of step 2: the synchronous fatal signals are delivered to the thread that caused the fault. To get that thread’s state, the code deliberately does not use PyThreadState_Get(). A source comment explains: that function “doesn’t give the state of the thread that caused the fault if the thread released the GIL” — so it reads thread-specific storage directly via PyGILState_GetThisThreadState(). This is the kind of correctness the module has to get exactly right, because in a crash there is no second chance.

Reentrancy is guarded by static volatile int reentrant = 0 in both faulthandler_dump_traceback() and faulthandler_dump_c_stack(): if dumping a traceback itself faults, the handler refuses to recurse rather than spinning forever.

How the Frames Are Read Without Allocating

The genuinely clever part lives in Python/traceback.c, in _Py_DumpTraceback / _Py_DumpTracebackThreads and the per-frame helper dump_frame(). The constraint is absolute: a signal handler may call only async-signal-safe functions. malloc/free are not async-signal-safe (a fault can occur while the allocator holds its internal lock; calling malloc again would deadlock), and obviously you cannot create Python objects or run bytecode. So the dumper must produce human-readable output touching only memory that already exists.

It starts from tstate->current_frame, the head of the linked list of interpreter frames (_PyInterpreterFrame *frame = tstate->current_frame;). It walks the chain by following frame->previous, reading the next pointer early — the source notes “Read frame→previous early since memory can be freed during” the walk — so a concurrently-freed frame cannot make it dereference garbage. For each frame, dump_frame():

  • Recovers the code object with _PyFrame_SafeGetCode(frame) — the “Safe” variants are the signal-safe accessors that tolerate a partially-torn-down frame.
  • Reads the file name straight from code->co_filename and the function name from code->co_name. These are already-allocated PyUnicodeObjects; the dumper does not build new strings, it reads the existing buffer.
  • Computes the line number from the last executed instruction: int lasti = _PyFrame_SafeGetLasti(frame); then lineno = _PyCode_SafeAddr2Line(code, lasti); — translating a bytecode offset to a source line by reading the code object’s line table, with no allocation.

Output goes through the same PUTS / _Py_write_noraise raw-write path, with integers rendered by _Py_DumpDecimal() (a hand-rolled, allocation-free integer-to-ASCII converter) and strings by _Py_DumpASCII().

These constraints are exactly why the documented traceback is deliberately impoverished compared to a normal Python traceback. The docs spell out the limitations, and now you can see each one is a direct consequence of signal-safety:

  • Only ASCII (backslashreplace on encode) — full Unicode rendering would need allocation and codec machinery.
  • Each string capped at 500 characters — #define MAX_STRING_LENGTH 500 in traceback.c; truncation avoids unbounded work on a corrupt string.
  • Only filename, function name, line number — no source-line text, because reading the .py file would require opening it (not signal-safe).
  • At most 100 frames and 100 threads — #define MAX_FRAME_DEPTH 100 and #define MAX_NTHREADS 100; beyond the frame limit it prints plus N frames.
  • Order reversed, most-recent first — matching how the frame chain is naturally walked.

This is the same fundamental technique an out-of-process profiler like Sampling Profilers and py-spy uses — read the frame structures directly rather than asking the interpreter to introspect itself — except faulthandler does it in-process from a signal handler. The constraints are stricter (it cannot pause and resume the interpreter the way py-spy peeks at another process), which is why the output is more limited.

Catching Hangs and Deadlocks: dump_traceback_later

A frozen process never receives a fatal signal, so the crash handler is useless against deadlocks. For that there is a watchdog thread:

faulthandler.dump_traceback_later(timeout, repeat=False, file=sys.stderr, exit=False)

It dumps the tracebacks of all threads after timeout seconds. If repeat=True it keeps dumping every timeout seconds; if exit=True it calls _exit(1) immediately after dumping (a hard exit that skips buffer flushing and cleanup — the right choice when the process is wedged and you want it gone). cancel_dump_traceback_later() cancels a pending dump, and a second call to dump_traceback_later() replaces the previous parameters and resets the timer.

Mechanically (faulthandler_dump_traceback_later() and faulthandler_thread() in faulthandler.c), the call allocates two locks, thread.running and thread.cancel_event, and starts a new OS thread running faulthandler_thread(). That thread sleeps by trying to acquire the cancel_event lock with a timeout: PyThread_acquire_lock_timed(thread.cancel_event, thread.timeout_us, 0). The lock is held armed (acquired at setup), so the acquire normally times out — and a timeout (PY_LOCK_FAILURE) is the signal to act: the thread writes its Timeout (H:MM:SS)! header and calls _Py_DumpTracebackThreads(thread.fd, thread.interp, NULL) to dump every thread. To cancel, cancel_dump_traceback_later() simply releases cancel_event; the watchdog’s pending acquire succeeds, and it exits cleanly instead of timing out. This lock-with-timeout pattern is itself a robust way to build an interruptible sleep without polling.

The typical use is a self-imposed deadline. Wrap a phase that should never take more than, say, 60 seconds; if it hangs, you get every thread’s traceback automatically:

import faulthandler
faulthandler.dump_traceback_later(60, exit=True)   # if we're not done in 60s, dump all threads and hard-exit
do_the_thing_that_might_deadlock()
faulthandler.cancel_dump_traceback_later()          # made it — disarm the watchdog

Because the dump runs in a separate, healthy thread, it works even when every other thread is blocked on the GIL or stuck in a C call — the watchdog never needs the GIL to write to a file descriptor.

Dumping On Demand in Production: register

For a long-running server that is misbehaving but not crashed — pegged CPU, mysterious slowness, suspected deadlock — you want to ask “what are you doing right now?” without restarting it. That is register:

faulthandler.register(signum, file=sys.stderr, all_threads=True, chain=False)

It installs faulthandler_user() as the handler for an arbitrary signal — conventionally signal.SIGUSR1, which has no other meaning to Python. From then on, kill -USR1 <pid> from the shell makes the process dump all thread tracebacks to file and keep running. chain=False means faulthandler’s handler replaces any prior one; chain=True calls the previously-installed handler afterward (the source restores the old handler, raise(signum)s to invoke it, then re-registers itself). unregister(signum) removes the handler and returns True if one was registered. register() and unregister() are not available on Windows, which lacks the relevant signal semantics.

The production pattern:

import faulthandler, signal
faulthandler.register(signal.SIGUSR1)   # at startup, in your server's main()
# ... server runs for days ...
# operator, on the box, when the server looks hung:
kill -USR1 $(pgrep -f myserver)
# tracebacks of every thread appear in the server's stderr / log — no restart, no attach

This gives you gdb-like “where is it stuck” visibility with nothing but a signal, which is invaluable when you cannot or dare not attach a debugger to a production process. For an even richer, attach-and-execute capability that can run code in the target, see the newer in-process remote debugging via PEP 768.

The File-Descriptor Caveat

enable(), dump_traceback_later() and register() all capture the integer file descriptor of their file argument, not the Python file object’s identity. The docs warn: “If the file is closed and its file descriptor is reused by a new file, or if os.dup2() is used to replace the file descriptor, the traceback will be written into a different file.” In a server that rotates logs by reopening file descriptors, a stale faulthandler could dump a crash report into whatever now occupies the old fd. The fix is to call these functions again whenever the underlying file is replaced. In practice, pointing faulthandler at sys.stderr (fd 2) and leaving stderr alone avoids the problem entirely.

Resolved (2026-06-01)

Confirmed against Modules/faulthandler.c at the v3.14.5 tag: all three entry points resolve the descriptor through the shared faulthandler_get_fileno() helper at call time — faulthandler_enable (line 587, then stores fatal_error.fd = fd at line 598), faulthandler_dump_traceback_later (line 779), and faulthandler_register_py (line 969, storing user->fd = fd at line 1001). So the integer fd is captured identically at call time across all three, exactly as the section above states; the stale-fd consequence follows directly. Source: faulthandler.c@v3.14.5.

Free-Threading Interaction (3.14)

The free-threaded (“no-GIL”) build of CPython 3.14 forced a safety change. In a GIL-protected build, dumping all threads is safe because the dumping thread implicitly has exclusive access. With the GIL disabled, other threads run truly in parallel and could be mutating their own frame chains while the handler walks them — a data race that could read freed memory. So, per the docs, “Changed in version 3.14: Only the current thread is dumped if the GIL is disabled to prevent the risk of data races.” In the source this shows up as the FT_IGNORE_ALL_THREADS sentinel: when the GIL is off, faulthandler_dump_traceback() prints <Cannot show all threads while the GIL is disabled> and dumps only the faulting thread. If you run on the free-threaded build, expect single-thread crash dumps. See The Global Interpreter Lock and the free-threading notes for the broader context of this trade-off.

Reading the Output

A minimal segfault, triggered deliberately with ctypes, illustrates the format:

$ python -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault

Current thread 0x00007fb899f39700 (most recent call first):
  File "/opt/python/Lib/ctypes/__init__.py", line 486 in string_at
  File "<stdin>", line 1 in <module>

Line by line: Fatal Python error: Segmentation fault is the PUTS header naming the signal. Current thread 0x... gives the thread identifier and reminds you the order is most recent call first (the inverse of a normal Python traceback). Each File "...", line N in funcname line is one dump_frame() call — note there is no source-code line printed beneath it, exactly because reading the file is not signal-safe. The faulting call string_at is at the top because it is the most recent frame. With c_stack=True on 3.14 a Current thread's C stack trace (most recent call first): section would follow, mapping native addresses to symbols. After printing all this, faulthandler re-raised the signal, so the shell would still report the process died by SIGSEGV.

Common Misunderstandings and Failure Modes

“faulthandler will let my program recover from a segfault.” No. It dumps and re-raises; the process still dies. It is a diagnostic, not a safety net. If you need a C fault to be survivable, that is a bug in the C extension to fix, not something faulthandler papers over.

“It catches Python exceptions too.” No. Python exceptions already produce tracebacks through the normal machinery; faulthandler is specifically for the cases where that machinery cannot run. Enabling faulthandler changes nothing about how ValueError is reported.

“The truncated output is a bug.” No — the 500-char strings, 100-frame cap, ASCII-only rendering and missing source lines are intentional consequences of async-signal-safety, as shown above. A richer dump would risk deadlocking or re-faulting in the handler.

Closed-fd misdirection. As covered in the caveat: if you point faulthandler at a log file you later rotate/close, a crash dump may land in the wrong file. Symptom: “faulthandler is enabled but I never see the crash report.” Diagnosis: confirm the fd is still the file you think it is, or just use stderr.

Windows gaps. register()/unregister() do not exist on Windows; the SIGUSR1-on-demand pattern is POSIX-only. On Windows, enable() additionally installs a handler for Windows structured exceptions (since 3.6).

Alternatives and When to Choose Them

faulthandler is the right tool when the failure is a C-level crash or a hang and you want zero-dependency, always-available, in-process diagnostics. For sibling needs, choose differently:

  • gdb with the CPython python-gdb.py helpers gives a far richer post-mortem (full C and Python stacks, local variables, the ability to poke at memory) from a core dump — but requires gdb, debug symbols, and the skill to drive it. faulthandler’s dump is the “first 80%” you get for free in the logs.
  • py-spy attaches to a running process from outside and reads its frames, so it needs no in-process setup at all and is excellent for “what is this hung production process doing” — but it is a separate binary you must have installed and permission to run. faulthandler’s register() covers the same on-demand-dump need with a signal and no external tool.
  • PEP 768 remote debugging (new in 3.14) goes further than faulthandler by injecting and running Python code in the target process, enabling a real interactive debugger session into a live process — at the cost of being a much heavier, code-executing mechanism rather than a read-only dump.
  • tracemalloc answers a different question (memory allocation sources), not crashes; -X dev mode bundles faulthandler with other checks.

Production Notes

The single highest-leverage move is to make the crash handler unconditional in any non-trivial deployment: set PYTHONFAULTHANDLER=1 (or pass -X faulthandler) in the service’s environment. It is nearly free at runtime — the handlers are dormant until a signal fires — and it converts an opaque Segmentation fault in your logs into an actionable Python traceback. This is especially valuable for services that load C extensions (numerical libraries, database drivers, serialization codecs), which are the usual source of segfaults in otherwise-pure-Python systems.

For deadlock-prone services, the register(signal.SIGUSR1) pattern at startup plus a documented kill -USR1 <pid> runbook step turns “the server is hung, restart it and hope” into “dump the stacks, see exactly which lock everyone is waiting on, then restart.” Pairing that with dump_traceback_later(timeout, exit=True) around any operation with a hard SLA gives you automatic stack dumps on the hangs you did not anticipate. Because the watchdog runs in its own thread and uses only raw fd writes, it remains effective precisely in the GIL-contention scenarios where the rest of the process is frozen — see The Global Interpreter Lock for why a stuck eval loop cannot otherwise be interrupted.

See Also

Graph View