vault

Remote Debugging in CPython

15 min read

Remote Debugging in CPython

Python 3.14 adds a safe, built-in way to attach a debugger or profiler to an already-running Python process — one you never instrumented in advance and must not restart. The public surface is a single function, sys.remote_exec(pid, script), which “executes script, a file containing Python code in the remote process with the given pid” and “returns immediately, [with] the code … executed by the target process’s main thread at the next available opportunity, similarly to how signals are handled” (per the sys docs, Added in version 3.14). The design comes from PEP 768 – Safe External Debugger Interface for CPython, Final, resolved March 2025. The single most important idea: the injecting tool only schedules a script path into the target’s interpreter state; the target itself, at one of its own eval-breaker safe points, notices the request and runs the script in its own thread. No code is forcibly executed at an arbitrary instant, which is exactly what makes it safe where the old hacks were not.

This note is the sibling of The pdb Debugger. pdb is the in-process interactive debugger; this note is about reaching into a process that is already running. The two meet in python -m pdb -p PID, which uses the mechanism described here as its transport.

The Problem It Solves

Before 3.14 there was no supported way to debug a running Python process you had not prepared. If a production service was wedged or behaving strangely, your options were all bad. You could not “attach pdb” — pdb (see The pdb Debugger) installs a trace function via sys.settrace from inside the program, which requires the program to have already called set_trace() or breakpoint(); there is no API to make a foreign process start tracing. The folk workarounds were genuinely dangerous: people attached gdb to the live process and called PyGILState_Ensure(); PyRun_SimpleString(...) by hand, or wired a SIGUSR1 handler in advance that dropped into pdb. Both inject and run arbitrary code at moments the interpreter never sanctioned — mid-allocation, mid-refcount-update, while a container object is half-built — and can corrupt the heap or deadlock on the GIL (The Global Interpreter Lock). PEP 768 exists to replace those with a mechanism that only ever runs injected code at an interpreter safe point. The What’s New summary frames it as a “zero-overhead debugging interface that allows debuggers and profilers to safely attach to running Python processes without stopping or restarting them” (per What’s New in 3.14).

“Zero-overhead” is the genuinely clever part: when no one is attached, the feature costs nothing. There is no background thread, no polling, no extra branch on the hot path beyond a flag that is already checked — the eval breaker — described next.

Mental Model: A Note Left in Shared State, Read at a Safe Point

Think of it as leaving a sticky note in a place the target already checks on its own schedule. The injecting process writes a script path and a “please run it” flag into a small structure inside the target’s interpreter state, then nudges the target’s existing eval breaker so the next time the interpreter reaches a safe point it will look at the note and act on it.

flowchart LR
    subgraph local["Attaching process (e.g. python -m pdb -p PID)"]
        RE["sys.remote_exec(pid, script_path)"]
    end
    subgraph os["Operating system"]
        PTRACE["cross-process memory write<br/>(ptrace / task_for_pid /<br/>WriteProcessMemory)"]
    end
    subgraph target["Target Python process"]
        TS["PyThreadState (main thread)<br/>_PyRemoteDebuggerSupport:<br/>debugger_script_path[]<br/>debugger_pending_call"]
        EB["eval breaker bit set"]
        SAFE{"safe point reached?<br/>(eval-breaker check)"}
        RUN["read path,<br/>execute script<br/>in target context"]
    end
    RE --> PTRACE --> TS
    PTRACE -.set bit.-> EB
    EB --> SAFE
    TS --> SAFE
    SAFE -->|yes| RUN

Diagram: sys.remote_exec does not run code in the target directly. It writes the script path and a pending-call flag into the target’s per-thread _PyRemoteDebuggerSupport structure and sets a bit in that thread’s eval breaker. The target, on its own, reaches the next eval-breaker safe point, sees the flag, and runs the script in its own main thread. The insight: control is cooperative and deferred — the target executes injected code only when it is in a known-consistent state, which is why this is safe where gdb-driven injection was not.

How It Works, Mechanically

The deferral point is the eval breaker: a per-thread flags word the bytecode interpreter checks at well-defined moments (between bytecode instructions / at back-edges), already used for delivering signals, releasing the GIL, and running pending calls. PEP 768 piggybacks on it so that “zero-overhead” is literal — no new check is added to the steady-state hot loop.

Per PEP 768 and the remote-debugging how-to, the steps are:

  1. The interpreter publishes layout via debug offsets. CPython exposes a _Py_DebugOffsets structure (a “cookie” plus a table of field offsets) at a known location relative to the PyRuntime symbol. An external tool reads it to learn, version-independently, where the relevant fields live — because a debugger written in C or Rust cannot assume the struct layout of the Python it is attached to.
  2. Locate a thread state. Using those offsets the tool walks PyRuntime → interpreters_head → threads_main to find the main thread’s PyThreadState.
  3. Write the request. Each PyThreadState carries a _PyRemoteDebuggerSupport sub-structure with (at least) a fixed-size debugger_script_path buffer and an integer debugger_pending_call flag. The tool writes the script’s filesystem path into the buffer and sets the flag to 1.
  4. Trip the eval breaker. The tool sets the eval-breaker bit for that thread so the interpreter will stop at its next safe point.
  5. The target acts. At the next eval-breaker check the interpreter sees debugger_pending_call, reads the path, and executes the file in the target’s own context. Any exception raised by the script is handled as an unraisable exception (it does not propagate into the target’s normal control flow).

The actual cross-process memory writes in steps 2–4 are not something Python code does — they require OS-level access (ptrace on Linux, task_for_pid() on macOS, WriteProcessMemory/ReadProcessMemory on Windows). sys.remote_exec performs them for you. Third-party tools that want their own protocol can read _Py_DebugOffsets and drive the same fields directly; CPython ships a private _remote_debugging C extension module that does exactly this (on this 3.14.5 build it is listed as a built module, and Lib/asyncio/tools.py imports RemoteUnwinder, FrameInfo from it to dump a running event loop’s tasks).

Resolved (2026-06-01)

The C-level shapes are now verified verbatim against the v3.14.5 tag. Include/cpython/pystate.h (lines 31–35) defines #define _Py_MAX_SCRIPT_PATH_SIZE 512 and the struct typedef struct { int32_t debugger_pending_call; char debugger_script_path[_Py_MAX_SCRIPT_PATH_SIZE]; } _PyRemoteDebuggerSupport;, and (line 210) embeds it on PyThreadState as remote_debugger_support — so the buffer is per-thread and the path buffer is exactly 512 bytes, confirming the figures above. The eval-breaker bit is #define _PY_EVAL_PLEASE_STOP_BIT (1U << 5) in Include/internal/pycore_ceval.h (line 328). Include/internal/pycore_debug_offsets.h (lines 230–234, 367–371) publishes these as _Py_DebugOffsets sub-fields remote_debugger_support (offset of the struct on PyThreadState), debugger_pending_call, debugger_script_path, and debugger_script_path_size (set to _Py_MAX_SCRIPT_PATH_SIZE). Sources: pystate.h@v3.14.5, pycore_ceval.h@v3.14.5, pycore_debug_offsets.h@v3.14.5.

I verified the observable behavior on this 3.14.5 system directly rather than trusting the prose: starting a child Python process that loops with frequent time.sleep calls, then from the parent calling sys.remote_exec(child_pid, '/tmp/inject.py'), the injected script ran inside the child — its print appeared in the child’s stdout and it wrote a file stamped with the child’s PID — while sys.remote_exec returned immediately in the parent. This confirms the signature sys.remote_exec(pid, script), the “returns immediately, runs at next safe point” semantics, and same-user attach working under Linux ptrace_scope=0.

API and Usage

The minimal API, exactly as documented:

import sys
import os
from tempfile import NamedTemporaryFile
 
# Write the code we want the target to run into a real file on disk.
with NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
    script_path = f.name
    f.write("import my_debugger; my_debugger.connect(%d)" % os.getpid())
 
# Schedule it into the process with PID 1234. Returns immediately.
sys.remote_exec(1234, script_path)

Line by line: the injected code must live in a file — remote_exec takes a path, not a code string, and “the caller is responsible for making sure that the file still exists whenever the remote process tries to read it and that it hasn’t been overwritten” (per the sys docs), because execution is deferred and asynchronous. A common pattern is to make the injected script connect back to the attaching process (open a socket to it) so the two can have a conversation; that is precisely what pdb does.

Version compatibility. “The remote process must be running a CPython interpreter of the same major and minor version as the local process. If either … is pre-release (alpha, beta, or release candidate) then the local and remote interpreters must be the same exact version” (per the sys docs), because the injected code and the debug offsets assume a matching layout. Availability is documented as Unix and Windows.

The pdb integration: python -m pdb -p PID. New in 3.14, “the pdb module now supports remote attaching to a running Python process using a new -p PID command-line option … This feature uses PEP 768 and the new sys.remote_exec() function to attach to the remote process and send the PDB commands to it” (per What’s New). Reading Lib/pdb.py shows the full choreography and explains why pdb needs more than a one-shot script: pdb.attach(pid, commands=()) opens a local listening socket on a random port, writes a small bootstrap connect_script to a temp file (pdb._connect(host="localhost", port=…, frame=sys._getframe(1), …)), chmods it world-readable so the target can read it, then calls sys.remote_exec(pid, connect_script.name). The target runs that bootstrap at its next safe point, which calls back to the listening socket; pdb then runs a _PdbServer inside the target and a _PdbClient in your terminal, shuttling commands and output over the socket. So remote_exec is used only as the transport to start a conversation — the interactive session afterward rides a normal socket. There is also a protocol-version handshake: _connect compares the local and remote pdb protocol versions and refuses to attach on mismatch (“incompatible with this PDB module”), and it refuses if another pdb is already attached. This socket/_PdbServer/_PdbClient layer is pdb-specific machinery and is covered as part of The pdb Debugger; here it matters only as the canonical consumer of remote_exec.

Security Model and Opt-Out

The trust boundary is the operating system, not Python. “On every supported platform the operating system gates cross-process memory access behind privilege checks (CAP_SYS_PTRACE, root, or administrator rights)” (per the how-to). Concretely: on Linux you can normally only attach to a process you own and can signal, and the Yama LSM may further restrict it — ptrace_scope=1 (the common default) limits tracing to descendant processes; echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope relaxes it. On macOS, even attaching to your own process typically needs root because of system security restrictions on task_for_pid(). On Windows you generally need administrative privileges / SeDebugPrivilege. In a container you add --cap-add=SYS_PTRACE. If permission is lacking, sys.remote_exec raises PermissionError; pdb’s CLI catches it and prints a pointer to the permission documentation. So a non-privileged attacker cannot use this to escalate — they already need the OS-level right to write another process’s memory, at which point Python’s safety is moot anyway.

Auditing. Two distinct audit events fire (per the sys docs): sys.remote_exec is raised in the calling process with the pid and script path; and cpython.remote_debugger_script is raised in the remote process, with the path, when the script is executed there. A security-conscious deployment can hook these via sys.addaudithook to log or block injection attempts.

Opting out entirely. PEP 768 provides three levels of disable. The environment variable PYTHON_DISABLE_REMOTE_DEBUG, “if … set to a non-empty string … disables the remote debugging feature described in PEP 768. This includes both the functionality to schedule code for execution in another process and the functionality to receive code for execution in the current process” (per cmdline docs, Added in 3.14). The command-line option -X disable-remote-debug does the same per-invocation and “is only available on some platforms and will do nothing if … not supported on the current system.” And redistributors can compile it out with the --without-remote-debug configure flag.

Resolved (2026-06-01)

The flag is spelled with a hyphen: -X disable-remote-debug. The two official pages disagreed (cmdline reference rendered underscore; What’s New rendered hyphen) — the cmdline reference page is the one with the typo, contrary to my earlier guess. Python/initconfig.c at v3.14.5 parses it via config_get_xoption(config, L"disable-remote-debug") (line 2097) and the built-in -X help text reads -X disable-remote-debug (line 312). Confirmed empirically on this 3.14.5 build: a child launched with -X disable-remote-debug rejected injection with RuntimeError('Remote debugging is not enabled in the remote process') and the injected script did not run, whereas a child launched with the underscore form silently ignored the (unknown) -X option and was successfully injected — Python discards unrecognized -X names rather than erroring, which is exactly why the underscore spelling appeared to “work.” Source: initconfig.c@v3.14.5.

Failure Modes

It seems to do nothing for a long time. Because execution waits for a safe point, “attaching to a process that is blocked in a system call or waiting for I/O will only work once the next bytecode instruction is executed or when the process receives a signal” (per the pdb docs). A process parked in a long time.sleep, blocking recv, or native C call will not pick up the request until it returns to the eval loop. This is inherent to the cooperative design, not a bug.

PermissionError. The most common real-world failure — usually Yama ptrace_scope, missing SYS_PTRACE in a container, or non-root on macOS. The fix is OS configuration, not Python.

Version mismatch. Attaching across different minor versions (or different pre-release builds) is refused; the injected layout assumptions would be wrong.

The script file vanished. Since execution is deferred, deleting or truncating the temp script before the target reads it leads to failure — keep it alive until you are sure the target has run it.

Race on interpreter state for external (non-remote_exec) tools. Tools that read/write interpreter memory directly (sampling profilers, _remote_debugging consumers) must contend with a moving target; the how-to recommends suspending the target (e.g. SIGSTOP/ptrace) before reading internal state to avoid tearing, then resuming. sys.remote_exec itself avoids this by only scheduling a flag and letting the target act when consistent.

Alternatives and When to Choose Them

sys.remote_exec (and python -m pdb -p PID on top of it) is the right tool when you need to interact with Python-level state in a running process — inspect objects, run expressions, attach a real debugger — and you have the OS privilege to do so. It pauses the target only when it is at a safe point and only for as long as your injected code runs. It is the wrong tool when you must not perturb the target at all and only want statistics: a sampling profiler such as py-spy reads the target’s stacks out of its memory without injecting any code and without requiring the program to cooperate, trading the inability to run arbitrary code for true non-intrusiveness. PEP 768 in fact narrows py-spy-style tools’ historical risk: external samplers already read foreign process memory via the same OS primitives, and the standardized _Py_DebugOffsets give them a supported, version-aware way to locate Python structures instead of reverse-engineering struct layouts per release. For timing within your own process you want cProfile; for interactive stepping you set up yourself, plain The pdb Debugger. Choose remote attach specifically for the “it’s already running and I can’t restart it” case.

Production Notes

The headline production use is debugging a wedged or misbehaving service without a restart — historically a choice between guessing and a risky gdb session. With 3.14 you python -m pdb -p <pid> (privileges permitting) and get a real prompt. The same _Py_DebugOffsets foundation powers python -m asyncio ps/pstree-style tooling: Lib/asyncio/tools.py uses the _remote_debugging module’s RemoteUnwinder to dump a live event loop’s task tree from outside, which is invaluable for diagnosing a hung asyncio server (see The asyncio Event Loop). For hardened deployments, decide deliberately: leave remote debugging on and monitor the sys.remote_exec / cpython.remote_debugger_script audit events, or disable it with PYTHON_DISABLE_REMOTE_DEBUG / --without-remote-debug if your threat model treats same-user code injection as unacceptable even given the OS privilege gate. Remember the latency caveat in any runbook: a process stuck in a blocking syscall will not respond to the attach until it next returns to the bytecode loop.

See Also

Graph View

Backlinks