vault

Sampling Profilers and py-spy

14 min read

Sampling Profilers and py-spy

A sampling profiler answers “where is my program spending time?” by periodically capturing the call stack — say a thousand times a second — and tallying how often each function appears, rather than instrumenting every call. The Python docs frame the contrast precisely: statistical profiling “randomly samples the effective instruction pointer, and deduces where time is being spent. The latter technique traditionally involves less overhead (as the code does not need to be instrumented), but provides only relative indications of where time is being spent” (per the profile docs). The flagship Python sampling profiler is py-spy, a tool written in Rust that reads the target process’s memory from the outside — no code change, no import, no restart — and can therefore attach to a long-running or production process safely (per the py-spy README). The trade-off versus deterministic profiling (Profiling with cProfile) is exact-counts-and-distortion versus statistical-estimate-and-near-zero-overhead: sampling never sees every call, so its numbers are estimates with a margin of error, but it does not suffer the per-call observer effect that inflates hot functions under cProfile, and its overhead is roughly constant regardless of how call-heavy the program is.

Mental Model

A deterministic profiler is a turnstile counting every person through a door; a sampling profiler is a security camera that photographs the room every second and counts who appears in the photos. If a function is on the stack in 30% of the photographs, it was running about 30% of the time — statistically. You never know the exact call count, but you get the time distribution cheaply, and the camera’s cost does not grow with how many times people walk through the door.

flowchart LR
    subgraph Target["Target Python process (unmodified)"]
        TS["PyThreadState"]
        TS --> F1["_PyInterpreterFrame"]
        F1 --> F2["_PyInterpreterFrame"]
        F2 --> F3["_PyInterpreterFrame"]
    end
    subgraph PySpy["py-spy (separate Rust process)"]
        TIMER["sampling timer<br/>(e.g. 100 Hz)"]
        READ["process_vm_readv (Linux)<br/>vm_read (macOS)<br/>ReadProcessMemory (Win)"]
        AGG["aggregate stacks<br/>by frequency"]
    end
    TIMER -->|"each tick"| READ
    READ -.->|"read memory<br/>at known struct offsets"| Target
    READ --> AGG
    AGG --> OUT["flame graph (SVG)<br/>top (live) · dump · speedscope"]

Figure: how py-spy samples a running CPython process externally. On every timer tick it reads the target’s memory through an OS call (process_vm_readv/vm_read/ReadProcessMemory), locates the interpreter and thread state, and walks the chain of _PyInterpreterFrame structs to reconstruct the Python call stack — all without the target executing any profiler code. The insight: because py-spy lives in its own process and merely reads the target, its overhead is paid by the profiler, not the program, so the program runs at full (specialized) speed and the measurement is non-intrusive — the opposite of cProfile’s in-process per-call hook.

How Sampling Profiling Works

The core loop is simple and the same across tools: set a timer, and on each tick suspend or peek at the target’s threads, record the current call stack of each, then resume. Over a run you accumulate a histogram: for each unique stack (the full chain of frames), how many samples landed there. Converting sample counts to time is statistical — the Tachyon docs give the estimator explicitly: time in function ≈ (samples in function / total samples) × duration, with the margin of error shrinking as you sample longer or faster.

This design has three consequences that define the technique:

  1. Overhead is low and roughly constant. It depends on the sampling rate (how many stacks per second) and the cost of reading one stack — not on how many function calls the program makes. A function-call-heavy workload that would cripple a deterministic profiler costs a sampling profiler nothing extra. This is the property the Python docs call “less overhead (as the code does not need to be instrumented).”
  2. No observer effect on hot code. Because nothing is injected into the program’s call path, a hot function runs at full speed — including with the specializing adaptive interpreter’s optimized bytecodes intact. Contrast Profiling with cProfile, where per-call bookkeeping systematically inflates cheap, frequently-called functions.
  3. Statistical, not exact. A function that runs many times but each time for less than the sampling interval may be undersampled or missed entirely. You get a time distribution, never call counts. For exact counts you need a deterministic profiler; for a 1–2% micro-difference you need timeit. The Tachyon docs explicitly say: use deterministic profiling when you “need exact call counts,” and prefer it for very short scripts where too few samples accumulate.

py-spy: Profiling from Outside the Process

py-spy is the canonical external sampling profiler for Python. The README’s framing: “py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way … written in Rust for speed and doesn’t run in the same process as the profiled Python program … py-spy is safe to use against production Python code” (per the README). As of this writing the current release is 0.4.2 (24 April 2026).

Reading another process’s memory

py-spy does not call into the Python interpreter; it reads the target process’s address space directly with the platform’s foreign-memory primitive: process_vm_readv on Linux, vm_read on macOS, ReadProcessMemory on Windows (per the README). This is the same family of mechanism a debugger uses to inspect a running process, and it is why py-spy needs the corresponding privilege: on Linux, ptrace/process_vm_readv permission (often requiring CAP_SYS_PTRACE or relaxing yama/ptrace_scope); on macOS, task_for_pid (typically root); on Windows, SeDebugPrivilege. Because the read is one-directional — py-spy never writes the target, only reads — it cannot corrupt the program it watches.

Walking the CPython stack from the outside

To turn raw bytes into a Python call stack, py-spy must know CPython’s internal data layout. Starting from the interpreter’s runtime state it finds the PyThreadState for each thread, and from each thread state it walks the chain of frames to read each frame’s code object, and from the code object the function name, filename, and line number.

Uncertain (reviewed 2026-06-01 — retained: py-spy is an external Rust tool, out of v3.14.5 CPython-tag scope)

Verify: py-spy’s exact current stack-walk on CPython 3.11+. Reason: the py-spy README describes “iterating over each PyFrameObject in each thread,” but that language predates the 3.11 frame redesign — since 3.11 the per-call frame is a lightweight _PyInterpreterFrame living on a per-thread data-stack chunk (an actual PyFrameObject is materialized lazily; see Stack Frames and the Frame Stack), so on a modern CPython py-spy must read the _PyInterpreterFrame chain from the PyThreadState, using per-version struct offsets it carries for each supported interpreter version. py-spy does support 3.11–3.14 in practice, which implies it tracks the new layout, but I did not pin the precise structs/offsets to a py-spy primary source (the relevant code is in its Rust python_interpreters/stack_trace modules). To resolve: read py-spy’s source for the version-specific frame structs. The external-read mechanism and the dependence on known struct offsets are confirmed by the README; only the precise post-3.11 frame field walk is unverified here. #uncertain

This external, layout-dependent approach has one operational catch worth stating: py-spy must understand the specific CPython version’s struct layout, so a new CPython release needs py-spy support before it can be profiled. It also means py-spy reads a snapshot that may be momentarily inconsistent (a stack being mutated as it is read); sampling profilers tolerate this because an occasional bad sample is statistically washed out, and py-spy offers a --nonblocking mode (read without pausing, faster but slightly more inconsistent) versus the default that briefly pauses threads for a clean read.

Output and usage

py-spy has three subcommands (per the README):

# record: sample for a while and write a flame graph (default) or other format
py-spy record -o profile.svg --pid 12345
py-spy record -o profile.svg -- python myprogram.py    # or launch the program itself
 
# top: a live, htop-style view of where time is going right now
py-spy top --pid 12345
 
# dump: a one-shot snapshot of every thread's current stack (great for hangs/deadlocks)
py-spy dump --pid 12345

record produces an interactive flame graph SVG by default, and can emit speedscope JSON (-f speedscope) or raw collapsed-stack data for other tools. --rate sets samples per second (default 100). --subprocesses follows children. --gil restricts samples to threads actually holding the GIL (so you measure real Python execution, not threads blocked in I/O). --idle includes idle threads. dump is the quiet hero for production incidents: a single read of every thread’s stack, which immediately shows where a hung process is stuck — no restart, no instrumentation.

Flame Graphs

A flame graph (invented by Brendan Gregg, per his write-up) is the standard visualization of sampled stacks. Each box is a function; the width of a box is proportional to how often that function (or its descendants) appeared in samples — i.e. how much time was spent there; stacked vertically shows the call hierarchy (a box sits on top of the function that called it). You read it horizontally for “what is wide is hot” and vertically for “who called whom.” Unlike a time-series, the x-axis is not time order — boxes at each level are merged and sorted, so width alone encodes cost. Flame graphs are the natural fit for sampling data precisely because sampling produces a frequency-per-stack histogram, which is exactly what the merged-and-stacked layout draws.

Sampling vs Deterministic: The Trade-off

AxisSampling (py-spy, Tachyon)Deterministic (cProfile)
MechanismPeriodic stack snapshotsEvent hook on every call/return
Where it runsExternal process (py-spy) / external readIn-process, in the call path
OverheadLow, ~constant (set by sample rate)High, grows with call count
Observer effect on hot codeNoneInflates cheap/hot functions
ResultStatistical time distributionExact call counts + precise per-function time
Attach to running processYes (no restart, production-safe)No (must launch under the profiler)
Best forLong-lived / production / call-heavy codeReproducible offline runs needing exact counts

The two are complementary, not competitors. The vault’s own decision framework captures the workflow: sample in production to find which path is hot, then reproduce that path under cProfile for exact per-function counts (see the MOC’s “Why is my Python slow?” framework). The observer effect — the reason sampling exists — is derived in full in Profiling with cProfile; this note does not repeat it.

Relationship to Remote Debugging and the 3.14/3.15 Story

External tools like py-spy historically had to reverse-engineer CPython’s memory layout, which is fragile across versions. CPython is now bringing that capability in-house. In CPython 3.14, PEP 768 added a safe external debugger interface — sys.remote_exec() and the underlying _remote_debugging machinery — letting an external process inject execution into a running interpreter at a safe point (the 3.14 What’s New lists exactly this; see Remote Debugging in CPython). This is the same “attach to a live process from outside” capability that sampling profilers depend on, now blessed and made safe by the runtime itself.

Uncertain — version-critical (reviewed 2026-06-01: 3.14 absence now confirmed; only the 3.15 final naming remains forward-looking)

The 3.14 half is resolved: no sampling profiler ships in the CPython 3.14 standard library. Confirmed four ways — the 3.14 What’s New mentions no profiling/sampling module (its only “attach to a running process” feature is PEP 768 sys.remote_exec() for debugging); Lib/profile/ does not exist at the v3.14.5 tag (raw fetch returns 404); Lib/cProfile.py is present but is the deterministic profiler; and on this live 3.14.5 build import profiling raises ModuleNotFoundError while the legacy single-file profile.py module is still present (not yet a package). The remaining forward-looking uncertainty is only the 3.15 stdlib sampling profiler — “Tachyon,” the profiling.sampling module. It was originally implemented under the name profile.sample (issue gh-135953, PR #135998) during the 3.15 cycle, and PEP 799 reorganized profiling into a profiling package (profiling.tracing for deterministic, profiling.sampling for statistical), deprecating the legacy profile module. The reason to flag: the naming churn (profile.sample → profiling.sampling) and the closeness of the 3.14/3.15 cycles make it easy to mis-attribute Tachyon to 3.14. As of June 2026, 3.15 is in beta (3.15.0b1 docs), so exact final names/flags may still shift before release. To resolve: re-check the 3.15 final What’s New on release. #uncertain

Tachyon (CPython 3.15 profiling.sampling) in brief

Because it is the stdlib successor to py-spy’s approach and the natural next note for a 3.14-tracking vault to watch, the shape of Tachyon (per the 3.15 docs, beta — flag above applies):

  • External, attach-by-PID sampling, built on the PEP 768 remote-debugging infrastructure (RemoteUnwinder, _remote_debugging) — so it walks the target’s stack across the process boundary using the runtime’s own blessed API rather than reverse-engineered offsets.
  • CLI: python -m profiling.sampling run script.py, ... attach 12345, ... dump 12345, ... replay profile.bin.
  • Sampling modes — --mode=wall (default, all elapsed time), cpu (only on-CPU), gil (only while holding the GIL), exception; threads and async (--async-aware) understood.
  • Output formats — pstats (default text table, but with sample counts rather than exact call counts), --collapsed (for Brendan Gregg’s flamegraph.pl), --flamegraph (self-contained interactive HTML), --gecko (Firefox Profiler JSON), --heatmap (per-line), --binary (record-and-replay).
  • A headline claim of sampling rates “up to 1,000,000 Hz” and near-zero overhead.

Uncertain (reviewed 2026-06-01 — retained: 3.15-beta/external-tool marketing figures, not pinnable to a v3.14.5 source)

Verify: the “up to 1,000,000 Hz” and “fastest sampling profiler for Python” claims for Tachyon, and py-spy’s exact default rate. Reason: the 1 MHz figure comes from the 3.15 beta docs / promotional framing and is a peak, not a sustained, rate; py-spy’s documented default is 100 Hz with a --rate flag, but I did not pin its maximum. These are marketing-adjacent claims about a still-in-beta 3.15 feature and an external tool, so they fall outside what the v3.14.5 tag can settle and should not be quoted as guarantees. To resolve: benchmark on the 3.15 final release, or read each tool’s implementation. #uncertain

Failure Modes

  • Permission denied on attach. The most common operational failure: process_vm_readv/ptrace is restricted. On Linux, yama/ptrace_scope may need lowering or the profiler run with CAP_SYS_PTRACE; in containers, the SYS_PTRACE capability must be granted. This is a feature (you should not be able to read arbitrary processes), not a bug.
  • Version mismatch. An external profiler that does not understand the target’s CPython version cannot decode its stacks. py-spy must be updated for new CPython releases; the stdlib profiling.sampling requires profiler and target to be the same minor version (and matching free-threaded/standard build), per the 3.15 docs.
  • Undersampling short functions. Functions that finish in less than the sampling interval are statistically underrepresented or invisible. Raise the rate or, for those cases, switch to a deterministic profiler.
  • Misreading a flame graph’s x-axis as time. Width is total time across all samples, merged — not chronological order. A function appearing once but expensively and a function appearing many times cheaply can have the same width.

Alternatives and When to Choose Them

  • Profiling with cProfile — the deterministic sibling: exact counts, precise call tree, but in-process overhead and the observer effect; the right tool for reproducible offline runs where you need to know how many times something was called.
  • py-spy — the go-to external sampling profiler for production and live processes on 3.14 (and earlier), where the stdlib has no sampler.
  • profiling.sampling (Tachyon) — on 3.15+, the in-stdlib sampling profiler that supersedes the need for a third-party tool for many use cases, integrated with PEP 768 remote debugging.
  • Scalene, pyinstrument, Austin — other Python sampling/statistical profilers (Scalene and Austin sample; pyinstrument samples in-process), each with different overhead/feature trade-offs; py-spy’s distinguishing trait is being fully external and Rust-based.

Production Notes

py-spy’s design — external, read-only, no restart — is what makes it the de-facto choice for production Python profiling: you can py-spy dump --pid a wedged web worker to see its stack, or py-spy record a live service for thirty seconds to find the hot path, all without touching the running code or its performance. The standard pattern is sampling-first triage: use a sampling profiler against the real workload to identify where time goes under realistic conditions (and with specialization intact), then, if you need exact counts for a specific suspect path, reproduce it locally under Profiling with cProfile. With CPython 3.14’s PEP 768 and 3.15’s profiling.sampling, this external-sampling capability is increasingly something the runtime supports natively rather than something a third-party tool must reverse-engineer.

See Also

Graph View

Backlinks