vault

Profile-Guided Optimization of CPython

13 min read

Profile-Guided Optimization of CPython

Profile-Guided Optimization (PGO) of CPython is a build-time technique for making the CPython interpreter binary itself run faster — it is not a runtime Python feature, and it has nothing to do with optimizing your Python code. It works at the level of the C compiler that compiles python.exe/libpython: you build an instrumented interpreter, run a representative workload through it to record which branches and functions are hot, then recompile the C sources feeding that profile back to the compiler so it can lay out hot paths, inline aggressively, and place branch hints optimally. In CPython’s build system this is one configure flag — ./configure --enable-optimizations — usually paired with --with-lto for link-time optimization (devguide: setup & building; configure docs). The payoff is a single-digit-to-low-double-digit percentage speedup that is completely orthogonal to, and stacks on top of, the Python-level engine work of Faster CPython and the JIT.

The Crucial Distinction Up Front

There are two utterly different things that get called “optimizing Python,” and conflating them is the single most common confusion this note exists to dispel:

  1. Optimizing the interpreter (this note). The CPython interpreter is a C program. Like any C program it can be compiled with PGO + LTO so the resulting machine code is faster. This happens once, when CPython is built, by whoever compiles the binary (a distro packager, a container image maintainer, or you). The same .py files run, just on a faster interpreter.
  2. Optimizing the Python program it runs. That is what the specializing interpreter, the Tier 2 JIT, and bytecode optimizations do at runtime — covered in The Faster CPython Project and its siblings. PGO does nothing here.

PGO is firmly category 1. A PGO-built interpreter makes every Python program faster by a few percent because the evaluation loop in ceval.c, the object allocator, and the hot type-slot calls are all compiled with better code layout — but it changes no Python semantics and no bytecode.

Mental Model — Measure the Compiler’s Guesses, Then Correct Them

An optimizing C compiler constantly makes guesses it cannot verify statically: which if branch is more likely? Is this function hot enough to inline? Which switch cases dominate? Without data it falls back to heuristics. PGO replaces guessing with measurement: compile once with instrumentation that counts how often each branch and edge is taken, run a workload, then compile again handing the compiler the real counts so its layout decisions are evidence-based.

flowchart LR
    A["C sources<br/>(ceval.c, object.c, ...)"] --> B["Build #1: instrumented<br/>-fprofile-generate"]
    B --> C["Instrumented python<br/>(slow, counts branches)"]
    C --> D["Run PROFILE_TASK<br/>(-m test --pgo)<br/>~40 curated test modules"]
    D --> E["Profile data<br/>(.gcda / .profraw counts)"]
    E --> F["Build #2: optimized<br/>-fprofile-use + LTO"]
    F --> G["Final python<br/>hot paths laid out optimally"]
    A --> F
    style D fill:#dae8fc
    style E fill:#ffe6cc
    style G fill:#d5e8d4

Figure: the three-phase PGO build pipeline. The insight is that the workload in the blue box defines what “hot” means — the compiler only optimizes for paths the profiling run actually exercised, so a profile that misses your usage pattern yields no benefit there (and can mildly pessimize cold paths). CPython ships a curated representative workload precisely so the profile generalizes to typical interpreter use.

How CPython’s PGO Build Actually Works

The configure flags

The CPython devguide’s standing recommendation for a fast build is exactly two flags together (devguide):

./configure --enable-optimizations --with-lto
make -j$(nproc)
  • --enable-optimizations turns on PGO. The configure documentation describes it precisely: “Enable Profile Guided Optimization (PGO) using PROFILE_TASK (disabled by default)” (configure docs). “Disabled by default” is the key clause — a plain ./configure build is not PGO-optimized, which is why development builds are quick and distribution builds are slow.
  • --with-lto turns on Link-Time Optimization (LTO): “Enable Link Time Optimization (LTO) in any build (disabled by default)” (configure docs). LTO defers a second round of optimization to link time, when the whole program is visible at once, so the optimizer can inline and propagate constants across .c file boundaries that ordinary per-translation-unit compilation cannot cross. It accepts --with-lto=[full|thin|no|yes]; ThinLTO support was added in 3.11, and since 3.12 ThinLTO is the default policy on Clang when the compiler accepts the flag (configure docs).

PGO and LTO are complementary, not redundant: PGO tells the compiler where the hot code is; LTO widens the scope over which it may act on that knowledge. Used together they compound.

There is a third, easily-missed effect bundled into --enable-optimizations. The configure docs note that it will also “disable semantic interposition in libpython if --enable-shared and GCC is used” — adding -fno-semantic-interposition to the compiler and linker flags (configure docs). This matters more than it looks. When CPython is built as a shared library (libpython3.14.so), the ELF dynamic-linking rules normally permit symbol interposition: a symbol exported by the library can be preempted — replaced at load time — by a definition from another object loaded earlier (this is what LD_PRELOAD exploits). Because the compiler must assume any exported function might be swapped out underneath it, it conservatively refuses to inline calls between functions inside libpython, and it cannot turn those calls into direct relative jumps. -fno-semantic-interposition tells GCC “no exported symbol in this library will be interposed,” which unlocks cross-function inlining and direct calls within libpython — a real win precisely because the interpreter’s hot loop calls many small internal helpers. The flag is GCC-specific here because Clang already does not apply semantic interposition by default.

The instrumentation → profile → recompile cycle

Mechanically, make under --enable-optimizations does three passes, mirroring the standard GCC/Clang PGO workflow:

  1. Instrumented build. Compile the C sources with -fprofile-generate (GCC) / -fprofile-instr-generate (Clang). The compiler injects counter increments at every branch and call edge, so the resulting python is slower than normal — it is doing extra bookkeeping on every basic-block transition. This build is throwaway; its only purpose is to be run.
  2. Training run. Execute the workload named by the PROFILE_TASK make variable. Its documented default is -m test --pgo --timeout=$(TESTTIMEOUT) (configure docs) — run the standard regression-test suite in its PGO mode. As the instrumented interpreter runs, the counters accumulate into on-disk profile files, and here the two toolchains diverge concretely. GCC emits one .gcda file per compiled object (the same gcov data format used for coverage), written out at process exit. Clang writes raw .profraw files that must then be merged and indexed by the llvm-profdata merge tool into a single .profdata file — which is exactly why the configure docs state that “the C compiler Clang requires llvm-profdata program for PGO” (and note that on macOS GCC is just an alias to Clang, so it needs it too) (configure docs).
  3. Optimized rebuild. Recompile the same sources from scratch with -fprofile-use (plus -fprofile-correction on GCC, which reconciles counter inconsistencies that arise from multithreaded execution during the training run, and plus LTO if requested), feeding the collected profile back. Armed with real counts the compiler now orders basic blocks hot-first, inlines hot callees, splits hot and cold code into separate regions, and emits branch hints that match observed reality instead of static heuristics.

The function that benefits most is the interpreter’s bytecode-dispatch routine in Python/ceval.c — the single hottest function in the entire interpreter, since every executed opcode passes through it. PGO lets the compiler lay out that giant function’s basic blocks so the common opcodes sit on the fall-through hot path and the rare ones are pushed to cold regions, and it informs the branch predictor’s static hints. This is also why PGO interacts so tightly with how the dispatch loop is written: the computed-goto vs switch vs tail-call structure of the loop (see Instruction Dispatch and Computed Gotos and The Tail-Call Interpreter) changes how much room PGO has to help — a point that became famous when an LLVM 19 layout regression in the computed-goto build was mistaken for a tail-call speedup (recounted in The Faster CPython Project).

What the training workload actually is

The brief’s hint that the profile workload is “pyperformance-derived” is wrong, and worth correcting precisely because it is plausible. The --pgo workload is a curated subset of CPython’s own standard test suite, defined in Lib/test/libregrtest/pgo.py as the PGO_TESTS list (pgo.py @ v3.14.5). It is roughly 40 test modules — test_array, test_base64, test_binascii, test_json, test_pickle, test_re, test_sqlite3, test_xml_etree, test_hashlib, and similar — chosen because they either exercise commonly-used C extension modules/types or run typical Python code, with long-running tests deliberately excluded since instrumentation already slows execution. There is no pyperformance dependency anywhere in this path. (pyperformance is the suite used to measure CPython’s speed for the Faster CPython project; it is not what trains the PGO profile.) The setup_pgo_tests() function supplies this default list whenever --pgo is passed without an explicit test set.

Verified from the git history (2026-06-01)

The “it used to be broader” framing is confirmed by the primary commit. Before GH-14702 / bpo-36044, “Reduce number of unit tests run for PGO build” (merged 2019-07-22, first shipped in CPython 3.8), the PGO generation task ran the full unit test suite; the commit message states it “Reduce[s] the number of unit tests run for the PGO generation task… speeds up the task by a factor of about 15x. Running the full unit test suite is slow,” and notes the old broad behaviour can still be restored via ./configure [..] PROFILE_TASK="-m test --pgo-extended". That commit also introduced Lib/test/libregrtest/pgo.py (the curated PGO_TESTS list). The current default and the curated list are likewise primary-verified (configure docs; pgo.py at v3.14.5).

The Speedup — Cite Carefully, Because the Docs Don’t

This is the figure the brief told me to verify hardest, and the honest finding is that the official CPython documentation gives no number at all. The devguide says only that optimized builds “can take a lot longer to build… and it’s usually not necessary,” and that it’s “essential if you want accurate benchmark results” — no percentage (devguide). The configure docs likewise quantify nothing for --enable-optimizations.

Uncertain

Verify (as of 2026-06-01): the “~10–20% faster” speedup commonly attributed to a PGO+LTO CPython build. Reason: no primary CPython source (devguide, configure docs, What’s New) states a percentage — re-confirmed this session that the devguide and configure docs quantify nothing. The figures available are all secondary and vary by methodology: ActiveState measured ~10% on CPU-bound code (13% on pybench, 10% on pyDes, but notes pybench is “the best case” since the optimization derives from it) (ActiveState); a nixpkgs commit reported a ~16% median across the pyperformance suite (nixpkgs commit). Pinning a figure requires a benchmark I cannot run in this environment — building CPython twice (plain vs --enable-optimizations --with-lto) on one machine and running pyperformance. Until then, treat “10–20%” as a secondary-sourced ballpark, not an official figure. uncertain

The defensible statement is therefore qualitative-with-a-range: a PGO+LTO build is meaningfully faster than a default build — secondary measurements cluster around ~10% for typical CPU-bound Python, with suite medians reported up to the mid-teens percent — but the exact number depends on workload, platform, and compiler, and CPython upstream declines to publish a single figure. The reason for the variance is structural: PGO only speeds the paths the training run exercised, so how much your program benefits depends on how well its hot paths overlap the PGO_TESTS profile.

The Build-Time Cost

The price of PGO is wall-clock build time, and it is steep because the interpreter is effectively built twice (instrumented, then optimized) with a full training run in between, and LTO adds a heavy whole-program optimization pass at link. The devguide’s warning is explicit: “It can take a lot longer to build CPython with optimizations enabled, and it’s usually not necessary to do so” (devguide). In practice this means an optimized build can run many minutes to tens of minutes against seconds-to-a-couple-minutes for a plain build — which is exactly why the workflow is: plain ./configure for day-to-day development and debugging, PGO+LTO only for release binaries and for benchmarking a proposed optimization. Distribution packagers (Fedora, Debian, the official python.org installers, Docker python images) ship PGO+LTO builds so end users get the fast binary without paying the build cost themselves.

How This Stacks With the Python-Level Work

PGO is orthogonal to everything in The Faster CPython Project, and the layers multiply rather than overlap:

  • PGO/LTO optimizes the C machine code of the interpreter binary — better branch layout in ceval.c’s dispatch loop, better inlining of pymalloc and refcount paths.
  • The specializing adaptive interpreter (The Specializing Adaptive Interpreter) optimizes the bytecode the interpreter executes, at Python runtime, regardless of how the C was compiled.
  • The JIT (The CPython JIT Compiler) emits fresh machine code for hot traces at runtime.

There is even a fourth C-level layer beyond PGO and LTO: BOLT (Binary Optimization and Layout Tool), a post-link optimizer that rewrites the already-linked binary using a second profile collected from the finished executable. CPython gained experimental BOLT support around 3.12, where the release notes credit it with a further 1–5% on top of PGO+LTO (3.12 What’s New). The progression PGO → LTO → BOLT is a sequence of ever-later optimization opportunities (compile time → link time → post-link), each squeezing a bit more from the same C sources without touching a line of Python.

A concrete intersection: the 3.14 tail-call interpreter’s release notes state that “enabling profile-guided optimization is highly recommended when using the new interpreter as it is the only configuration that has been tested and validated for improved performance” (3.14 What’s New). In other words, the Python-level dispatch redesign and the build-time C optimization are tuned together — the tail-call design assumes the compiler will be given a profile to lay out its many small opcode functions well. That is the clearest illustration that PGO is not a competitor to the Faster CPython work but a foundation it builds on.

Common Misunderstandings

  • “PGO optimizes my Python code.” No. It optimizes the compiled C interpreter. Your bytecode is unchanged.
  • “./configure already does PGO.” No — PGO is disabled by default; you must pass --enable-optimizations (configure docs).
  • “The profile is built from pyperformance.” No — it’s the PGO_TESTS subset of the standard regression suite (pgo.py).
  • “LTO and PGO are the same thing.” No — PGO supplies profile data; LTO widens the scope of optimization to whole-program at link time. They are independent flags that compound.
  • “I should always build with --enable-optimizations.” Not for development — the doubled build time and slower-than-normal instrumented intermediate stage make it counterproductive unless you are producing a release binary or benchmarking.

See Also

Graph View