vault

Tier 2 IR and the Micro-Op Optimizer

14 min read

Tier 2 IR and the Micro-Op Optimizer

Above CPython’s tier-1 bytecode — the specialized, adaptive bytecode that the main interpreter executes — sits a tier-2 intermediate representation (IR) made of micro-operations (micro-ops, written µops or uops). When a stretch of tier-1 code runs hot, CPython projects it into a flat, branch-free linear sequence of uops called a trace, wraps that trace in an executor object, and runs an optimizer (Python/optimizer.c, with the abstract-interpretation passes generated from Python/optimizer_bytecodes.c) over it. The optimizer performs data-flow analysis to remove redundant guards and _CHECK_* ops, constant-fold pure operations, and narrow types, producing a leaner uop trace. That optimized trace is then either compiled to machine code by the copy-and-patch JIT (PEP 744) or, when the JIT is off, executed directly by the tier-2 micro-op interpreter. The whole tier-2 machinery has shipped disabled by default in every build since Python 3.13, and the JIT remains experimental and off by default in 3.14 (PEP 744; What’s New 3.14).

Resolved (2026-06-01)

The identifiers below are confirmed present at the v3.14.5 tag: uop_optimize, _Py_uop_analyze_and_optimize, translate_bytecode_to_trace, make_executor_from_uops, _PyOptimizer_Optimize, BRANCH_TO_GUARD, ENTER_EXECUTOR, and MAX_CHAIN_DEPTH were re-grepped in Python/optimizer.c v3.14.5 (MAX_CHAIN_DEPTH is #defined to 4 there); the struct/type names _PyExecutorObject, _PyUOpExecutor_Type, _PyUOpInstruction, and _PyExitData are confirmed in Include/internal/pycore_optimizer.h v3.14.5. The three optimization passes (remove_globals, optimize_uops, remove_unneeded_uops) are named in cpython#133171. These remain private internals with no API stability and may move in later releases — re-verify at the tag you build.

Where Tier 2 Sits — the Two-Tier Execution Model

CPython’s modern execution engine, the product of the Faster CPython effort, is organized into two tiers, a structure borrowed from JIT-compiled language runtimes (PEP 744; pydevtools, “What is CPython’s JIT Compiler?”).

Tier 1 is the ordinary bytecode interpreter you already know — the loop in _PyEval_EvalFrameDefault that fetches a bytecode, dispatches to the C code implementing it, and repeats (see The CPython Evaluation Loop). Since Python 3.11 this tier is the specializing adaptive interpreter (PEP 659): it watches the types flowing through each instruction and rewrites generic opcodes in place into type-specialized variants — BINARY_OP becomes BINARY_OP_ADD_INT once it sees two integers — backed by inline caches. Tier 1 is where all Python runs by default; tier 2 is an optional accelerator on top.

Tier 2 is a different representation of the same program, used only for code that has proven hot. A specialized tier-1 instruction like BINARY_OP_ADD_INT is not one atomic action — under the hood it is a guard (“are both operands really int?”) followed by work (“add them”) followed by bookkeeping (reference counts, the instruction pointer). Tier 2 makes those sub-steps explicit: each one becomes a separate micro-op. So a single tier-1 specialized opcode expands into several tier-2 uops, e.g. _GUARD_BOTH_INT, then _BINARY_OP_ADD_INT, then a _CHECK_* / _SET_IP housekeeping op (PEP 744, which describes “translation … into micro-ops”). The point of breaking opcodes apart is that the smaller, simpler pieces are far easier for an optimizer to reason about and eliminate.

flowchart TD
    A["Tier 1: specialized adaptive bytecode<br/>(BINARY_OP_ADD_INT, ...)"] -->|"hot loop detected<br/>(JUMP_BACKWARD counter)"| B["Projection / trace builder<br/>expand opcodes → linear uop trace"]
    B --> C["Raw tier-2 uop trace<br/>_GUARD_BOTH_INT, _BINARY_OP_ADD_INT, _CHECK_*, ..."]
    C --> D["Micro-op OPTIMIZER<br/>(uop_optimize → _Py_uop_analyze_and_optimize)<br/>remove redundant guards, const-fold, narrow types"]
    D --> E["Optimized uop trace in an EXECUTOR"]
    E -->|"JIT on"| F["Copy-and-patch JIT → machine code (PEP 744)"]
    E -->|"JIT off"| G["Tier-2 micro-op interpreter"]

Diagram: the tier-2 pipeline. Hot tier-1 code is projected into a linear uop trace, optimized, stored in an executor, and then either JIT-compiled or interpreted. The insight: the optimizer and the trace are shared — the JIT is just one of two possible back-ends for the same optimized tier-2 IR. Turning the JIT off does not turn tier 2 off.

Mechanical Walk-through — From Hot Loop to Optimized Trace

1. Hotness detection and projection. Tier-1 instructions that close a loop — chiefly JUMP_BACKWARD — carry a counter, managed via a backoff scheme (initial_temperature_backoff_counter() in the source). When it expires, the runtime decides the loop is hot and _PyOptimizer_Optimize builds a trace starting there. The trace builder translate_bytecode_to_trace projects tier-1 bytecode into tier-2 by walking instructions and, for each, emitting its constituent uops; it follows the predicted path (e.g. it assumes a branch goes the way it has been going), turning control flow into a straight-line superblock terminated by guards that bail out (“side-exit” / deopt) if reality diverges from the prediction. Conditional branches are rewritten into typed guard uops via a static BRANCH_TO_GUARD lookup table, so a two-way branch becomes “assume the predicted side, guard, continue.” The trace is bounded — the projector caps the chain length (MAX_CHAIN_DEPTH) and special-cases the first instruction with a comment that the first and the MAX_CHAIN_DEPTH’th executor “must make progress in order to avoid infinite loops or excessively-long side-exit chains” (all per the raw Python/optimizer.c v3.14.5). The decision to predict using a CONFIDENCE_CUTOFF (set to 333, confirmed at v3.14.5) keeps low-confidence branches from being baked into a trace.

2. The executor. The trace is built by translate_bytecode_to_trace and turned into an executor object by make_executor_from_uops (both in Python/optimizer.c); the object type is _PyExecutorObject, with the uop flavor _PyUOpExecutor_Type. Each uop in it is a fixed-size record (_PyUOpInstruction) holding an opcode, an oparg, a target back-pointer to the originating tier-1 instruction (needed to deopt correctly), and inline operand fields used to carry cached values from the tier-1 inline caches into the trace (the substrate for constant propagation); side-exit metadata lives in _PyExitData (raw optimizer.c v3.14.5). The executor is then installed at the hot tier-1 instruction: the original opcode is replaced by an ENTER_EXECUTOR op that, when reached, jumps into the trace instead of continuing tier-1; the whole-program direction of “all executors execute tier-2 micro-ops, run by the tier-2 interpreter or the JIT” is laid out in the design issue faster-cpython#113860.

Resolved (2026-06-01)

The shipped _PyUOpInstruction is 24 bytes in a standard build (32 bytes with Py_STATS), not the “32-bit” design-note figure nor the “~16 bytes” sometimes quoted. Per Include/internal/pycore_optimizer.h v3.14.5 (struct at lines ~51–67): an 8-byte header (uint16_t opcode:15 + format:1, uint16_t oparg, then a 4-byte union of uint32_t target or {uint16_t jump_target; uint16_t error_target}), followed by two uint64_t cache fields operand0/operand1 (16 bytes). A third uint64_t execution_count is compiled in only under #ifdef Py_STATS, taking the size to 32 bytes in stats builds. The early “32-bit instruction” proposal (faster-cpython#580) was a starting point, not what shipped.

3. Optimization — the heart of this note. Before the executor is used, the uop optimizer runs over the raw trace. The entry point is uop_optimize, which calls _Py_uop_analyze_and_optimize; that analyzer is composed of three passes named in CPython’s own thread-safety triage — remove_globals, optimize_uops, and remove_unneeded_uops (cpython#133171). The core pass optimize_uops is an abstract interpreter: it walks the uops symbolically, tracking — instead of concrete runtime values — what it can prove about each value on the stack: its type, and where possible its constant value (Jin, “uops optimizer” notes). The uop DSL classifies every uop as pure (no side effects, deterministic), guard (a runtime check that can deopt), or impure. Three optimizations fall out of this analysis:

  • Guard / _CHECK_* elimination. A guard is redundant if the property it checks has already been proven upstream. After a _GUARD_BOTH_INT, the abstract interpreter records that those two stack values are int; any later _GUARD_BOTH_INT on values it can prove are the same (and thus still int) is deleted: “We eliminate guards when we see they are redundant using constant or type information” (Jin’s notes). The canonical example is a hot loop body like y + y; x + x where x = y: the first add’s _GUARD_BOTH_INT proves int-ness, value-numbering proves x and y are the same value, so the second add’s guard is removed entirely. Because a guard is the only thing standing between an interpreted check and straight-line work, removing guards is where most of the tier-2 win lives.
  • Constant propagation / folding. For pure uops whose inputs are known constants, “the abstract interpreter automatically evaluates the bodies of pure instructions with their constant values” (Jin’s notes) — the operation is computed at optimization time and replaced by its result, so it costs nothing at run time.
  • Type narrowing / specialization without re-checking. Types are “propagated forwards,” so once a value is known to be a specific type the trace can use the type-specialized uop without re-guarding (Jin’s notes). The JIT additionally uses this proven liveness/type information for reference-count elimination (skipping refcount churn when a value is provably alive for the trace’s duration) and register allocation of short-lived values (pydevtools).

A worked intuition: a tier-1 hot loop adding integers looks, in raw tier-2, like a repeating _GUARD_BOTH_INT → _BINARY_OP_ADD_INT → housekeeping per iteration. After optimization, the redundant per-iteration _GUARD_BOTH_INTs collapse to (ideally) a single guard hoisted toward the top, the integer-typed nature of the accumulator is propagated, and refcount bookkeeping the optimizer can prove unnecessary is dropped. The loop body shrinks to nearly pure arithmetic — and that is what the back-end turns into fast code.

4. Execution — two back-ends for the same IR. The optimized executor is run in one of two ways. If CPython was built with the JIT (--enable-experimental-jit), the copy-and-patch JIT turns each uop into a precompiled machine-code stencil (compiled by LLVM at CPython build time, not at your program’s runtime) and patches in runtime addresses/constants, producing native code for the trace (PEP 744; pydevtools). If the JIT is off but tier 2 is on, the tier-2 micro-op interpreter (Python/executor_cases.c.h, generated from the same DSL) executes the uops one by one — slower than native code but still benefiting from the optimizer’s guard removal (faster-cpython#580). Crucially, the JIT compiles the optimized tier-2 trace, not raw tier-1 bytecode — it sits “somewhere between the ‘baseline’ and ‘optimizing’ compiler tiers” of other runtimes (PEP 744).

The 3.14 Status — Pin It Carefully

This is the part most secondary write-ups get wrong, so be precise (all as of CPython 3.14.x, 2026-05-30):

  • Tier-2 machinery is built into every CPython, but disabled by default. “All CPython builds have included this exact micro-op translation, optimization, and execution machinery” since 3.13, “though it remains disabled by default” (PEP 744). It is not a separate download; it is dormant code.
  • The JIT is experimental and off by default — and likely to stay opt-in. PEP 744: “The JIT is currently not part of the default build configuration, and it is likely to remain that way.” It becomes non-experimental only after clearing criteria including “a meaningful performance improvement for at least one popular platform (realistically, on the order of 5%)” (PEP 744).
  • Performance today is roughly break-even. The JIT is “about as fast as the existing specializing interpreter on most platforms,” at “about 10–20% more memory” (PEP 744). The value right now is the infrastructure, not a headline speedup.
  • Binary availability. 3.14’s Windows and macOS binary releases now ship support for the experimental JIT (What’s New 3.14); enabling it from source uses --enable-experimental-jit (values below).
  • Incompatible with free-threading in 3.14. The JIT is not yet thread-safe, so the free-threaded build (--disable-gil) and the JIT cannot run together. Before this was fixed, configuring both would build a JIT that, in the maintainers’ words, “is built, but never actually used”; CPython therefore made configure error out when both are requested. That change — PR GH-133179, “Prevent combinations of --disable-gil and --enable-experimental-jit (for now)” — was merged 2 May 2025, before the 3.14.0 release, so the configure-time error ships in 3.14. Making the JIT thread-safe is tracked for 3.15 (cpython#133171, “Make the JIT thread-safe”). See Free-Threaded CPython.

Building and Toggling Tier 2

# Build with the JIT enabled (Tier 2 + copy-and-patch). Needs LLVM/Clang for build.
./configure --enable-experimental-jit
make -j
 
# Build the JIT but leave it OFF unless PYTHON_JIT=1 at runtime:
./configure --enable-experimental-jit=yes-off
 
# Build ONLY the tier-2 micro-op interpreter (no native codegen) — for debugging tier 2:
./configure --enable-experimental-jit=interpreter

Line-by-line, from the official configure docs (configure docs): --enable-experimental-jit (≡ =yes) builds and enables the JIT, runtime-toggleable with PYTHON_JIT=0; =yes-off builds it but defaults off (opt in with PYTHON_JIT=1); =interpreter enables the “JIT interpreter” — i.e. the tier-2 micro-op interpreter without machine-code generation, described as “only useful for those debugging the JIT itself”; =no (the default) builds neither. This interpreter value is exactly the “tier 2 on, JIT off” configuration where the micro-op interpreter executes optimized uop traces.

Failure Modes and Misunderstandings

  • “The JIT compiles my bytecode.” No — it compiles the optimized tier-2 uop trace of hot code, after projection and optimization. Cold code never leaves tier 1.
  • “Tier 2 == the JIT.” They are different layers. Tier 2 is the IR and its optimizer; the JIT is one back-end for it. --enable-experimental-jit=interpreter gives you tier 2 with no JIT. Conflating them leads people to think turning the JIT off disables all the tier-2 work — it doesn’t.
  • “It’ll speed up my code a lot in 3.14.” As of 3.14 the JIT is roughly break-even with the specializing interpreter and uses more memory; it is shipped for evaluation, not as a free win (PEP 744).
  • “I can combine the JIT with the free-threaded build.” Not in 3.14 — configure rejects it; planned for 3.15 (cpython#133171).
  • Deopt / side-exit surprises. Because traces are built on predicted paths, a value that suddenly changes type (the guard fails) forces a deopt back to tier 1 at the guard’s target. Pathologically polymorphic hot loops can thrash between trace-and-deopt and see little benefit — the trade-off of trace-based specialization (the _PyExitData / side-exit machinery in optimizer.c v3.14.5).

Alternatives and When to Choose Them

Tier 2 is one layer of the Faster CPython stack and rarely the right first lever:

  • Tier-1 specialization (PEP 659, on by default since 3.11) — already gives type-specialized bytecode with zero configuration; most real speedups today come from here, not tier 2.
  • Tail-call interpreter (3.14, opt-in) — an orthogonal dispatch-layer optimization of tier 1; it speeds the loop, it is not a trace optimizer. It shares only a musttail/Clang dependency with the JIT.
  • PyPy’s tracing JIT — a far more mature, meta-tracing JIT delivering much larger speedups on long-running pure-Python workloads; the relevant alternative implementation when raw Python throughput dominates (see CPython vs PyPy and Alternative Implementations).
  • Native extensions / Cython / NumPy — for numeric hot loops, dropping to C-level code still beats any CPython-3.14 JIT result.

Choose to enable CPython’s tier-2 JIT today mainly to experiment, benchmark, and report back — its purpose in 3.14 is to mature the infrastructure.

Production Notes

As of CPython 3.14.x (2026-05-30), tier 2 and its JIT are experimental, opt-in, and roughly performance-neutral; the realistic posture is “interesting to track, not yet a production speed lever.” The architecture is the strategic bet: a copy-and-patch JIT generated from the same DSL as the interpreter means CPython gets JIT support for a new opcode “for free” whenever the bytecode definition changes, lowering the maintenance cost that has historically sunk Python JIT efforts (PEP 744). A known operational caveat: “profilers and debuggers for C code are currently unable to trace back through JIT frames,” so native-level debugging of JIT-compiled traces is degraded (PEP 744). The free-threading incompatibility is the most consequential near-term limitation — the two flagship Faster-CPython directions (no-GIL parallelism and the JIT) do not yet compose, and reconciling them is 3.15 work (cpython#133171).

See Also

Graph View

Backlinks