vault

CPython Compilation Pipeline

13 min read

CPython Compilation Pipeline

Before CPython can run a line of Python, it must compile it. The compilation pipeline is the sequence of passes that turns a stream of source characters into a code object — the immutable container of bytecode that the evaluation loop actually executes. The pipeline has five conceptual stages: tokenize the text into a stream of tokens, parse those tokens (with the PEG grammar) directly into an abstract syntax tree, build symbol tables that classify every name as local/global/free, generate code into a control-flow graph of pseudo-instructions, and finally assemble that graph into a code object holding real bytecode plus its metadata tables. The defining fact — and the thing that surprises people coming from C, Go, or Rust — is that all of this happens lazily at runtime: a module is compiled the first time it is imported (or whenever you call exec/compile/eval), one module at a time, not in an offline build step. This note is the orientation map for §2 of the Python Internals MOC; it frames each stage and hands off the deep mechanics to its sibling notes.

Scope of this note

This is the overview. Each stage has its own deep-dive: the lexer in Python Tokenizer, the grammar engine in The PEG Parser, the tree in Python Abstract Syntax Tree, name classification in Symbol Table Construction, the codegen+optimize+assemble back end in Bytecode Compilation, the peephole and flowgraph passes in Compiler Optimization Passes, the output container in Code Objects, and the .pyc cache in Bytecode Caching and pyc Files. This note explains how they connect and when they run — it does not re-teach any of them.

Mental Model: A Lazy, Per-Module Front End

Think of CPython as carrying its own compiler around inside the interpreter. There is no separate pyc build step you must run; the compiler is invoked on demand, by the import machinery or by the built-in compile(), the instant a piece of source first needs to become executable. The unit of compilation is a code block — most commonly a whole module, but also the body of exec("..."), an expression handed to eval(), or the string compiled by compile(src, "<string>", "exec"). Each block becomes one code object (functions and classes nested inside it get their own nested code objects, compiled in the same pass).

The pipeline is a classic compiler front end with a twist. A C compiler runs lex → parse → semantic-analysis → optimize → codegen ahead of time, emits a machine-code object file, and is never seen again at runtime. CPython runs an analogous sequence — but the “target” is portable bytecode for an abstract stack machine, not native code, and the whole thing fires while your program is starting up. The cost is amortized by caching the result in a .pyc file keyed to the source’s hash or mtime (see Bytecode Caching and pyc Files), so the second import skips straight to loading the cached code object.

graph TD
    SRC["Source text<br/>(.py file / exec / compile string)"] --> TOK["1. Tokenizer<br/>characters → token stream"]
    TOK --> PARSE["2. PEG Parser<br/>tokens → AST (grammar actions<br/>build nodes directly)"]
    PARSE --> AST["mod_ty AST<br/>(Module / Expression / Interactive)"]
    AST --> SYM["3. Symbol Table<br/>classify every name:<br/>local / global / cell / free"]
    SYM --> GEN["4. Code Generation<br/>AST → CFG of basic blocks<br/>of PSEUDO-instructions"]
    GEN --> OPT["4b. CFG Optimizer<br/>peephole, dead-code,<br/>jump threading"]
    OPT --> ASM["5. Assembler<br/>pseudo-ops → real bytecode;<br/>build location + exception tables"]
    ASM --> CODE["PyCodeObject<br/>(co_code, co_consts, co_names,<br/>co_positions, exception table)"]
    CODE --> CACHE[".pyc cache<br/>(skipped on re-import)"]
    CODE --> EVAL["Evaluation loop<br/>(ceval.c)"]

Diagram: the five-stage front end. The boundary that matters for understanding compile.c is the AST node: stages 1–2 (in Parser/) turn text into a tree; stages 3–5 (in Python/) turn that tree into a code object. The compiler’s own internal entry point, _PyAST_Compile, takes the finished AST as input — so “the compiler” in CPython source terms is really stages 3–5, and tokenize+parse are an upstream front-of-front-end. The insight: the AST is the contract between the parser and the compiler proper.

The Five Stages, End to End

Stage 1 — Tokenize

The tokenizer (lexer) reads the raw source — a byte stream that it first decodes according to the file’s encoding declaration — and chops it into a flat sequence of tokens: NAME, NUMBER, STRING, OP, the structural NEWLINE, and the synthetic INDENT/DEDENT tokens that encode Python’s significant whitespace. Indentation handling is the lexer’s signature responsibility: it tracks an indent stack and emits INDENT/DEDENT tokens at block boundaries so the grammar can be written without whitespace-counting rules. The tokenizer also handles line continuations, the f-string sub-tokenization that became part of the formal grammar in 3.12, and reports the precise (line, col) span of each token — spans that propagate all the way to the location table and ultimately to error messages. The full mechanism is in Python Tokenizer.

Stage 2 — Parse to an AST

Since Python 3.9, parsing uses a PEG (Parsing Expression Grammar) parser generated from Grammar/python.gram; the old LL(1) pgen parser was removed in 3.10 (PEP 617, PEP 617). PEG’s ordered-choice and (memoized) backtracking semantics let the grammar express constructs that defeated the old LL(1) parser. Crucially for the pipeline’s shape, the PEG grammar uses grammar actions — embedded C expressions attached to each rule that construct AST nodes directly as the rule matches. There is no separate concrete syntax tree (parse tree) materialized and then walked; the parser emits the abstract syntax tree in one pass (per PEP 617). The output is a mod_ty — one of Module, Interactive, Expression, or FunctionType, depending on the compile mode. The node types are defined in Parser/Python.asdl (ASDL, the Abstract Syntax Description Language) and code-generated into C structs. The deep treatment is in The PEG Parser and the tree itself in Python Abstract Syntax Tree.

AST is also a public API

The same tree is exposed to Python through the ast module: ast.parse(src) runs stages 1–2 and hands you the tree, and compile(tree, ...) will run stages 3–5 on an AST you supply. This is how tools like linters and ast-rewriting decorators hook into the pipeline between parse and codegen.

Stage 3 — Build the Symbol Table

With the AST in hand, the compiler-proper begins. Its first pass is not code generation — it is symbol-table construction (Python/symtable.c, entry point _PySymtable_Build). The compiler walks the entire AST of each block before emitting any instructions, recording every name that is bound (by assignment, def, class, import, parameters, for/with/except targets, the walrus operator, global/nonlocal declarations, etc.) and every name that is merely used. From this it classifies each name into a scope: local, global (explicit or implicit), cell (a local captured by a nested function, requiring a closure cell), or free (a name used here but bound in an enclosing function). This classification is what lets the code generator later choose LOAD_FAST (an array index into the frame’s fast-locals) versus LOAD_GLOBAL versus LOAD_DEREF for the same syntactic name reference. The full mechanism — including why this pass must complete before codegen, and how it implements the static-scoping rules of the Python Execution Model — is in Symbol Table Construction.

There is also a preliminary pass that runs even before the symbol table: the compiler checks for __future__ statements (from __future__ import annotations, etc.), because a future import can change how the rest of the module compiles. The top-of-file comment in Python/compile.c (v3.14.5) lists the post-AST passes explicitly:

The compiler makes several passes to build the code object: 1. Checks for future statements. 2. Builds a symbol table. 3. Generate an instruction sequence. 4. Generate a control flow graph and run optimizations on it. 5. Assemble the basic blocks into final code.

Note that this list — the one inside compile.c — starts after tokenize and parse, because _PyAST_Compile(mod_ty mod, ...) receives the AST as a parameter. The five-stage view at the top of this note is the end-to-end pipeline (tokenize → assemble); the five-pass view in compile.c is the compiler-proper subset (future-check → assemble). They are two valid framings of the same machinery seen from different entry points; conflating them is a common source of confusion.

Stage 4 — Generate Code (Pseudo-Instructions → CFG → Optimize)

The code generator (compiler_codegen and the compiler_visit_* family in Python/compile.c) recursively walks the AST node by node, emitting instructions via macros such as ADDOP, ADDOP_I, and ADDOP_JUMP. But it does not emit final bytecode. It emits pseudo-instructions — an abstract instruction sequence that may reference symbolic jump targets (named labels rather than byte offsets) and may use pseudo-opcodes that have no runtime existence. This abstraction is what makes the next step tractable.

That instruction sequence is then assembled into a control-flow graph (CFG): a set of basic blocks, each a straight-line run of instructions with a single entry and a single exit (a jump or fall-through), connected by edges (_PyCfg_FromInstructionSequence in Python/flowgraph.c). On this graph the compiler runs its optimizations — peephole rewrites, dead-code elimination, jump threading, redundant-load removal (_PyCfg_OptimizeCodeUnit). Working on a graph rather than a flat list is precisely what lets these passes reason about reachability and control flow safely. The catalog of these passes is in Compiler Optimization Passes, and the codegen mechanics in Bytecode Compilation.

Stage 5 — Assemble into a Code Object

Finally the assembler (Python/assemble.c, _PyAssemble_MakeCodeObject) flattens the optimized CFG back into a linear instruction sequence and lowers pseudo-instructions to real bytecode: it resolves every symbolic jump label to a concrete relative byte offset, packs opcodes and arguments into the co_code bytes object (inserting EXTENDED_ARG prefixes for arguments too large for a single byte, and reserving inline CACHE slots for specializable opcodes — see Inline Caches and Quickening), and builds the side tables: the location table (co_linetable/co_positions, the compressed per-instruction (start_line, end_line, start_col, end_col) data from PEP 657, PEP 657) and the exception table (the zero-cost-exceptions side table that maps instruction ranges to handlers — see Zero-Cost Exception Handling). The result is wrapped in a PyCodeObject with all its co_* attributes populated. The orchestration is compiler_mod → _PyCompile_OptimizeAndAssemble → optimize_and_assemble_code_unit (per Python/compile.c v3.14.5). The output container is dissected in Code Objects.

Where It Happens in Source

The arena allocator (Python/pyarena.c) underpins the whole compiler: all AST nodes and intermediate compiler structures are allocated from a single arena, so the entire transient compilation state is freed in one operation once the code object is built. The major files:

StageFile(s)Key entry point
TokenizeParser/ (tokenizer.c, lexer/)tokenizer state machine
Parse → ASTParser/ (generated parser.c from Grammar/python.gram)PEG rule functions
Symbol tablePython/symtable.c_PySymtable_Build
Code generationPython/compile.c (driver), Python/codegen.c (visitors)compiler_codegen (compile.c), codegen_visit_* (codegen.c)
CFG + optimizePython/flowgraph.c_PyCfg_OptimizeCodeUnit
AssemblePython/assemble.c_PyAssemble_MakeCodeObject
DriverPython/compile.c_PyAST_Compile

(File list verified against the v3.14.5 tag and InternalDocs/compiler.md. codegen.c was split out from compile.c during the 3.13 development cycle to separate AST→pseudo-instruction generation from the CFG/assembly back end; in older trees this lived entirely in compile.c.)

Resolved (2026-06-01)

Verified at the v3.14.5 tag. The thin driver compiler_codegen lives in Python/compile.c (line 822) and dispatches into _PyCodegen_Module / _PyCodegen_Expression. The actual AST visitors — codegen_visit_stmt (line 2991), codegen_visit_expr (line 5175), codegen_visit_keyword, codegen_body, and the _PyCodegen_* entry points — all live in Python/codegen.c. compile.c keeps only the driver (_PyAST_Compile, line 1478) and the CFG/assemble back end (optimize_and_assemble_code_unit, line 1412). Note the visitors are named codegen_visit_*, not compiler_visit_*.

A Worked Trace

Consider compiling the one-liner module print(len([1, 2])):

  1. Tokenize → NAME 'print', OP '(', NAME 'len', OP '(', OP '[', NUMBER '1', OP ',', NUMBER '2', OP ']', OP ')', OP ')', NEWLINE, ENDMARKER.
  2. Parse → an Expr statement wrapping a Call(func=Name('print'), args=[Call(func=Name('len'), args=[List([Constant(1), Constant(2)])])]), inside a Module.
  3. Symbol table → print and len are used but never bound in this module. At module top level the compiler emits LOAD_NAME (which checks locals → globals → builtins at runtime); only inside a function body would unbound uses become LOAD_GLOBAL.
  4. Codegen → pseudo-instructions to load the callables, build the list argument, perform the nested calls, discard the result, and return None.
  5. Optimize + assemble → jumps (none here) are resolved and the location/exception tables are built.

The real 3.14.5 output of stage 5 (dis.dis(compile('print(len([1, 2]))', '<s>', 'exec'), show_offsets=True)):

  0          0  RESUME                   0
  1          2  LOAD_NAME                0 (print)
             4  PUSH_NULL
             6  LOAD_NAME                1 (len)
             8  PUSH_NULL
            10  LOAD_SMALL_INT           1
            12  LOAD_SMALL_INT           2
            14  BUILD_LIST               2
            16  CALL                     1
            24  CALL                     1
            32  POP_TOP
            34  LOAD_CONST               1 (None)
            36  RETURN_VALUE

Two things worth noting from the actual output. First, the list literal is not constant-folded: the compiler emits LOAD_SMALL_INT 1, LOAD_SMALL_INT 2, BUILD_LIST 2 because a list is mutable — folding it to a constant would let callers mutate a shared object (the optimizer folds tuple literals and arithmetic on constants, but never a list/set/dict display). Second, co_positions() confirms the location table: the inner len(...) call instruction carries the span (start_line=1, end_line=1, start_col=6, end_col=17) — columns 6–17 of the source, i.e. exactly len([1, 2]). You can watch stages 1–2 with ast.dump(ast.parse(src)) and stage 5’s output with dis.dis(src) — see The dis Module and Bytecode Disassembly and Python Abstract Syntax Tree.

Why Lazy, Per-Module Compilation Matters

Three consequences flow from “compile at runtime, one module at a time”:

  • Import is compilation. The first import foo triggers stages 1–5 for foo.py (unless a fresh .pyc exists). This is why a syntax error in a module you import only surfaces when you import it, not when the interpreter starts — and why import time, not just execution time, includes compilation cost. See The Python Import System.
  • .pyc caching is essential. Because recompiling every module on every run would be wasteful, CPython writes the code object to __pycache__/foo.cpython-314.pyc, validated by either the source’s mtime+size or (with hash-based pyc, PEP 552) its hash. The second import deserializes the cached code object and skips the front end entirely. See Bytecode Caching and pyc Files.
  • compile()/exec()/eval() expose the pipeline directly. compile(src, filename, mode) runs stages 1–5 and returns the code object without executing it; exec(code) and eval(code) then run it. Passing an ast tree instead of a string lets you inject a transformed AST between parse and codegen — the mechanism behind many metaprogramming tools.

Contrast with a C/Go Compiler

AspectC / Go (ahead-of-time)CPython
When compilation runsOffline build stepLazily, at import/exec, per module
OutputNative machine code (.o/binary)Portable bytecode in a PyCodeObject
Whole-program viewLinker sees all unitsEach module compiled in isolation
Optimization scopeAggressive, inter-proceduralModest peephole/CFG passes only
Where speed is recoveredAt compile timeAt run time, by the specializing interpreter and JIT

The key asymmetry: CPython does little optimization in this front-end pipeline on purpose. The heavy lifting happens at runtime in the evaluation loop, which observes actual types and rewrites hot bytecode in place (specialization). The compiler’s job is to produce correct, compact bytecode fast — not optimal bytecode.

See Also

Graph View

Backlinks