Python Abstract Syntax Tree

The Abstract Syntax Tree (AST) is CPython’s structured, in-memory representation of a parsed program: every statement, expression, and operator becomes a typed node in a tree. The PEG parser builds it directly from the token stream, and it is the AST — not the source text — that the rest of the compiler consumes: symbol-table construction walks it to classify names, and the code generator walks it to emit bytecode. The shape of every node — its name, its child fields, their types — is defined declaratively in one file, Parser/Python.asdl, written in the Abstract Syntax Description Language (ASDL), from which a code generator produces both the C structs the parser fills in and the Python classes the ast standard-library module exposes (docs).

This note describes what the tree is and how to manipulate it. Its sibling, The PEG Parser, describes how the tree gets built (the grammar actions like { _PyAST_BinOp(...) } that call the node constructors). For the phases on either side, see Python Tokenizer and Bytecode Compilation.

Mental Model

Think of the AST as the program with all syntactic noise stripped away, leaving only semantic structure. Parentheses, whitespace, comments, and the order in which you wrote things are gone; what remains is “this is an assignment, whose target is the name x in Store context, and whose value is a binary-add of the names a and b.” It is abstract precisely because it discards the concrete surface form — unlike a concrete syntax tree (CST), it has no node for “the ( token.” Two source lines that mean the same thing (a+b and a + b) produce identical ASTs.

flowchart TD
    ASDL["Parser/Python.asdl<br/>(ASDL grammar — node definitions)"]
    ASDL -->|"make regen-ast<br/>via asdl_c.py"| CSTRUCT["C: pycore_ast.h structs<br/>+ _PyAST_* constructors"]
    ASDL -->|"asdl_c.py also emits<br/>Python classes"| PYCLASS["Python: ast.Module, ast.BinOp,<br/>ast.Name, ... (the ast module)"]
    PARSER[["PEG parser<br/>(The PEG Parser)"]] -->|"actions call<br/>_PyAST_BinOp(...)"| TREE
    CSTRUCT -.defines.-> TREE["AST instance:<br/>Module(body=[Assign(...)])"]
    TREE -->|consumed by| SYM["Symbol Table Construction"]
    SYM --> GEN["Bytecode Compilation"]

Figure: one declarative file, Python.asdl, is the single source of truth. The asdl_c.py generator turns it into both the C node structs (filled in by the parser) and the Python ast.* classes (for introspection). The parser builds a tree of these nodes; symbol-table construction and code generation then consume it. The insight: node types are never hand-written twice — change the ASDL, regenerate, and both the C side and the Python side stay in lockstep.

ASDL — The Definition Language

ASDL is a tiny domain-specific language for describing tree-shaped data. CPython’s entire node taxonomy lives in Parser/Python.asdl. Its four built-in primitive types are declared at the top (verbatim): identifier, int, string, constant. Everything else is built from sum types (a choice of constructors, like a tagged union) and product types (a single record of fields).

A sum type lists alternative constructors separated by |. The top-level mod type — the four possible roots of a tree — is the simplest example (verbatim from Python.asdl):

mod = Module(stmt* body, type_ignore* type_ignores)
    | Interactive(stmt* body)
    | Expression(expr body)
    | FunctionType(expr* argtypes, expr returns)

Reading it: mod is one of Module, Interactive, Expression, or FunctionType. A Module (the root for normal exec-mode parsing) has a field body that is a list of stmt (the * means “zero or more”), plus type_ignores. An Expression (the root for eval mode) has a single expr body. Each constructor name (Module, BinOp, Name…) becomes a concrete node class; each left-hand-side type name (mod, stmt, expr…) becomes an abstract base class. So ast.BinOp is a subclass of ast.expr, which is a subclass of ast.AST (docs).

Field syntax encodes cardinality:

expr value — exactly one (required).
expr? value — optional; defaults to None if absent.
expr* values — a list (possibly empty).
expr?* keys — a list whose elements may individually be None (used by Dict, where {**x} has a None key).

The expr sum type is the workhorse — the operators you write every day. An excerpt (verbatim):

    expr = BoolOp(boolop op, expr* values)
         | NamedExpr(expr target, expr value)
         | BinOp(expr left, operator op, expr right)
         | UnaryOp(unaryop op, expr operand)
         | Lambda(arguments args, expr body)
         | IfExp(expr test, expr body, expr orelse)
         | Dict(expr?* keys, expr* values)
         ...
         | Call(expr func, expr* args, keyword* keywords)
         | Constant(constant value, string? kind)
         | Attribute(expr value, identifier attr, expr_context ctx)
         | Name(identifier id, expr_context ctx)

So BinOp has three fields in order — left, op, right — where op is itself a small sum type, operator = Add | Sub | Mult | MatMult | Div | Mod | Pow | LShift | RShift | BitOr | BitXor | BitAnd | FloorDiv. These operator constructors are leaf nodes with no fields: ast.Add() is a complete, empty node that just tags which arithmetic operation a BinOp represents. The other tiny sum types follow the same shape: boolop = And | Or, unaryop = Invert | Not | UAdd | USub, cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn, and expr_context = Load | Store | Del.

The expr_context deserves attention because it surfaces in every example: a Name node carries a ctx saying whether the name is being read (Load), assigned (Store), or deleted (Del). x on the right of = is Name(id='x', ctx=Load()); x on the left is Name(id='x', ctx=Store()). The parser computes this from position.

Product types have no constructor name — they are bare records, used where there is only one shape. Examples (verbatim):

    comprehension = (expr target, expr iter, expr* ifs, int is_async)
    arguments = (arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs,
                 expr?* kw_defaults, arg? kwarg, expr* defaults)
    keyword = (identifier? arg, expr value)
    alias = (identifier name, identifier? asname)
    withitem = (expr context_expr, expr? optional_vars)

A product type is “just one shape,” so it needs no tag. The arguments record above is worth seeing in a real tree, because it is where the sum-vs-product distinction becomes tangible and where a function’s whole signature is encoded. Parsing def f(a, b=1, *args, **kw): pass on CPython 3.14.5 yields (real ast.dump output):

Module(
  body=[
    FunctionDef(
      name='f',
      args=arguments(
        args=[
          arg(arg='a'),
          arg(arg='b')],
        vararg=arg(arg='args'),
        kwarg=arg(arg='kw'),
        defaults=[
          Constant(value=1)]),
      body=[
        Pass()])])

Reading it against the ASDL: FunctionDef is a stmt sum constructor, so it prints with its name. Its args field holds a single arguments product node — note it has no alternative constructors, it is the one record shape for a parameter list. Inside, args=[arg(arg='a'), arg(arg='b')] are the two positional parameters (each an arg product node whose own arg field is the identifier); vararg and kwarg hold the *args and **kw parameters; and defaults=[Constant(value=1)] is the single default value 1. The defaults list is shorter than the parameter list — Python stores defaults right-aligned (the last N parameters), so a one-element defaults against two args means only b has a default. The empty fields (posonlyargs, kwonlyargs, kw_defaults) are suppressed by ast.dump’s default show_empty=False. This single tree exercises three product types (arguments, arg, and the implicit Constant) and shows how the parser flattens a rich surface syntax into a uniform node structure.

Attributes — source locations. After a stmt or expr block, ASDL declares attributes(...) — extra fields every node of that type carries beyond its structural children (verbatim): attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset). These are the source positions the parser attaches via the EXTRA macro (see The PEG Parser). A comment in the file pins down the subtle one: “col_offset is the byte offset in the utf8 string the parser uses” — column offsets are UTF-8 byte offsets, not character counts, which matters for non-ASCII source.

Version markers in the 3.14 ASDL. The grammar tracks the language. As of 3.14, expr includes TemplateStr(expr* values) and Interpolation(expr value, constant str, int conversion, expr? format_spec) — the nodes for template strings (t-strings, the t"..." literal), new in 3.14 (docs). The stmt type carries TypeAlias and type_param* fields (PEP 695 generics, 3.12), Match (3.10), and TryStar (except* exception groups, 3.11). The type_param sum type — TypeVar | ParamSpec | TypeVarTuple — was added in 3.12. This is why the AST is version-sensitive: parsing the same source under different Python versions can yield structurally different trees.

From ASDL to Code — `asdl_c.py`

Python.asdl is not used at runtime; it is a build-time input. The generator Parser/asdl_c.py reads it and emits three files, run via make regen-ast (recipe verbatim from Makefile.pre.in):

regen-ast:
	# Regenerate 3 files using Parser/asdl_c.py:
	# - Include/internal/pycore_ast.h
	# - Include/internal/pycore_ast_state.h
	# - Python/Python-ast.c
	$(PYTHON_FOR_REGEN) $(srcdir)/Parser/asdl_c.py \
		$(srcdir)/Parser/Python.asdl \
		-H $(srcdir)/Include/internal/pycore_ast.h.new \
		-I $(srcdir)/Include/internal/pycore_ast_state.h.new \
		-C $(srcdir)/Python/Python-ast.c.new

pycore_ast.h holds the C struct for each node and the _PyAST_* constructor prototypes (e.g. _PyAST_BinOp, which the PEG parser’s grammar actions call). Python-ast.c holds the constructor bodies and the glue that exposes each node as a Python class on the ast module. So the exact same ASDL produces the C representation the parser fills and the Python classes you introspect — they cannot drift apart, because both are generated from one file. Each generated Python class gets a _fields tuple listing its child field names in order; ast.BinOp._fields is ('left', 'op', 'right'), matching the ASDL line above (docs).

The `ast` Module — Building and Inspecting Trees

ast.parse(source, filename='<unknown>', mode='exec', *, type_comments=False, feature_version=None, optimize=-1) is the front door: it runs the same parser the interpreter uses and hands back the tree (docs). The mode selects the start rule and thus the root node type:

'exec' (default) → ast.Module — a sequence of statements (a whole file).
'eval' → ast.Expression — a single expression.
'single' → ast.Interactive — one interactive statement (REPL semantics).
'func_type' → ast.FunctionType — a PEP 484 function-type comment like (int, str) -> bool.

feature_version=(major, minor) parses under an older grammar (e.g. (3, 9) rejects match statements); the minimum supported is (3, 7). type_comments=True enables PEP 484/526 type-comment parsing, populating type_comment fields and a type_ignores list.

The relationship to compile() is direct: ast.parse(src) is essentially compile(src, filename, mode, flags=ast.PyCF_ONLY_AST). The PyCF_ONLY_AST flag tells compile() to stop after building the AST and return it instead of a code object. Conversely, you can pass an AST back to compile() to finish the job: compile(tree, '<ast>', 'exec') runs symbol-table construction and code generation on your (possibly modified) tree and returns a code object you can exec(). 3.13 added PyCF_OPTIMIZED_AST (and optimize > 0 in parse) to return the AST after the constant-folding optimizer has run (docs). See Compiler Optimization Passes.

ast.dump(node, annotate_fields=True, include_attributes=False, *, indent=None, show_empty=False) renders a tree as a readable string — the standard way to see an AST. For example (docs):

>>> import ast
>>> print(ast.dump(ast.parse('x = a + b'), indent=2))
Module(
  body=[
    Assign(
      targets=[
        Name(id='x', ctx=Store())],
      value=BinOp(
        left=Name(id='a', ctx=Load()),
        op=Add(),
        right=Name(id='b', ctx=Load())))])

This dump is the mental model made concrete: the Module root, one Assign statement, its target x in Store context, its value a BinOp of two Load-context names with an Add() operator. include_attributes=True would additionally show the lineno/col_offset positions; show_empty=False (3.13+) suppresses empty optional lists for readability.

Node attributes and construction

Every node subclass exposes _fields (its child names) and, since 3.13, _field_types (a dict mapping each field to its declared type). Nodes are constructed positionally or by keyword: ast.BinOp(left=a, op=ast.Add(), right=b). Location attributes — lineno, col_offset, end_lineno, end_col_offset — are the ASDL attributes(...) made into instance attributes; for a single-line node, source_line[node.col_offset : node.end_col_offset] slices out the exact source span (recall offsets are UTF-8 byte offsets). Since 3.13, omitting a required field on construction raises a DeprecationWarning (to become an error in 3.15); optional fields default to None, list fields to [], and expr_context fields to Load(). In 3.14, __repr__() on a node now includes its field values (docs).

Traversal — `NodeVisitor` and `NodeTransformer`

Two base classes implement the visitor pattern over the tree. ast.NodeVisitor is read-only: its visit(node) method dispatches to a visit_<ClassName> method if you defined one (e.g. visit_Name), otherwise to generic_visit, which recurses into every child. You subclass it to inspect a tree — e.g. collect every name used:

class NameCollector(ast.NodeVisitor):
    def visit_Name(self, node):          # called for every ast.Name node
        print(node.id, type(node.ctx).__name__)
        self.generic_visit(node)         # keep descending into children

ast.NodeTransformer subclasses NodeVisitor to rewrite a tree. Each visit_* method’s return value replaces the visited node: return the node unchanged for no-op, return a new node to substitute, return None to delete it, or return a list (in a statement context) to splice in several nodes. After editing, call ast.fix_missing_locations(tree) so the new nodes inherit source positions from their parents — compile() requires every node to have locations (docs):

class ConstFolder(ast.NodeTransformer):
    def visit_BinOp(self, node):
        self.generic_visit(node)                       # transform children first
        if (isinstance(node.op, ast.Add)
                and isinstance(node.left, ast.Constant)
                and isinstance(node.right, ast.Constant)):
            return ast.Constant(node.left.value + node.right.value)
        return node
 
tree = ast.parse('y = 1 + 2')
tree = ConstFolder().visit(tree)
ast.fix_missing_locations(tree)
exec(compile(tree, '<gen>', 'exec'))                   # y == 3

This is the foundation of real tools: linters (flake8, pylint), formatters, pytest’s assertion rewriting, and macro-like libraries all subclass these.

Other helpers worth knowing

ast.literal_eval(s) — safely evaluate a literal (numbers, strings, tuples, lists, dicts, sets, True/False/None); does not run arbitrary code like eval(), though malformed input can still exhaust the stack (docs).
ast.unparse(node) (3.9+) — regenerate source code from a tree (round-trips, though not byte-for-byte: ast.unparse(ast.parse('x + 1', mode='eval')) yields '(x + 1)').
ast.walk(node) — yield every descendant in unspecified order; ast.iter_child_nodes(node) — direct children only; ast.iter_fields(node) — (name, value) pairs.
ast.copy_location, ast.increment_lineno, ast.get_source_segment, ast.get_docstring.
ast.compare(a, b, *, compare_attributes=False) (3.14+) — structural equality of two trees.

The AST as Compiler Input

The AST is the pivot of the CPython Compilation Pipeline. Once the parser returns it, two phases consume it. First, symbol-table construction walks the tree to discover every name binding and classify each name in each scope as local, global, free (closure), or cell — without executing anything, purely from structure. Then the code generator walks the tree (now annotated by the symbol table) and emits bytecode for a code object, with optimization passes such as constant folding applied along the way. Because every consumer reads the AST and nothing reads the source text again, the AST is the real interface between “your program as written” and “your program as executed.”

Common Misunderstandings

“The AST keeps formatting.” No — it is abstract. Comments, blank lines, and parenthesization are discarded; ast.unparse reconstructs valid source, not your original source. Tools needing to preserve formatting (e.g. black, LibCST) use a concrete syntax tree library instead.

“ast.Num/ast.Str still work.” Removed in 3.14. The old literal node classes ast.Num, ast.Str, ast.Bytes, ast.NameConstant, and ast.Ellipsis were deprecated in 3.8 and removed in 3.14; all literals are now ast.Constant (verified against the 3.14 docs). Correspondingly, the NodeVisitor methods visit_Num/visit_Str/visit_Bytes/visit_NameConstant/visit_Ellipsis are no longer called in 3.14+ — define visit_Constant instead. (ast.Index/ast.ExtSlice were deprecated in 3.9 but, as of 3.14.5, not yet removed.)

“col_offset is a character index.” It is a UTF-8 byte offset, per the ASDL comment. For ASCII source the two coincide; for source with multibyte characters they diverge.

“Editing a node is enough.” After a NodeTransformer inserts nodes, you must ast.fix_missing_locations before compile(), or compilation fails on the missing lineno.

vault

Explorer

Python Abstract Syntax Tree

Python Abstract Syntax Tree

Mental Model

ASDL — The Definition Language

From ASDL to Code — `asdl_c.py`

The `ast` Module — Building and Inspecting Trees

Node attributes and construction

Traversal — `NodeVisitor` and `NodeTransformer`

Other helpers worth knowing

The AST as Compiler Input

Common Misunderstandings

See Also

Graph View

Table of Contents

Backlinks

vault

Explorer

Python Abstract Syntax Tree

Python Abstract Syntax Tree

Mental Model

ASDL — The Definition Language

From ASDL to Code — asdl_c.py

The ast Module — Building and Inspecting Trees

Node attributes and construction

Traversal — NodeVisitor and NodeTransformer

Other helpers worth knowing

The AST as Compiler Input

Common Misunderstandings

See Also

Graph View

Table of Contents

Backlinks

From ASDL to Code — `asdl_c.py`

The `ast` Module — Building and Inspecting Trees

Traversal — `NodeVisitor` and `NodeTransformer`