vault

Comprehension Scoping

14 min read

Comprehension Scoping

A list, set, or dictionary comprehension introduces a small scope of its own: the loop variable and any name bound inside the comprehension do not leak into the enclosing scope. Writing [x for x in range(3)] at module level leaves no x behind — query 'x' in dir() afterwards and it is False. The language reference states the rule directly: “aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don’t ‘leak’ into the enclosing scope” (per the expressions reference). What changed is how CPython delivers that guarantee. Up to Python 3.11 each comprehension compiled to a hidden nested function with its own frame; since Python 3.12, PEP 709 inlines list/set/dict comprehensions straight into the enclosing code object — no separate function, no extra frame — while preserving the no-leak semantics through a save-and-restore trick built on the [[Fast Locals and the LOAD_FAST Family|LOAD_FAST_AND_CLEAR]] opcode. Generator expressions are deliberately left out of the inlining and still get their own scope.

The Scoping Rule, and Why It Exists

Comprehensions are expressions, but they contain a binding construct — the for x in ... target. The design question Python answered long ago — list comprehensions stopped leaking their target in Python 3 — is: should that x be a variable of the surrounding function, or its own thing? Leaking is hostile. If [x for x in data] overwrote an outer x, then a one-line comprehension dropped into the middle of a function could silently clobber a variable forty lines away, and the bug would be invisible at the call site. So the rule is that the comprehension’s target list and any intermediate names are isolated: they exist only for the duration of the comprehension and vanish afterward, leaving any same-named outer variable untouched.

There is one deliberately carved-out region that is not isolated. The expressions reference is precise: “The iterable expression in the leftmost for clause is evaluated directly in the enclosing scope and then passed as an argument to the implicitly nested scope.” That is, in [f(y) for x in EXPR if cond], the EXPR of the outermost for is evaluated in the enclosing scope (so it can see the enclosing locals normally, and any error in it is raised at the comprehension’s definition point), but everything else — the target x, nested for clauses, the if filter, the result expression — runs in the isolated scope. This split matters for the implementation: the outermost iterable is evaluated before isolation is established.

A quick demonstration of the rule, including that the outer variable’s prior value survives:

>>> x = "outer"
>>> [x for x in range(3)]          # comprehension target also named x
[0, 1, 2]
>>> x                              # outer x is untouched, not clobbered to 2
'outer'
>>> 'tmp' not in dir()             # intermediate names don't leak either
True

The Pre-3.12 Implementation: A Hidden Nested Function

Before Python 3.12, CPython achieved isolation the obvious way: it compiled each comprehension into a hidden anonymous function — a separate code object named <listcomp>, <setcomp>, or <dictcomp> — and called it. The comprehension’s loop variable was simply a local of that function, so it could not possibly leak, for the same reason a local of any function does not leak: it lives in a different frame. The outermost iterable was evaluated in the enclosing scope and passed in as the function’s single argument (conventionally named .0).

This was correct but expensive. Every comprehension, no matter how trivial, paid the full cost of a Python function call: build a function object, build a frame, push it on the call stack, run it, tear it down, return the result. For a comprehension that itself does very little per element, that fixed per-comprehension overhead dominated. The motivation section of PEP 709 quantifies the win from removing it (below). Generator expressions worked — and still work — the same way, because they genuinely need a separate frame: a generator suspends and resumes, so its loop state must live in a frame that outlives each next() call. (Generator expression frame mechanics are covered in Generator and Coroutine Frame Internals.)

The 3.12 Change: PEP 709 Inlining

PEP 709 (“Inlined comprehensions”, Final, Python-Version 3.12) removed the hidden function for list, set, and dict comprehensions. The comprehension’s body is now compiled directly into the enclosing function’s code object — no <listcomp> code object is generated, no separate frame is built, and there is no function call. The loop becomes ordinary inline bytecode in the surrounding function. You can confirm the disappearance of the nested code object directly:

>>> def f(items): return [n * 2 for n in items]
>>> any(hasattr(c, 'co_code') for c in f.__code__.co_consts)
False          # no nested code object — the comprehension body is inline

The performance claim, stated verbatim in the PEP, is: “up to 2x faster for a microbenchmark of a comprehension alone, translating to an 11% speedup for one sample benchmark derived from real-world code that makes heavy use of comprehensions.” The headline “~2×” is the microbenchmark figure — the comprehension in isolation — and the realistic, whole-program figure is far smaller (the cited 11%); quote both honestly rather than implying every comprehension-heavy program halves its time.

graph TD
    subgraph pre["Pre-3.12: hidden function"]
        A["enclosing frame"] -->|"build hidden listcomp func<br/>+ frame, CALL"| B["hidden listcomp frame<br/>n is a local here →<br/>can't leak"]
        B -->|"return list"| A
    end
    subgraph post["3.12+ (PEP 709): inlined"]
        C["enclosing frame"] -->|"LOAD_FAST_AND_CLEAR n<br/>(save + clear outer n)"| D["inline loop bytecode<br/>in the SAME frame"]
        D -->|"STORE_FAST n<br/>(restore outer n)"| C
    end

    style B fill:#fde8e8,stroke:#c0392b
    style D fill:#e8f5e9,stroke:#27ae60

Figure: the same isolation guarantee, two implementations. Pre-3.12 (top) builds a whole nested frame so n is structurally separate — correct but a function call per comprehension. 3.12+ (bottom) runs the loop in the enclosing frame and gets isolation instead by saving and clearing the enclosing n around the loop and restoring it afterward. The insight: PEP 709 trades a structural barrier (a separate frame) for a bookkeeping barrier (save/restore of the one slot), eliminating the call overhead while keeping the observable scoping rule identical.

How Inlining Preserves Isolation: LOAD_FAST_AND_CLEAR

If the comprehension now runs in the enclosing frame and writes the loop variable into one of that frame’s fast-local slots, how does it avoid clobbering a same-named outer local? The answer is a new opcode introduced specifically for this, LOAD_FAST_AND_CLEAR, paired with a restoring store. The PEP describes the pair: isolation “is achieved by the combination of the new LOAD_FAST_AND_CLEAR opcode … which saves any outer value of x on the stack before running the comprehension, and … STORE_FAST, which restores the outer value of x (if any) after running the comprehension.”

LOAD_FAST_AND_CLEAR does two things atomically with respect to the slot: it pushes the slot’s current contents onto the value stack (saving whatever the outer scope had bound to x — or NULL if the name was unbound), then sets the slot to NULL (clearing it, so the comprehension starts with a fresh, unbound x). The comprehension then runs, freely binding and rebinding x in that slot. When the comprehension finishes, a STORE_FAST pops the saved value back into the slot, restoring the outer binding exactly as it was — including restoring it to unbound if it was unbound before (the restoring store is STORE_FAST_MAYBE_NULL in Python/codegen.c, which the disassembler renders as STORE_FAST). Because the save lives on the value stack — not in any named slot — there is nothing for the outer scope to observe.

Here is the actual 3.14 disassembly of a list comprehension, annotated:

def f(items):
    return [n * 2 for n in items]
 12   LOAD_FAST_BORROW      0 (items)
      GET_ITER                          # evaluate outermost iterable in enclosing scope
      LOAD_FAST_AND_CLEAR   1 (n)        # SAVE outer n to stack, CLEAR slot 1
      SWAP                  2
 L1:  BUILD_LIST            0            # accumulator for the result list
      SWAP                  2
 L2:  FOR_ITER             11 (to L3)
      STORE_FAST_LOAD_FAST 17 (n, n)     # n = next item; then load it
      LOAD_SMALL_INT        2
      BINARY_OP             5 (*)
      LIST_APPEND           2
      JUMP_BACKWARD        13 (to L2)
 L3:  END_FOR
      POP_ITER
 L4:  SWAP                  2
      STORE_FAST            1 (n)        # RESTORE outer n from stack
      RETURN_VALUE

Notice three things. First, GET_ITER on items happens before LOAD_FAST_AND_CLEAR n — the outermost iterable is evaluated in the enclosing scope, matching the language rule. Second, the entire loop body is inline in f’s own bytecode; there is no CALL and no <listcomp> code object. Third, the LOAD_FAST_AND_CLEAR 1 (n) at the top and STORE_FAST 1 (n) at the bottom bracket the loop — that pair is the isolation. The SWAPs shuffle the saved value and the partial result around on the stack so the comprehension’s product ends up as the top-of-stack return value while the saved n waits underneath to be restored.

There is also an exception path (the ExceptionTable in real disassembly maps the loop range to a cleanup handler): if the loop raises, the handler still runs the restore so the outer n is put back even when the comprehension blows up — the save/restore is not skipped on error. In Python/codegen.c, restore_inlined_comprehension_locals emits the restoring stores on both the normal-exit and the exception-cleanup paths for exactly this reason.

The Cell Sub-Case

One subtlety connects this note to its sibling Cell Variables and Closures. If the comprehension’s loop variable is itself captured by a function nested inside the comprehension — e.g. [(lambda: row) for row in rows] — then the loop variable must be a cell, not a plain fast local, so the lambda can close over it. The codegen handles this: after LOAD_FAST_AND_CLEAR, it emits MAKE_CELL to box the now-cleared slot for the comprehension, and restores the original (which may itself be a cell) afterward. The source comment in codegen.c spells it out: “in the case of a cell, this will actually push the cell itself to the stack, then we’ll create a new one for the comprehension and restore the original one after.” The disassembly shows the extra MAKE_CELL:

 14   LOAD_FAST_BORROW      0 (rows)
      GET_ITER
      LOAD_FAST_AND_CLEAR   1 (row)      # save the outer slot (possibly a cell)
      MAKE_CELL             1 (row)      # fresh cell for the comprehension's row
      SWAP                  2
      ...
      STORE_DEREF           1 (row)      # loop writes through the cell
      ...                                # lambda closes over that cell

This is also where the loop-variable late-binding surprise surfaces inside a comprehension: [(lambda: row) for row in range(3)] returns three lambdas that all yield 2, because they share the single comprehension row cell. That gotcha is Late Binding Closures; the cell mechanism is Cell Variables and Closures.

The Walrus Exception: := Deliberately Leaks

There is one deliberate hole in comprehension isolation, and it predates and is orthogonal to PEP 709. An assignment expression (the walrus operator :=, PEP 572) used inside a comprehension binds its target in the enclosing scope, not the comprehension’s isolated scope. PEP 572 states the rule verbatim: “an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as ‘comprehensions’) binds the target in the containing scope,” and gives the example:

if any((comment := line).startswith('#') for line in lines):
    print("First comment:", comment)

Here comment is usable after the comprehension, because the walrus deliberately leaks it. This is by design: the whole point of allowing := in a comprehension is to capture a value computed during iteration for use afterward — which would be pointless if it were isolated like the loop variable. The contrast is sharp and worth internalising: the for target is isolated; the := target is not. Confirm it:

>>> data = [(w := k + 1) for k in range(3)]
>>> w           # leaked on purpose
3
>>> k           # NameError — the for-target did not leak
Traceback (most recent call last):
NameError: name 'k' is not defined

In the bytecode this shows up plainly: the walrus target gets a STORE_FAST to a slot that is not bracketed by the save/restore pair, and is read back with LOAD_FAST_CHECK in the enclosing code after the loop — it is a genuine local of the enclosing function, never cleared.

Uncertain

Verify (as of 2026-06-01): the interaction between PEP 572 walrus leaking and PEP 709 inlining at the symbol-table level — specifically that the walrus target is allocated in the enclosing scope’s varnames (and is not added to the comprehension’s save/restore set), which is what lets it survive the restore. Reason: confirmed empirically on 3.14.5 — disassembly shows a STORE_FAST to a slot outside the save/restore bracketing and a later LOAD_FAST_CHECK in the enclosing code, and both PEP texts were read — but neither PEP discusses the other’s mechanism, so the codegen reconciliation is inferred from observed bytecode, not stated in one primary source. The empirical evidence is strong; this flag remains only because the symbol-table source was not read. To resolve: read compile/symtable.c (the symtable_handle_namedexpr / comprehension-inlining path) at v3.14.5 and confirm the := target is recorded in the enclosing function’s symtable entry rather than the comprehension’s. uncertain

Generator Expressions Are Still Separate

PEP 709 explicitly excludes generator expressions: “Generator expressions are currently not inlined in the reference implementation of this PEP.” A genexpr still compiles to its own <genexpr> code object and is invoked as a generator, because a generator’s execution is suspended and resumed across next() calls — its loop state genuinely must live in a frame that persists between yields, which an inlined-into-the-caller body cannot provide. The disassembly makes the difference obvious:

def g(items):
    return (n * 2 for n in items)
 17   LOAD_CONST   0 (<code object <genexpr> ...>)   # separate code object!
      MAKE_FUNCTION
      LOAD_FAST_BORROW 0 (items)
      GET_ITER
      CALL         0                                  # actually called
      RETURN_VALUE

So in 3.14, list/set/dict comprehensions are inline and frameless, while generator expressions retain the pre-3.12 nested-function model. All four still obey the same observable scoping rule — the loop variable does not leak — but only the first three pay zero call overhead.

Common Misunderstandings

“PEP 709 changed comprehension scoping.” It did not change the semantics at all — the loop variable was isolated before 3.12 and is isolated after. PEP 709 changed only the implementation (function-call isolation → save/restore isolation). Code that relied on the documented no-leak behaviour is unaffected.

“Comprehensions can now see enclosing locals because they’re inlined.” Mostly no. The body still cannot assign to an enclosing local and have it stick (the restore wipes it). It can read enclosing names — but it always could, via the nested function’s free-variable mechanism pre-3.12. The one genuinely new observable consequence of inlining is in tracebacks and sys.settrace: there is no longer a separate <listcomp> frame, so a comprehension no longer shows up as its own stack frame — a point flagged in the PEP’s “Backwards Compatibility” discussion.

“The walrus and the for-target behave the same in a comprehension.” The opposite — that contrast is the whole subtlety. The for target is isolated; the := target leaks to the enclosing scope, by deliberate PEP 572 design.

Production Notes

The inlining is invisible to almost all code, which was the goal. The few places it surfaces: (1) tools that introspect frames or stack depth see one fewer frame for comprehensions; (2) sys.settrace-based coverage and profiling tools had to stop expecting a <listcomp> call event; and (3) micro-optimised hot paths that build many small comprehensions got the documented speedup for free on upgrading to 3.12+. The PEP authors note that the change was extensively tested against the standard library precisely because “implicitly nested scope” was a long-standing, subtly load-bearing detail. For the rare code that needs the old separate-frame behaviour (e.g. relying on a <listcomp> traceback frame), the workaround is to write an explicit nested function or use a generator expression, which is still framed.

See Also

Graph View

Backlinks