vault

Generators and the yield Mechanism

16 min read

Generators and the yield Mechanism

A generator is the object CPython hands back when you call a function whose body contains a yield expression. Instead of running the body to completion and returning a value, the call returns a generator-iterator: a resumable object that runs the body in fits and starts, pausing at each yield, returning the yielded value to whoever advanced it, and remembering everything — local variables, the instruction pointer, the in-progress evaluation stack, and active exception handlers — so that the next advance picks up exactly where it left off. The single most important idea is that yield turns a function (one entry, one exit, stack-allocated activation record) into a coroutine-like state machine (many entries, many exits, a heap-stored activation record that outlives any single call). This note covers the generator-level contract — the object, its lifecycle, what yield/send/throw/close/yield from do — and defers the frame-suspension mechanics (how the activation record physically survives a pause) to its sibling Generator and Coroutine Frame Internals.

Generators were introduced by PEP 255 (“Simple Generators”, targeting Python 2.2), which added the yield statement and specified that “a generator function is an ordinary function object in all respects, but has the new CO_GENERATOR flag set in the code object’s co_flags member” (PEP 255). PEP 342 (“Coroutines via Enhanced Generators”, Python 2.5) promoted yield from a statement to an expression, added send(), throw(), and close(), and introduced the GeneratorExit exception (PEP 342). PEP 380 (“Syntax for Delegating to a Subgenerator”, Python 3.3) added yield from for transparent delegation (PEP 380). Everything below is verified against the CPython v3.14.5 source tag (as of 10 May 2026).

Mental Model

Think of a generator as a bookmark in a paused function. An ordinary function call is a round trip: control enters, runs straight through, and the activation record (the frame — locals, evaluation stack, instruction pointer) is destroyed on return. A generator call instead manufactures an object that holds a paused frame and returns immediately without running a single line of the body. Each next()/send() is a “resume the bookmark, run until the next page-marker (yield), put the bookmark back” operation. The body never knows it was paused; from inside, a yield expression looks like an ordinary call that “returns” whatever value the next send() supplies.

stateDiagram-v2
    [*] --> GEN_CREATED: gen = f()  (body not yet run)
    GEN_CREATED --> GEN_RUNNING: first next()/send(None)
    GEN_RUNNING --> GEN_SUSPENDED: hits a yield
    GEN_SUSPENDED --> GEN_RUNNING: next()/send(v)/throw(e)
    GEN_RUNNING --> GEN_CLOSED: return / falls off end (raises StopIteration)
    GEN_RUNNING --> GEN_CLOSED: uncaught exception
    GEN_SUSPENDED --> GEN_CLOSED: close()  (throws GeneratorExit)
    GEN_CREATED --> GEN_CLOSED: close() before first run
    note right of GEN_RUNNING
      Only ever one frame
      EXECUTING at a time —
      re-entrancy raises
      ValueError.
    end note

Diagram: the user-visible generator lifecycle as reported by inspect.getgeneratorstate(). The insight: a generator spends most of its life in GEN_SUSPENDED, oscillating with GEN_RUNNING once per advance; GEN_CREATED is the brief window before the first run, and GEN_CLOSED is terminal. These four names are Python-level labels, not the internal C state machine — the mapping to the real gi_frame_state field is given below.

Two Layers of State: inspect Labels vs. gi_frame_state

A subtle but important point that trips people up: the four GEN_* names are not the internal state machine. They are string constants returned by inspect.getgeneratorstate(), computed at the Python level from cheaper attributes (Lib/inspect.py):

def getgeneratorstate(generator):
    if generator.gi_running:           # gi_frame_state == FRAME_EXECUTING
        return GEN_RUNNING
    if generator.gi_suspended:         # FRAME_STATE_SUSPENDED(gi_frame_state)
        return GEN_SUSPENDED
    if generator.gi_frame is None:     # FRAME_STATE_FINISHED(gi_frame_state)
        return GEN_CLOSED
    return GEN_CREATED                  # the remaining case: FRAME_CREATED

Internally, every PyGenObject carries a single int8_t gi_frame_state field whose values come from the PyFrameState enum (pycore_frame.h):

typedef enum _framestate {
    FRAME_CREATED = -3,              // allocated, never run; first RESUME pending
    FRAME_SUSPENDED = -2,            // paused at a plain yield
    FRAME_SUSPENDED_YIELD_FROM = -1, // paused inside a `yield from`/await delegation
    FRAME_EXECUTING = 0,             // currently running on the interpreter
    FRAME_COMPLETED = 1,             // returned/raised; frame contents still around
    FRAME_CLEARED = 4               // frame torn down, locals released
} PyFrameState;

So the real machine has six states, not four, and it distinguishes a plain suspension from one inside a delegation (FRAME_SUSPENDED_YIELD_FROM) — a distinction the GEN_* view collapses. The mapping is: GEN_RUNNING ↔ FRAME_EXECUTING; GEN_SUSPENDED ↔ FRAME_SUSPENDED or FRAME_SUSPENDED_YIELD_FROM; GEN_CLOSED ↔ FRAME_COMPLETED or FRAME_CLEARED; GEN_CREATED ↔ FRAME_CREATED. Two helper macros encode the grouping: FRAME_STATE_SUSPENDED(S) is true for the two negative-one/negative-two states, and FRAME_STATE_FINISHED(S) is (S) >= FRAME_COMPLETED. Throughout genobject.c the code branches on gi_frame_state, never on the GEN_* strings — those exist only for human introspection. The gi_frame_state field lives inside the embedded frame’s owner object; why a generator owns its own frame at all is the subject of Generator and Coroutine Frame Internals.

The PyGenObject Structure

A generator object is defined by the _PyGenObject_HEAD macro in pycore_interpframe_structs.h. The gi_ prefix is a mnemonic for “generator-iterator”:

#define _PyGenObject_HEAD(prefix)                       \
    PyObject_HEAD                                       \
    PyObject *prefix##_weakreflist;     /* weak refs */ \
    PyObject *prefix##_name;            /* __name__ */  \
    PyObject *prefix##_qualname;        /* __qualname__ */ \
    _PyErr_StackItem prefix##_exc_state; /* saved exc_info */ \
    PyObject *prefix##_origin_or_finalizer;             \
    char prefix##_hooks_inited;                         \
    char prefix##_closed;                               \
    char prefix##_running_async;                        \
    int8_t prefix##_frame_state;        /* PyFrameState */ \
    _PyInterpreterFrame prefix##_iframe; /* THE frame */
 
struct _PyGenObject { _PyGenObject_HEAD(gi) };

Reading the fields with their roles: gi_weakreflist lets the generator be the target of weakref.ref. gi_name/gi_qualname mirror the originating function’s names (so repr(gen) reads <generator object f at 0x...>). gi_exc_state is an _PyErr_StackItem — when a generator is running, the thread’s “currently handled exception” (sys.exc_info()) is swapped to point at this saved slot, so a generator’s exception context does not leak into or out of its caller; on suspension the previous item is restored (this swap is visible in gen_send_ex2, below). gi_origin_or_finalizer holds either the captured creation traceback (for coroutines, used in “coroutine was never awaited” warnings) or the async-generator finalizer hook. gi_frame_state is the lifecycle field discussed above. The final field, gi_iframe, is the embedded _PyInterpreterFrame — the entire activation record lives inside the generator object. That embedding is the whole trick of suspension and is covered in Generator and Coroutine Frame Internals; here we treat it as a black box that “remembers where we paused.”

Note that struct _PyCoroObject and struct _PyAsyncGenObject use the identical _PyGenObject_HEAD macro with cr/ag prefixes — coroutines and async generators are the same machinery with different type objects (PyCoro_Type, PyAsyncGen_Type) and different exhaustion exceptions. The differences are explored in Coroutines and the async await Protocol.

What yield Compiles To

When the compiler sees a yield in a function body, two things happen. First, it sets the CO_GENERATOR bit (0x0020) in the resulting code object’s co_flags (code.h; the value matches the data-model docs’ “0x20: set if the function is a generator”). The sibling flags are CO_COROUTINE = 0x0080 (an async def function), CO_ITERABLE_COROUTINE = 0x0100 (a generator decorated with @types.coroutine), and CO_ASYNC_GENERATOR = 0x0200 (an async def containing yield). These flags are how the interpreter decides, at the call site, to manufacture a generator/coroutine object rather than run the body.

Second, the body’s bytecode acquires three special opcodes. RETURN_GENERATOR is emitted as the very first real instruction of a generator’s code; it is what runs when you call the function, and it builds the generator object and returns it instead of executing the body (covered mechanically in Generator and Coroutine Frame Internals). YIELD_VALUE is emitted at each yield. RESUME sits at the top of the body and after each YIELD_VALUE, marking a re-entry point. The relevant fragment of YIELD_VALUE from bytecodes.c:

inst(YIELD_VALUE, (retval -- value)) {
    // NOTE: It's important that YIELD_VALUE never raises an exception!
    frame->instr_ptr++;                                  // advance past the yield
    PyGenObject *gen = _PyGen_GetGeneratorFromFrame(frame); // recover gen from frame addr
    gen->gi_frame_state = FRAME_SUSPENDED + oparg;       // -2 plain, -1 if yield-from
    _PyStackRef temp = retval;
    SAVE_STACK();                                        // persist the eval stack
    tstate->exc_info = gen->gi_exc_state.previous_item;  // restore caller's exc_info
    gen->gi_exc_state.previous_item = NULL;
    _Py_LeaveRecursiveCallPy(tstate);
    _PyInterpreterFrame *gen_frame = frame;
    frame = tstate->current_frame = frame->previous;     // pop back to caller's frame
    gen_frame->previous = NULL;                          // unlink — survive suspension
    ...
    value = PyStackRef_MakeHeapSafe(temp);               // hand the yielded value out
}

Read line by line: the instruction pointer is bumped first, so a future resume restarts after the yield, not on it. _PyGen_GetGeneratorFromFrame recovers the owning PyGenObject by pointer arithmetic (the frame is embedded at a known offset). The frame state becomes FRAME_SUSPENDED + oparg — oparg is 0 for a plain yield (giving FRAME_SUSPENDED = -2) and 1 for a yield that is part of a yield from/await delegation (giving FRAME_SUSPENDED_YIELD_FROM = -1), which is exactly the comment FRAME_SUSPENDED_YIELD_FROM == FRAME_SUSPENDED + 1 in the source. The thread’s exc_info is restored to the caller’s, undoing the swap. Then the generator’s frame is unlinked from the call chain (frame->previous) so it can outlive this call, and control returns to the caller, carrying the yielded value out. The comment “YIELD_VALUE never raises” matters: the compiler treats any exception there as a failed close()/throw(), so the opcode is engineered to be infallible. The frame->previous linking/unlinking is the physical suspension mechanism and is detailed in Generator and Coroutine Frame Internals and The Value Stack and Frame Evaluation.

The Resume Path: gen_send_ex2

Every advance of a generator — next(), send(), throw(), and the implicit for-loop step — funnels through one C function, gen_send_ex2 in genobject.c. It is the canonical implementation of “resume the bookmark.” The core:

static PySendResult
gen_send_ex2(PyGenObject *gen, PyObject *arg, PyObject **presult,
             int exc, int closing)
{
    PyThreadState *tstate = _PyThreadState_GET();
    _PyInterpreterFrame *frame = &gen->gi_iframe;
 
    if (gen->gi_frame_state == FRAME_CREATED && arg && arg != Py_None) {
        PyErr_SetString(PyExc_TypeError,
            "can't send non-None value to a just-started generator");
        return PYGEN_ERROR;
    }
    if (gen->gi_frame_state == FRAME_EXECUTING) {
        PyErr_SetString(PyExc_ValueError, "generator already executing");
        return PYGEN_ERROR;
    }
    if (FRAME_STATE_FINISHED(gen->gi_frame_state)) {
        if (arg && !exc) { *presult = Py_NewRef(Py_None); return PYGEN_RETURN; }
        return PYGEN_ERROR;          // exhausted: re-raise StopIteration upstream
    }
    /* Push arg onto the frame's value stack */
    PyObject *arg_obj = arg ? arg : Py_None;
    _PyFrame_StackPush(frame, PyStackRef_FromPyObjectNew(arg_obj));
    /* swap thread exc_info to the generator's saved slot */
    gen->gi_exc_state.previous_item = tstate->exc_info;
    tstate->exc_info = &gen->gi_exc_state;
    if (exc) { _PyErr_ChainStackItem(); }   // throw(): set up the exception
    gen->gi_frame_state = FRAME_EXECUTING;
    PyObject *result = _PyEval_EvalFrame(tstate, frame, exc);
    ...
    if (result) {
        if (FRAME_STATE_SUSPENDED(gen->gi_frame_state)) {
            *presult = result; return PYGEN_NEXT;   // hit another yield
        }
        ...
    }
    *presult = result;
    return result ? PYGEN_RETURN : PYGEN_ERROR;       // body returned / raised
}

The guard clauses encode the data-model contract precisely. Sending to a just-started generator (FRAME_CREATED) with a non-None value is a TypeError, because there is no suspended yield expression to receive the value — matching PEP 342’s rule that the first send() must pass None. Re-entering a running generator (FRAME_EXECUTING) is a ValueError("generator already executing"); generators are not re-entrant. Advancing an exhausted generator (FRAME_STATE_FINISHED) either yields None/StopIteration for send or signals end-of-iteration for next.

The heart is the middle: the argument (the value passed to send(), or None for next()) is pushed onto the frame’s own value stack with _PyFrame_StackPush. This is the elegant part — the resuming RESUME/post-YIELD_VALUE bytecode simply reads the top of stack as “the result of the yield expression,” so value = yield x works by the yield leaving a slot the next push fills. Then the thread’s exception state is swapped to the generator’s saved gi_exc_state (so the generator’s try/except context is restored), the state flips to FRAME_EXECUTING, and _PyEval_EvalFrame runs the frame until the next YIELD_VALUE (which sets the state back to suspended and returns the yielded value) or until the body returns/raises. The PySendResult return value — PYGEN_NEXT (yielded again), PYGEN_RETURN (body returned), PYGEN_ERROR (propagating exception) — is the three-way fork that the wrapper gen_send_ex translates into the Python-visible “value or StopIteration.”

send(), throw(), close(), and the StopIteration Convention

The public send() is gen_send → gen_send_ex(gen, arg, exc=0, closing=0). When gen_send_ex2 returns PYGEN_RETURN, the wrapper converts a normal generator return into the iterator-protocol signal: a bare return (value None) becomes StopIteration with no value, and return v becomes StopIteration(v) via _PyGen_SetStopIterationValue. This is the bridge between “a function returned” and “an iterator is exhausted” — for loops catch the StopIteration and stop. Per the data-model docs, “the send() method returns the next value yielded by the generator, or raises StopIteration if the generator exits without yielding another value” (expressions reference).

throw() ultimately calls gen_send_ex(gen, Py_None, exc=1, closing=0), which sets up the supplied exception (chaining it onto the current exception context with _PyErr_ChainStackItem) and then resumes the frame as if the yield raised that exception. If the generator’s try/except catches it and yields again, throw() returns the next yielded value; if it escapes, it propagates to the caller. The classic three-argument form throw(type, value, tb) is deprecated as of Python 3.12 in favor of throw(exception_instance).

close() (gen_close) raises GeneratorExit at the suspension point — “equivalent to calling throw(GeneratorExit).” If the generator was never started (FRAME_CREATED), it transitions straight to FRAME_COMPLETED with no code run. If already finished, close() is a no-op returning None. Otherwise it sets GeneratorExit and resumes via gen_send_ex(gen, Py_None, exc=1, closing=1). The contract: if the generator swallows GeneratorExit and yields a value, close() raises RuntimeError("generator ignored GeneratorExit"); if it lets GeneratorExit (or StopIteration) propagate or simply returns, close() is satisfied. Changed in Python 3.13: if a generator returns a value while being closed, that value is now returned by close() (previously discarded) — the _PyGen_FetchStopIterationValue(&retval) tail of gen_close retrieves it. gen_close also has a fast path: if the frame is suspended right at a RESUME with shallow exception depth and no debugging/monitoring tools are active, it just tears the frame down without resuming, avoiding the cost of re-entering the interpreter to immediately exit.

yield from — Delegation (PEP 380)

yield from EXPR delegates the entire iteration protocol to a subiterator. PEP 380 specifies it by an exact expansion: RESULT = yield from EXPR is semantically equivalent to a loop that obtains iter(EXPR), then in a loop yields each value the subiterator produces, forwards send() values into the subiterator’s send(), forwards throw() exceptions into its throw(), forwards close() into its close(), and — crucially — when the subiterator finishes by raising StopIteration, takes the value attribute of that StopIteration as the result of the whole yield from expression (PEP 380). The data-model docs put it succinctly: “When the underlying iterator is complete, the value attribute of the raised StopIteration instance becomes the value of the yield expression.” That is why result = yield from subgen() captures whatever subgen returned.

CPython does not implement this by literally running the PEP’s Python expansion; it implements it natively via the SEND opcode and the FRAME_SUSPENDED_YIELD_FROM state. When a delegating generator is suspended inside a yield from, gi_frame_state == FRAME_SUSPENDED_YIELD_FROM (-1), and the helper _PyGen_yf (in genobject.c) peeks the top of the delegating frame’s value stack to recover the subiterator currently being driven:

PyObject * _PyGen_yf(PyGenObject *gen) {
    if (gen->gi_frame_state == FRAME_SUSPENDED_YIELD_FROM) {
        _PyInterpreterFrame *frame = &gen->gi_iframe;
        return PyStackRef_AsPyObjectNew(_PyFrame_StackPeek(frame));
    }
    return NULL;   // not delegating
}

gen_close and _gen_throw use _PyGen_yf to find the subiterator and forward close/throw into it first (via gen_close_iter), so a close() on the outer generator cleanly shuts down the whole delegation chain top-down. This is also the mechanism that makes await work: an await in a coroutine compiles to a yield from-style delegation onto the awaited object’s __await__ iterator (see Coroutines and the async await Protocol). The SEND opcode that drives a subgenerator inline — pushing the sent value onto the subgenerator’s frame stack and linking frames — is examined in Generator and Coroutine Frame Internals.

Failure Modes and Common Misunderstandings

“send() failed with TypeError on a fresh generator.” You called gen.send(x) with x is not None before the first next(). There is no paused yield to receive x. Prime the generator first with next(gen) or gen.send(None).

“ValueError: generator already executing.” You re-entered a generator that is on the stack above you — e.g., a generator that, directly or indirectly, advances itself. The FRAME_EXECUTING guard forbids this; there is exactly one live frame per generator.

return value inside a generator does not return value to the iterator consumer. It raises StopIteration(value). A for loop discards that value; only yield from (or manual StopIteration.value inspection) sees it. Conflating the two is a classic bug. Also note PEP 479 (since Python 3.7) makes a StopIteration that escapes a generator body a RuntimeError, to prevent silent truncation.

Forgetting to close()/exhaust a generator with a finally. Cleanup code in a try/finally around a yield runs only when the generator is closed or garbage-collected. If a generator holding a file handle is abandoned mid-iteration, the finally runs at an unpredictable GC time (and for async generators, possibly during event-loop shutdown — hence aclose()).

“My generator’s exceptions leak into the caller’s sys.exc_info().” They should not, because of the gi_exc_state swap. If you observe leakage, it is almost always because you manually inspected sys.exc_info() from inside the running generator, where the swap is in effect by design.

Alternatives and When to Choose Them

A generator is the right tool when you want lazy, stateful, single-pass iteration — streaming large data, pipelines, or producer/consumer coroutines. Compared with writing a class that implements __iter__/__next__ by hand, a generator is dramatically less code and keeps the iteration state implicitly in local variables and the instruction pointer rather than in explicit instance attributes — but you give up random access and reuse (a generator is exhausted after one pass). Compared with returning a list, a generator avoids materializing everything in memory, at the cost of not being indexable or re-iterable. For concurrency, generators were historically the substrate for coroutines (the @asyncio.coroutine/yield from style), but modern async code uses native async def/await (Coroutines and the async await Protocol) — which is the same frame machinery with a distinct type and StopAsyncIteration semantics. When you need genuine parallelism rather than cooperative suspension, generators do nothing for you: see Python Threading Model and multiprocessing and Process-Based Parallelism.

Production Notes

Generators are pervasive in the standard library: itertools is built on them, os.walk is a generator, and every comprehension-style lazy pipeline uses generator expressions (which are generators whose code object also carries CO_GENERATOR). The contextlib @contextmanager decorator is a striking application: it wraps a single-yield generator so that the code before yield is __enter__, the yielded value is the as target, and the code after yield (driven by gen.send/gen.throw on context exit) is __exit__ — throw() is how an exception inside the with block is injected back at the yield. The send()/throw()/close() protocol is therefore not academic; contextlib exercises all of it. Because each generator embeds a full frame, a program holding millions of live generators pays a per-generator memory cost proportional to the frame size (locals + stack); this is the trade-off analyzed in Generator and Coroutine Frame Internals.

See Also

Graph View

Backlinks