vault

The Iterator Protocol

14 min read

The Iterator Protocol

The iterator protocol is the single mechanism behind every for loop, comprehension, *-unpacking, and call to iter()/next() in Python. It rests on two methods: an iterable defines __iter__ returning a fresh iterator, and that iterator defines __next__, which returns the next item or raises StopIteration to signal exhaustion (per the data-model and stdtypes “Iterator Types”). Crucially, an iterator’s own __iter__ returns itself, which is what lets a for loop accept either a container or an already-running iterator. At the C level a for loop compiles to GET_ITER (call iter() on the iterable) followed by a FOR_ITER loop head (call tp_iternext, jump out on exhaustion). This note traces that machinery end to end: the protocol contract, the iterable-vs-iterator distinction, the GET_ITER/FOR_ITER opcodes and their adaptive specializations, the legacy __getitem__-based fallback iterator, StopIteration handling and the PEP 479 change, and the two-argument iter(callable, sentinel) form.

Mental Model

Keep two roles distinct. An iterable is anything you can start iterating — it answers iter() by handing back an iterator. An iterator is the stateful cursor that actually produces values one at a time via next() and knows when it is done. A list is iterable but is not its own iterator (calling next() on a list fails); iter(my_list) returns a separate list_iterator object holding an index. By contrast a generator and a file object are their own iterators. The protocol unifies the two roles with one rule: an iterator’s __iter__ returns self, so code can call iter() on anything and get back something next()-able, whether it started with a container or a cursor.

flowchart TD
    A["for x in obj:"] --> B["GET_ITER  ->  iter(obj)  ->  PyObject_GetIter"]
    B --> C{"type has tp_iter ?"}
    C -- yes --> D["call __iter__()  -> iterator"]
    C -- no --> E{"PySequence_Check<br/>(has sq_item) ?"}
    E -- yes --> F["PySeqIter_New(obj)<br/>legacy __getitem__ iterator"]
    E -- no --> G["TypeError: object is not iterable"]
    D --> H["FOR_ITER loop head"]
    F --> H
    H --> I["call tp_iternext (__next__)"]
    I -- "returns item" --> J["push item, run loop body, JUMP_BACKWARD"]
    I -- "returns NULL + StopIteration set" --> K["clear StopIteration,<br/>jump past END_FOR, POP_ITER"]
    J --> H

Diagram: the compiled shape of a for loop. The insight is that there are exactly two ways to become iterable — define __iter__ (the tp_iter slot), or define integer __getitem__ and let CPython synthesize a sequence iterator — and that exhaustion is signaled by StopIteration, which the loop head catches and silently converts into a jump out of the loop, not into a Python-visible exception.

The Protocol Contract

The stdtypes “Iterator Types” section gives the formal contract, quoted here:

  • container.__iter__() — “Return an iterator object. … This method corresponds to the tp_iter slot of the type structure.”
  • iterator.__iter__() — “Return the iterator object itself. This is required to allow both containers and iterators to be used with the for and in statements.” Also tp_iter.
  • iterator.__next__() — “Return the next item from the iterator. If there are no further items, raise the StopIteration exception. This method corresponds to the tp_iternext slot.”

There is one more binding rule, stated verbatim: “Once an iterator’s __next__() method raises StopIteration, it must continue to do so on subsequent calls. Implementations that do not obey this property are deemed broken.” This monotonic exhaustion guarantee is what lets consumers (e.g. zip, itertools) trust that once a StopIteration arrives, the iterator is permanently spent — they will not poll it again and accidentally get more values.

The collections.abc ABCs formalize the two roles minimally (collections.abc docs): Iterable has the single abstract method __iter__ and no mixins; Iterator inherits Iterable, adds the abstract method __next__, and provides __iter__ as a mixin (returning self). So subclassing collections.abc.Iterator and supplying only __next__ yields a correct iterator whose __iter__ already returns self.

Iterable vs Iterator — Why the Split Matters

The separation is not pedantry; it is what makes iterators reusable in the right way and not in the wrong way. Because a container’s __iter__ returns a fresh iterator each call, you can loop over a list twice and each loop starts from the beginning — two independent cursors. Because an iterator’s __iter__ returns self, looping over an already-started iterator continues from where it was and exhausts it permanently. This is the source of the classic bug:

nums = iter([1, 2, 3])
print(list(nums))   # [1, 2, 3]  -- drains the iterator
print(list(nums))   # []         -- iterator is now exhausted, monotonic StopIteration

The second list(nums) calls iter(nums), which returns nums itself (already at the end), so it immediately gets StopIteration and yields an empty list. A list does not have this problem because each iter(some_list) is a new cursor. The rule of thumb that follows: store the iterable if you need to re-iterate; store the iterator only to consume it once.

The Compiled for Loop: GET_ITER and FOR_ITER

A for loop disassembles (verified on Python 3.14.5) to:

LOAD_NAME    0 (s)
GET_ITER
L1:  FOR_ITER  3 (to L2)
     STORE_NAME 1 (x)
     JUMP_BACKWARD 5 (to L1)
L2:  END_FOR
     POP_ITER

GET_ITER replaces the iterable on the stack with iter(iterable); the dis docs define it as “Implements STACK[-1] = iter(STACK[-1]).” Its bytecodes.c body (v3.14.5) is thin:

inst(GET_ITER, (iterable -- iter)) {
    PyObject *iter_o = PyObject_GetIter(PyStackRef_AsPyObjectBorrow(iterable));
    PyStackRef_CLOSE(iterable);
    ERROR_IF(iter_o == NULL);
    iter = PyStackRef_FromPyObjectSteal(iter_o);
}

It calls PyObject_GetIter, the abstract-layer function discussed below, and pushes the resulting iterator. FOR_ITER is the loop head: each iteration it calls the iterator’s tp_iternext and either pushes the produced value (leaving the iterator beneath it) or, on exhaustion, jumps forward by delta past the END_FOR. The _FOR_ITER body (v3.14.5) shows exactly how StopIteration is handled at the C level:

replaced op(_FOR_ITER, (iter -- iter, next)) {
    PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter);
    iternextfunc func = Py_TYPE(iter_o)->tp_iternext;
    if (func == NULL) {
        _PyErr_Format(tstate, PyExc_TypeError,
                      "'%.100s' object is not an iterator", ...);
        ERROR_NO_POP();
    }
    PyObject *next_o = func(iter_o);              // call __next__
    if (next_o == NULL) {
        if (_PyErr_Occurred(tstate)) {
            int matches = _PyErr_ExceptionMatches(tstate, PyExc_StopIteration);
            if (!matches) {                        // some OTHER exception -> propagate
                ERROR_NO_POP();
            }
            _PyEval_MonitorRaise(tstate, frame, this_instr);
            _PyErr_Clear(tstate);                  // swallow StopIteration
        }
        /* iterator ended normally */
        JUMPBY(oparg + 1);                         // jump past END_FOR
        DISPATCH();
    }
    next = PyStackRef_FromPyObjectSteal(next_o);
}

Reading it: tp_iternext (the C slot behind __next__) is fetched and called. A non-NULL result is the next value, pushed for the loop body. A NULL result means either an exception or normal exhaustion. The code checks whether the pending exception is StopIteration: if it is something else (say a KeyError from inside a generator), it propagates via ERROR_NO_POP; if it is StopIteration, the interpreter clears it (_PyErr_Clear) and jumps out of the loop. This is the precise mechanism by which StopIteration terminates a for loop without ever surfacing as a Python exception in the loop’s surrounding code.

Two version notes (verified via dis docs): FOR_ITER was “Changed in version 3.12: Up until 3.11 the iterator was popped when it was exhausted” — modern FOR_ITER leaves the exhausted iterator on the stack, which is why a separate POP_ITER opcode (“Added in version 3.14”) now removes it after END_FOR. On 3.11 and earlier you would not see POP_ITER in the disassembly.

Adaptive Specializations of FOR_ITER

bytecodes.c declares a FOR_ITER specialization family: FOR_ITER_LIST, FOR_ITER_TUPLE, FOR_ITER_RANGE, and FOR_ITER_GEN (see Adaptive Specialization Families for how the specializing interpreter rewrites a generic opcode into one of these after observing a stable iterator type). The speedup is dramatic because the generic path calls tp_iternext through a function pointer and goes through the StopIteration machinery, whereas the specialized path reads the iterator’s internal index directly. For example, _ITER_NEXT_LIST (part of the FOR_ITER_LIST macro) on the default build is simply:

assert(it->it_index < PyList_GET_SIZE(seq));
next = PyStackRef_FromPyObjectNew(PyList_GET_ITEM(seq, it->it_index++));

It indexes the underlying list array and bumps the iterator’s it_index — no function call, no exception check. A companion guard (_ITER_JUMP_LIST) checks it_index >= len and jumps out of the loop, again with no exception. FOR_ITER_RANGE computes the next integer arithmetically — its _ITER_NEXT_RANGE body (v3.14.5) reads the range iterator’s running start, advances it by step, decrements the remaining len, and boxes the value with PyLong_FromLong — so it never materializes a list, and the loop-exit check is just r->len <= 0. FOR_ITER_GEN pushes an interpreter frame for the generator directly rather than going through the generic tp_iternext/send path (cross-link Generators and the yield Mechanism). These specializations are why iterating a list, range, or generator in a tight for loop is fast in 3.11+.

The Legacy __getitem__ Sequence Iterator

PyObject_GetIter in abstract.c (v3.14.5) shows the second way to be iterable:

PyObject *
PyObject_GetIter(PyObject *o)
{
    PyTypeObject *t = Py_TYPE(o);
    getiterfunc f = t->tp_iter;
    if (f == NULL) {
        if (PySequence_Check(o))         // has tp_as_sequence->sq_item?
            return PySeqIter_New(o);     // synthesize a sequence iterator
        return type_error("'%.200s' object is not iterable", o);
    }
    else {
        PyObject *res = (*f)(o);         // call __iter__
        if (res != NULL && !PyIter_Check(res)) {
            // __iter__ returned a non-iterator -> TypeError
            ...
        }
        return res;
    }
}

If a type has no tp_iter (no __iter__) but does pass PySequence_Check (it has integer __getitem__ via sq_item), CPython manufactures a PySeqIter object. That iterator’s tp_iternext (iter_iternext in iterobject.c) is:

result = PySequence_GetItem(seq, it->it_index);   // calls __getitem__(it_index)
if (result != NULL) {
    it->it_index++;
    return result;
}
if (PyErr_ExceptionMatches(PyExc_IndexError) ||
    PyErr_ExceptionMatches(PyExc_StopIteration)) {
    PyErr_Clear();
    it->it_seq = NULL;       // exhausted
    Py_DECREF(seq);
}
return NULL;

It calls __getitem__ with it_index starting at 0, incrementing each call, and treats an IndexError (or StopIteration) as the end. This is why a class defining only integer __getitem__ — with no __iter__ — is still iterable, supports for, in, and list(). It is a backward-compatibility bridge from before __iter__ existed (Python 2.1 and earlier). Note: PyObject_GetIter also validates that a real __iter__ returns an actual iterator (PyIter_Check) — returning a non-iterator from __iter__ raises “iter() returned non-iterator.” The same PySeqIter mechanism is what The Sequence and Mapping Protocols refers to when it says in and iteration fall back to __getitem__.

iter(callable, sentinel) — the Two-Argument Form

The built-in iter() has a less-known second form (functions docs): “If the second argument, sentinel, is given, then the first argument must be a callable object. The iterator created in this case will call callable with no arguments for each call to its __next__() method; if the value returned is equal to sentinel, StopIteration will be raised, otherwise the value will be returned.” Its iterator’s tp_iternext (calliter_iternext in iterobject.c) confirms the mechanism:

result = _PyObject_CallNoArgs(it->it_callable);   // call the callable()
if (result != NULL && it->it_sentinel != NULL) {
    int ok = PyObject_RichCompareBool(it->it_sentinel, result, Py_EQ);
    if (ok == 0) {
        return result;                            // not sentinel -> yield it
    }
    if (ok > 0) {                                 // == sentinel -> stop
        Py_CLEAR(it->it_callable);
        Py_CLEAR(it->it_sentinel);
    }
}

Each __next__ calls the zero-argument callable and compares the result to the sentinel with ==. The canonical use is reading fixed-size blocks until end-of-file: for block in iter(partial(f.read, 64), b''): ... (the docs’ own example) reads 64-byte chunks until f.read returns the empty bytes sentinel.

StopIteration, PEP 479, and the Generator Leak

StopIteration is the universal “I’m done” signal, raised by __next__/tp_iternext. The built-in next(iterator) re-raises it to the caller, but next(iterator, default) swallows it and returns default instead (functions docs). Historically there was a dangerous coupling: because generators are iterators and signal completion with StopIteration, a StopIteration accidentally raised inside a generator body (e.g. from an inner next() that ran dry) would propagate out and be misread by the enclosing for/next() as the generator finishing normally — silently truncating iteration and masking the real bug.

PEP 479 fixed this. Its abstract: “When StopIteration is raised inside a generator, it is replaced with RuntimeError.” Concretely, “if a StopIteration is about to bubble out of a generator frame, it is replaced with RuntimeError, which causes the next() call (which invoked the generator) to fail.” The change was opt-in via from __future__ import generator_stop in 3.5, warned in 3.6, and became the default in Python 3.7. The mechanism lives in the generator’s frame-finalization path, not in FOR_ITER (which still treats a plain StopIteration from a real exhausted iterator normally) — see Generators and the yield Mechanism for where the substitution happens. The practical upshot today: a stray StopIteration inside a generator is a loud RuntimeError, not a silent early stop.

StopIteration is also a data-carrying signal, not merely a flag. The exception has a .value attribute, and when a generator finishes with return expr (rather than falling off the end), the StopIteration raised on the final __next__/tp_iternext carries that expr in .value — this is how a delegating yield from (the SEND/RETURN_GENERATOR machinery) harvests a sub-generator’s return value. The data-model reference notes that when a generator function “executes a return statement or falls off the end, a StopIteration exception is raised” (datamodel); the return-with-value form populates StopIteration.value, whereas falling off the end leaves it None. A plain for loop discards this value (it only cares that iteration ended), but result = yield from subgen binds it, and you can observe it directly:

def gen():
    yield 1
    return 99            # becomes StopIteration(99)
 
it = gen()
next(it)                 # -> 1
try:
    next(it)
except StopIteration as e:
    print(e.value)       # -> 99

This is why StopIteration is a full exception class rather than a sentinel singleton: it must ferry the generator’s return value back to whoever drove it. The next(it, default) form, by contrast, swallows the exception entirely and so discards .value — only the try/except StopIteration (or yield from) path can recover it.

Who Else Rides the Protocol

for is the most visible consumer, but the same GET_ITER + tp_iternext machinery underlies far more of the language. Iterable unpacking — a, b, c = obj or first, *rest = obj — compiles to UNPACK_SEQUENCE / UNPACK_EX, which internally obtain an iterator and pull exactly the expected number of items, raising ValueError: too many values to unpack if the iterator yields too many and ValueError: not enough values if StopIteration arrives too early. List/set/dict comprehensions and generator expressions each compile their for clause to the identical GET_ITER/FOR_ITER pair. The built-ins list(), tuple(), set(), dict(), sorted(), min(), max(), sum(), any(), all(), and the entire itertools module are all defined purely in terms of “call iter(), then next() until StopIteration.” Argument unpacking with *args at a call site, and the star-expression inside a literal like [*a, *b], likewise consume the protocol. This is the payoff of having one protocol: implement __iter__/__next__ once and your type plugs into every one of these for free.

There is also an asynchronous sibling. async for and aiter()/anext() use __aiter__ (returning an async iterator) and __anext__ (a coroutine that resolves to the next item or raises StopAsyncIteration). The shape mirrors this protocol exactly, but the methods are awaitable and the exhaustion sentinel is StopAsyncIteration rather than StopIteration; the compiled form uses GET_AITER/GET_ANEXT instead of GET_ITER/FOR_ITER. The synchronous protocol described here is the foundation the async one is patterned on.

Failure Modes and Common Misunderstandings

Calling next() on a non-iterator iterable. next([1,2,3]) raises TypeError: 'list' object is not an iterator. A list is iterable, not an iterator; you must next(iter([1,2,3])). The FOR_ITER body’s tp_iternext == NULL check is the C-level source of this error.

An iterator that doesn’t return self from __iter__. If a custom iterator’s __iter__ returns a fresh object, for x in it: first calls iter(it) (getting the wrong object) — wrap-it-in-a-for then re-iterate it manually and the two see different state. Always return self (or inherit collections.abc.Iterator).

Re-iterating an exhausted iterator. As shown earlier, list(it) twice gives data then empty. This is correct per the monotonic-exhaustion rule, but surprises people who treat an iterator like a re-usable collection.

Forgetting StopIteration is swallowed only at the loop boundary. Inside arbitrary code, a raised StopIteration is an ordinary exception; only FOR_ITER/next()-without-default treat it specially. And inside a generator, PEP 479 turns it into RuntimeError. A bare raise StopIteration to “break” out of a function is an anti-pattern that PEP 479 specifically defeats inside generators.

An infinite iterator with no exhaustion. itertools.count() never raises StopIteration; for x in count(): loops forever. Iterators are not obligated to terminate — termination is purely the StopIteration convention.

See Also

Graph View

Backlinks