vault

Python Tokenizer

23 min read

Python Tokenizer

The tokenizer (also called the lexer or lexical analyzer) is the first stage of CPython’s compilation pipeline: it reads raw source — a stream of bytes — and turns it into a flat stream of tokens, the atomic lexical units (NAME, NUMBER, STRING, OP, NEWLINE, …) that the PEG parser consumes to build an abstract syntax tree. Two things make Python’s tokenizer unusual among programming languages: it synthesizes INDENT/DEDENT tokens so that significant whitespace can encode block structure (there are no { } braces around blocks), and since Python 3.12 it tokenizes f-strings with explicit start/middle/end tokens rather than as opaque strings (per PEP 701). The token kinds themselves are declared once in Grammar/Tokens and code-generated into both a C header (Include/internal/pycore_token.h) and the Python token module (token docs). As of CPython 3.14.5.

Mental Model

Think of the tokenizer as a state machine that scans left-to-right, one character at a time, classifying runs of characters into tokens. Most tokens fall out of simple maximal-munch rules: a run of identifier characters becomes a NAME, a run of digits becomes a NUMBER, a quoted run becomes a STRING. The hard parts — the parts that make a Python tokenizer different from a textbook lexer — are the three pieces of non-local state it must maintain across lines:

  1. An indentation stack that remembers how deeply nested the current block is, so it can emit INDENT when a line moves right and DEDENT when it moves left.
  2. A bracket-nesting counter (level) that tracks open ( [ {, so that newlines inside brackets are treated as insignificant whitespace rather than statement terminators.
  3. A mode stack (since 3.12) that switches the lexer between “regular Python” mode and “inside an f-string” mode, so the expressions embedded in f"{...}" can themselves contain strings, quotes, and braces.
flowchart TD
    SRC["Raw bytes<br/>(source file)"] -->|"detect_encoding:<br/>BOM or PEP 263 cookie"| DEC["Decode to text"]
    DEC --> SCAN["Character scanner<br/>(maximal munch)"]
    SCAN --> IND{"At start<br/>of line?<br/>(atbol)"}
    IND -->|"col vs indstack[indent]"| STACK["Indent stack<br/>push -> pendin++ (INDENT)<br/>pop  -> pendin-- (DEDENT)"]
    IND -->|"level &gt; 0"| SKIP["Inside brackets:<br/>skip indentation,<br/>newline -> NL"]
    SCAN --> TOK["Emit token:<br/>NAME / NUMBER / STRING /<br/>OP / NEWLINE / NL / COMMENT"]
    STACK --> TOK
    TOK --> PARSER["Token stream -> PEG parser"]
    TOK -->|EOF, level==0| END["DEDENT* then ENDMARKER"]

Figure: the source-to-token-stream flow. The insight to take away is that the scanner is mostly local (one character at a time), but three pieces of carried state — the indent stack, the bracket level, and (not shown) the f-string mode stack — are what encode Python’s whitespace-sensitive, brace-free block structure into a flat token stream the parser can consume.

Where the Token Kinds Are Defined

There is a single source of truth for every token type: the file Grammar/Tokens (3.14.5). It is a plain list — bare names like ENDMARKER, NAME, NUMBER, STRING, NEWLINE, INDENT, DEDENT, followed by the operator/delimiter tokens with their literal spellings (LPAR '(', COLON ':', PLUSEQUAL '+=', and so on), and finally the “meta” tokens OP, COMMENT, NL, ERRORTOKEN, the f-string trio, the t-string trio, and ENCODING. A build-time script, Tools/build/generate_token.py, reads this file and code-generates two artifacts that must always agree: the C header Include/internal/pycore_token.h, whose top line is literally the comment // Auto-generated by Tools/build/generate_token.py, and the pure-Python Lib/token.py. Each token gets a small integer id. From the 3.14.5 header, the contiguous numbering is ENDMARKER 0, NAME 1, NUMBER 2, STRING 3, NEWLINE 4, INDENT 5, DEDENT 6, then the operators 7–54, OP 55, SOFT_KEYWORD 58, the f-string tokens FSTRING_START 59 / FSTRING_MIDDLE 60 / FSTRING_END 61, the template-string tokens TSTRING_START 62 / TSTRING_MIDDLE 63 / TSTRING_END 64, COMMENT 65, NL 66, and ERRORTOKEN 67 (pycore_token.h 3.14.5).

A subtlety that matters for understanding the two interfaces below: the very last entry in Grammar/Tokens is preceded by the comment “These aren’t used by the C tokenizer but are needed for tokenize.py”, and that entry is ENCODING. Accordingly the C header stops numbering at ERRORTOKEN 67 and defines N_TOKENS 69 and NT_OFFSET 256 — it never assigns a constant for ENCODING at all. The Python token module does: token.ENCODING == 68. So ENCODING is a fiction invented by the Python-level tokenizer to report the source encoding; the C tokenizer that actually feeds the parser has no such token. (NT_OFFSET 256 is the boundary above which numbers denote grammar non-terminals rather than terminals; the macro ISTERMINAL(x) is (x) < NT_OFFSET. The tokenizer only ever produces terminals.)

Each emitted token, at the Python level, is a TokenInfo named 5-tuple with fields type, string, start, end, line (tokenize docs): the integer type, the literal string matched, the start and end positions as (row, column) pairs (rows 1-based, columns 0-based), and line, the full physical source line the token came from. For an OP token, the property exact_type resolves the generic OP (55) to the precise operator id — e.g. LPAR (7) for ( — which is what python -m tokenize -e prints.

Significant Whitespace: the INDENT/DEDENT Stack

Python has no braces or begin/end keywords; block structure is carried entirely by indentation, and it is the tokenizer — not the parser — that translates layout into structure. It does so by emitting two synthetic tokens, INDENT and DEDENT, that have no textual spelling of their own. The mechanism is an explicit stack of column numbers held in the tokenizer state. From Parser/lexer/state.h (3.14.5), the relevant fields of struct tok_state are:

int tabsize;                 /* Tab spacing */
int indent;                  /* Current indentation index (top of stack) */
int indstack[MAXINDENT];     /* Stack of indents */
int atbol;                   /* Nonzero if at begin of new line */
int pendin;                  /* Pending indents (if > 0) or dedents (if < 0) */
int altindstack[MAXINDENT];  /* Stack of alternate indents */

MAXINDENT is 100, so Python source cannot nest blocks more than ~100 levels deep before the tokenizer fails with E_TOODEEP. The algorithm runs at the beginning of every logical line — when the atbol (“at beginning of line”) flag is set. The scanner walks the leading whitespace and computes a column count col, expanding tabs to the next multiple of tabsize (default 8) and resetting on form-feed (\014, “for Emacs users”). Once it reaches the first non-whitespace character, it compares col to the top of the indent stack, indstack[indent] (lexer.c lines 574–610):

if (col == tok->indstack[tok->indent]) {
    /* No change */
}
else if (col > tok->indstack[tok->indent]) {
    /* Indent -- always one */
    tok->pendin++;
    tok->indstack[++tok->indent] = col;       /* push new column */
    tok->altindstack[tok->indent] = altcol;
}
else /* col < tok->indstack[tok->indent] */ {
    /* Dedent -- any number, must be consistent */
    while (tok->indent > 0 && col < tok->indstack[tok->indent]) {
        tok->pendin--;                         /* pop, one DEDENT each */
        tok->indent--;
    }
    if (col != tok->indstack[tok->indent]) {
        tok->done = E_DEDENT;                  /* dedent to a non-existent level */
        return MAKE_TOKEN(ERRORTOKEN);
    }
}

Reading it line by line: if the new column equals the current top, nothing changes. If it is greater, this is an indent — always exactly one INDENT regardless of how many spaces were added — so pendin is incremented and the new column is pushed onto indstack. If it is less, this is a dedent of possibly many levels: the while loop pops columns off the stack, decrementing pendin once per popped level, until the stack top is <= col. The crucial consistency check follows the loop: after popping, col must exactly match some column that was previously on the stack. If it does not — you dedented to a column that was never an indentation level — the tokenizer sets tok->done = E_DEDENT and returns ERRORTOKEN, which surfaces to the user as the familiar IndentationError: unindent does not match any outer indentation level.

pendin (“pending indents”) is the elegant trick that lets one scan of leading whitespace produce several tokens. The whitespace scan only updates the counter; a separate block, run on each call to fetch the next token, drains it (lexer.c lines 617–637):

if (tok->pendin != 0) {
    if (tok->pendin < 0) {
        tok->pendin++;
        return MAKE_TOKEN(DEDENT);   /* emit one DEDENT, come back for more */
    }
    else {
        tok->pendin--;
        return MAKE_TOKEN(INDENT);
    }
}

So a line that closes three nested blocks at once sets pendin = -3, and the next three token requests each peel off one DEDENT. Note that INDENT/DEDENT tokens carry empty or whitespace strings and zero-width positions for DEDENT — they mark a transition, not a span of text. You can see this in the worked example below, where every DEDENT has string='' and identical start/end.

The altindstack: tabs-vs-spaces consistency

The second stack, altindstack, exists to catch ambiguous tab/space mixing. Alongside the real col (tabs expanded to tabsize, default 8), the scanner computes altcol using ALTTABSIZE (1). If two lines compute the same col but different altcol — meaning they look equal only because of how tabs were expanded — the tokenizer raises an indentation error via _PyTokenizer_indenterror, which surfaces as TabError. TabError subclasses IndentationError, which subclasses SyntaxError (verified: TabError.__mro__ is (TabError, IndentationError, SyntaxError, Exception, …) on 3.14.5). This is why “inconsistent use of tabs and spaces in indentation” is its own distinct error.

NEWLINE versus NL: logical versus non-logical line ends

Python distinguishes a newline that terminates a statement from a newline that is merely cosmetic. The tokenizer encodes this in two different tokens:

  • NEWLINE — a logical newline that ends a statement. The parser depends on these to delimit statements; they are semantically meaningful.
  • NL — a non-logical newline: a blank line, a comment-only line, or a line break inside an open bracket. These carry no grammatical weight.

The decision is made at the \n handler in the lexer (lexer.c lines 808–828):

if (c == '\n') {
    tok->atbol = 1;
    if (blankline || tok->level > 0) {
        if (tok->tok_extra_tokens) {
            return MAKE_TOKEN(NL);    /* blank/comment line, or inside brackets */
        }
        goto nextline;                /* otherwise silently skip */
    }
    ...
    return MAKE_TOKEN(NEWLINE);       /* a real statement terminator */
}

Two conditions produce an NL (or, when not asking for extra tokens, no token at all): blankline is set when the line held only whitespace and/or a comment, and tok->level > 0 means we are inside an open (, [, or {. In both cases the newline does not end a statement. Everywhere else, the tokenizer emits NEWLINE. Note the tok->tok_extra_tokens guard: NL and COMMENT are only produced as tokens when that flag is on. The C tokenizer feeding the parser leaves it off — the parser does not want comment or blank-line noise — so the C path silently swallows them with goto nextline. The tokenize module turns the flag on (see below), which is why your tooling sees NL and COMMENT but the compiler never does.

Line Joining: explicit and implicit continuation

A single logical line can span several physical lines, and there are two ways to do it. Explicit continuation is a backslash immediately before the line break: x = 1 + \ continued on the next line. Implicit continuation happens automatically inside any open bracket pair: as long as level > 0, physical newlines become NL and indentation is ignored, so

x = (1 +
     2)

is one logical statement. The bracket counter lives in tok->level, with a parenstack recording which bracket opened each level. On (/[/{ the tokenizer does tok->parenstack[tok->level] = c; tok->level++; (lexer.c ~line 1300); on a closer it decrements level and checks that the closer matches the opener, raising closing parenthesis ')' does not match opening parenthesis '[' style errors on mismatch. MAXLEVEL is 200, so brackets cannot nest deeper than that. If the file ends while level > 0 — an unclosed bracket — the EOF handler returns ERRORTOKEN rather than a clean ENDMARKER, and the error surfaces as a SyntaxError about an unexpected EOF / unclosed bracket. The worked example earlier shows the implicit case: the \n after 1 + inside the parentheses becomes an NL, and only the \n after the closing ) becomes a real NEWLINE.

The ENCODING token, PEP 263, and bytes-to-text

Python source is bytes on disk; the tokenizer must decide how to decode those bytes to text before it can scan characters. The rules come from PEP 263. First, a UTF-8 byte-order mark (BOM, the three bytes \xef\xbb\xbf) at the very start forces UTF-8. Otherwise the tokenizer looks for an encoding cookie: a comment on the first or second physical line matching the regex ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+), which recognizes all of # coding=utf-8, the Emacs form # -*- coding: utf-8 -*-, and the Vim form # vim: set fileencoding=utf-8 :. If neither a BOM nor a cookie is present, CPython 3 defaults to UTF-8. PEP 263 itself was written in the Python 2 era and specifies an ASCII default; the change to a UTF-8 default for Python 3 is the subject of a separate standard, PEP 3120 “Using UTF-8 as the default source encoding”, which landed in Python 3.0. The behavior is confirmed live on 3.14.5: tokenize.detect_encoding returns 'utf-8' for b'x=1\n' (no cookie), 'utf-8-sig' when a BOM is present, and 'iso-8859-1' when the source carries a # -*- coding: iso-8859-1 -*- cookie.

Resolved (2026-06-01)

The Python-3 UTF-8 default is set by PEP 3120, not PEP 263 (which is the ASCII-default Python-2 standard). BOM → utf-8-sig, latin-1 cookie → iso-8859-1 verified on a live 3.14.5 interpreter.

If a file has both a BOM and a cookie, the only encoding the cookie may name is utf-8; any disagreement raises SyntaxError. This decoding logic is reflected in struct tok_state by enum decoding_state { STATE_INIT, STATE_SEEK_CODING, STATE_NORMAL } and the char *encoding field. In the C tokenizer, the encoding is applied (used to decode) but never reported as a token. In the Python tokenize module it surfaces as the synthetic ENCODING token, always the very first token of the stream, with position (0, 0) (row 0 — below the 1-based source lines) and string equal to the encoding name. You can see it in the byte-mode example below.

Two Surfaces, One Engine: the C tokenizer and the tokenize module

The brief framing of “a fast C tokenizer versus a separate pure-Python tokenize module” is outdated for 3.12+, and getting it right matters. There are not two independent lexers. As of CPython 3.14 there is one engine — the C tokenizer in Parser/tokenizer/ and Parser/lexer/ — exposed through two surfaces.

The C engine itself is split across small files: Parser/lexer/lexer.c holds the core tok_get scanning loop and the indent/bracket/f-string logic; Parser/lexer/state.c and state.h define and manage struct tok_state; Parser/lexer/buffer.c handles the input buffer; and Parser/tokenizer/ provides the input sources — file_tokenizer.c (read from a FILE*), string_tokenizer.c (tokenize a char*), readline_tokenizer.c (drive a Python readline callable), and utf8_tokenizer.c (dir listing, 3.14.5). This is the path the compiler uses: it scans tokens straight into the PEG parser, with tok_extra_tokens off, so it never materializes COMMENT, NL, or ENCODING tokens.

The tokenize module (Lib/tokenize.py) is the tooling-facing surface — used by linters, formatters (Black), syntax highlighters, 2to3-style rewriters, and anything that must inspect or rewrite source text faithfully. Crucially, it does not re-implement lexing. Both its public entry points delegate to the same C engine through a thin C extension, _tokenize.TokenizerIter (Lib/tokenize.py 3.14.5):

def tokenize(readline):                       # bytes input
    ...
    yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')   # synthesize ENCODING first
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
 
def generate_tokens(readline):                # str input
    return _generate_tokens_from_c_tokenizer(readline, extra_tokens=True)
 
def _generate_tokens_from_c_tokenizer(source, encoding=None, extra_tokens=False):
    it = _tokenize.TokenizerIter(source, encoding=encoding, extra_tokens=extra_tokens)
    for info in it:
        yield TokenInfo._make(info)

Two differences from the compiler’s use of the same engine are visible here. First, extra_tokens=True is passed, flipping on tok_extra_tokens so the engine does yield COMMENT and NL tokens — exactly what a faithful source tool needs and what the tok->tok_extra_tokens guards in the C code gate on. Second, tokenize.tokenize (the bytes API) prepends the synthetic ENCODING token in Python before delegating; generate_tokens (the unicode-string API) does not, because there are no bytes to decode. This is the only behavioral difference between the two public functions worth memorizing: same tokens otherwise, but tokenize leads with ENCODING and generate_tokens does not.

The historical regex-based pure-Python tokenizer still leaves vestiges in Lib/tokenize.py: the module-level PseudoToken regular expression is still defined (line 136 of the v3.14.5 file), but it is not on the live tokenizing path — both tokenize (line 472) and generate_tokens (line 500) route through _generate_tokens_from_c_tokenizer (line 581), which constructs _tokenize.TokenizerIter. PseudoToken survives only as a public name external tools may import. The unification of the two implementations onto the single C engine is the work of PEP 701 “Syntactic formalization of f-strings”, whose Python-Version header is 3.12 — that release rewrote the tokenizer and routed the tokenize module through the C engine.

The module also offers untokenize(iterable), which is genuinely pure-Python: it reconstructs source text from a token iterable and is the basis of round-tripping. The documented guarantee (tokenize docs) is that the result is lossless with respect to token type and string — untokenize(tokenize(readline)) tokenizes back to the same stream — though exact whitespace may differ if you only feed two-element (type, string) tuples. Helpers detect_encoding(readline) (returns (encoding, [lines_read]), reading at most two lines) and tokenize.open(filename) (opens a file with its declared encoding) round out the public API. The exception raised on an incomplete construct — an unterminated multi-line string or an unclosed bracket at EOF — is TokenError (not TokenizeError); the delegating wrapper above catches the engine’s SyntaxError, rewrites the message, and re-raises it as TokenError.

PEP 701: F-String Tokenization (3.12+)

Before Python 3.12, the tokenizer treated an entire f-string as a single opaque STRING token; the structure inside {...} was teased apart later, by special-case code in the parser/compiler operating on the string’s text. That post-hoc approach is why pre-3.12 f-strings had irritating, arbitrary restrictions: you could not reuse the same quote character inside an expression (f"{d["k"]}" was a syntax error), you could not put a backslash inside the expression part, nesting depth was capped, and # comments were forbidden inside braces. PEP 701 “formalized” f-strings by making the tokenizer understand them, and shipped in Python 3.12.

The tokenizer now emits three distinct tokens for an f-string (PEP 701; tokenize docs):

  • FSTRING_START — the prefix (f, F, rf, fr, …) together with the opening quote(s).
  • FSTRING_MIDDLE — a run of literal text that is neither an opening/closing brace nor part of an embedded expression.
  • FSTRING_END — the closing quote(s).

The expressions between the { and } are tokenized as ordinary Python — real NAME, OP, STRING, etc. tokens — because the lexer literally switches back into regular mode for them. The mechanism is the tokenizer mode stack in struct tok_state: tokenizer_mode tok_mode_stack[MAXFSTRINGLEVEL] with tok_mode_stack_index (state.h 3.14.5). Each tokenizer_mode records whether it is TOK_REGULAR_MODE or TOK_FSTRING_MODE, the quote character and size, and curly_bracket_depth. When the scanner sees f", it pushes a new f-string mode; inside, a { pushes back into regular tokenization for the embedded expression and the matching } pops back; the closing quote pops the f-string mode entirely. MAXFSTRINGLEVEL is 150, so f-strings can nest 150 deep. Because the embedded expression is tokenized by the same regular machinery — including its own quote and bracket handling — the old restrictions simply evaporate: same-quote reuse, backslashes, comments, and arbitrary nesting like f"{f"{f"{1+1}"}"}" all became legal in 3.12 as a direct consequence of this design (PEP 701).

The worked example below shows this concretely: f"Hi {name}!" becomes FSTRING_START('f"'), FSTRING_MIDDLE('Hi '), OP('{'), NAME('name'), OP('}'), FSTRING_MIDDLE('!'), FSTRING_END('"').

Python 3.14 adds an exact parallel for template strings (PEP 750): the t"..." prefix tokenizes into TSTRING_START (62), TSTRING_MIDDLE (63), and TSTRING_END (64), using the identical mode-stack machinery (the f-string mode’s enum string_kind_t is either FSTRING or TSTRING). The difference is purely semantic downstream — a t"..." evaluates to a Template object rather than a str — but at the tokenizer level they are the same construct. Verified on 3.14.5: t"Hi {name}" yields TSTRING_START('t"'), TSTRING_MIDDLE('Hi '), OP('{'), NAME('name'), OP('}'), TSTRING_END('"').

Worked Example: an indented function on a real 3.14.5 interpreter

The following is the verbatim output of tokenize.generate_tokens on a small function, captured from CPython 3.14.5. Annotations follow.

src = '''def greet(name):
    if name:
        return f"Hi {name}!"
    return None
'''
import tokenize, io
for tok in tokenize.generate_tokens(io.StringIO(src).readline):
    print(tok)
NAME    'def'    (1,0)-(1,3)
NAME    'greet'  (1,4)-(1,9)
OP      '('      (1,9)-(1,10)
NAME    'name'   (1,10)-(1,14)
OP      ')'      (1,14)-(1,15)
OP      ':'      (1,15)-(1,16)
NEWLINE '\n'     (1,16)-(1,17)
INDENT  '    '   (2,0)-(2,4)        <- block opens: push col 4
NAME    'if'     (2,4)-(2,6)
NAME    'name'   (2,7)-(2,11)
OP      ':'      (2,11)-(2,12)
NEWLINE '\n'     (2,12)-(2,13)
INDENT  '        ' (3,0)-(3,8)      <- deeper block: push col 8
NAME    'return' (3,8)-(3,14)
FSTRING_START  'f"'    (3,15)-(3,17)   <- f-string begins
FSTRING_MIDDLE 'Hi '   (3,17)-(3,20)
OP             '{'     (3,20)-(3,21)   <- back to regular mode
NAME           'name'  (3,21)-(3,25)
OP             '}'     (3,25)-(3,26)
FSTRING_MIDDLE '!'     (3,26)-(3,27)
FSTRING_END    '"'     (3,27)-(3,28)   <- f-string ends
NEWLINE '\n'     (3,28)-(3,29)
DEDENT  ''       (4,4)-(4,4)        <- pop col 8 (zero-width, no text)
NAME    'return' (4,4)-(4,10)
NAME    'None'   (4,11)-(4,15)
NEWLINE '\n'     (4,15)-(4,16)
DEDENT  ''       (5,0)-(5,0)        <- pop col 4 at EOF
ENDMARKER ''     (5,0)-(5,0)

Walking it: line 1 tokenizes as you would expect, ending in a NEWLINE (a logical statement terminator — def greet(name): is a complete header). Moving to line 2’s body emits one INDENT (note its string is the actual ' ' and it spans the leading whitespace). Line 3 indents again — one more INDENT, column 8 pushed — and contains the f-string broken into its five PEP-701 tokens around the regular-mode {name}. Line 4 dedents back to column 4: a single DEDENT with an empty string and a zero-width position (4,4)-(4,4), because a dedent marks a transition, not text. At EOF, the tokenizer drains the remaining indentation: it pops the last level (a final DEDENT at (5,0)) and emits the terminal ENDMARKER. Every well-formed stream ends in ENDMARKER, and the indent stack is always fully unwound to column 0 first.

A second example, fed as bytes via tokenize.tokenize, shows the tokens the compiler never sees — ENCODING, COMMENT, and the implicit-continuation NL:

src = b'''# comment line
x = (1 +
     2)
'''
for t in tokenize.tokenize(io.BytesIO(src).readline):
    print(t)
ENCODING 'utf-8'  (0,0)-(0,0)     <- synthetic, only from tokenize() not generate_tokens()
COMMENT  '# comment line' (1,0)-(1,14)
NL       '\n'     (1,14)-(1,15)   <- comment-only line: NL, not NEWLINE
NAME     'x'      (2,0)-(2,1)
OP       '='      (2,2)-(2,3)
OP       '('      (2,4)-(2,5)     <- level -> 1
NUMBER   '1'      (2,5)-(2,6)
OP       '+'      (2,7)-(2,8)
NL       '\n'     (2,8)-(2,9)     <- newline INSIDE brackets: NL (implicit continuation)
NUMBER   '2'      (3,5)-(3,6)
OP       ')'      (3,6)-(3,7)     <- level -> 0
NEWLINE  '\n'     (3,7)-(3,8)     <- now a real statement terminator
ENDMARKER ''      (4,0)-(4,0)

The ENCODING token leads the stream with the detected encoding and a row-0 position. The comment line and the line break inside the open parenthesis both become NL; only the newline after the closing ) — when level is back to 0 — is a NEWLINE. This is the single clearest demonstration of the NEWLINE/NL distinction and of implicit line joining.

Failure Modes and How Errors Surface

Tokenizer-level errors do not produce a special “tokenizer exception” to the end user — they are mapped to SyntaxError (or one of its subclasses) before they reach you. Inside the C tokenizer the failure is recorded as an error code in tok->done and an ERRORTOKEN is returned: E_TOODEEP for more than MAXINDENT (100) indentation levels, E_DEDENT for a dedent that matches no outer level, E_TABSPACE (via _PyTokenizer_indenterror) for inconsistent tab/space use, and a returned ERRORTOKEN at EOF while level > 0 for an unclosed bracket. These become, respectively, IndentationError (“too many levels of indentation” / “unindent does not match any outer indentation level”), TabError (“inconsistent use of tabs and spaces in indentation”), and SyntaxError about an unexpected EOF or unclosed bracket. All of IndentationError and TabError are subclasses of SyntaxError, so a blanket except SyntaxError catches them all.

When you use the tokenize module on incomplete input — say an unterminated triple-quoted string or a bracket that never closes before EOF — the module raises tokenize.TokenError instead, carrying the message and the (lineno, offset) where the construct began. A separate trap: the docs explicitly warn that tokenize is only defined on syntactically valid Python; on invalid input its behavior “is undefined and can change at any point” (tokenize docs). Tools that try to lex broken code mid-edit (as IDEs do) must therefore be defensive and not rely on the exact tokens produced for malformed source.

Alternatives and Boundaries

The tokenizer is deliberately not the parser. It performs no grammar matching, no precedence resolution, no scope analysis — it only classifies characters into a flat token stream. Everything structural happens one stage later in the PEG parser, which consumes this stream. The split is the classic lexer/parser separation, and it is why the same token stream can feed both the compiler and the tokenize module’s tooling consumers. For inspecting or rewriting source you reach for tokenize; for inspecting the parsed structure you reach for the [[Python Abstract Syntax Tree|ast module]]; for the compiled result you reach for [[The dis Module and Bytecode Disassembly|dis]]. A subtle point on choosing between tokenize and ast: tokenize preserves comments and exact text (great for formatters), whereas ast discards them (better for semantic analysis). Use the one whose level of abstraction matches the task.

Production Notes

The tokenize module is the backbone of Python’s source-tooling ecosystem precisely because it is a faithful, lossless view of the lexical layer. Auto-formatters and linters lean on its comment and NL preservation; the standard library’s own python -m tokenize -e file.py (exact-type mode, added in 3.3) is a handy way to see what the lexer makes of a file. The PEP 701 rework was driven by the Faster CPython team (see The Faster CPython Project) and the f-string maintainers, and the payoff was twofold: simpler, faster f-string handling in the compiler, and the removal of long-standing arbitrary restrictions — a rare change that made the language both faster and more expressive at once. The historical migration of Lib/tokenize.py from an independent regex lexer onto the shared C engine (in 3.12) eliminated a perennial class of bugs where the two implementations disagreed on edge cases — a reminder that “two implementations of the same spec” is a liability, and unifying them onto one engine with two surfaces is the more robust design.

See Also

Graph View

Backlinks