vault

Bytecode Caching and pyc Files

13 min read

Bytecode Caching and pyc Files

Compiling Python source to bytecode is not free — it means tokenizing, PEG-parsing, building symbol tables, and emitting a code object. To avoid paying that cost on every import, CPython caches the compiled code object on disk as a .pyc (“Python compiled”) file, serialized with the marshal protocol and stamped with a small header that lets the import machinery decide, cheaply, whether the cache is still valid for the current source and interpreter. Since PEP 3147 (Python 3.2) these files live in a per-directory __pycache__/ folder under an implementation-and-version tag like foo.cpython-314.pyc, so multiple Python versions can coexist over one source tree without clobbering each other’s caches. PEP 552 (Python 3.7) then added an optional hash-based validation mode so a .pyc can be a deterministic function of the source content — essential for reproducible builds. The whole mechanism is implemented in pure Python in Lib/importlib/_bootstrap_external.py.

Mental Model

A .pyc file is “a marshalled code object with a 16-byte sanity header bolted on the front.” The header exists for exactly one job: to answer “is this cache safe to use, or must I recompile?” — without ever decoding the bytecode. Two independent questions must both pass: (1) Was this .pyc produced by a compatible bytecode format? — answered by the magic number. (2) Does the cached bytecode still match the current source? — answered either by the source’s modification time (timestamp mode) or by a hash of the source bytes (hash mode). Only if both checks pass does the import machinery marshal.loads() the rest of the file into a live code object; otherwise it recompiles and rewrites the cache.

graph TD
    IMP["import foo"] --> SRC{"foo.py exists?"}
    SRC -->|"yes"| PYC{"__pycache__/foo.cpython-314.pyc<br/>exists?"}
    PYC -->|"no"| COMPILE["compile foo.py → code object"]
    PYC -->|"yes"| MAGIC{"header magic ==<br/>this interpreter's magic?"}
    MAGIC -->|"no (stale format)"| COMPILE
    MAGIC -->|"yes"| MODE{"flags bit0:<br/>timestamp or hash?"}
    MODE -->|"timestamp"| TS{"header mtime+size ==<br/>source stat?"}
    MODE -->|"hash (checked)"| HASH{"header hash ==<br/>siphash(source)?"}
    TS -->|"no"| COMPILE
    TS -->|"yes"| LOAD["marshal.loads(body) → code object"]
    HASH -->|"no"| COMPILE
    HASH -->|"yes"| LOAD
    COMPILE -->|"write back"| WRITE["marshal.dumps + header<br/>→ atomic write to __pycache__"]
    WRITE --> LOAD

Figure: the get_code() decision tree from _bootstrap_external.py. The insight: every “no” branch funnels to recompile-then-rewrite, and the magic/mtime/hash checks are deliberately ordered cheapest-first (a 4-byte compare, then a stat compare, and only in hash mode the more expensive source read + SipHash). The bytecode is never decoded until all gates pass.

The __pycache__ Directory and the Cache Tag

Before PEP 3147, .pyc files sat next to their .py source (foo.py → foo.pyc). That broke the moment two Python versions shared a source tree: each would overwrite the other’s foo.pyc because the magic numbers differed, so they would “fight over the pyc file and rewrite it each time the source is compiled” (per PEP 3147). The fix is a dedicated subdirectory plus a version tag in the filename, so every implementation/version gets its own cache file:

mypkg/
    __init__.py
    util.py
    __pycache__/
        __init__.cpython-314.pyc
        util.cpython-314.pyc
        util.cpython-313.pyc      ← 3.13 cache coexists peacefully

The tag (cpython-314) comes from sys.implementation.cache_tag, which PEP 3147 specifies “should contain the implementation name and a version number shorthand, e.g. cpython-32.” On a 3.14 interpreter sys.implementation.cache_tag == 'cpython-314' (verified on 3.14.5). The mapping between a source path and its cache path is computed by importlib.util.cache_from_source(path) and reversed by source_from_cache(path); the loaded module also gets a __cached__ attribute pointing at the actual .pyc used. When the source is missing (sdist shipping only .pycs), the __pycache__ files are ignored and only a legacy foo.pyc directly beside where the source would be is honoured — the “sourceless distribution” path.

Optimization levels add a second tag segment. Running under -O (drop asserts / __debug__) or -OO (also strip docstrings) yields foo.cpython-314.opt-1.pyc / foo.cpython-314.opt-2.pyc, so optimized and unoptimized caches don’t collide. The cache_from_source code builds this name explicitly: it joins the base name, the cache tag, an optional .opt-N segment, and .pyc.

The 16-Byte .pyc Header — Field by Field

PEP 552 expanded the header from PEP 3147’s 12 bytes to 16 bytes — four 32-bit little-endian words. The exact layout, and the validation that reads it, are in _bootstrap_external.py. The _classify_pyc function reads it:

magic = data[:4]                       # word 0: magic number
if magic != MAGIC_NUMBER: raise ImportError(...)   # wrong bytecode format
if len(data) < 16: raise EOFError(...)             # header truncated
flags = _unpack_uint32(data[4:8])      # word 1: bit field
if flags & ~0b11: raise ImportError(...)           # only 2 flag bits defined
  • Bytes 0–3 — magic number. A 4-byte signature identifying the bytecode format. If it doesn’t match the running interpreter’s, the cache is from an incompatible version and is discarded. Detailed below.
  • Bytes 4–7 — flag bit field. Only the two low bits are defined (flags & ~0b11 is rejected). Bit 0 = invalidation mode: 0 → timestamp-based, 1 → hash-based. Bit 1 = the check_source flag (only meaningful when bit 0 is set): 1 → “checked” hash pyc, 0 → “unchecked”.
  • Bytes 8–15 — mode-dependent. Their meaning depends on bit 0:
    • Timestamp mode (bit 0 = 0): bytes 8–11 = source mtime (low 32 bits, seconds), bytes 12–15 = source size (low 32 bits). This is the PEP 3147 layout, still the default.
    • Hash mode (bit 0 = 1): bytes 8–15 = the 8-byte (64-bit) SipHash of the source bytes. There is no mtime or size.

The two writers make the layout unambiguous (from the same file):

def _code_to_timestamp_pyc(code, mtime=0, source_size=0):
    data = bytearray(MAGIC_NUMBER)          # bytes 0-3
    data.extend(_pack_uint32(0))            # bytes 4-7: flags = 0 (timestamp)
    data.extend(_pack_uint32(mtime))        # bytes 8-11
    data.extend(_pack_uint32(source_size))  # bytes 12-15
    data.extend(marshal.dumps(code))        # body
    return data
 
def _code_to_hash_pyc(code, source_hash, checked=True):
    data = bytearray(MAGIC_NUMBER)          # bytes 0-3
    flags = 0b1 | checked << 1              # bit0=hash; bit1=checked
    data.extend(_pack_uint32(flags))        # bytes 4-7
    assert len(source_hash) == 8
    data.extend(source_hash)                # bytes 8-15: 64-bit SipHash
    data.extend(marshal.dumps(code))        # body
    return data

Everything after byte 16 is the marshal.dumps() of the code object — the payload.

The Magic Number and Why It Changes

The magic number is the format guard. PEP 3147 puts it well: when Python “encounters a .pyc file with a mismatched magic number” it recompiles. The value changes whenever the bytecode the compiler emits can no longer be understood by an older eval loop — typically when an opcode is added, removed, or renumbered. In 3.14 the canonical value moved into a C header, Include/internal/pycore_magic_number.h, and is exposed to the Python layer as _imp.pyc_magic_number_token; importlib.util.MAGIC_NUMBER is derived from it. The header documents the modern numbering scheme:

“Starting with Python 3.11, Python 3.n starts with magic number 2900+50n. Within each minor version, the magic number is incremented by 1 each time the file format changes.”

So 3.14 starts at 2900 + 50×14 = 3600, and each bytecode-format change during the 3.14 development cycle bumped it by one. The final shipped 3.14 value is 3627 (the last entry in the header’s history, “Python 3.14rc3 3627 (Fix miscompilation of some module-level annotations)”):

#define PYC_MAGIC_NUMBER 3627
/* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes
   (little-endian) and then appending b'\r\n'. */
#define PYC_MAGIC_NUMBER_TOKEN \
    ((uint32_t)PYC_MAGIC_NUMBER | ((uint32_t)'\r' << 16) | ((uint32_t)'\n' << 24))

The on-disk 4 bytes are therefore 3627 as a 2-byte little-endian integer (0x2B 0x0E, since 3627 = 0x0E2B), followed by \r\n (0x0D 0x0A). Verified on 3.14.5: importlib.util.MAGIC_NUMBER == b'\x2b\x0e\x0d\x0a' (bytes [43, 14, 13, 10]). The trailing carriage-return/line-feed is a deliberate booby-trap, explained in the header’s comment: “if you ever read or write a .pyc file in text mode the magic number will be wrong” — a text-mode transfer would mangle the \r\n and the corruption would be caught immediately by the magic check rather than producing a subtly broken bytecode load. (3.15 will start at 3650, per the same header.)

Timestamp vs Hash-Based Invalidation

Timestamp invalidation (the default) is what _validate_timestamp_pyc enforces: it compares the header’s stored mtime against the source file’s current modification time (truncated to 32 bits), and, if a size was stored, the source size too. If either differs, “bytecode is stale” → recompile. This is cheap (just a stat) but has a real weakness: mtime is volatile metadata unrelated to the source content. Checking out the same commit on two machines, or unpacking a tarball, can give the source a fresh mtime even though the bytes are identical — invalidating a perfectly good cache — and conversely, restoring an old mtime can make a changed file look unchanged.

Hash invalidation (PEP 552) fixes determinism. Instead of mtime+size, the header stores a 64-bit SipHash of the source bytes (reusing CPython’s PEP 456 hashing with a fixed key — “Security isn’t the primary concern; fast, auditable hashing is prioritized”). Because the .pyc now depends only on the source content (plus the magic number), identical source compiles to a byte-identical .pyc — the property reproducible-build systems (Debian, Nix, Bazel) need. Hash pycs come in two sub-flavours, selected by header bit 1:

  • Checked (check_source = 1): on import, Python reads the source, recomputes the SipHash, and compares — re-validating against the source like a content-aware timestamp check. Safe default; still pays a source read + hash per import.
  • Unchecked (check_source = 0): Python loads the .pyc without reading the source at all, trusting an external system (a package manager) to keep them in sync. Fastest, but a stale source goes unnoticed. PEP 552: “For hash-based pycs with the check_source unset, Python will simply load the pyc without checking the hash of the source file.”

The _validate_hash_pyc check is just if data[8:16] != source_hash: raise ImportError. The interpreter-wide override is --check-hash-based-pycs (a.k.a. _imp.check_hash_based_pycs), a tristate: default respects each pyc’s check_source bit; always forces hash re-validation even on unchecked pycs; never skips hash validation entirely (timestamp pycs are unaffected). You can see all three consulted in get_code():

hash_based = flags & 0b1 != 0
if hash_based:
    check_source = flags & 0b10 != 0
    if (_imp.check_hash_based_pycs != 'never' and
        (check_source or _imp.check_hash_based_pycs == 'always')):
        source_hash = _imp.source_hash(_imp.pyc_magic_number_token, source_bytes)
        _validate_hash_pyc(data, source_hash, fullname, exc_details)
else:
    _validate_timestamp_pyc(data, source_mtime, st['size'], fullname, exc_details)

How marshal Serializes the Code Object

The body of a .pyc (everything past byte 16) is marshal.dumps(code_object). marshal is CPython’s internal binary serialization format — distinct from pickle — purpose-built for the small set of types that appear in a code object. The marshal docs enumerate the supported types: the numeric types int/bool/float/complex; str and bytes; the containers tuple, list, set, frozenset, and (since format version 5) slice; the singletons None, Ellipsis, and StopIteration; and code objects themselves “if allow_code is true” — plus references between them. It is deliberately not a general or stable format: the docs state “This is not a general ‘persistence’ module,” and “Details of the format are undocumented on purpose; it may change between Python versions (although it rarely does).” They add the key warning for .pyc purposes: “The format of code objects is not compatible between Python versions, even if the version of the format is the same” — which is exactly why the magic number, not the marshal version, guards a .pyc. Each value is written as a one-byte type tag followed by the type’s payload (e.g. a tuple tag, then a length, then each element recursively). The current format is Py_MARSHAL_VERSION = 5 (per Include/marshal.h); marshal.version “indicates the format that the module uses,” and the writer deduplicates repeated objects via a FLAG_REF back-reference scheme to keep the output compact. This contrasts sharply with the self-describing, schema-driven [[The struct Module and Binary Layout|struct module]] (fixed C-layout packing of numbers) and with [[The pickle Protocol|pickle]] (a general, versioned object-graph format): marshal trades all generality and stability for speed and simplicity, betting that the magic number makes cross-version compatibility a non-issue.

A subtle but important step happens on load: when marshal.loads() reconstructs the code object, the trailing co_code_adaptive array is a flat byte sequence with no initialised inline caches. So creating a code object — including from a .pyc — runs _PyCode_Quicken() to initialise those caches “because the on-disk format is a sequence of bytes, and some of the caches need to be initialized with 16-bit values” (per the code_objects.md internal doc). See Code Objects for what those fields are.

Writing Caches Eagerly: compileall, py_compile, -X Flags

By default the cache is written lazily as a side effect of the first import (the get_code() write-back path, an atomic temp-file-then-os.replace). Several tools force it eagerly or change its mode:

  • compileall byte-compiles whole directory trees ahead of time — the standard way distro/Docker builds warm the cache so the first run isn’t slow. Its --invalidation-mode flag selects the pyc mode: timestamp (default), checked-hash, or unchecked-hash — mapping to PEP 552’s TIMESTAMP / CHECKED_HASH / UNCHECKED_HASH.
  • py_compile.compile(file, invalidation_mode=...) compiles a single file with the same mode choice; this is the programmatic entry point.
  • PYTHONDONTWRITEBYTECODE=1 / -B / sys.dont_write_bytecode suppress cache writing entirely (read-only filesystems, ephemeral containers).
  • PYTHONPYCACHEPREFIX / sys.pycache_prefix redirect all __pycache__ output under one mirror tree instead of scattering it through the source — handy for read-only source mounts.
  • -X no_debug_ranges (and PYTHONNODEBUGRANGES) omits the per-instruction column information from co_positions() to shrink .pyc size; a .pyc compiled this way carries no column data, and co_positions() will yield Nones (per the data model docs).

Failure Modes and Gotchas

  • bad magic number in 'foo' on import: the .pyc was produced by a different CPython version (or a non-CPython implementation). Harmless — Python recompiles — but a sign you copied .pycs between versions. Delete __pycache__.
  • Stale cache after editing on a fast filesystem. Timestamp resolution is per-second-ish; an edit-and-immediately-rerun within the same mtime tick can leave the cache looking fresh. The size field (bytes 12–15) catches most such cases, but identical-length edits are the classic trap. Hash mode eliminates it.
  • Shipping .pyc without source as “obfuscation.” A sourceless distribution works (legacy foo.pyc beside the source location), but marshal is trivially reversible with dis — it is not protection. And it’s version-locked: a .pyc only loads on the matching magic number.
  • Reproducibility broken by timestamp pycs. Two CI runs of the same commit produce different .pyc bytes (different mtimes) → cache-busting downstream. Build with compileall --invalidation-mode checked-hash (or unchecked) for byte-stable artifacts. This was PEP 552’s entire motivation.
  • Clock skew / restored mtimes. Restoring an old mtime onto an edited file can make a changed source look unchanged in timestamp mode — a real footgun for rsync/backup-restore workflows; hash mode is the cure.

See Also

Graph View

Backlinks