vault

Frozen and Built-in Modules

16 min read

Frozen and Built-in Modules

Some modules are not on disk at all — they live inside the python executable. CPython distinguishes two such families. A built-in module is a C extension compiled and statically linked into the interpreter binary; its code is machine code, found through a fixed table (_PyImport_Inittab) and surfaced in sys.builtin_module_names. A frozen module is an ordinary Python module whose compiled bytecode was marshalled at build time and embedded in the binary as a C byte array; at run time the FrozenImporter unmarshals that array into a code object and executes it, with no filesystem access. Both families exist to solve a bootstrap chicken-and-egg problem — the import machinery itself is written in Python, so it cannot be imported from disk by the very machinery it implements — and, since CPython 3.11, freezing the stdlib startup modules also makes interpreter startup measurably faster (per the import reference and Python/frozen.c). This note describes both families as of CPython 3.14.5.

Mental Model: Two Ways to Be “Inside the Binary”

The everyday import path reads a .py file from disk, compiles it, and caches the result in a .pyc (see Bytecode Caching and pyc Files). Frozen and built-in modules short-circuit that path entirely. The distinction between them is what is embedded:

  • A built-in module embeds compiled C — the module’s behavior is native machine code produced by your C compiler when CPython was built. sys, builtins, _thread, gc, marshal, time, and itertools are built-ins. There is no Python source for them at all (the .py you might find, like Lib/_collections_abc.py, is a different module; the C built-ins have names like _collections, _thread, _io).
  • A frozen module embeds marshalled Python bytecode — the exact same bytes a .pyc would hold, but stored as a const unsigned char[] array compiled into the binary rather than as a file. At run time it is unmarshalled (deserialized) into a code object and executed like any other Python module. importlib._bootstrap, os, site, and abc are frozen.
flowchart TD
    IMP["import X<br/>(__import__ walks sys.meta_path)"] --> BI{"BuiltinImporter<br/>_imp.is_builtin(X)?"}
    BI -->|yes| BIC["create_builtin:<br/>scan _PyImport_Inittab,<br/>call PyInit_X() (C code)"]
    BI -->|no| FI{"FrozenImporter<br/>_imp.find_frozen(X)?"}
    FI -->|yes| FIC["get_frozen_object:<br/>unmarshal _Py_M__X[] byte array<br/>into a code object, then exec"]
    FI -->|no| PF["PathFinder:<br/>search sys.path on disk<br/>(.py / .pyc / extension .so)"]
    BIC --> MOD["module object in sys.modules"]
    FIC --> MOD
    PF --> MOD

Figure: the first two meta-path finders handle in-binary modules before the disk-based PathFinder ever runs. The insight: built-in and frozen modules are resolved by membership in a static table or array, not by any filesystem search — which is precisely why they are available before sys.path is even set up. See Finders and Loaders for the full meta-path protocol.

Built-in Modules: The Inittab Table

A built-in module is registered in a C array of struct _inittab entries. The canonical table is _PyImport_Inittab, generated from Modules/config.c.in by the makesetup build script (which reads Modules/Setup to decide which optional modules are statically linked versus built as shared .so extensions versus disabled). Each entry pairs a module name with the address of its module-initialization function:

struct _inittab _PyImport_Inittab[] = {
    /* This module lives in marshal.c */
    {"marshal", PyMarshal_Init},
    /* This lives in import.c */
    {"_imp", PyInit__imp},
    /* This lives in Python/Python-ast.c */
    {"_ast", PyInit__ast},
    /* These entries are here for sys.builtin_module_names */
    {"builtins", NULL},
    {"sys", NULL},
    /* This lives in gcmodule.c */
    {"gc", PyInit_gc},
    /* ... */
    {0, 0}  /* Sentinel */
};

Reading this verbatim from config.c.in: each {name, initfunc} pair names a module and points at a PyInit_<name> C function. Two entries are special — {"builtins", NULL} and {"sys", NULL} have a NULL initfunc. Those two modules are constructed by the runtime before the import system exists (they are needed to even start the interpreter), so they cannot be re-initialized through the normal path; the comment in the source — “These entries are here for sys.builtin_module_names” — says exactly why they appear at all: so that they show up in the list of built-in names even though they are bootstrapped specially. The {0, 0} sentinel terminates the table.

The Python-visible side of this is the BuiltinImporter meta-path finder. Its find_spec is a one-liner — it asks the C layer whether the name is in the table:

class BuiltinImporter:
    _ORIGIN = "built-in"
 
    @classmethod
    def find_spec(cls, fullname, path=None, target=None):
        if _imp.is_builtin(fullname):
            return spec_from_loader(fullname, cls, origin=cls._ORIGIN)
        else:
            return None
 
    @staticmethod
    def create_module(spec):
        if spec.name not in sys.builtin_module_names:
            raise ImportError(f'{spec.name!r} is not a built-in module', name=spec.name)
        return _call_with_frames_removed(_imp.create_builtin, spec)
 
    @staticmethod
    def exec_module(module):
        _call_with_frames_removed(_imp.exec_builtin, module)

Walking this line by line: find_spec calls _imp.is_builtin(fullname), a C function that scans _PyImport_Inittab for a matching name; if found, it returns a module spec whose loader is BuiltinImporter itself and whose origin string is "built-in". create_module defends against names that slipped through (if spec.name not in sys.builtin_module_names) and then delegates to _imp.create_builtin. exec_module calls _imp.exec_builtin. The _call_with_frames_removed wrapper hides the importlib internals from tracebacks so user-facing stack traces are not cluttered with bootstrap frames.

_imp.create_builtin is implemented by create_builtin in Python/import.c. The heart of it walks the inittab:

struct _inittab *found = NULL;
for (struct _inittab *p = INITTAB; p->name != NULL; p++) {
    if (_PyUnicode_EqualToASCIIString(info.name, p->name)) {
        found = p;
    }
}
/* ... */
PyModInitFunction p0 = (PyModInitFunction)found->initfunc;
if (p0 == NULL) {
    /* Cannot re-init internal module ("sys" or "builtins") */
    mod = import_add_module(tstate, info.name);
    goto finally;
}
mod = import_run_extension(tstate, p0, &info, spec, get_modules_dict(tstate, true));

It loops over INITTAB looking for a name match; when initfunc is NULL (the sys/builtins case) it just retrieves the already-created module from sys.modules via import_add_module; otherwise it calls import_run_extension, which invokes the C PyInit_<name>() function. That init function is the same multi-phase or single-phase extension-init machinery that any C extension uses — a built-in module is, mechanically, a statically linked C extension. Note that BuiltinImporter.get_code and get_source both return None: a built-in has no Python code object and no source, which is why inspect.getsource(itertools) raises.

sys.builtin_module_names is the tuple of every name in the inittab, documented as “the names of all modules that are compiled into this Python interpreter” (per the sys docs). On the 3.14.5 build inspected here it held 37 names, including _abc, _io, _thread, builtins, gc, itertools, marshal, posix, sys, and time. The exact set varies by platform and build configuration — posix versus nt, the presence of pwd, and which optional modules Modules/Setup chose to link statically all shift the list.

Frozen Modules: Marshalled Bytecode in a Byte Array

A frozen module is described by struct _frozen (Include/cpython/import.h):

struct _frozen {
    const char *name;             /* ASCII encoded string */
    const unsigned char *code;
    int size;
    int is_package;
};

Walking the fields: name is the module’s dotted name; code points at a static byte array holding the module’s marshalled code object; size is its length; is_package flags whether the module is a package (it gets a __path__). Python/frozen.c collects these into three tables — bootstrap_modules, stdlib_modules, and test_modules:

static const struct _frozen bootstrap_modules[] = {
    {"_frozen_importlib", _Py_M__importlib__bootstrap, (int)sizeof(_Py_M__importlib__bootstrap), false},
    {"_frozen_importlib_external", _Py_M__importlib__bootstrap_external, (int)sizeof(_Py_M__importlib__bootstrap_external), false},
    {"zipimport", _Py_M__zipimport, (int)sizeof(_Py_M__zipimport), false},
    {0, 0, 0} /* bootstrap sentinel */
};
static const struct _frozen stdlib_modules[] = {
    /* stdlib - startup, without site (python -S) */
    {"abc", _Py_M__abc, (int)sizeof(_Py_M__abc), false},
    {"codecs", _Py_M__codecs, (int)sizeof(_Py_M__codecs), false},
    {"io", _Py_M__io, (int)sizeof(_Py_M__io), false},
    /* stdlib - startup, with site */
    {"_collections_abc", ...}, {"_sitebuiltins", ...}, {"genericpath", ...},
    {"ntpath", ...}, {"posixpath", ...}, {"os", ...}, {"site", ...}, {"stat", ...},
    /* runpy - run module with -m */
    {"importlib.util", ...}, {"importlib.machinery", ...}, {"runpy", ...},
    {0, 0, 0} /* stdlib sentinel */
};

This is the authoritative answer to “exactly which stdlib modules are frozen in 3.14.5” — read directly from the source. The bootstrap_modules table holds the three modules needed to make the import system itself work: importlib._bootstrap (exposed under the internal name _frozen_importlib), importlib._bootstrap_external (_frozen_importlib_external), and zipimport. The stdlib_modules table holds the modules pulled in during a normal startup: the bare-startup trio abc, codecs, io; the site-startup set _collections_abc, _sitebuiltins, genericpath, ntpath, posixpath, os, site, stat; and the python -m (runpy) set importlib.util, importlib.machinery, runpy. The test_modules table holds toy modules (__hello__, __phello__ and friends) used by the test suite to exercise the freezing machinery; they print “famous words” when imported.

The arrays themselves (_Py_M__abc and friends) are #included from generated headers in Python/frozen_modules/, e.g. #include "frozen_modules/abc.h". Those headers are build artifacts — they are git-ignored and regenerated, so they are not present in a fresh checkout.

The Python-side loader is FrozenImporter. Its find_spec asks _imp.find_frozen, and its exec_module retrieves and runs the code object:

class FrozenImporter:
    _ORIGIN = "frozen"
 
    @classmethod
    def find_spec(cls, fullname, path=None, target=None):
        info = _call_with_frames_removed(_imp.find_frozen, fullname)
        # ... builds a spec with loader=FrozenImporter, origin="frozen"
 
    @staticmethod
    def exec_module(module):
        name = module.__spec__.name
        code = _call_with_frames_removed(_imp.get_frozen_object, name)
        exec(code, module.__dict__)

The load-bearing step is _imp.get_frozen_object(name): in C it finds the matching _frozen entry, then calls PyMarshal_ReadObjectFromString on the embedded byte array to deserialize the code object — the inverse of the marshal step that a .pyc file goes through (contrast Bytecode Caching and pyc Files, which marshals to disk). exec(code, module.__dict__) then runs that code object against the new module’s namespace. Crucially, find_spec sets origin="frozen" and the spec has no file location, so a frozen module’s __file__ is absent unless CPython can resolve the original source path for diagnostics. You can observe the distinction live: on a 3.14.5 interpreter, os.__spec__.loader is <class '_frozen_importlib.FrozenImporter'> and os.__spec__.origin is 'frozen'.

How the C Arrays Are Generated

Two tools cooperate. Tools/build/freeze_modules.py holds the authoritative list of what to freeze — a FROZEN table grouped into sections ('import system', 'stdlib - startup, without site (python -S)', 'stdlib - startup, with site', 'runpy - run module with -m', and the test section). Running make regen-frozen invokes this script, which (1) runs Programs/_freeze_module on each target, (2) rewrites the #includes and the tables in Python/frozen.c, and (3) updates the Makefile and the Windows project files. The list at the top of the script is the single place a contributor edits to add or remove a frozen module.

The actual byte-array generation lives in Programs/_freeze_module.c. Its compile_and_marshal compiles the source and marshals the resulting code object:

PyObject *code = Py_CompileStringExFlags(text, filename, Py_file_input, NULL, 0);
PyObject *marshalled = PyMarshal_WriteObjectToString(code, Py_MARSHAL_VERSION);

and write_code emits it as a comma-separated list of byte literals:

fprintf(outfile, "const unsigned char %s[] = {\n", varname);
for (size_t n = 0; n < data_size; n += 16) {
    /* ... */
    fprintf(outfile, "%u,", (unsigned int) data[i]);
}
fprintf(outfile, "};\n");

So _Py_M__abc[] is literally the bytes of abc’s marshalled code object written out as 73,99,0,0,.... The variable name is derived from the module name by get_varname, which prefixes _Py_M__ and replaces dots with underscores (so importlib.util becomes _Py_M__importlib_util). A telling detail: _freeze_module is built as a stand-alone executable with a deliberately empty frozen-module table (no_modules) and config._install_importlib = 0, “to avoid unintentional import of a stale version of _frozen_importlib” — the freezer must not depend on the frozen importlib it is about to regenerate. That is the bootstrap chicken-and-egg made concrete in the build.

The Bootstrap Chicken-and-Egg

The import system is implemented in Lib/importlib/_bootstrap.py and _bootstrap_external.py — in Python. But you cannot import importlib._bootstrap from disk, because importing-from-disk is the very thing _bootstrap.py implements. Freezing breaks the cycle: importlib._bootstrap is frozen as _frozen_importlib, so during interpreter startup the C runtime can load it directly from its embedded byte array (no finders, no sys.path, no filesystem). Once _frozen_importlib is running, it installs BuiltinImporter and FrozenImporter onto sys.meta_path and the full import machinery comes online. This is why the bootstrap modules are a separate table: they are loaded by a hard-coded C path during _PyimportZip/init, before the meta-path even exists. See importlib Internals for the full startup sequence and The CPython Source Tree Layout for where these files live in the tree.

The 3.11 Freeze Work and the Deep-Freeze That Was Removed

Two distinct mechanisms are easily conflated, and the difference matters for any version claim.

Plain freezing (marshalled bytecode in a byte array, unmarshalled at startup) is old — importlib._bootstrap has been frozen since Python 3.3, when importlib became the import implementation. What 3.11 added was freezing the stdlib startup modules (os, site, abc, io, the path modules, etc. — the bpo-45661 meta-issue), so that those modules skip the read-.pyc-and-unmarshal-from-disk path. The 3.11 What’s New reports “Interpreter startup is now 10–15% faster in Python 3.11.”

Deep-freezing was a separate, more aggressive technique also added in 3.11 (bpo-45696): instead of embedding marshalled bytecode that still has to be unmarshalled into a code object at startup, the Tools/build/deepfreeze.py script generated C source that statically allocated the code objects themselves, so startup did “Statically allocated code object → Evaluate” with no unmarshal step at all. The 3.11 What’s New text — “their Code Objects (and bytecode) are statically allocated by the interpreter” — describes deep-freezing.

Deep-freezing was turned off in Python 3.13, not 3.14. Issue gh-108716 (“Remove deep-freezing of code objects and modules”) was implemented by PR #108722 (“Turn off deep-freezing of modules”), merged into the 3.13 development branch (3.13.0a0) on 8 September 2023, with cleanup PRs (#110078, #116919, #117141) following. The rationale, from the issue: deep-freezing “is slow to build,” “does not fit into the normal build system” (make would not regenerate deep-frozen modules), “only makes sense if the objects are immutable, but code objects are not,” and “gets in the way of other optimizations, notably faster loading from pyc files.” Critically, the arrival of immortal objects (PEP 683, 3.12 — see Immortal Objects) removed deep-freezing’s main remaining benefit (avoiding refcount churn on the statically allocated objects). So in CPython 3.14.5, modules are still frozen (marshalled byte arrays, unmarshalled at startup — write_code in _freeze_module.c still emits exactly that), but no longer deep-frozen. The Tools/build/deepfreeze.py script still exists in the tree but is not invoked by the build; the Makefile only references Python/deepfreeze in a clean stanza, and deepfreeze.c is no longer a build target.

Uncertain

Verify: how much of the 3.11 “10–15% faster startup” figure survived the 3.13 deep-freeze removal. Reason: the figure is a single bundled 3.11 number. The 3.11 What’s New text attributes it to the “Statically allocated code object → Evaluate” path — i.e. the deep-freeze mechanism — which was the part removed in 3.13, so the survival fraction is exactly what is unpublished. No primary 3.13/3.14 benchmark isolating plain freezing (the unmarshal-from-byte-array path that remains) was found, and the 3.14 What’s New headlines no startup regression. To resolve: find the gh-108716 / PR #108722 benchmark comments or a python-dev pyperformance startup comparison across 3.12→3.13. Do not infer that 3.14 startup regressed 10–15% — modules remain frozen, so the bulk of the benefit (skipping the disk .pyc read + unmarshal-from-file) is retained; only the marginal unmarshal-into-code-object step came back. uncertain

The -X frozen_modules Flag

Frozen modules can be disabled at startup with -X frozen_modules=off (or the environment-equivalent). In Python/import.c and Python/initconfig.c, the default is on: initconfig.c initializes config->use_frozen_modules = 1 (and the help text states “-X frozen_modules=[on|off]: whether to use frozen modules; the default is ‘on’”). When set to off, FrozenImporter is bypassed for the stdlib modules and they are loaded from their .py/.pyc files on disk instead — useful when debugging the importlib source, because edits to Lib/importlib/_bootstrap.py otherwise have no effect until you rebuild and re-freeze. Empirically on 3.14.5: under normal startup os.__spec__.loader is the FrozenImporter, while under python -X frozen_modules=off it becomes a SourceFileLoader reading os.py from disk. There is also an internal test-only escape hatch, _imp._override_frozen_modules_for_tests, used by the test suite (Lib/test/support/import_helper.py).

-X frozen_modules=off disables only the stdlib and test tables — the bootstrap modules stay frozen unconditionally, which is why disabling the flag never breaks startup. This is explicit in look_up_frozen in v3.14.5 Python/import.c: the function always iterates _PyImport_FrozenBootstrap first (with the comment “We always use the bootstrap modules”), and only walks _PyImport_FrozenStdlib and _PyImport_FrozenTest when use_frozen() returns true. use_frozen() consults the per-interpreter override and then interp->config.use_frozen_modules, the value the -X flag sets. The same structure appears in list_frozen_module_names: bootstrap entries are appended unconditionally, stdlib/test entries only when enabled. The FROZEN_DISABLED status (set when a frozen lookup misses because freezing is off) carries the message “Frozen modules are disabled and the frozen object named %R is not essential” — “essential” here being precisely the bootstrap table, which the lookup can never report as disabled.

Failure Modes and Diagnosis

The most common confusion is mistaking a built-in for a frozen module or vice versa. A quick diagnosis: name in sys.builtin_module_names tells you it is a C built-in (no source, get_source returns None); module.__spec__.origin == 'frozen' tells you it was loaded from an embedded byte array. A frozen module does have a code object (module.__spec__.loader.get_code(name) returns one); a built-in never does.

A subtler failure: editing Lib/importlib/_bootstrap.py or Lib/os.py in a CPython source tree and seeing no effect at run time. The cause is that the running interpreter is using the frozen copy embedded in the binary, not your edited file. The fixes are either make regen-frozen && make to re-embed, or run with -X frozen_modules=off to force disk loading.

A third: assuming inspect.getsourcefile() or __file__ always exists. Built-in modules have no __file__ at all (sys.__file__ raises AttributeError); frozen modules may have __file__ set for diagnostics only if CPython could resolve the original stdlib path (FrozenImporter._resolve_filename), and None otherwise.

Alternatives and When to Choose Them

For application freezing (shipping a self-contained executable that embeds your own Python modules), the in-tree struct _frozen machinery is exposed via the public, mutable PyImport_FrozenModules pointer — “Embedding apps may change this pointer to point to their favorite collection of frozen modules” (per frozen.c). Historically the Tools/freeze tool used this. In practice, third-party packagers — PyInstaller, cx_Freeze, Nuitka — solve the same “ship without a Python install” problem differently (bundling a real .pyc tree or compiling to C), and most applications should reach for those rather than CPython’s internal freezing, which exists primarily to bootstrap the interpreter itself. For making C functionality available, a normal dynamically loaded extension .so/.pyd (loaded by ExtensionFileLoader from disk) is the default; statically linking it as a built-in via Modules/Setup is reserved for modules that must be present before sys.path works, or for fully static interpreter builds.

See Also

Graph View

Backlinks