vault

The CPython C API

16 min read

The CPython C API

The CPython C API (also written Python/C API) is the C-language interface that CPython — the reference implementation everyone actually runs — exposes to programs written in C and C++ “who want to write extension modules or embed Python” (Python/C API Reference Manual, 3.14). It is the substrate beneath the entire native-extension ecosystem: NumPy’s array core, lxml’s libxml2 binding, every database driver, every pip install-able wheel with a .so inside — all of them talk to the interpreter through this one header set (#include <Python.h>). The API’s defining idea is uniformity: almost every function takes and returns PyObject *, “a pointer to an opaque data type representing an arbitrary Python object” (c-api intro), so a single set of functions manipulates integers, lists, classes, and modules alike. That same uniformity is double-edged: because the API hard-codes reference counting and a single-threaded object model into its contract, it is also the reason the Global Interpreter Lock and reference counting have been so hard to remove.

This is the orientation note for §15. It maps the API surface and explains why the C API simultaneously enables the ecosystem and constrains the interpreter. Two adjacent concerns are deliberately deferred: the borrowed-vs-owned reference discipline that causes most extension bugs lives in its sibling Reference Counting in C Extensions, and the binary-compatibility story (Limited API, stable ABI, Py_LIMITED_API) lives in The Stable ABI and Limited API. How you actually build a module is Writing C Extension Modules.

Mental Model — One Currency, Layered Protocols

The right way to picture the C API is a set of concentric layers, all trading in one currency. The currency is PyObject * (see PyObject and the Object Header): every value the interpreter knows is a heap block whose first bytes are a PyObject header, so any of them can be passed around as a PyObject * and cast back when its concrete type is needed. The layers are the abstraction levels at which you operate on that currency.

graph TD
    subgraph generic["Abstract Objects Layer — type-agnostic"]
        OP["Object Protocol<br/>PyObject_GetAttr, PyObject_Call,<br/>PyObject_GetItem, PyObject_Repr"]
        NP["Number / Sequence / Mapping<br/>PyNumber_Add, PySequence_GetItem"]
        IT["Iterator + Buffer Protocols"]
    end
    subgraph concrete["Concrete Objects Layer — type-specific"]
        CO["PyLong_FromLong, PyList_New,<br/>PyDict_GetItem, PyUnicode_FromString"]
    end
    subgraph infra["Runtime services"]
        RC["Reference Counting<br/>Py_INCREF / Py_DECREF"]
        EX["Exceptions<br/>PyErr_SetString / PyErr_Occurred"]
        GIL["Thread State + GIL<br/>PyGILState_Ensure / Py_BEGIN_ALLOW_THREADS"]
    end
    TYPE["PyTypeObject (tp_* slots)<br/>defines behavior of every object"]
    OP --> TYPE
    NP --> TYPE
    CO --> TYPE
    OP -.dispatches through.-> CO
    TYPE -.every call touches.-> RC
    OP -.every call may set.-> EX
    RC -.only safe while holding.-> GIL

Diagram: the C API as layers over one currency. The Abstract Objects Layer (PyObject_*, PyNumber_*, PySequence_*, PyMapping_*) is type-agnostic — it works on any object by dispatching through that object’s PyTypeObject slots, exactly as the bytecode interpreter does. The Concrete Objects Layer (PyLong_*, PyList_*, PyDict_*, PyUnicode_*) is type-specific and faster but requires you to know the object’s type. Both rest on three pervasive runtime services — reference counting, the exception indicator, and the thread-state/GIL machinery — that every call implicitly interacts with. The insight: there is no separate “object system” in C; the API is the object system, exposed function by function, and every layer ultimately bottoms out at the type object’s slots.

The API Surface

PyObject * as the universal currency

The single fact that organizes everything: “Most Python/C API functions have one or more arguments as well as a return value of type PyObject*” (c-api intro). Because every Python object begins with the same header — a reference count and a type pointer (see PyObject and the Object Header) — the API can be type-blind at its top layer. A function like PyObject_GetAttr(obj, name) does not care whether obj is an instance, a module, or a class; it reads obj->ob_type, finds the tp_getattro slot, and calls through it. This is the C-level mirror of “everything is a PyObject”: the slogan is not philosophy, it is the calling convention.

The Object Protocol and the abstract layers

The Abstract Objects Layer is the type-agnostic API. Its workhorse is the Object Protocol, the PyObject_* family: PyObject_GetAttr / PyObject_SetAttr, PyObject_GetItem / PyObject_SetItem, PyObject_Call, PyObject_Repr, PyObject_RichCompare, PyObject_IsTrue, PyObject_Hash. Layered beside it are the Number Protocol (PyNumber_Add, PyNumber_Multiply, …), the Sequence Protocol (PySequence_GetItem, PySequence_Concat, PySequence_Length), the Mapping Protocol (PyMapping_Keys, PyMapping_GetItemString), the Iterator Protocol (PyObject_GetIter, PyIter_Next), and the Buffer Protocol (PyObject_GetBuffer, for zero-copy access to raw memory). These functions are the C entry points for the same dunder-method machinery the interpreter uses: PyNumber_Add(a, b) ultimately consults a’s nb_add slot (and b’s, for the reflected case), exactly as the BINARY_OP bytecode does. Abstract-layer functions have a crucial property for memory discipline — they “always create a new strong reference … of the object they return” (c-api intro) — which is the cornerstone of the sibling note Reference Counting in C Extensions.

Type objects and the concrete-type APIs

Beneath the abstract layer is the Concrete Objects Layer: APIs that are specific to one built-in type and therefore require you to know the type up front. PyLong_FromLong(42) builds an int; PyLong_AsLong(obj) extracts a C long; PyList_New(3), PyList_GetItem, PyList_SetItem, PyList_Append manipulate lists; PyDict_New, PyDict_GetItem, PyDict_SetItem manipulate dicts; PyUnicode_FromString builds a str. These bypass slot dispatch (they cast straight to the concrete struct and touch its fields), so they are faster but unforgiving — calling PyList_GetItem on a non-list is undefined behavior, whereas PySequence_GetItem on a non-sequence raises a Python exception. The behavior of every object — abstract or concrete — is ultimately defined by its PyTypeObject, a large static struct of C function pointers called slots (tp_repr, tp_hash, tp_call, tp_getattro, nb_add, sq_item, …; see Type Objects and PyTypeObject and Type Slots and the Protocol Tables). Writing a new C type is filling in a PyTypeObject.

Exceptions

The C API has no try/except; it uses a per-thread error indicator and a return-value convention. A function that can fail signals failure by returning NULL (for PyObject * returns) or -1 (for int returns) and setting the error indicator with a function like PyErr_SetString(PyExc_ValueError, "bad"). The caller checks the sentinel, and may interrogate the indicator with PyErr_Occurred() (returns the current exception type or NULL) or PyErr_ExceptionMatches(PyExc_KeyError), and clear it with PyErr_Clear(). The discipline is strict: a returned NULL/-1 must be propagated or handled, because the next C API call assumes no error is pending. This convention threads through the worked examples in Reference Counting in C Extensions (the incr_item goto error pattern). Deeper exception mechanics are in Exception Handling Internals.

The GIL and thread-state APIs

The C API is not, by default, thread-safe: “only a thread that holds the GIL may operate on Python objects or invoke Python’s C API” (c-api threads, 3.14). Three API clusters manage this. First, when a C extension is about to do a blocking call that needs no Python objects (a read(), a network round-trip, a long compute loop), it should release the GIL so other Python threads can run, using the bracket macros:

Py_BEGIN_ALLOW_THREADS
... blocking I/O or pure-C work, no PyObject access ...
Py_END_ALLOW_THREADS

These expand to a saved thread-state dance — _save = PyEval_SaveThread(); … PyEval_RestoreThread(_save); — that detaches the thread state, releasing the GIL, and reattaches it (blocking until the GIL is reacquired) afterward (c-api threads). Second, a thread created outside Python (e.g. by a C library’s own thread pool) that wants to call into Python must first acquire a thread state and the GIL with the GIL-state API:

PyGILState_STATE gstate = PyGILState_Ensure();
/* ... safe to call Python C API here ... */
PyGILState_Release(gstate);

PyGILState_Ensure() “ensures that the current thread is ready to call the Python C API regardless of the current state of Python” (c-api threads), creating a thread state if needed. The whole subsystem — PyThreadState, the switch interval, why this design exists — is the subject of The Global Interpreter Lock and Thread State and the GIL Handshake.

Calling Conventions: tp_call and Vectorcall

How does C “call” a Python callable? Historically through one slot, tp_call, whose signature packs every call into a tuple and a dict:

PyObject *tp_call(PyObject *callable, PyObject *args, PyObject *kwargs);

This is the C form of callable(*args, **kwargs): positional arguments arrive as a tuple, keywords as a dict (or NULL if none) (c-api call, 3.14). The cost is obvious — every call had to allocate a tuple (and possibly a dict) just to pass arguments, even a one-argument call in a hot loop.

Vectorcall, introduced by PEP 590 and added in Python 3.9 (stable ABI since 3.12), is the faster protocol CPython now prefers for internal calls (c-api call). Instead of a tuple, it passes a flat C array of argument pointers:

typedef PyObject *(*vectorcallfunc)(PyObject *callable,
                                    PyObject *const *args,
                                    size_t nargsf,
                                    PyObject *kwnames);

Walking the signature symbol by symbol: callable is the object being called; args is a contiguous C array holding the positional arguments followed by the keyword-argument values; nargsf is the number of positional arguments, possibly OR-ed with the flag PY_VECTORCALL_ARGUMENTS_OFFSET (use PyVectorcall_NARGS(nargsf) to recover the real count); and kwnames is a tuple of the keyword names (the keys), or NULL when there are no keywords (c-api call). Because the arguments are already laid out as an array — typically the interpreter’s own value stack — no tuple has to be built. The PY_VECTORCALL_ARGUMENTS_OFFSET flag is the clever bit: it tells the callee it may temporarily borrow the slot at args[-1] (restoring it before return), which lets a bound method cheaply prepend its self argument without copying the whole array (c-api call).

A type opts in by setting the Py_TPFLAGS_HAVE_VECTORCALL type flag and storing the offset of its vectorcallfunc pointer in tp_vectorcall_offset. The rule is that vectorcall is strictly an optimization: “A class supporting vectorcall must also implement tp_call with identical semantics” (c-api call), the recommended fallback being tp_call = PyVectorcall_Call. To invoke a callable from C you normally use PyObject_Vectorcall(callable, args, nargsf, kwnames) (most efficient when the callee supports it) or PyObject_VectorcallMethod for method calls; PyObject_Call(callable, args_tuple, kwargs_dict) remains the tuple/dict form. Note one subtlety verified in the docs: as of 3.12, Py_TPFLAGS_HAVE_VECTORCALL is automatically cleared if a class’s __call__ is later reassigned, since the fast path would no longer match the Python-level callable.

The 3.14 PyObject Header Repack and Its Impact on Extensions

CPython 3.14 repacked the PyObject header: on 64-bit builds the reference count is now a 32-bit ob_refcnt packed alongside new ob_flags/ob_overflow sub-fields in a single 64-bit word, rather than the older Py_ssize_t ob_refcnt (the byte-level layout is dissected in PyObject and the Object Header and Reference Counting Mechanics; the ABI consequences are The Stable ABI and Limited API). For this note the relevant question is what changes for extension authors. Two concrete, sourced effects:

First, smaller refcount values break the Py_REFCNT(op) == 1 idiom. In 3.14 “the interpreter internally avoids some reference count modifications when loading objects onto the operands stack by borrowing references when possible” (the LOAD_FAST_BORROW optimization), which “can lead to smaller reference count values compared to previous Python versions” (What’s New in 3.14, porting notes, §C-API porting). Extensions that used Py_REFCNT(arg) == 1 to decide “I am the sole owner, so I may mutate this object in place” can now see a 1 where older Pythons showed 2 — or vice versa — and must instead use the new PyUnstable_Object_IsUniqueReferencedTemporary() (gh-133164, Sam Gross). Relatedly, Py_REFCNT() itself now warns that you should “not rely on the returned value to be accurate, other than a value of 0 or 1,” and on free-threaded builds even 1 is insufficient — use PyUnstable_Object_IsUniquelyReferenced() instead (c-api refcounting, 3.14).

Second, the Limited API now hides these fields behind opaque calls. “In the limited C API version 3.14 and newer, Py_TYPE and Py_REFCNT are now implemented as an opaque function call to hide implementation details” (What’s New in 3.14, gh-120600), precisely so that a future header repack will not break already-compiled Limited-API extensions — the trade-off being a function-call cost where there used to be an inline field read. The lesson generalizes: any extension that reaches into the header directly is betting on a layout that CPython now actively reserves the right to change.

Resolved (2026-06-01)

Confirmed against Include/object.h at the v3.14.5 tag. The free-threaded (Py_GIL_DISABLED) struct _object does not store a single packed refcount: it has a distinct uint32_t ob_ref_local (local reference count) and Py_ssize_t ob_ref_shared (shared/atomic count), alongside ob_tid, ob_flags, a per-object PyMutex ob_mutex, and ob_gc_bits — the split-counting/biased-refcounting layout of Biased Reference Counting. The default GIL build instead packs a 32-bit ob_refcnt with ob_overflow/ob_flags into one 64-bit word (union with ob_refcnt_full). The two are entirely different object headers.

The PyUnstable_ Tier vs Stable vs Limited

The C API is best understood as three stability tiers, even though the documentation’s framing names only the two deviations from the default (c-api stable, 3.14):

  1. The default / general API. Everything not otherwise marked. It is source-compatible across minor releases under CPython’s backwards-compatibility policy, PEP 387: a function may not be removed without a deprecation period, but the binary layout (struct sizes, inline-function bodies) may change, so an extension built against 3.13 must be recompiled for 3.14. This is what 99% of extensions use.

  2. The PyUnstable_ tier. Introduced by PEP 689 (Final, targeting 3.12), this tier carries the PyUnstable_ prefix (no leading underscore) and exposes CPython implementation internals. Its contract: “may change in every minor release (e.g. from 3.9 to 3.10) without any deprecation warnings,” but “will not change in a bugfix release” (c-api stable; PEP 689 abstract). It is “generally intended for specialized, low-level tools like debuggers” — the 3.14 additions PyUnstable_IsImmortal, PyUnstable_Object_EnableDeferredRefcount, and the two IsUniqueReferenced* functions above are textbook cases: deep internals exposed deliberately, with a renaming-on-every-release escape hatch so CPython is not frozen by them. PEP 689 also clarified the other end: a leading underscore (_Py...) marks a function internal — it may “change or disappear without any notice” and must not be used by extensions at all (PEP 689).

  3. The Limited API / stable ABI. Opt-in by defining Py_LIMITED_API before #include <Python.h>. This restricts you to a curated subset and, in exchange, produces a binary (an abi3 wheel) that loads unmodified across many minor versions. The full story — what’s in it, how the abi3 tag works, what it costs — is The Stable ABI and Limited API.

The takeaway: the prefix on a function name is its stability contract. PyList_GetItem is general (recompile per minor version); PyUnstable_Object_IsUniquelyReferenced may vanish next minor; _PyList_Extend is off-limits.

Why the C API Both Enables and Constrains the Interpreter

This is the conceptual heart of the note. The C API is CPython’s greatest asset and its heaviest anchor, for the same reason: it froze implementation details into a public contract that a vast ecosystem now depends on.

It enables because it is the reason Python is the lingua franca of scientific and systems computing. NumPy, SciPy, pandas, PyTorch, TensorFlow, lxml, psycopg, cryptography — all are C/C++ libraries wearing a Python skin, and they wear it through this API. Without a stable, performant way to drop into C, Python’s interpreter (one of the slower mainstream runtimes) would be unusable for numerical work; with it, the slow interpreter merely orchestrates fast C kernels.

It constrains because the contract is not neutral — it bakes in CPython’s two most consequential implementation choices. (1) Reference counting. The API exposes Py_INCREF/Py_DECREF and a borrowed-vs-owned discipline directly to extension authors (the subject of Reference Counting in C Extensions). Millions of lines of C across the ecosystem manually increment and decrement refcounts. Any scheme that replaced refcounting with, say, a tracing collector would invalidate all of it. So refcounting is not merely an internal choice — it is a published API, which is why CPython cannot simply drop it. (2) The GIL. Because refcount mutation (ob_refcnt++) is not atomic, and because the API let extensions assume single-threaded object access “for free,” removing the GIL means making every refcount operation correct under concurrency without breaking the existing API. That is precisely what free-threading (PEP 703) had to engineer — biased and deferred counting, per-object locks — and why it took years and a parallel ABI (Py_GIL_DISABLED). The 3.14 free-threaded build still requires extensions to define Py_GIL_DISABLED and adopt new APIs like PyList_GetItemRef because the old borrowed-reference contract is unsafe when another thread can mutate the container concurrently (c-api list, 3.14; What’s New in 3.14).

In short: the GIL and refcounting are hard to remove because the C API made them promises. The API is the ecosystem’s foundation and the interpreter’s cage, and every modern CPython performance effort is, at bottom, an attempt to renovate the cage without evicting the tenants. Newer, more abstraction-tolerant interfaces — HPy and the gradual hardening of the Limited API — are the long-term bet on a C API that hides these choices well enough that they could one day change.

Failure Modes

  • Forgetting to check NULL/-1. Using a returned PyObject * without checking for NULL dereferences garbage when a call failed; ignoring a pending exception corrupts the next call’s error check. Every fallible call must be checked.
  • Touching PyObjects without the GIL. Calling almost any C API function (including Py_INCREF) from a thread that does not hold the GIL is a data race; symptoms are intermittent crashes and refcount corruption. Use PyGILState_Ensure from foreign threads; only release the GIL via Py_BEGIN_ALLOW_THREADS around code that touches no Python objects.
  • Relying on Py_REFCNT(op) == 1. As of 3.14 this is unreliable (borrowed-reference stack loads, immortal objects, free-threading); use PyUnstable_Object_IsUniqueReferencedTemporary / PyUnstable_Object_IsUniquelyReferenced (c-api refcounting).
  • Concrete-type API on the wrong type. PyList_GetItem on a non-list (no type check) is undefined behavior; reach for the abstract PySequence_GetItem when the type is not guaranteed.
  • The borrowed-vs-owned bug class. The single largest source of extension bugs — using a borrowed reference after its owner dropped it, leaking owned references, or double-stealing — is large enough to have its own note: Reference Counting in C Extensions.

See Also

Graph View

Backlinks