vault

Capsules and Opaque Pointers

13 min read

Capsules and Opaque Pointers

A PyCapsule is a Python object that wraps a raw C void * pointer so it can be passed through Python code and handed to other C code that knows what to do with it. Concretely a capsule bundles three things: the opaque void * pointer itself, an optional name string, and an optional destructor callback (per the capsule C-API reference). Its defining use is to let one extension module export a C-level API — typically a table of function pointers — that a second extension imports and calls directly, without the two ever being linked against each other at the C level. The pointer rides over from exporter to importer inside an ordinary Python module attribute, so the regular import machinery does the plumbing. The name field carries a built-in safety check that stops a consumer from extracting a pointer of the wrong type.

Mental Model

Think of a capsule as a tamper-evident, labelled envelope for a C pointer. Python objects cannot hold a bare void * — everything visible to Python must be a PyObject *. A capsule is the smallest possible PyObject whose entire job is to carry one such bare pointer across the Python boundary opaquely: Python code can store it in a variable, stash it as a module attribute, and pass it around, but cannot dereference or inspect it. The label on the envelope is the capsule’s name — a C string like "spam._C_API" — and any C code that opens the envelope must state the expected label; if it doesn’t match, the open fails. The optional destructor is a note on the envelope saying “when this is thrown away, run this cleanup” (e.g. free() the pointed-to memory).

flowchart LR
    subgraph EXP["Exporter module 'spam'"]
      T["static void *PySpam_API[]<br/>(function-pointer table)"] --> C["PyCapsule_New(table,<br/>'spam._C_API', NULL)"]
      C --> A["module attr<br/>spam._C_API"]
    end
    subgraph IMP["Client module 'client'"]
      I["import_spam():<br/>PyCapsule_Import('spam._C_API')"] --> P["void **PySpam_API"]
      P --> CALL["PySpam_System(cmd)<br/>= (*PySpam_API[0])(cmd)"]
    end
    A -. "regular import resolves<br/>spam, reads attribute" .-> I
    style C fill:#cfe9ff
    style I fill:#cfe9ff

Figure: the capsule “C API export/import” pattern. The insight is that the exporter never links against the client and vice versa — the function-pointer table travels as data inside a Python module attribute, and PyCapsule_Import (which internally just imports spam and reads its _C_API attribute) is what turns that attribute back into a usable C pointer. The name string 'spam._C_API' must match on both ends or the import is rejected.

Why Capsules Exist: the Linking Problem

When extension B wants to call a C function PySpam_System defined in extension A, the obvious approach — have B’s C code call PySpam_System directly — does not work, because each extension is an independently dlopened shared object with no link-time relationship to the others. B has no symbol for PySpam_System; resolving it would require linking B against A, which the dynamic-loading model deliberately avoids. The capsule sidesteps linking entirely. A collects its public C functions into a static array of void * pointers, wraps that array’s address in a capsule, and publishes the capsule as a normal attribute on its module object. B, at its own initialization time, performs an ordinary import of A (resolving it through sys.path like any Python import), extracts the pointer from the capsule, and casts each table slot back to the right function-pointer type. From then on B calls A’s functions through that table — direct C calls, no per-call overhead, no linking (extending tutorial, “Providing a C API for an Extension Module”).

The PyCapsule API

PyCapsule is “this subtype of PyObject [that] represents an opaque value, useful for C extension modules which need to pass an opaque value (as a void* pointer) through Python code to other C code” (capsule C-API reference). It was added in Python 3.1 and is part of the Stable ABI. The core functions:

PyObject *PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor) creates a capsule wrapping pointer. The pointer “may not be NULL.” The name may be NULL or a valid C string; if non-NULL, the string must outlive the capsule (the capsule stores the pointer to it, not a copy — though you may free it from inside the destructor). The docs give the key convention: “If this capsule will be stored as an attribute of a module, the name should be specified as modulename.attributename. This will enable other modules to import the capsule using PyCapsule_Import()” (capsule C-API reference). It returns a new reference, or NULL with an exception set on failure.

void *PyCapsule_GetPointer(PyObject *capsule, const char *name) retrieves the wrapped pointer. This is where the name-matching safety check lives: “The name parameter must compare exactly to the name stored in the capsule. If the name stored in the capsule is NULL, the name passed in must also be NULL. Python uses the C function strcmp() to compare capsule names” (capsule C-API reference). On any mismatch it sets an exception and returns NULL. The comparison is a string comparison (strcmp), not a pointer comparison — so two distinct char[] arrays holding the same characters match, which is what lets the exporter and importer hard-code the same literal independently.

void *PyCapsule_Import(const char *name, int no_block) is the importer’s one-shot helper: “Import a pointer to a C object from a capsule attribute in a module.” It takes a dotted name like "spam._C_API", “splits name on the . character, and imports the first element” (i.e. imports module spam), then walks the remaining dotted components as attribute lookups to reach the capsule attribute, and finally returns the capsule’s internal pointer. The stored capsule name “must match this string exactly” (capsule C-API reference). The historical no_block parameter “has no effect anymore” as of Python 3.3 — pass 0.

int PyCapsule_IsValid(PyObject *capsule, const char *name) is the guarded gate: a capsule is valid if it is non-NULL, passes PyCapsule_CheckExact(), holds a non-NULL pointer, and “its internal name matches the name parameter.” If it returns true, “calls to any of the accessors (any function starting with PyCapsule_Get) are guaranteed to succeed.” It “will not fail” — it returns nonzero for valid/matching and 0 otherwise, never raising (capsule C-API reference).

The remaining accessors mirror the constructor’s fields. PyCapsule_GetName, PyCapsule_GetDestructor, and PyCapsule_GetContext read back the name, destructor, and context; each “is legal” to be NULL, “which makes a NULL return code somewhat ambiguous; use PyCapsule_IsValid() or PyErr_Occurred() to disambiguate.” The setters PyCapsule_SetPointer (pointer “may not be NULL”), PyCapsule_SetName, PyCapsule_SetDestructor, and PyCapsule_SetContext mutate those fields in place, each returning 0 on success and nonzero with an exception on failure (capsule C-API reference).

The context field deserves a note: it is a second void * slot independent of the main pointer, intended for bookkeeping the destructor might need (the main pointer is the payload; the context is private to whoever set up the capsule). Most capsules leave it NULL.

The Destructor and Cleanup

The destructor type is typedef void (*PyCapsule_Destructor)(PyObject *) (capsule C-API reference). If you pass a non-NULL destructor to PyCapsule_New, “it will be called with the capsule as its argument when it is destroyed” — i.e. when the capsule’s own reference count hits zero. The destructor receives the capsule, not the raw pointer, so it typically calls PyCapsule_GetPointer to recover the payload and then frees it:

static void
my_capsule_destructor(PyObject *capsule)
{
    void *buf = PyCapsule_GetPointer(capsule, "mymod.buffer");
    free(buf);                    /* release the C-level resource */
}

This is the mechanism that makes capsules safe for wrapping owned C resources passed through Python: when the last Python reference drops, the resource is reclaimed automatically by Python’s reference counting. For the C-API-export use case below, the table is a static array that lives forever, so the destructor is NULL — there is nothing to free.

Worked Example: Exporting a C API

This is the canonical pattern from the extending tutorial. The exporter spam builds a static array of its public function pointers and publishes it as a capsule in its Py_mod_exec slot (extending tutorial):

static int
spam_module_exec(PyObject *m)
{
    static void *PySpam_API[PySpam_API_pointers];   /* table, lives for the process */
    PyObject *c_api_object;
 
    PySpam_API[PySpam_System_NUM] = (void *)PySpam_System;   /* slot 0 = the function */
    c_api_object = PyCapsule_New((void *)PySpam_API, "spam._C_API", NULL);
 
    if (PyModule_Add(m, "_C_API", c_api_object) < 0) {   /* attach to module */
        return -1;
    }
    return 0;
}

Line by line: PySpam_API is a static array of void *, so its address is stable for the life of the process — exactly what we want behind a capsule with no destructor. We store the address of the C function PySpam_System into slot PySpam_System_NUM (an enum-like index). PyCapsule_New wraps the array’s address with the name "spam._C_API" and a NULL destructor. PyModule_Add attaches the capsule to the module as the attribute _C_API (and steals the reference, so we don’t manage it — see Module Objects and Namespaces). After this, spam._C_API is, from Python, an opaque capsule object; from C, the doorway to spam’s function table.

The contract between exporter and importers lives in a shared header (spammodule.h), which uses the C preprocessor to present PySpam_System as if it were a real function to client code, while behind the macro it is actually an indirection through the imported table (extending tutorial):

#define PySpam_System_NUM 0
#define PySpam_System_RETURN int
#define PySpam_System_PROTO (const char *command)
#define PySpam_API_pointers 1
 
#ifdef SPAM_MODULE
/* compiling the exporter itself: declare the real function */
static PySpam_System_RETURN PySpam_System PySpam_System_PROTO;
#else
/* compiling a client: PySpam_System is a call through the imported table */
static void **PySpam_API;
 
#define PySpam_System \
 (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM])
 
static int
import_spam(void)
{
    PySpam_API = (void **)PyCapsule_Import("spam._C_API", 0);
    return (PySpam_API != NULL) ? 0 : -1;
}
#endif

The #ifdef SPAM_MODULE split is the clever part. The exporter compiles with SPAM_MODULE defined, so it sees a plain forward declaration of the real PySpam_System. Every client compiles without it, so for them PySpam_System expands to (*(int (*)(const char*)) PySpam_API[0]) — a cast of table slot 0 to the function-pointer type, dereferenced and ready to call. The client calls import_spam() once during its own initialization; PyCapsule_Import("spam._C_API", 0) imports spam, reads its _C_API capsule, validates the name, and returns the table pointer, which import_spam stashes in the client-local PySpam_API. From then on PySpam_System("ls") in client code is a direct C call into spam’s function — no linking, name-checked once at import (extending tutorial):

static int
client_module_exec(PyObject *m)
{
    if (import_spam() < 0) {   /* resolve spam's C API at our own init time */
        return -1;
    }
    return 0;
}

Real-World Use: NumPy and datetime

Two heavily-used capsules anchor this pattern in practice. The standard library’s datetime module exports its C API through a capsule whose name is given by the PyDateTime_CAPSULE_NAME macro (the docs call this “internal usage only” and direct consumers to the macro below rather than the literal string). A consumer includes datetime.h and calls the PyDateTime_IMPORT macro, which “on success, populate[s] the PyDateTimeAPI pointer” — the file-static imported table — “on failure, set[s] PyDateTimeAPI to NULL and set[s] an exception,” so the caller must check PyErr_Occurred() (datetime C-API reference). Once PyDateTimeAPI is populated, accessor macros like PyDate_FromDate resolve through it — the exact structure of the worked example above. Notably, the docs warn PyDateTime_IMPORT “is not compatible with subinterpreters,” underscoring the static-pointer caveat discussed under failure modes. NumPy uses the same mechanism on a much larger scale: import_array() is the macro every NumPy-using extension must call to populate NumPy’s imported function-pointer table, which is why forgetting it produces the infamous segfault on the first array API call.

Uncertain (reviewed 2026-06-01)

Verify: that NumPy’s import_array() is implemented via PyCapsule_Import of numpy.core.multiarray._ARRAY_API (rather than a legacy PyCObject path on very old NumPy). Reason: NumPy is a third-party project and the exact capsule name/implementation are not a CPython-version fact; its source was not fetched this round. To resolve: read NumPy’s generated __multiarray_api.h / the import_array macro definition in the installed NumPy. (The datetime capsule facts above are verified against the CPython 3.14 datetime C-API docs and are unaffected.) uncertain

Capsules as Opaque Handles vs the C-API-Export Pattern

There are two distinct ways capsules get used, and conflating them causes confusion. The C-API-export pattern above is one C extension publishing a pointer table for other C extensions — the capsule is an implementation detail that Python code never touches meaningfully. The second use is the opaque handle: a capsule wrapping some C resource (a database connection, a file handle, an allocated buffer) that is handed back to Python code as a return value, passed around by Python, and later handed to another C function that unwraps it. Here the capsule is a genuine value in the Python program, used to give Python a safe, non-dereferenceable handle to a C object it must not poke at directly. The destructor matters far more in this second mode, because the resource’s lifetime is now tied to the capsule’s Python reference count rather than to a process-lifetime static array. Both modes use the same PyCapsule_* functions; the difference is who consumes the pointer and whether a destructor is needed.

Failure Modes

The name check is the most common source of confusing errors: a capsule created with name "spam._C_API" but read with PyCapsule_GetPointer(cap, "spam.C_API") (typo) returns NULL with a ValueError, because strcmp fails — and the message is generic enough that a one-character mismatch can cost real debugging time. NULL-vs-name ambiguity: because a valid capsule may legitimately have a NULL name, destructor, or context, a NULL return from a Get accessor does not by itself mean error; the docs prescribe PyCapsule_IsValid or PyErr_Occurred to disambiguate. Dangling name strings: passing a non-NULL name that does not outlive the capsule (e.g. a stack buffer) leaves the capsule holding a pointer into freed memory, and the eventual strcmp reads garbage. Forgetting import_spam() in a client — or calling NumPy APIs before import_array() — leaves the table pointer NULL, so the first “function call” dereferences NULL[index] and segfaults. Sub-interpreter sharing: a static void **PySpam_API in a client is a process-global, so a client built for sub-interpreter isolation should re-import the capsule per module rather than rely on a shared static — consistent with the per-module-state discipline in Writing C Extension Modules.

Alternatives and When to Choose Them

If you only need to hand a Python program an opaque handle to a C object, a capsule is the lightest tool. If you need to expose a rich, evolving API surface to Python (not C), define a proper extension type instead — capsules carry no methods. If you are wrapping an existing shared library and can link against it, ctypes and cffi reach its symbols without any capsule. The capsule’s niche is specifically “C extension A must call C extension B’s functions, with no link-time coupling and a runtime type-name check” — and for that niche there is no simpler standard mechanism.

See Also

Graph View