vault

The Buffer Protocol

20 min read

The Buffer Protocol

The buffer protocol (specified by PEP 3118, “Revising the Buffer Protocol,” authored by Travis Oliphant and Carl Banks) is CPython’s low-level mechanism for one object to expose its raw memory block to another object without copying it (PEP 3118). It is the reason a bytes, a bytearray, an array.array, and a NumPy ndarray can all be handed to the same C function — or to the same memoryview — and have that function read or write their bytes in place. The protocol carries not just a pointer and a length but a full description of the memory: how many dimensions it has, how big each element is, what C type each element is, and how the elements are laid out (the strides). The single most important idea is zero-copy: passing a hundred-megabyte image between an image library and a numerical library should cost a few pointer assignments, not an O(n) memcpy of a hundred megabytes (PEP 3118 rationale).

This note covers the C-level structure (Py_buffer), the type slots that implement it (tp_as_buffer / bf_getbuffer / bf_releasebuffer), the Python-facing memoryview object, the format-string mini-language that describes element types, the lifetime/locking discipline that keeps exported memory valid, and the NumPy interoperability story. All Python-level behavior below was verified on a live CPython 3.14.x interpreter; version-sensitive C-API claims are cited to the 3.14 documentation and the PEPs.

Why It Exists

Python 2 had an older, weaker buffer protocol (PyBufferProcs with getreadbuffer/getwritebuffer). PEP 3118 lists its fatal inadequacies, every one of which the new protocol was designed to fix (PEP 3118):

  1. No “I’m done” notification. The old protocol gave a consumer a pointer but never told the exporter when the consumer finished. The exporter therefore could never safely free or resize its memory — a dangling pointer was always possible. The new protocol pairs every acquisition with a mandatory release, so the exporter can lock its memory for the lifetime of the export and unlock it afterward.
  2. No type information. The old buffer was just void * + length. A consumer receiving a megabyte of memory had no way to know whether it held 32-bit floats, 64-bit ints, or RGBA pixels. The new protocol carries a format string describing one element.
  3. No shape. Array-like libraries (NumPy, the Python Imaging Library, wxPython, PyGTK) all model multi-dimensional data, but the old protocol had no standard way to communicate (rows, cols). The new protocol carries ndim and a shape array.
  4. No discontiguous-memory support. A slice like image[::2, ::2] is a view into every other pixel — its bytes are not contiguous. The old protocol couldn’t describe such a view, forcing a copy. The new protocol carries strides (byte gaps between elements) and suboffsets (pointer indirection), so even a strided or pointer-of-pointers layout can be shared without copying.

The motivating use case is interoperation between numerical libraries. Before PEP 3118, moving an array from one C library to another meant serializing to bytes and re-parsing — an O(n) copy and a loss of shape/type metadata. With the buffer protocol, NumPy can hand a 2-D float array straight to a C routine that reads it in place. The protocol became “fundamental to Python’s scientific computing ecosystem” (PEP 3118).

Mental Model

Think of the protocol as a contract between an exporter (the object that owns the memory) and a consumer (the object or C code that wants to read/write it). The consumer asks for a buffer, optionally stating requirements via flags (“I need it writable,” “I can only handle contiguous memory,” “fill in the format string”). The exporter either satisfies the request — filling a Py_buffer description struct and bumping its own reference count — or refuses with a BufferError. When the consumer is done, it must release the buffer, which decrements the exporter’s refcount and unlocks the memory.

flowchart LR
    subgraph Exporter["Exporter (owns memory)"]
        MEM["raw bytes<br/>e.g. bytearray, ndarray"]
        SLOT["tp_as_buffer:<br/>bf_getbuffer / bf_releasebuffer"]
    end
    subgraph Consumer["Consumer (wants memory)"]
        VIEW["Py_buffer view struct<br/>buf, len, format,<br/>ndim, shape, strides"]
    end
    Consumer -->|"PyObject_GetBuffer(exporter, &view, flags)"| SLOT
    SLOT -->|"fill view, incref self,<br/>lock against resize"| VIEW
    VIEW -.->|"view.buf points into"| MEM
    Consumer -->|"PyBuffer_Release(&view)"| SLOT
    SLOT -->|"decref self, unlock"| MEM

Diagram: the export handshake. The consumer requests a buffer through the exporter’s bf_getbuffer slot, receives a Py_buffer whose buf field points directly into the exporter’s memory (no copy), and the exporter increments its own refcount and locks against resizing for as long as the view lives. PyBuffer_Release reverses both. The insight: the Py_buffer is metadata describing borrowed memory — the bytes are never duplicated, only described.

The Py_buffer Struct, Field by Field

The heart of the protocol is the Py_buffer C struct. A consumer allocates one (usually on its C stack) and passes its address to PyObject_GetBuffer; the exporter fills every field. Here is each field, with its exact type and meaning from the 3.14 C-API reference (Buffer Protocol):

  • void *buf — A pointer to the start of the logical structure. It need not be the start of the physical block: with negative strides it may point to the end of the memory (the logical first element). For contiguous arrays it points to the beginning. This is the pointer the consumer dereferences (with stride arithmetic) to reach each element.
  • PyObject *obj — A new reference to the exporting object. The consumer owns this reference; PyBuffer_Release decrements it and sets the field to NULL. This is what keeps the exporter alive while the view exists, and what the release machinery uses to find the exporter’s bf_releasebuffer slot.
  • Py_ssize_t len — product(shape) * itemsize. For contiguous arrays this is the length of the underlying memory block in bytes. For non-contiguous arrays it is the length the data would occupy if copied to a contiguous layout — not the number of bytes physically spanned. (Py_ssize_t is CPython’s signed pointer-sized integer, the type used for all sizes and indices.)
  • Py_ssize_t itemsize — The size in bytes of one element. It equals struct.calcsize(format). For a double array this is 8; for raw bytes it is 1.
  • int readonly — 1 if the buffer must not be written, 0 if it is writable. Controlled by the PyBUF_WRITABLE request flag: if the consumer set that flag the exporter must provide writable memory or fail.
  • int ndim — The number of dimensions. If ndim == 0, buf points to a single scalar item and shape/strides/suboffsets are all NULL. The maximum is PyBUF_MAX_NDIM, currently 64 (C-API buffer).
  • char *format — A NUL-terminated string in struct-module syntax describing one element’s type (see the format mini-language below). If NULL, 'B' (unsigned byte) is assumed. Controlled by the PyBUF_FORMAT flag.
  • Py_ssize_t *shape — An array of length ndim giving the size in elements along each dimension. The invariant shape[0] * ... * shape[ndim-1] * itemsize == len must hold.
  • Py_ssize_t *strides — An array of length ndim giving, per dimension, the number of bytes to skip to advance one element along that axis. Stride values may be any integer, including negative (reverse iteration) or zero (broadcasting). This is what makes a sliced view zero-copy: a step-2 slice has a stride of 2 * itemsize instead of pointing at a copied compacted block.
  • Py_ssize_t *suboffsets — An array of length ndim, or NULL. If suboffsets[n] >= 0, the values along dimension n are pointers that must be dereferenced, and the suboffset says how many bytes to add after dereferencing. A negative entry means no dereference. This supports the Python Imaging Library’s “array of row pointers” layout, where rows are scattered in memory.
  • void *internal — Reserved for the exporter’s private bookkeeping. The consumer must not read or alter it.

To reach element [i0, i1, ..., in-1] of a strided (but non-indirect) buffer, the consumer computes (C-API buffer):

char *ptr = (char *)view.buf
          + indices[0] * view.strides[0]
          + ...
          + indices[n-1] * view.strides[n-1];
item = *((ItemType *)ptr);

When suboffsets are present, each step may additionally dereference a pointer and add the suboffset — the PIL-style access pattern from the docs:

char *pointer = (char *)buf;
for (int i = 0; i < ndim; i++) {
    pointer += strides[i] * indices[i];
    if (suboffsets[i] >= 0)
        pointer = *((char **)pointer) + suboffsets[i];
}

Each line: start at buf; for every dimension, advance by stride × index; if that dimension is indirect, follow the pointer stored there and add the suboffset. This single loop unifies contiguous, strided, and pointer-of-pointers layouts under one access model.

Contiguity: C, Fortran, and Strided

Three layouts matter. The Python glossary defines them precisely (glossary: contiguous):

A buffer is considered contiguous exactly if it is either C-contiguous or Fortran contiguous. Zero-dimensional buffers are C and Fortran contiguous. In one-dimensional arrays, the items must be laid out in memory next to each other, in order of increasing indexes starting from zero. In multidimensional C-contiguous arrays, the last index varies the fastest when visiting items in order of memory address. However, in Fortran contiguous arrays, the first index varies the fastest.

In plain terms: a C-contiguous 2-D array stores its data row-by-row (the rightmost index moves fastest), the layout C and Python default to. A Fortran-contiguous array stores column-by-column (the leftmost index moves fastest), the convention of Fortran, MATLAB, and numpy.asfortranarray. A strided (non-contiguous) view — produced by, say, a[::2] — is neither: there are gaps between consecutive logical elements. Verified live, a 2×3 C-contiguous float64 array reports strides == (24, 8) (skip 24 bytes for the next row, 8 for the next column); its Fortran transpose reports strides == (8, 16) and f_contiguous == True; and a bytearray[::2] view reports c_contiguous == False, strides == (2,).

A consumer that can only handle contiguous memory requests it via flags; if it asks for PyBUF_C_CONTIGUOUS and the exporter’s data is strided, the export fails rather than silently copying — the consumer must then decide to copy explicitly.

Request Flags

The flags argument to PyObject_GetBuffer is a bitmask telling the exporter what the consumer needs and can handle. The structural flags form a ladder of increasing capability (C-API buffer):

FlagshapestridessuboffsetsMeaning
PyBUF_SIMPLENULLNULLNULLJust a flat byte block; consumer can’t handle shape
PyBUF_NDyesNULLNULLC-contiguous N-D array, no strides
PyBUF_STRIDESyesyesNULLStrided array (implies shape)
PyBUF_INDIRECTyesyesif neededFull generality incl. pointer indirection

PyBUF_WRITABLE (sets readonly handling — exporter must give writable memory or fail) and PyBUF_FORMAT (exporter must fill format) are OR-ed onto the structural flags. PyBUF_FORMAT must be combined with something other than PyBUF_SIMPLE, since PyBUF_SIMPLE already implies the format is 'B'. Convenience combinations exist: PyBUF_CONTIG (writable, C-contiguous, no format), PyBUF_CONTIG_RO (read-only variant), PyBUF_STRIDED, PyBUF_RECORDS (strided + format), PyBUF_FULL (strides + suboffsets if needed + format + writable), and their _RO read-only siblings. The mental rule: the consumer requests the least it can tolerate; the exporter provides at least that or raises BufferError.

The C-Level Slots: tp_as_buffer, bf_getbuffer, bf_releasebuffer

A type advertises buffer support through the tp_as_buffer slot of its PyTypeObject (see Type Slots and the Protocol Tables for how the tp_as_* sub-tables work). That slot points at a PyBufferProcs struct with exactly two function pointers (C-API type objects):

typedef struct {
    int  (*bf_getbuffer)(PyObject *exporter, Py_buffer *view, int flags);
    void (*bf_releasebuffer)(PyObject *exporter, Py_buffer *view);
} PyBufferProcs;

bf_getbuffer (type getbufferproc) is the export handshake. Its contract, from the documentation of the public wrapper PyObject_GetBuffer (C-API buffer):

Send a request to exporter to fill in view as specified by flags. If the exporter cannot provide a buffer of the exact type, it MUST raise BufferError, set view->obj to NULL and return -1. On success, fill in view, set view->obj to a new reference to exporter and return 0.

So a correct bf_getbuffer must: (1) check the requested flags against what it can provide and fail with BufferError if it cannot; (2) fill every relevant Py_buffer field; (3) Py_INCREF the exporter and store it in view->obj (this is the new reference that keeps the exporter alive); (4) return 0. The PyBuffer_FillInfo helper does most of this for the common one-dimensional contiguous case.

bf_releasebuffer (type releasebufferproc) is called — through PyBuffer_Release — when the consumer is finished. It is the exporter’s chance to undo any per-export bookkeeping, such as decrementing an “export count” that locks the object against resizing. It returns void and may be NULL if the exporter needs no cleanup beyond the automatic Py_DECREF of view->obj (which PyBuffer_Release always does). The documentation stresses the pairing: “Successful calls to PyObject_GetBuffer() must be paired with calls to PyBuffer_Release(), similar to malloc() and free().”

Several core built-ins expose buffers through these slots: bytes and bytearray implement bf_getbuffer (with bytearray additionally tracking exports to forbid resizing — see CPython Bytes and Bytearray Internals), as do array.array and memoryview itself.

Lifetime and Locking — Why You Can’t Resize an Exported bytearray

The release-pairing contract has a visible Python-level consequence. While a buffer is exported, the exporter’s memory must not move. A bytearray that grew (and thus reallocated its backing store via realloc) while a consumer held a pointer into the old store would hand that consumer a dangling pointer. CPython prevents this by maintaining an export count on bytearray: every bf_getbuffer increments it, every bf_releasebuffer decrements it, and any operation that would resize the buffer checks the count first.

Verified live on 3.14.x — opening a memoryview exports the buffer, and a subsequent resize raises:

>>> ba = bytearray(b'test')
>>> m = memoryview(ba)          # exports a buffer; export count -> 1
>>> ba.append(1)
BufferError: Existing exports of data: object cannot be re-sized
>>> ba.clear()
BufferError: Existing exports of data: object cannot be re-sized
>>> m.release()                 # decrements export count -> 0
>>> ba.append(1)                # now permitted
>>> ba
bytearray(b'test\x01')

The exact message — "Existing exports of data: object cannot be re-sized" — is the symptom you will see in real code that forgets to release() a view before resizing. This is also why memoryview is a context manager: with memoryview(ba) as m: guarantees the export count returns to zero on block exit even if an exception fires. Reading and in-place writing through the view are always fine; only operations that would reallocate (append, extend, clear, slice-assignment that changes length) are blocked. Immutable bytes has no such issue because it never resizes.

The Python-Facing Object: memoryview

memoryview is the protocol made first-class in Python. memoryview(obj) acquires a buffer from any object that exports one — bytes, bytearray, array.array, NumPy arrays, and any class implementing __buffer__ — and exposes it without copying (stdtypes: memoryview). Its attributes mirror the Py_buffer fields: obj, nbytes (= product(shape) * itemsize), readonly, format, itemsize, ndim, shape, strides, suboffsets, plus the booleans c_contiguous, f_contiguous, and contiguous.

Its methods: tobytes() (copy the data out to a bytes — this does copy), hex(), tolist() (unpack each element per format into a Python list), toreadonly() (a read-only view of the same memory), release() (release the buffer early), cast(format[, shape]), and __eq__. It is also a context manager (__enter__/__exit__ call release on exit).

Slicing is zero-copy. A slice of a memoryview returns a new memoryview over a subset of the same buffer — no bytes are copied, and writes through the slice mutate the original. Verified live:

>>> ba = bytearray(b'abcdef')
>>> m = memoryview(ba)
>>> s = m[2:5]            # view of bytes 2..4, NOT a copy
>>> bytes(s)
b'cde'
>>> s[0] = ord('Z')      # write through the slice
>>> ba                    # underlying bytearray mutated in place
bytearray(b'abZdef')

This is the property that makes memoryview valuable for performance: parsing a network packet by repeatedly slicing a memoryview allocates no intermediate buffers, whereas slicing bytes copies each time.

.cast() reinterprets the bytes under a different element type or shape, again without copying (stdtypes: memoryview). It has two hard restrictions: the source must be C-contiguous, and casts are allowed only between formats with the same itemsize, or to/from 'B' (raw bytes). Verified live:

>>> m = memoryview(b'\x00\x01\x02\x03\x04\x05\x06\x07')
>>> m.cast('I').tolist()              # reinterpret 8 bytes as 2 uint32
[50462976, 117835012]
>>> m.cast('B', shape=[2, 4]).tolist()  # same bytes, 2x4 layout
[[0, 1, 2, 3], [4, 5, 6, 7]]
>>> memoryview(bytearray(range(10)))[::2].cast('B')
TypeError: memoryview: casts are restricted to C-contiguous views

The last line shows the restriction biting: a strided ([::2]) view is not C-contiguous, so cast refuses with that exact TypeError. nbytes versus len() is another live gotcha: for an array.array('i', [1,2,3]), m.nbytes == 12 (three 4-byte ints) but len(m) == 3 (the element count), since len reports the first dimension’s size, not the byte count.

The Format-String Mini-Language

The format field describes one element using the struct module’s syntax, which PEP 3118 extends. The base codes are the struct standard codes (struct: format characters): b/B (signed/unsigned char, 1 byte), ? (_Bool, 1), h/H (short, 2), i/I (int, 4), l/L (long, 4 standard), q/Q (long long, 8), n/N (ssize_t/size_t), e/f/d (16/32/64-bit float), s/p (char arrays), P (void*), c (1-byte). A leading byte-order/size/alignment character may prefix the string (struct docs): @ = native byte order, native size, native alignment (the default); = = native byte order, standard size, no alignment; < = little-endian; > = big-endian; ! = network order (always big-endian).

PEP 3118 adds codes to this base for the buffer protocol’s richer needs (PEP 3118 additions): 't' (a bit), 'g' (long double), 'c'/'u'/'w' (UCS-1/2/4 text), 'O' (pointer to a Python object), 'Z' as a complex prefix (Zf = complex float, Zd = complex double), '&' (pointer to another type), 'T{...}' (a nested struct with a field layout inside the braces), '(k1,k2,...)' (a multi-dimensional sub-array), ':name:' (a field name annotation), 'X{}' (a function pointer), and '^' as an “unaligned native” byte-order char.

Resolved (2026-06-01)

Enumerated directly on live CPython 3.14.5 and cross-checked against Modules/_struct.c at the v3.14.5 tag. Two facts settle the superset/subset question: (1) The struct module’s native_table accepts x b B c s p h H i I l L n N q Q ? e f d F D P — note F/D (complex float/double) are the codes struct actually implements for complex; the PEP 3118 'Z'-prefix forms are not struct-typeable: struct.calcsize('Zf') raises struct.error, while struct.calcsize('F') returns 8 and struct.unpack('F', struct.pack('F', 1+2j)) round-trips. (2) memoryview.cast() accepts only native single-character formats (optionally @-prefixed): cast('Zf') and cast('T{i:x:}') both raise ValueError: memoryview: destination format must be a native single character format.... So the buffer format mini-language is a strict superset of what struct/memoryview.cast accept at runtime — composite forms (Zf, T{...}, (k,...)) are produced by exporters such as NumPy and consumed by readers of Py_buffer.format, not constructible via cast or struct.pack.

The Python-Level Buffer Protocol (PEP 688)

Until Python 3.12 the buffer protocol was reachable only from C. PEP 688 (“Making the Buffer Protocol Accessible in Python,” added in 3.12) exposed it to Python code (PEP 688). A pure-Python class can now implement two dunder methods (PEP 688 Python-level protocol):

def __buffer__(self, flags: int, /) -> memoryview: ...
def __release_buffer__(self, buffer: memoryview, /) -> None: ...

__buffer__ “is called to create a buffer from a Python object, for example by the memoryview() constructor” and must return a memoryview. __release_buffer__ “should be called when a caller no longer needs the buffer” and is “an optional part of the buffer protocol” (PEP 688). PEP 688 also adds collections.abc.Buffer — an abstract base class requiring __buffer__, intended for type annotations and isinstance checks — and inspect.BufferFlags, an enum.IntFlag mirroring the C request flags. Verified live on 3.14.x:

>>> import collections.abc, inspect
>>> isinstance(b'x', collections.abc.Buffer)
True
>>> inspect.BufferFlags.SIMPLE, inspect.BufferFlags.WRITABLE, inspect.BufferFlags.FORMAT
(<BufferFlags.SIMPLE: 0>, <BufferFlags.WRITABLE: 1>, <BufferFlags.FORMAT: 4>)
>>> class MyBuf:
...     def __init__(self, data): self.data = bytearray(data)
...     def __buffer__(self, flags): return memoryview(self.data)
...     def __release_buffer__(self, view): view.release()
>>> bytes(memoryview(MyBuf(b'hello')))
b'hello'

The flags integer the interpreter passes to __buffer__ is exactly the C request bitmask, decodable with inspect.BufferFlags, so a Python implementation can honor WRITABLE/FORMAT requests just as a C exporter would.

NumPy Interop — Two Related Protocols

NumPy is both the original motivation for PEP 3118 and its biggest beneficiary. There are two distinct zero-copy mechanisms, and they are easy to conflate.

1. The buffer protocol itself. A NumPy ndarray is a buffer exporter, and np.frombuffer is a buffer consumer. Verified live on CPython 3.14.x with NumPy 2.3.5:

>>> import numpy as np
>>> ba = bytearray(b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00')
>>> arr = np.frombuffer(ba, dtype=np.int32)   # zero-copy view onto ba
>>> arr
array([1, 2, 3], dtype=int32)
>>> ba[0:4] = (99).to_bytes(4, 'little')       # mutate the bytearray
>>> arr
array([99,  2,  3], dtype=int32)               # numpy view sees the change

Conversely, memoryview(ndarray) makes NumPy the exporter — the memoryview reports NumPy’s format, itemsize, shape, and strides directly. A 2×3 float64 C-contiguous array yields format='d', shape=(2,3), strides=(24,8); calling np.asfortranarray and re-wrapping yields f_contiguous=True, strides=(8,16). Writes through the memoryview land in the NumPy array (memoryview(np.zeros(4, np.uint8))[0] = 200 sets arr[0]), demonstrating bidirectional zero-copy sharing.

2. The array interface (__array_interface__). This is a separate, older NumPy-specific protocol — a Python dictionary (or a C-struct variant) that describes the same memory through keys data (a (pointer, readonly) tuple), shape, typestr, strides, descr, and version. Verified live:

>>> np.arange(4, dtype=np.int16).__array_interface__
{'data': (94745275473824, False), 'strides': None,
 'descr': [('', '<i2')], 'typestr': '<i2', 'shape': (4,), 'version': 3}

The array interface predates PEP 3118 and lives entirely in NumPy/Python space (it exposes a raw integer pointer rather than going through tp_as_buffer). For modern interop within the scientific stack the buffer protocol is preferred, but __array_interface__ remains supported and is what some libraries still probe.

Uncertain (reviewed 2026-06-01)

Verify: that __array_interface__ is still the recommended cross-library mechanism vs. the newer __dlpack__ / DLPack protocol (and the array-API standard) that much of the ecosystem has moved toward. Reason: this is a third-party (NumPy) convention, not a CPython-version fact, and the current numpy.org interop-protocols docs were not fetched this round. To resolve: consult the current numpy.org “interoperability” page and the array-API standard for the present recommendation; the buffer-protocol mechanics this note teaches are unaffected either way. #uncertain

Failure Modes and Common Misunderstandings

  • Forgetting to release a view, then resizing. The classic bug: hold a memoryview on a bytearray, try to append/extend/clear it, and get BufferError: Existing exports of data: object cannot be re-sized. Fix: scope the view with with, or call .release() before resizing.
  • Assuming tobytes() is free. Slicing and cast are zero-copy, but tobytes(), tolist(), and hex() materialize a copy. Reaching for tobytes() in a hot loop reintroduces the O(n) cost the protocol was meant to avoid.
  • cast on a non-contiguous view. cast requires C-contiguity and matching itemsize (or 'B'); a strided slice raises TypeError: memoryview: casts are restricted to C-contiguous views. To cast strided data you must first copy it contiguous.
  • len(m) vs m.nbytes. len is the first-dimension element count, nbytes is the byte total. Confusing them on a multi-byte-element array (array('i', ...) → len 3, nbytes 12) is a frequent off-by-N.
  • Refcount leaks in C. Every PyObject_GetBuffer that returns 0 must be balanced by PyBuffer_Release, or the exporter’s refcount (and its memory lock) leaks. This mirrors the broader owned-reference discipline of Reference Counting in C Extensions.

See Also

Graph View

Backlinks