vault

CPython Bytes and Bytearray Internals

14 min read

CPython Bytes and Bytearray Internals

bytes and bytearray are CPython’s two flat sequences of 8-bit integers, and the difference between them is the entire story: bytes is immutable, so its characters live in a flexible array member glued onto the object header in a single heap allocation, its hash can be cached, and length-0 and length-1 instances are shared immortal singletons; bytearray is mutable, so it owns a separately allocated, over-allocated, resizable buffer (with a logical-start offset that lets it pop from the front in O(1)), it can never be hashed or used as a dict key, and it must refuse to resize while another object holds a borrowed view of its memory through the buffer protocol. This note dissects both layouts as-of CPython 3.14.0 (Objects/bytesobject.c, Objects/bytearrayobject.c, and the matching headers).

Mental Model

The cleanest way to hold the two types in your head is “frozen string of bytes” versus “growable byte vector.” Because bytes never changes, CPython can be aggressive: bake the characters into the same allocation as the header, cache the hash forever, and hand out shared singletons for the trivial cases. Because bytearray does change, it needs the machinery of a dynamic array — spare capacity so most appends don’t reallocate, a start offset so deletions from the front don’t shift the whole buffer, and an export counter so it knows when it’s unsafe to move that buffer out from under a borrower.

flowchart TD
    subgraph BYTES["bytes — immutable, one allocation"]
        BH["PyObject_VAR_HEAD<br/>ob_shash (cached hash)<br/>ob_sval[ ] flexible array<br/>... + trailing NUL"]
    end
    subgraph BA["bytearray — mutable, two allocations"]
        BAH["PyObject_VAR_HEAD<br/>ob_alloc (capacity)<br/>ob_bytes ───────────┐<br/>ob_start ─┐         |<br/>ob_exports          |"]
        BUF["heap buffer:  [ slack ][ ob_start .. live bytes .. ][ slack ][ NUL ]"]
        BAH -->|ob_bytes points to base| BUF
        BAH -.->|ob_start points inside| BUF
    end

Diagram: the two layouts side by side. The insight: bytes is one block — header and data are inseparable, which is what makes immutability cheap and sharing safe. bytearray is a small fixed header plus a heap buffer it can grow, shrink, and slide within; ob_start living inside the ob_bytes allocation is what makes deleting from the front O(1) (just advance the start pointer) instead of memmoving the tail.

The bytes Layout

The struct is tiny, verbatim from Include/cpython/bytesobject.h (3.14.0):

typedef struct {
    PyObject_VAR_HEAD
    Py_DEPRECATED(3.11) Py_hash_t ob_shash;
    char ob_sval[1];
 
    /* Invariants:
     *     ob_sval contains space for 'ob_size+1' elements.
     *     ob_sval[ob_size] == 0.
     *     ob_shash is the hash of the byte string or -1 if not computed yet.
     */
} PyBytesObject;

Field by field:

  • PyObject_VAR_HEAD — the variable-length object header: reference count, type pointer, and ob_size. For bytes, ob_size is the number of bytes, and len(b) simply reads it (O(1)).
  • ob_shash — the cached hash, -1 until computed. It is marked Py_DEPRECATED(3.11) because direct field access from C extensions is discouraged (use the accessor functions), but the field itself is alive and used internally. Caching is legal precisely because bytes is immutable.
  • ob_sval[1] — the flexible array member. Declared as length 1 in C (a C89-compatible idiom predating true flexible array members), it actually extends to hold all the bytes. The invariant comment is load-bearing: the allocation reserves ob_size + 1 bytes, and ob_sval[ob_size] == 0 — an embedded trailing NUL. That extra NUL byte means a bytes object’s data can be passed straight to C string APIs that expect null-termination, provided the data contains no interior NUL. The NUL is not counted in ob_size or len().

The single-allocation design is visible in the constructor. From _PyBytes_FromSize in Objects/bytesobject.c (verbatim):

    /* Inline PyObject_NewVar */
    if (use_calloc)
        op = (PyBytesObject *)PyObject_Calloc(1, PyBytesObject_SIZE + size);
    else
        op = (PyBytesObject *)PyObject_Malloc(PyBytesObject_SIZE + size);
    if (op == NULL) {
        return PyErr_NoMemory();
    }
    _PyObject_InitVar((PyVarObject*)op, &PyBytes_Type, size);
    set_ob_shash(op, -1);
    if (!use_calloc) {
        op->ob_sval[size] = '\0';
    }
  • PyBytesObject_SIZE + size is the total request: the fixed header portion plus size data bytes (the +1 from the flexible member’s declaration is folded into PyBytesObject_SIZE, giving room for the NUL). Header and data come from one allocation.
  • set_ob_shash(op, -1) initializes the hash to “not computed.”
  • op->ob_sval[size] = '\0' writes the trailing NUL when the buffer wasn’t zero-filled by calloc.

The empty and single-byte caches

CPython pre-allocates the two most common bytes values as immortal singletons in _Py_SINGLETONS (the per-interpreter static-object table). From bytesobject.c (verbatim):

#define CHARACTERS _Py_SINGLETON(bytes_characters)
#define CHARACTER(ch) \
     ((PyBytesObject *)&(CHARACTERS[ch]));
#define EMPTY (&_Py_SINGLETON(bytes_empty))
 
// Return a reference to the immortal empty bytes string singleton.
static inline PyObject* bytes_get_empty(void)
{
    PyObject *empty = &EMPTY->ob_base.ob_base;
    assert(_Py_IsImmortal(empty));
    return empty;
}
  • bytes_empty is the single shared b'' object — immortal, so its reference count never drops it.
  • bytes_characters is an array of 256 one-byte bytes objects, one per possible byte value 0–255, also immortal. CHARACTER(ch) indexes it.

The constructor routes the trivial sizes to these singletons. From PyBytes_FromStringAndSize (verbatim):

    if (size == 1 && str != NULL) {
        op = CHARACTER(*str & 255);
        assert(_Py_IsImmortal(op));
        return (PyObject *)op;
    }
    if (size == 0) {
        return bytes_get_empty();
    }

So b'' and any b'\x41'-style single byte are not freshly allocated — they hand back the shared immortal object. Consequence: b'' is b'' is True, and a single-byte slice like b'hello'[0:1] returns the cached object. This is a singleton cache for sizes 0 and 1 only — it is not general byte-string interning. There is no global table that deduplicates arbitrary bytes; two independently built b'spam' objects are distinct objects (though == compares their contents).

There is no free list for bytes in 3.14 — deallocated multi-byte bytes objects return their memory to the allocator via PyObject_Free; they are not pooled for reuse.

Why bytes can be hashed

Because the data never changes, the hash is stable and worth caching. From bytes_hash (verbatim):

static Py_hash_t
bytes_hash(PyObject *self)
{
    PyBytesObject *a = _PyBytes_CAST(self);
    Py_hash_t hash = get_ob_shash(a);
    if (hash == -1) {
        hash = Py_HashBuffer(a->ob_sval, Py_SIZE(a));
        set_ob_shash(a, hash);
    }
    return hash;
}
  • First call: ob_shash is -1, so Py_HashBuffer runs the buffer hash (SipHash family, the same randomized algorithm used for str, seeded by PYTHONHASHSEED) over the bytes, then caches the result in ob_shash.
  • Subsequent calls return the cached value directly. This makes bytes cheap and legal dict keys and set members.

In the free-threaded build the load/store of ob_shash go through relaxed atomics (get_ob_shash/set_ob_shash switch on Py_GIL_DISABLED), so concurrent first-hash races are benign — every thread computes the same value.

The bytearray Layout

The mutable sibling, verbatim from Include/cpython/bytearrayobject.h (3.14.0):

typedef struct {
    PyObject_VAR_HEAD
    Py_ssize_t ob_alloc;   /* How many bytes allocated in ob_bytes */
    char *ob_bytes;        /* Physical backing buffer */
    char *ob_start;        /* Logical start inside ob_bytes */
    Py_ssize_t ob_exports; /* How many buffer exports */
} PyByteArrayObject;
  • PyObject_VAR_HEAD — header; ob_size here is the logical length (number of live bytes), and len() reads it.
  • ob_alloc — the capacity: total bytes in the ob_bytes allocation. Almost always larger than ob_size because of over-allocation.
  • ob_bytes — pointer to the start of the physical heap buffer (the thing that gets PyMem_Malloc’d / PyMem_Realloc’d / PyMem_Free’d).
  • ob_start — pointer to the logical first byte, which may sit inside ob_bytes past some unused leading slack. This is the trick that makes deleting from the front O(1): advance ob_start instead of shifting every remaining byte.
  • ob_exports — a counter of outstanding buffer-protocol exports (borrowed views). When non-zero, the buffer must not move.

Notice the data is not glued to the header — bytearray has the two-allocation shape (small fixed header + separate ob_bytes buffer) precisely so the buffer can be reallocated independently as the array grows.

Over-allocation and the resize policy

The heart of bytearray’s amortized-O(1) append is bytearray_resize_lock_held in bytearrayobject.c. The growth and shrink decisions (verbatim):

    if (size + logical_offset + 1 <= alloc) {
        /* Current buffer is large enough to host the requested size,
           decide on a strategy. */
        if (size < alloc / 2) {
            /* Major downsize; resize down to exact size */
            alloc = size + 1;
        }
        else {
            /* Minor downsize; quick exit */
            Py_SET_SIZE(self, size);
            PyByteArray_AS_STRING(self)[size] = '\0'; /* Trailing null */
            return 0;
        }
    }
    else {
        /* Need growing, decide on a strategy */
        if (size <= alloc * 1.125) {
            /* Moderate upsize; overallocate similar to list_resize() */
            alloc = size + (size >> 3) + (size < 9 ? 3 : 6);
        }
        else {
            /* Major upsize; resize up to exact size */
            alloc = size + 1;
        }
    }

Walking the four cases:

  • Already fits, minor shrink (size + logical_offset + 1 <= alloc and size >= alloc/2): the existing buffer is kept, just update ob_size and re-write the trailing NUL. No allocation at all — this is the common fast path when appending into spare capacity or trimming a little.
  • Already fits, major shrink (size < alloc/2): the array has shrunk to less than half its capacity, so reclaim memory by reallocating down to size + 1 (exact size plus the NUL).
  • Needs growth, moderate (size <= alloc * 1.125, i.e. within ~12.5% over current capacity): over-allocate with the formula alloc = size + (size >> 3) + (size < 9 ? 3 : 6). The size >> 3 is size / 8 (the ~12.5% growth headroom), and the constant (3 for tiny arrays, 6 otherwise) guarantees a minimum slack even for small sizes. This is the same strategy list uses, and it is what makes a sequence of appends amortized O(1): most appends find room in the slack and hit the no-allocation fast path above.
  • Needs growth, major (a big jump beyond 1.125× — e.g. bytearray(b).extend(huge)): skip over-allocation and request exactly size + 1, since the caller clearly knows the final size.

After deciding alloc, the function either memcpys into a fresh PyMem_Malloc’d buffer (when logical_offset > 0, i.e. ob_start had drifted forward and we want to re-base it) or PyMem_Reallocs the existing buffer in place. Then it resets ob_bytes = ob_start = sval, sets the new size, stores ob_alloc, and writes the trailing NUL (obj->ob_bytes[size] = '\0'). So a bytearray, like bytes, keeps a NUL-terminator one past its logical end.

The ob_start offset

ob_start is what lets del b[0] or b.pop(0) avoid an O(n) shift. Operations that consume from the front advance ob_start within the existing ob_bytes allocation, leaving “dead” slack at the head of the buffer. The logical_offset in the resize code (ob_start - ob_bytes) measures that slack; when a resize happens and the offset is non-zero, the data is memcpy’d to a freshly allocated buffer with ob_start re-based to the front, reclaiming the leading slack. PyByteArray_AS_STRING (from the header) returns ob_start (or a shared empty-string sentinel when the size is zero), so all the byte-access logic transparently sees the logical view, not the physical base.

Why bytearray cannot be hashed

Hashability requires a value that never changes, and a bytearray’s contents can change at any moment — so a stale hash in a dict would corrupt lookups. CPython enforces this by giving the type no hash function. In the PyByteArray_Type definition (verbatim from bytearrayobject.c):

    0,                                  /* tp_hash */

A tp_hash slot of 0 (combined with the type not inheriting one) makes instances unhashable; hash(bytearray(b'x')) raises TypeError: unhashable type: 'bytearray', and a bytearray cannot be a dict key or set member. This is the same mechanism that makes list, dict, and set unhashable.

The Buffer Protocol Connection

Both types expose their raw memory through the buffer protocol — the C-level mechanism (tp_as_buffer slot, bf_getbuffer/bf_releasebuffer functions) that lets one object lend a contiguous view of its memory to another without copying. This is how numpy, memoryview, struct, and zero-copy I/O read and write byte data directly. The full machinery is described in The Buffer Protocol; here only the interplay with the two types’ mutability matters.

For bytes (immutable) the exported view is read-only, and because the object never moves its data, exports are unconditionally safe. For bytearray (mutable) the view can be writable, and here the ob_exports counter earns its keep. Each export increments it; each release decrements it (the source shows obj->ob_exports++ and obj->ob_exports-- with assert(obj->ob_exports >= 0)). A resize is then guarded by _canresize (verbatim):

    if (self->ob_exports > 0) {
        PyErr_SetString(PyExc_BufferError,
            "Existing exports of data: object cannot be re-sized");

If any view is outstanding, resizing is refused with a BufferError. This prevents a use-after-free: if a bytearray reallocated its buffer (changing ob_bytes) while a memoryview held a pointer into the old buffer, that view would dangle. The export count makes the unsafe operation impossible rather than merely discouraged.

>>> b = bytearray(b'spam')
>>> mv = memoryview(b)      # exports a view; ob_exports -> 1
>>> b.append(0x21)          # tries to resize while exported
Traceback (most recent call last):
  ...
BufferError: Existing exports of data: object cannot be re-sized
>>> mv[0] = 0x53            # writing through the view is fine (same memory)
>>> b
bytearray(b'Spam')
>>> mv.release()            # ob_exports -> 0
>>> b.append(0x21)          # now resizing is allowed again
  • Creating the memoryview bumps ob_exports to 1.
  • append would need to (potentially) reallocate, so _canresize blocks it with BufferError.
  • Writing through the view mutates the shared buffer in place — no resize, so it’s allowed and is visible in b.
  • After mv.release() the export count is back to 0 and resizing works again.

Failure Modes and Common Misunderstandings

“bytes and str are interchangeable.” No — bytes is a sequence of integers 0–255; str is a sequence of Unicode code points. Indexing a bytes gives an int (b'A'[0] == 65), not a one-byte bytes. Crossing between them always requires an explicit .encode() / .decode() with a named codec; see CPython String Internals for the str side.

“A bytes object’s data isn’t null-terminated, so I can’t pass it to C.” It is — the invariant guarantees ob_sval[ob_size] == 0. The real caveat is interior NULs: b'a\x00b' will look like "a" to a C string function that stops at the first NUL. Use the length-aware accessors.

“I can put a bytearray in a set if I’m careful.” No — it is structurally unhashable (tp_hash == 0); there is no “careful” path. Freeze it to bytes(my_bytearray) first.

“Appending to a bytearray is always O(1).” Amortized O(1), not worst-case. Most appends use spare capacity (the no-allocation fast path); occasionally one triggers a reallocation-and-copy. The over-allocation policy keeps the average cost constant.

“memoryview keeps my bytearray from being garbage-collected and that’s the only effect.” The more important effect is that the live view freezes the size — any resize raises BufferError until every view is released. Forgetting to release() (or let go of) a memoryview is a classic cause of mysterious BufferErrors on a later append/extend.

Alternatives and When to Choose Them

Reach for bytes when the data is fixed: network payloads you only read, dictionary keys, hashable cache keys, anything you want shared and immutable. Reach for bytearray when you build up or edit binary data: assembling a frame incrementally, in-place transforms, or as the writable backing store behind a memoryview for zero-copy parsing. For a fixed-size mutable region, or to interpret bytes as typed numbers, array.array and struct build on the same buffer protocol. When you need a sliceable, multi-dimensional, or read-only view over someone else’s bytes without copying, use memoryview directly rather than copying into a new bytes/bytearray.

Production Notes

The single-allocation bytes layout means each object carries a fixed header (~33 bytes on 64-bit, header + ob_shash) plus the data plus one NUL, so millions of tiny bytes are header-dominated — the size-0/size-1 singleton cache exists exactly to blunt this for the most common trivial values. For bytearray, the over-allocation means sys.getsizeof reports capacity (ob_alloc-driven), not logical length, so a bytearray that grew and shrank can hold more memory than its current contents suggest until a major-downsize resize reclaims it. In high-throughput I/O code, prefer growing a single bytearray with extend (amortized growth) over repeated bytes concatenation (each + allocates a fresh object and copies everything), and use memoryview slicing to parse without copying — just remember to let the views go before you next resize the backing bytearray.

See Also

Graph View

Backlinks