malloc Arenas and Thread Caching

A single global lock around the heap would make malloc a serialization point that throttles every thread. glibc’s ptmalloc avoids that with two layers of sharding. Arenas are independent heaps, each with its own mutex and its own set of bins; threads spread across arenas so they rarely contend. On top of that, since glibc 2.26, each thread keeps a small lock-free per-thread cache (tcache) that satisfies most small allocations with no lock at all. The main_arena is backed by [[brk and mmap as Allocator Backends|brk]]; secondary arenas are mmap’d on demand when contention is detected, and their number is capped at 8 × CPU cores on 64-bit by default. This two-level scheme is what makes glibc malloc scale — and, as Allocator Trade-offs and Selection explains, it is also the source of the notorious multithreaded RSS bloat. As of glibc 2.43 the tcache is no longer merely the fast path: it is the only per-size-class cache left, because 2.43 deleted the fastbins outright and raised the per-bin cache depth from 7 to 16 to absorb their role.

Authority, versions, and what this note owns

Every claim here is glibc-side and is read from the glibc-2.43 release tag on sourceware.org (malloc/malloc.c, malloc/arena.c, elf/dl-tunables.list, manual/tunables.texi), cross-checked against the glibc-2.42 and glibc-2.44 tags to date changes. All measurements in this note were run on the host that wrote it: glibc 2.43 (Fedora glibc-2.43-8.fc44, ldd --version → 2.43), GCC 16.2.1, Linux 7.1.13, 32 logical CPUs, x86-64. Kernel-side behaviour of brk/mmap is deliberately not covered here — that is brk and mmap as Allocator Backends.

Division of labour with the sibling notes. ptmalloc and glibc malloc owns chunk layout, the boundary tags, the bin taxonomy (unsorted / small / large / top), the binmap, and the overall allocation algorithm. This note owns everything above and around those bins: the arena as a unit of concurrency (how many exist, how a thread is bound to one, how they are recycled), and the tcache as a lock-free layer in front of them (its data structure, its capacity, its hardening, and the 2.43 fastbin removal that made it the sole cache). Where the two touch — for example “a tcache miss falls through into the arena’s small bins” — this note describes the hand-off, not the bins themselves.

Two Levels of Sharding

The mental model is a hierarchy. At the bottom sits one main_arena. As threads start contending on it, glibc spins up additional secondary arenas, each a fully independent heap. Each arena has a mutex; a thread that owns an arena’s lock can allocate and free within it without touching any other thread’s memory. Sitting above the arenas, every thread has a private tcache — an array of tiny free lists — that needs no lock because only that one thread ever touches it. The fast path of malloc therefore checks the thread’s own tcache first and, on a hit, returns without acquiring any lock or entering the arena machinery at all.

flowchart TB
  subgraph T1["Thread 1"]
    TC1["tcache (private, lock-free)<br/>64 small bins × 7 chunks"]
  end
  subgraph T2["Thread 2"]
    TC2["tcache (private, lock-free)"]
  end
  subgraph T3["Thread 3"]
    TC3["tcache (private, lock-free)"]
  end
  TC1 -->|"miss → lock"| A1
  TC2 -->|"miss → lock"| A1
  TC3 -->|"miss → lock"| A2
  subgraph ARENAS["Arenas (shared, one mutex each)"]
    A1["main_arena<br/>(brk-backed heap)"]
    A2["secondary arena #1<br/>(mmap-backed heap)"]
  end
  A1 --> BRK["sbrk / brk"]
  A2 --> MMAP["mmap heaps"]

The two-level allocator hierarchy. What it shows: each thread hits its own lock-free tcache first; only on a miss does it take an arena mutex, and threads are distributed across a bounded set of arenas (main + secondaries). The insight to take: the common case — a small allocation whose size is cached — never touches a lock or the kernel; arenas exist only to keep the uncommon (tcache-miss) path from serializing, and each private cache/arena is memory the other threads cannot reclaim.

The escalation ladder

It helps to read malloc as a ladder of increasingly expensive steps, where each rung is only reached because the one above it failed. Reading __libc_malloc and _int_malloc in malloc/malloc.c at the glibc-2.43 tag, the rungs are: the thread’s own tcache bin; the arena’s bins (small, unsorted, large, top); and finally sysmalloc, which is the only place that talks to the kernel.

flowchart TB
  M["malloc(n)<br/>nb = checked_request2size(n)"]
  T1{"nb &lt; mp_.tcache_max_bytes<br/>and that bin non-empty?"}
  T2["tcache_get(tc_idx)<br/>pop LIFO head, unmask next pointer<br/><b>no lock, no arena, ~a dozen instructions</b>"]
  S{"SINGLE_THREAD_P?"}
  AG["arena_get: lock thread_arena<br/>(or arena_get2 if unbound)"]
  MA["_int_malloc(&main_arena, n)<br/>no lock taken at all"]
  B1["small bin exact fit<br/>+ stash siblings into tcache"]
  B2["unsorted bin scan<br/>+ opportunistic tcache prefill"]
  B3["large bin best fit / binmap scan"]
  B4["split the top chunk"]
  SM["sysmalloc<br/>brk / mmap / grow_heap"]
  RETRY["retry in another arena<br/>(arena_get_retry)"]
  M --> T1
  T1 -->|hit| T2
  T1 -->|miss| S
  S -->|yes| MA
  S -->|no| AG
  AG --> B1
  MA --> B1
  B1 -->|empty| B2
  B2 -->|no fit| B3
  B3 -->|no fit| B4
  B4 -->|top too small| SM
  SM -->|NULL| RETRY

The malloc escalation ladder in glibc 2.43. What it shows: each rung is tried in order and the vast majority of calls stop at the first one; only a sysmalloc call reaches the kernel, and only the rungs below arena_get hold a mutex. The insight to take: the tcache is not an optimisation bolted onto the side of the allocator, it is the normal path — everything below it, including the entire bin machinery of ptmalloc and glibc malloc, is the slow path that a well-behaved program touches rarely. Note also the SINGLE_THREAD_P shortcut: a program that has never created a thread skips arena locking entirely, which is why single-threaded malloc microbenchmarks look unrealistically good.

Main arena versus thread arena, side by side

The two kinds of arena differ in almost every mechanical respect, and most confusion about glibc’s memory behaviour comes from applying facts about one to the other. All values below are read from glibc-2.43 (malloc/arena.c and malloc/malloc.c) for 64-bit Linux.

Property	main_arena	Secondary (“thread”) arena
Where the malloc_state lives	static object in libc’s .data	inside the first heap_info of its own mapping
How many	exactly one, always	0 … arena_max, created lazily on contention
Backing memory	the process heap, grown with sbrk/brk via MORECORE	one or more mmap’d “sub-heaps”
Growth call	MORECORE(size) in sysmalloc	grow_heap() → mprotect of already-reserved address space
Address-space reservation	none; the break just moves	HEAP_MAX_SIZE = 2 * DEFAULT_MMAP_THRESHOLD_MAX = 64 MiB reserved PROT_NONE per sub-heap
Alignment	none (wherever the break happens to be)	each sub-heap is aligned to 64 MiB, deliberately
Finding a chunk’s arena on free	NON_MAIN_ARENA bit clear → it’s main_arena	mask the pointer down to the 64 MiB boundary → heap_info->ar_ptr
Shrinking	systrim() lowers the program break	shrink_heap() → MADV_DONTNEED or a PROT_NONE re-map
Can it be discarded entirely?	no	yes — heap_trim can munmap an empty non-first sub-heap
Chunks can be served by mmap directly	yes (over the mmap threshold)	yes (same threshold, same code)
Visible in /proc/pid/maps as	[heap]	unnamed 64 MiB-aligned rw-p region followed by a ---p tail

The 64 MiB alignment is the single cleverest thing in the design, and it exists purely to make free cheap. A chunk header has no back-pointer to its arena — there is no room for one. Instead arena_for_chunk does:

static __always_inline heap_info *
heap_for_ptr (void *ptr)
{
  size_t max_size = heap_max_size ();       /* 64 MiB on 64-bit */
  return PTR_ALIGN_DOWN (ptr, max_size);    /* one AND instruction */
}
 
static __always_inline struct malloc_state *
arena_for_chunk (mchunkptr ptr)
{
  return chunk_main_arena (ptr) ? &main_arena : heap_for_ptr (ptr)->ar_ptr;
}

Line by line: chunk_main_arena(ptr) tests the NON_MAIN_ARENA status bit that ptmalloc and glibc malloc describes, and if it is clear the answer is the statically-known main_arena with no memory access at all. Otherwise PTR_ALIGN_DOWN masks off the low 26 bits of the address — because every sub-heap begins at a 64 MiB boundary, that arithmetic lands exactly on the heap_info header at the start of the sub-heap, whose ar_ptr field names the owning arena. So free(p) resolves “which lock do I need?” in two instructions, for any pointer, from any thread. The price is that a sub-heap can never exceed 64 MiB and must reserve that much address space up front.

Arenas: main vs. Secondary, and How Many

The main_arena is a static structure in libc’s data segment; its heap is the classic process heap grown with sbrk/brk (MORECORE). It is created once and always exists. Secondary arenas are different: each is allocated as an mmap’d region and managed through a heap_info structure. A secondary heap is a contiguous mmap that “always starts at an address aligned to HEAP_MAX_SIZE” (malloc/arena.c), which on 64-bit is 2 * DEFAULT_MMAP_THRESHOLD_MAX = 64 MiB (HEAP_MIN_SIZE is 32 KiB). The alignment is a trick: given any chunk’s address, masking off the low HEAP_MAX_SIZE bits yields the heap_info at the start, and from there the owning arena — so free can find a chunk’s arena in O(1) without storing a back-pointer in every chunk (the NON_MAIN_ARENA flag bit tells it whether to do this lookup at all).

Arenas are created lazily, per thread, on that thread’s first allocation — not, as is often repeated, “when contention is detected”. This distinction matters and is easy to check in the source. arena_get is a two-line macro:

#define arena_get(ptr, size) do {                    \
      ptr = thread_arena;                            \
      arena_lock (ptr, size);                        \
  } while (0)
 
#define arena_lock(ptr, size) do {                   \
      if (ptr)                                       \
        __libc_lock_lock (ptr->mutex);   /* blocks */ \
      else                                           \
        ptr = arena_get2 ((size), NULL); /* bind */   \
  } while (0)

Read it literally: if the thread is already bound to an arena (the thread-local thread_arena is non-NULL), it takes that arena’s mutex and waits if the mutex is held. It does not go looking for a less busy arena, and it does not create one. New arenas appear only down the else branch — the path taken exactly once per thread, the first time that thread calls malloc. Contention only enters the picture after the arena limit has been reached, inside reused_arena (below).

That limit is the crucial tunable. From arena_get2 in malloc/arena.c, if mp_.arena_max is set (via mallopt(M_ARENA_MAX, …), the glibc.malloc.arena_max tunable, or the MALLOC_ARENA_MAX environment variable) it is a hard cap and arena_test is ignored entirely; otherwise, once the arena count exceeds mp_.arena_test (default 8 on 64-bit, 2 on 32-bit — the static initializer is .arena_test = NARENAS_FROM_NCORES (1), and the tunables manual states the same defaults), glibc finally calls __get_nprocs() and sets the limit to NARENAS_FROM_NCORES(n):

#define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))

So on a 64-bit machine (sizeof(long) == 8) the default ceiling is 8 arenas per CPU core, and on 32-bit it is 2 per core — a fact confirmed both by glibc’s own documentation and by production write-ups (Heroku states “the glibc default number of memory pools on 64bit systems is 8 times the number of CPU cores” — Heroku). On a 16-core server that is up to 128 arenas, each potentially holding tens of megabytes of un-returned free chunks — the mechanism behind the RSS explosion discussed in Allocator Trade-offs and Selection. Once the limit is reached, further threads must share arenas: reused_arena round-robins through the existing arenas (via a next_to_use cursor), locking whichever it lands on, so many threads may serialize on one arena’s mutex. Arenas whose threads exit are placed on a global free_list (protected by free_list_lock) for reattachment.

Arena limits, verified at the glibc-2.43 tag

Every number in this table was read from source at the pinned tag rather than from memory or from the manual, because the manual and the code disagree elsewhere in this same subsystem (see The documentation is stale, below). “Env alias” is the legacy environment variable; every one of these is also settable through GLIBC_TUNABLES=glibc.malloc.<name>=<value>.

Knob	Tunable / mallopt	Env alias	Default	Where the default comes from
Hard arena cap	glibc.malloc.arena_max / M_ARENA_MAX	MALLOC_ARENA_MAX	0 (= “no explicit cap”)	mp_ static init; minval: 1 in elf/dl-tunables.list
Arena count at which the CPU-based limit is computed	glibc.malloc.arena_test / M_ARENA_TEST	MALLOC_ARENA_TEST	8 (64-bit), 2 (32-bit)	.arena_test = NARENAS_FROM_NCORES (1)
Implied cap once arena_test is passed	—	—	8 × online CPUs (64-bit), 2 × (32-bit)	#define NARENAS_FROM_NCORES(n) ((n) * (sizeof (long) == 4 ? 2 : 8))
Sub-heap reservation / alignment	— (compile-time)	—	64 MiB	HEAP_MAX_SIZE = 2 * DEFAULT_MMAP_THRESHOLD_MAX, and DEFAULT_MMAP_THRESHOLD_MAX = 4 * 1024 * 1024 * sizeof(long)
Minimum sub-heap	— (compile-time)	—	32 KiB	#define HEAP_MIN_SIZE (32 * 1024) in malloc/arena.c

Two subtleties that the bare numbers hide. First, arena_max and arena_test are not two ways of saying the same thing: arena_test only decides when to bother asking the kernel how many CPUs there are. Below that count glibc creates arenas freely; the comment in arena_get2 explains the deliberate integer underflow (n <= narenas_limit - 1 with narenas_limit == 0) that makes the test always fail while the limit is still uncomputed. Second, NARENAS_FROM_NCORES is evaluated once, and __get_nprocs() reports CPUs online, not the cgroup CPU quota — so a container pinned to 2 CPUs on a 128-core host still gets a 1024-arena ceiling unless MALLOC_ARENA_MAX is set. That mismatch is the usual root cause of “our container’s RSS is far above its working set”.

What happens when the cap is hit: reused_arena

Once narenas has reached the limit, arena_get2 calls reused_arena, and only here does contention actually steer anything. The function keeps a static mstate next_to_use cursor and walks the circular list of all arenas from there, calling __libc_lock_trylock on each:

  result = next_to_use;
  do {
      if (!__libc_lock_trylock (result->mutex))
        goto out;                 /* found an idle arena, keep the lock */
      result = result->next;
  } while (result != next_to_use);
 
  if (result == avoid_arena)       /* don't retry the arena that just failed us */
    result = result->next;
 
  __libc_lock_lock (result->mutex); /* everything is busy: block on one */
out:
  /* ... detach from the old arena, attach to this one ... */
  thread_arena = result;
  next_to_use = result->next;

So the policy is first idle arena, else block on the next one in rotation, with the cursor advanced so consecutive late-arriving threads spread out instead of piling onto the same arena. The attach/detach bookkeeping under free_list_lock maintains attached_threads per arena, which is what lets arena_thread_freeres decide, at thread exit, whether the arena should go back on the free list. Note the two FIXME: … data race comments glibc itself carries on next_to_use and on the result->next walk: the cursor is deliberately raced on, because being occasionally wrong only costs a slightly worse arena choice.

sequenceDiagram
  autonumber
  participant T as New thread
  participant TLS as thread_arena (TLS)
  participant AG as arena_get2
  participant FL as free_list
  participant NEW as _int_new_arena
  participant RU as reused_arena
  T->>TLS: read thread_arena
  TLS-->>T: NULL (never allocated here)
  T->>AG: arena_get2(size, NULL)
  AG->>FL: get_free_list()
  alt an arena was left by an exited thread
    FL-->>AG: recycled arena (warm bins)
    AG->>TLS: thread_arena = that arena
  else free list empty and narenas < limit
    AG->>NEW: _int_new_arena(size)
    NEW->>NEW: new_heap(): mmap 128 MiB PROT_NONE,<br/>trim to a 64 MiB-aligned window,<br/>mprotect the used prefix RW
    NEW-->>AG: fresh arena, its own mutex
    AG->>TLS: thread_arena = fresh arena
  else free list empty and at the limit
    AG->>RU: reused_arena(avoid)
    loop over the circular arena list from next_to_use
      RU->>RU: trylock(arena->mutex)
    end
    RU-->>AG: first idle arena, else block on next_to_use
    AG->>TLS: thread_arena = shared arena
  end
  Note over T,TLS: from now on this thread always locks<br/>that one arena and never looks for another

Binding a thread to an arena, once, on its first malloc. What it shows: the three mutually exclusive outcomes of arena_get2 — inherit a recycled arena, create a new one, or share an existing one — and the fact that the decision is made exactly once per thread. The insight to take: thread-to-arena assignment is sticky and essentially arbitrary; glibc never rebalances a running thread onto a quieter arena. If a workload has one hot allocator thread and many idle ones, and they happen to land on the same arena, glibc will not fix it for you — the tcache is what saves you, not the arena layer.

The tcache: Lock-Free Per-Thread Caching

Introduced in glibc 2.26 (2017), the thread-local cache is the single biggest reason modern glibc malloc is fast for small objects. Each thread has a tcache_perthread_struct holding an array of singly-linked LIFO free lists — one per size class. The classic dimensions, unchanged from 2.26 through glibc 2.41, were TCACHE_MAX_BINS = 64 bins covering chunk sizes in 16-byte steps, each bin holding at most TCACHE_FILL_COUNT = 7 chunks. Both of those numbers changed in the 2.42/2.43 pair and the widely-copied “64 bins × 7 chunks” description is now wrong on a current system; the verified values are tabulated below.

The per-thread structure, as it is in 2.43

typedef struct tcache_entry {
  struct tcache_entry *next;   /* overlaid on the freed chunk's user data */
  uintptr_t key;               /* set to tcache_key while cached: double-free tripwire */
} tcache_entry;
 
typedef struct tcache_perthread_struct {
  uint16_t      num_slots[TCACHE_MAX_BINS];   /* remaining capacity, counts DOWN */
  tcache_entry *entries  [TCACHE_MAX_BINS];   /* LIFO list heads */
} tcache_perthread_struct;
 
static __thread tcache_perthread_struct *tcache =
  (tcache_perthread_struct *) &__tcache_dummy.inactive;

Three design details are worth unpacking, because each one exists to shave instructions off the hot path.

num_slots counts down, not up. Older glibc stored a per-bin counts[] and compared it against mp_.tcache_count on every free. 2.43 stores the remaining capacity and the free fast path becomes if (tcache->num_slots[tc_idx] != 0) tcache_put(...) — a compare against a constant zero instead of a load-and-compare against a global. The counter is uint16_t, which is exactly why the tunable’s ceiling is MAX_TCACHE_COUNT = UINT16_MAX (65,535): a larger value would overflow the slot counter.

tcache is never NULL. Instead of a null check on every call, the thread-local pointer starts out aimed at a shared, all-zero dummy object, __tcache_dummy.inactive. Because every num_slots entry in it is zero and every entries head is NULL, the ordinary fast-path tests fail naturally and control falls through to the slow path, which then notices tcache_inactive() and allocates a real one. A second dummy, __tcache_dummy.disabled, sits one byte further along in the same union so the two have distinct addresses; a thread that is shutting down, or whose tcache allocation failed, or that ran with tcache_count=0, is pointed at that one and will never retry initialisation. This is a neat trick: an entire “is the cache usable?” state machine is encoded in the value of a single pointer, at zero cost to the fast path.

The double-free key is random per process. tcache_key_initialize pulls it from getrandom(2) (GRND_NONBLOCK, via __getrandom_nocancel_nostatus_direct) and then rejects values that are too small, too large, or have an unbalanced population count, looping until it gets one with at least __WORDSIZE / 4 one-bits and at most three quarters ones. The comment says why: a key that collides with values commonly found in user data would send free down the expensive tcache_double_free_verify scan of all 76 bins on every innocent free. It is a performance guard as much as a security one.

stateDiagram-v2
  [*] --> Inactive: thread starts<br/>tcache = &__tcache_dummy.inactive
  Inactive --> Live: first malloc/free reaches the slow path<br/>tcache_init() allocates 760 bytes
  Inactive --> Disabled: tcache_init() with mp_.tcache_count == 0<br/>(tunable set to 0)
  Inactive --> Disabled: tcache_init() allocation returned NULL<br/>(never retried)
  Live --> Disabled: tcache_thread_shutdown()<br/>drain all 76 bins, free the struct
  Disabled --> [*]: thread exits
  note right of Inactive
    all-zero shared dummy:
    every fast-path test fails
    naturally, no NULL check needed
  end note
  note right of Live
    entries[] / num_slots[] usable
    lock-free malloc and free
  end note

The three states of a thread’s tcache pointer in glibc 2.43. What it shows: tcache is never NULL; it points at one of two shared all-zero dummy objects or at a real per-thread structure, and the transitions are one-way. The insight to take: the “is my cache usable?” question is answered by the pointer’s value, so the fast path contains no state check at all — and a failed tcache_init is permanent for that thread, which is why an allocation failure during thread start-up quietly costs you the cache for that thread’s entire life.

tcache parameters, verified at glibc-2.43

Constant / tunable	Value at 2.43	Value at 2.41	Meaning
TCACHE_SMALL_BINS	64	64 (as TCACHE_MAX_BINS)	linear 16-byte-step size classes
TCACHE_LARGE_BINS	12	— (did not exist)	logarithmic classes, up to 4 MiB chunks
TCACHE_MAX_BINS	76	64	total bins in tcache_perthread_struct
TCACHE_FILL_COUNT	16	7	default chunks per bin
MAX_TCACHE_COUNT	65,535 (UINT16_MAX)	65,535	ceiling for tcache_count
MAX_TCACHE_SMALL_SIZE	tidx2csize(63) = 1040 bytes of chunk	same	largest linearly-indexed chunk
mp_.tcache_max_bytes (default)	MAX_TCACHE_SMALL_SIZE + 1 = 1041	same	chunk-size cut-off tested by nb < mp_.tcache_max_bytes
Largest usable size cached by default	1032 bytes on 64-bit (516 on 32-bit)	same	1040 − SIZE_SZ
glibc.malloc.tcache_max	raises the cut-off, max 4,194,304	capped at the small-bin boundary	opt-in large-block caching (new in 2.42)
glibc.malloc.tcache_count	per-bin depth; 0 disables the cache	same	env alias: none — tunable only
glibc.malloc.tcache_unsorted_limit	0 = unlimited	same	caps how far the unsorted-bin scan will go while prefilling

Note the asymmetry in that table: tcache_max and tcache_count have no environment-variable alias. Unlike MALLOC_ARENA_MAX, they can only be set through GLIBC_TUNABLES=glibc.malloc.tcache_count=…, which is verifiable directly in elf/dl-tunables.list (the env_alias: line is absent for exactly these three). Guides that tell you to export MALLOC_TCACHE_COUNT_ are describing something that has never existed.

Indexing is split. For chunk sizes below the small boundary, csize2tidx(x) = (x - MINSIZE) / MALLOC_ALIGNMENT — a subtract and a shift. Above it, large_csize2tidx uses a leading-zero count to build a logarithmic index, TCACHE_SMALL_BINS + __builtin_clz(MAX_TCACHE_SMALL_SIZE) - __builtin_clz(nb), so each of the 12 large bins spans a power-of-two size range. Because a large bin therefore holds chunks of different sizes, the large-bin helpers (tcache_put_large, tcache_get_large, tcache_location_large) keep each list sorted by size and walk it, which is why large caching is opt-in: it trades a constant-time pop for a short list walk.

The hot path, instruction by instruction

The payoff is on the hot path. __libc_malloc in 2.43 is, in its entirety, the cache lookup plus a tail call:

void *__libc_malloc (size_t bytes) {
  size_t nb = checked_request2size (bytes);       /* user size -> chunk size, overflow-checked */
  if (nb < mp_.tcache_max_bytes) {                /* one compare against a global */
      size_t tc_idx = csize2tidx (nb);            /* subtract + shift */
      if (__glibc_likely (tc_idx < TCACHE_SMALL_BINS)) {
          if (tcache->entries[tc_idx] != NULL)    /* one load from TLS */
            return tag_new_usable (tcache_get (tc_idx));   /* pop, unmask, return */
      } else { /* ... large-bin variant ... */ }
  }
  return __libc_malloc2 (bytes);                  /* everything else */
}

No mutex, no arena walk, no syscall, no branch into the bin machinery. free is the mirror image: check the chunk’s key against tcache_key (double-free tripwire), compute the index, and if (tcache->num_slots[tc_idx] != 0) return tcache_put (p, tc_idx);.

Only on a tcache miss — an empty bin on allocation, a full bin on free — does control fall through to the arena machinery of ptmalloc and glibc malloc. And even then glibc tries to make the miss pay for itself: when _int_malloc satisfies a small request from a small bin, it immediately drains more chunks of the same size out of that bin into the tcache while it still holds the arena lock —

  /* While bin not empty and tcache not full, copy chunks over.  */
  while (tcache->num_slots[tc_idx] != 0
         && (tc_victim = last (bin)) != bin)
    { ...; tcache_put (tc_victim, tc_idx); }

— so one lock acquisition can amortise up to tcache_count subsequent allocations. The same prefill happens opportunistically while scanning the unsorted bin, bounded by glibc.malloc.tcache_unsorted_limit when that tunable is set (it is 0, meaning unbounded, by default). Raising TCACHE_FILL_COUNT from 7 to 16 in 2.43 makes this batching roughly twice as effective, which is precisely the mechanism the fastbin-removal patch relied on to recover its lost performance.

What the cache costs you

The tcache is not free of memory cost, and the cost grew in 2.42/2.43. The tcache_perthread_struct is itself allocated on the heap, with malloc, the first time a thread touches the allocator: sizeof (tcache_perthread_struct) = 76 × 2 bytes of num_slots plus 76 × 8 bytes of list heads = 760 bytes, which becomes a 768-byte chunk. It is released by tcache_thread_shutdown when the thread exits, which first walks all 76 bins and returns every cached chunk to its owning arena with _int_free_chunk (arena_for_chunk (p), p, chunksize (p), 0) — note arena_for_chunk, not “this thread’s arena”: chunks go home to whichever arena they came from, so a thread that freed other threads’ memory does not corrupt anyone’s accounting on exit.

More significantly, the contents of the cache are memory held out of circulation. The glibc manual computes the classic worst case as “approximately 236 KB on 64-bit systems” — but that figure assumes tcache_count = 7. With the 2.43 default of 16 the same arithmetic (64 bins, mean chunk size (32+1040)/2 = 536 bytes, 16 deep) gives 64 × 16 × 536 ≈ 536 KiB per thread, and if you opt into large-block caching by raising tcache_max, twelve more bins each holding up to sixteen chunks of up to 4 MiB puts the theoretical ceiling in the hundreds of megabytes per thread. These are worst cases, not typical ones — a bin only fills if the program actually frees that many chunks of that size class — but the direction of travel is clear: since 2.42, glibc trades more per-thread footprint for less locking. Across many threads this is a real contributor to RSS, a smaller cousin of the arena-bloat problem in Allocator Trade-offs and Selection.

tcache Hardening

Because tcache entries are singly-linked through pointers stored inside freed chunks — the next and key fields are overlaid on what was the user’s data — the cache is an obvious exploitation target, and glibc has hardened it twice.

Key-based double-free detection. Each cached chunk’s second word is set to the process-wide random tcache_key while it sits in the cache (e->key = tcache_key in tcache_put_n) and cleared when it is handed out (e->key = 0 in tcache_get_n). free checks that word before anything else: if it matches, the chunk is probably already cached, so tcache_double_free_verify walks all TCACHE_MAX_BINS lists to confirm. If it finds the chunk, it aborts with free(): double free detected in tcache 2; if it does not — the value was a coincidence, or the chunk is in another thread’s cache — it clears the key and retries the free normally. The same walk also enforces a length bound (free(): too many chunks detected in tcache) and alignment (free(): unaligned chunk detected in tcache 2), so a corrupted list is caught here too. Earlier glibc used a fixed key derived from the tcache_perthread_struct address; since the 2.34-era rework it is randomised from getrandom(2) at first use, as described above.

Safe-linking (glibc 2.32). The next pointers are stored obfuscated:

#define PROTECT_PTR(pos, ptr) \
  ((__typeof (ptr)) ((((size_t) pos) >> 12) ^ ((size_t) ptr)))
#define REVEAL_PTR(ptr)  PROTECT_PTR (&ptr, ptr)

Walking the symbols: pos is the address of the slot the pointer is written into, ptr is the pointer being stored. Shifting pos right by 12 discards the page offset and leaves the page frame number, which under ASLR is not known to an attacker who has only a relative write; XOR-ing that into the stored value means a forged next pointer cannot be aimed anywhere useful without first leaking a heap address. REVEAL_PTR is self-inverse because XOR is, and it recovers pos from the address of the field itself. tcache_get_n additionally runs misaligned_mem(e) on every chunk it pops, so a mangled pointer that does not decode to a 16-byte-aligned address is caught immediately (malloc(): unaligned tcache chunk detected). The mechanism and its limits are analysed in the Check Point write-up.

When safe-linking landed it protected both the tcache and the fastbins, and most descriptions of it still say so. On glibc 2.43 there are no fastbins left, so PROTECT_PTR/REVEAL_PTR are a tcache-only mechanism; the doubly-linked bins that remain in ptmalloc and glibc malloc are protected by the older “safe unlinking” consistency checks instead, which is a different mitigation with different properties.

Large-Block tcache (glibc 2.42, 2025)

Historically the tcache only cached small chunks (≤ ~1032 bytes). glibc 2.42 (August 2025) extended it: the source now defines TCACHE_SMALL_BINS = 64 plus TCACHE_LARGE_BINS = 12, giving TCACHE_MAX_BINS = 76, with the large bins using logarithmic size classes (large_csize2tidx, via a __builtin_clz leading-zero count) to span chunks up to 4 MiB. Per the NEWS, “the thread-local cache in malloc (tcache) now supports caching of large blocks … enabled by setting the tunable glibc.malloc.tcache_max to a larger value (max 4194304)” (glibc NEWS; patch v12). The feature is opt-in — the default tcache_max still corresponds to the classic small-bin boundary, so out-of-the-box behavior in 2.42 matches earlier releases unless you raise the tunable.

Resolved

The two open questions in the earlier draft of this note are now settled against the glibc-2.43 tag and against the running system. (1) The default glibc.malloc.tcache_max is unchanged — mp_.tcache_max_bytes is statically initialised to MAX_TCACHE_SMALL_SIZE + 1 = 1041, i.e. the classic ~1032-byte usable boundary, so large-block caching really is opt-in. (2) TCACHE_FILL_COUNT is 16 in released 2.43, not 7; that is verified three ways — #define TCACHE_FILL_COUNT 16 at glibc-2.43:malloc/malloc.c:317, the malloc: set default tcache fill count to 16 commit in the fastbin-removal series, and a measurement on this host (below). The glibc manual still says 7; the manual is wrong.

The Fastbins Are Gone: glibc 2.43 Deleted a Whole Allocator Layer

This is the largest structural change to glibc malloc since the tcache itself arrived in 2.26, and it belongs in this note rather than in ptmalloc and glibc malloc because it is fundamentally a tcache story: the fastbins were removed because the tcache had made them redundant.

What a fastbin was

For twenty-odd years, ptmalloc kept a set of fastbins in every arena: singly-linked, LIFO, per-size-class lists of small freed chunks that were deliberately not coalesced with their neighbours. A fastbin chunk kept its PREV_INUSE bit set so the surrounding chunks would not try to merge with it, which made free of a small chunk O(1) — push and return — at the cost of leaving the heap fragmented until a malloc_consolidate() sweep flushed every fastbin into the unsorted bin. The cutoff was global_max_fast, settable with mallopt(M_MXFAST, …) or the glibc.malloc.mxfast tunable, defaulting to 128 bytes of request size on 64-bit. Fastbins sat below the tcache in priority: malloc consulted the tcache first, then the fastbin; free tried the tcache first, then the fastbin.

What 2.43 does instead

Nothing. The layer is gone. Measured by grepping malloc/malloc.c at consecutive release tags:

Tag	fastbin occurrences in malloc/malloc.c	malloc_consolidate	mallopt(M_MXFAST)
glibc-2.42	106	present (20 references)	functional
glibc-2.43	0	removed	do_set_mxfast is { return 1; }
glibc-2.44	0	removed	do_set_mxfast is { return 1; }

mallopt(M_MXFAST, n) still returns success (1) for any n, because failing would break callers that check the return value; it simply has no effect. The glibc.malloc.mxfast tunable still exists in elf/dl-tunables.list and still appears in ld.so --list-tunables, but __ptmalloc_init in 2.43 no longer contains a TUNABLE_GET (mxfast, …) line, so setting it is a no-op too. mallinfo’s smblks and fsmblks fields — which reported fastbin counts — are now hard-coded to 0 for backward compatibility, and malloc_info(3)’s XML has lost its <total type="fast" …/> element. Watch for that last one: any monitoring script that parses malloc_info output for the fast total silently stops finding it on 2.43.

Why: the maintainers’ own argument

Rather than infer the rationale, it is worth reading it. The change landed as Dev Jain’s ten-patch series “[PATCH v2 00/10] Remove fastbins”, posted to libc-alpha on 2025-12-08 and reviewed by DJ Delorie, Florian Weimer and Adhemerval Zanella. The cover letter states the case directly (libc-alpha archive):

The small tcache data structure is similar to the fastbin layer - we have size classes in which we store same size chunks. After the tcache was introduced in malloc, the tcache precedes the fastbin when we try to find a victim to allocate. Similarly, the tcache precedes the fastbin when we free a chunk … It is, therefore, easy to observe that post the introduction of the tcache, the fastbin layer has become redundant because it serves exactly the same functionality as the tcache, with the extra overhead of arena locks. Therefore, it makes sense to completely remove this allocation layer.

And on the performance question, with the numbers:

Upon just disabling fastbins, we get a very minor regression (~0.5%). Upon setting the default TCACHE_FILL_COUNT to 16, we get almost the same performance.

That is the whole story in two sentences, and it is worth restating plainly. A fastbin and a tcache bin are the same data structure — a LIFO singly-linked list of same-size chunks — but the fastbin lives inside the arena and therefore requires the arena mutex, while the tcache lives in thread-local storage and requires nothing. The only thing the fastbin offered over the tcache was depth: when a thread’s 7-deep tcache bin filled up, the overflow had somewhere cheap to go. Doubling the tcache depth to 16 removes that advantage, and the measurement in the cover letter says the exchange is performance-neutral. The series is broken into ten commits (remove fastbin tests, remove malloc_consolidate, remove allocation from fastbin, and trim_fastbins, remove mallopt fastbin stats, …, set default tcache fill count to 16) and nets out at 836 lines deleted against 61 added.

Second-order consequences

Deleting a layer that had been load-bearing for two decades changes several behaviours that are worth knowing about:

• FASTBIN_CONSOLIDATION_THRESHOLD was renamed ATTEMPT_TRIMMING_THRESHOLD (still 64 KiB, still “half the default trim threshold”). Its meaning changed with its name: it used to mean “a chunk this big was freed, so flush the fastbins and coalesce”, and now means only “a chunk this big was freed, so it is worth trying to trim the arena”. See brk and mmap as Allocator Backends for what that trimming actually does.
• Coalescing is now immediate for every non-cached chunk. Without fastbins there is no class of freed chunk that deliberately refuses to merge. A freed chunk either lands in the tcache or goes straight through _int_free_chunk, which coalesces with its neighbours at once. Heaps should fragment slightly less and consolidate more predictably; there is no longer a “phantom” state where mallinfo reports memory free but coalescing has not happened yet.
• A class of heap exploits lost its primitive. The fastbin dup family of attacks — free the same chunk twice into a fastbin, then corrupt the fd pointer to make malloc return an arbitrary address — has no fastbin to work with any more. The analogous tcache attacks remain, but they face the key-based double-free check and safe-linking described below. glibc removed malloc/tst-safe-linking.c and malloc/tst-mxfast.c in the same series.
• Nothing was announced. The NEWS file for 2.43 does not mention fastbins, M_MXFAST, or mallinfo.smblks anywhere. A user-visible ABI-adjacent behaviour change — two mallinfo fields permanently zeroed, one malloc_info XML element gone, one mallopt parameter silently neutered — shipped without a release note. If you maintain tooling that reads any of those, you will find out by observation, which is exactly how the fact was established for this note.

Uncertain

Verify: whether any distribution ships a 2.43 build that reinstates fastbins via a downstream patch. Reason: this note checked upstream tags plus one distribution build (Fedora glibc-2.43-8.fc44, where the removal is present — malloc_info emits no <total type="fast">); other vendors were not checked. To resolve: on the target system, run a program that calls malloc_info(0, stdout) and look for a fast total, or check whether mallinfo2().smblks is ever non-zero. uncertain

Worked Example: Tuning Arenas

The single most common production intervention is capping arenas to curb memory. Both are equivalent ways to set the cap:

# Environment variable, read at process startup (no recompile, no code change):
export MALLOC_ARENA_MAX=2
./my_server
 
# Or programmatically, before threads spawn, from C:
#   #include <malloc.h>
#   mallopt(M_ARENA_MAX, 2);

MALLOC_ARENA_MAX=2 tells glibc to create at most two arenas regardless of core count, so free memory pools far less and RSS drops — at the cost of more lock contention on those two arenas. This is exactly what Heroku sets by default for dynos (since 2019) and what heavily-threaded runtimes like Ruby and the JVM recommend when native RSS balloons; the full trade-off and measurements are in Allocator Trade-offs and Selection.

Measuring It Yourself

Everything above is observable from user space with nothing more exotic than /proc/self/maps and malloc_info(3). All output below is real, produced on glibc 2.43 (Fedora glibc-2.43-8.fc44), GCC 16.2.1, Linux 7.1.13, x86-64, 32 logical CPUs.

Counting arenas: /proc/self/maps and malloc_info(3)

Secondary-arena sub-heaps are the only anonymous mappings in a normal process that begin on a 64 MiB boundary, so counting them is a two-line scan. Spawn eight threads, have each allocate enough to force a real heap, hold them at a barrier, and look:

static void *worker(void *arg) {
    void *keep[64];
    for (int i = 0; i < 64; i++) { keep[i] = malloc(4096); memset(keep[i], 1, 4096); }
    pthread_barrier_wait(&b);   /* hold the arenas open while main inspects */
    pthread_barrier_wait(&b);
    return NULL;
}
/* in main, after the first barrier: scan /proc/self/maps for
   anonymous rw-p regions whose start address is 64 MiB-aligned */

--- 64 MiB-aligned rw-p regions (secondary arena heaps) ---
  00007f47f0000000-00007f47f0041000 rw-p  size=260K
  00007f47f4000000-00007f47f4041000 rw-p  size=260K
  00007f47f8000000-00007f47f8041000 rw-p  size=260K
  00007f4800000000-00007f4800041000 rw-p  size=260K
  00007f4808000000-00007f4808041000 rw-p  size=260K
  00007f480c000000-00007f480c041000 rw-p  size=260K
  00007f4810000000-00007f4810041000 rw-p  size=260K
  00007f4814000000-00007f4814041000 rw-p  size=260K
  => 8 secondary-arena heaps found

Eight threads, eight secondary arenas, plus main_arena for the initial thread — one arena per thread, exactly as arena_get2 predicts, and well under the ceiling (32 cores × 8 = 256). Every start address ends in 0000000 and the gaps between them are multiples of 64 MiB. The 260K is what is currently mprotected read/write, not what is reserved; the reservation is visible if you print the next line of /proc/self/maps too:

7f4964000000-7f4964034000 rw-p 00000000 00:00 0        <- committed prefix, 208 KiB
7f4964034000-7f4968000000 ---p 00000000 00:00 0        <- PROT_NONE tail

0x7f4968000000 − 0x7f4964000000 is exactly 0x4000000 = 64 MiB. That ---p region is the reserved-but-uncommitted remainder of the sub-heap, and it is the reason a many-threaded glibc process shows an alarming VmSize while its VmRSS stays modest. alloc_new_heap in malloc/arena.c gets there by mapping HEAP_MAX_SIZE << 1 (128 MiB) PROT_NONE, carving out the 64 MiB-aligned window inside it, munmapping the slack, and then mprotecting only the prefix it needs.

malloc_info(3) is the better tool and is almost always overlooked. It dumps one XML <heap> element per arena on a stream of your choosing, with no allocation of its own:

<malloc version="1">
<heap nr="0">                                     <!-- main_arena -->
<sizes>
</sizes>
<total type="rest" count="1" size="126464"/>
<system type="current" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="1">                                     <!-- first secondary arena -->
<total type="rest" count="1" size="912"/>
<system type="current" size="266240"/>
<aspace type="mprotect" size="266240"/>
<aspace type="subheaps" size="1"/>
</heap>
...
<total type="rest" count="9" size="133760"/>      <!-- process-wide totals -->
<total type="mmap" count="0" size="0"/>
<system type="current" size="2265088"/>
</malloc>

Nine <heap> elements confirms nine arenas. <aspace type="subheaps"> tells you how many 64 MiB sub-heaps an arena has chained together, and its absence from heap nr="0" is how you identify the main arena. Note what is missing: on glibc ≤ 2.42 each heap also carried a <total type="fast" count="…" size="…"/> line. On 2.43 it is gone, because the fastbins are gone — the diff is directly observable at runtime, without reading any source.

Setting MALLOC_ARENA_MAX=2 on the identical binary collapses the picture to one secondary arena (main_arena counts toward the cap), and the eight worker threads share it:

--- 64 MiB-aligned rw-p regions (secondary arena heaps) ---
  00007f0d60000000-00007f0d60102000 rw-p  size=1032K
  => 1 secondary-arena heaps found

Note that the single shared sub-heap grew to 1032 KiB where each private one was 260 KiB — the same total working set, now concentrated. Process-wide <system type="current"> fell only from 2,265,088 to 2,138,112 bytes here, because this toy workload’s live data dominates; the RSS win from capping arenas shows up in programs whose arenas accumulate free chunks, which is the case described in Allocator Trade-offs and Selection.

Measuring the tcache depth without touching glibc internals

TCACHE_FILL_COUNT is a compile-time constant inside a static object; there is no API to read it. But it is inferable, because the tcache is strictly LIFO and the arena’s bins are not. Allocate a run of same-size chunks, free them front to back, then allocate them again: the first tcache_count of them come back in exact reverse order (they were pushed onto and popped off the cache), and the remainder come back in the order the arena’s bins chose.

#define K 40
void *a[K], *b[K];
for (int i = 0; i < K; i++) a[i] = malloc(64);
for (int i = 0; i < K; i++) free(a[i]);     /* a[0] freed first */
for (int i = 0; i < K; i++) b[i] = malloc(64);
/* report, for each b[i], which a[j] it is */

=== default (glibc 2.43) ===
realloc order (index into the original array): 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 16 17 18 19 ...
leading LIFO run = 16  -> tcache bin capacity = 16

=== GLIBC_TUNABLES=glibc.malloc.tcache_count=7 ===
realloc order (index into the original array): 6 5 4 3 2 1 0 7 8 9 10 11 12 13 14 15 16 17 18 19 ...
leading LIFO run = 7  -> tcache bin capacity = 7

=== GLIBC_TUNABLES=glibc.malloc.tcache_count=0 (disabled) ===
realloc order (index into the original array): 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ...
leading LIFO run = 1  -> tcache bin capacity = 1

Read the first line carefully: 15 14 13 … 0 is a perfect sixteen-element reversal, then 16 17 18 19 … in forward order. The first sixteen frees filled the tcache bin and were served back from it LIFO; the seventeenth free onward overflowed into the arena and came back FIFO from the bins. Forcing tcache_count=7 reproduces the pre-2.43 behaviour exactly, which both validates the method and demonstrates that 16 is the default on this system. With the cache disabled the reversal disappears entirely.

Demonstrating that a tcache hit takes no lock

The claim “the tcache avoids the arena mutex” can be tested directly by removing the arenas’ ability to spread the load. Run eight threads doing nothing but malloc(64)/free on a small rotating ring of pointers, and compare four configurations. (Compile with -fno-builtin-malloc -fno-builtin-free, or GCC will elide the allocation pairs and you will measure nothing.)

[default (8 arenas, tcache)   ] 8 threads x 2000000 malloc/free(64):      21 ms  ( 759.6 Mops/s)
[1 arena, tcache on           ] 8 threads x 2000000 malloc/free(64):      24 ms  ( 668.7 Mops/s)
[8 arenas, tcache OFF         ] 8 threads x 2000000 malloc/free(64):     104 ms  ( 154.3 Mops/s)
[1 arena,  tcache OFF         ] 8 threads x 2000000 malloc/free(64):    5902 ms  (   2.7 Mops/s)

The four numbers tell the whole story of this note.

• Rows 1 → 2 (tcache on, arenas 8 → 1): throughput falls about 12%. Collapsing eight independent heaps onto one shared mutex costs almost nothing, because with the cache warm the threads essentially never reach for the mutex.
• Rows 3 → 4 (tcache off, arenas 8 → 1): throughput falls 57×, from 154 Mops/s to 2.7. This is what a contended arena mutex actually costs, and it is what glibc looked like before 2.26.
• Rows 1 → 3 (8 arenas, cache on → off): even with eight arenas and eight threads — one arena each, so no mutex contention at all — turning off the cache still costs 5×. That difference is pure per-call overhead: the uncontended lock/unlock pair, the size-class lookup in the bins, the binmap walk, the coalescing checks.

Together, rows 2 and 4 are the fastbin-removal argument made visible: an allocation layer that lives inside the arena can only ever be as fast as row 4’s lock allows, no matter how cheap its own data structure is, and that is why a redundant lock-protected LIFO list was worth deleting.

The documentation is stale — check the source, not the manual

Working through the above turned up three places where glibc’s own documentation at the 2.43 tag no longer matches glibc’s own code at the 2.43 tag. This is not a criticism so much as a warning about method: for this subsystem, manual/tunables.texi and mallopt(3) are secondary sources.

Claim in the 2.43 docs	Reality in the 2.43 code	How to check
glibc.malloc.tcache_count — “The default is 7.” (manual/tunables.texi)	#define TCACHE_FILL_COUNT 16	the LIFO-run measurement above
glibc.malloc.mxfast — a full paragraph describing fastbins (manual/tunables.texi); M_MXFAST likewise in mallopt(3)	no fastbins exist; no TUNABLE_GET (mxfast, …) in __ptmalloc_init; do_set_mxfast returns 1 unconditionally	malloc_info has no fast total
glibc.malloc.tcache_max — “The default (and maximum) value is 1032 bytes”	the default is still 1032 usable bytes, but the maximum is now 4,194,304 (2.42’s large-block tcache)	NEWS entry for 2.42
“approximate maximum overhead of the per-thread cache is … approximately 236 KB on 64-bit systems”	that figure assumes 7 chunks per bin; at 16 it is roughly 536 KiB	arithmetic: 64 × 16 × 536

The one place the manual is right and a naive reading of the source is wrong is glibc.malloc.top_pad, which brk and mmap as Allocator Backends works through in detail — malloc/malloc.c appears to say the default is 0, and it is actually 128 KiB.

Arena Lifecycle and Thread Exit

The full lifecycle explains where allocations land over a program’s life. When a thread first allocates, arena_get checks its thread_arena (initially NULL), falls through to arena_get2, and either finds an arena on the global free_list to attach to, creates a new arena (if under the limit), or — if at the limit — picks one via reused_arena’s round-robin. Once bound, the thread caches that arena in thread_arena and keeps using it, so a thread’s allocations concentrate in one arena for its lifetime (barring the arena being busy, which can transiently push it elsewhere). When a thread exits, its arena is not destroyed — it is pushed onto the free_list (under free_list_lock) so a future thread can reattach to it and inherit its warmed-up heap. This reattachment is why thread churn (spawn/exit cycles) does not endlessly grow arena count: arenas are pooled and recycled, not created per-thread. It also means a short-lived thread’s allocations can outlive it in the recycled arena’s bins, which is usually harmless but occasionally surprising when tracking down where memory sits.

The exit path is worth reading in full, because the ordering is deliberate. __malloc_arena_thread_freeres (the TLS destructor registered for the allocator) does exactly two things, in this order:

void __malloc_arena_thread_freeres (void)
{
  /* Shut down the thread cache first.  This could deallocate data for
     the thread arena, so do this before we put the arena on the free list.  */
  tcache_thread_shutdown ();
 
  mstate a = thread_arena;
  thread_arena = NULL;
  if (a != NULL) {
      __libc_lock_lock (free_list_lock);
      assert (a->attached_threads > 0);
      if (--a->attached_threads == 0) {     /* last thread using it? */
          a->next_free = free_list;         /* then park it for reuse */
          free_list = a;
      }
      __libc_lock_unlock (free_list_lock);
  }
}

The cache must be drained before the arena is parked, because draining it allocates nothing but frees plenty — every cached chunk is returned with _int_free_chunk into whichever arena owns it, which may be this arena or another thread’s. And the arena is parked only when attached_threads drops to zero: when several threads are sharing an arena (because the cap was reached), the arena stays live until the last of them leaves. Nothing is ever munmapped here — the sub-heaps, their bins and their contents survive intact on the free_list, waiting for get_free_list to hand the whole warm structure to the next thread that calls malloc.

stateDiagram-v2
  [*] --> Unbound: new thread<br/>thread_arena == NULL
  Unbound --> Bound: arena_get2 -> new / recycled / shared
  Bound --> Bound: every malloc locks this one arena
  Bound --> Parked: LAST attached thread exits<br/>--attached_threads == 0<br/>pushed onto free_list
  Bound --> Bound: a NON-last thread exits<br/>attached_threads just decremented,<br/>arena stays live
  Parked --> Bound: get_free_list() hands the<br/>warm arena to the next new thread
  note right of Parked
    memory is NOT returned:
    sub-heaps, bins and free
    chunks all survive
  end note

The lifecycle of a secondary arena. What it shows: an arena is bound to a thread once, stays bound for that thread’s life, and on the last attached thread’s exit is parked rather than destroyed. The insight to take: arena count is bounded by peak concurrent threads, not by total threads created — a server that spawns and reaps ten thousand short-lived threads does not create ten thousand arenas. The flip side is that a short-lived thread’s freed memory sits in a parked arena indefinitely, invisible to top as anything but RSS, until some future thread inherits it.

Failure Modes

Arena explosion and RSS bloat. A program with many threads each doing bursty allocation ends up with one arena per concurrent thread, up to 8 × online CPUs, and every one of them retains free chunks that no other thread can use. RSS stays high long after everything has been freed. The diagnostic is cheap and definitive: rerun with MALLOC_ARENA_MAX=2 (or 1) and watch RSS. If it drops sharply, arenas were the cause. The container case is the nasty one, because __get_nprocs() reports CPUs online on the host, not the cgroup’s CPU quota — a 2-vCPU container on a 128-core node computes a ceiling of 1024 arenas and will happily create as many as it has concurrent allocating threads. Counting <heap> elements in malloc_info(0, stderr) output tells you how many it actually made.

Lock contention after capping. The cure has a dose-response curve. MALLOC_ARENA_MAX=1 on a many-threaded allocation-heavy program can collapse throughput, because every tcache miss now funnels through a single mutex. The measurements above quantify the range: with the tcache warm, going from 8 arenas to 1 cost 12%; with the tcache off, it cost 57×. Real programs sit between those poles depending on how well their allocation sizes fit the cache. Values of 2–4 are the usual compromise, and the right way to pick one is to measure both RSS and p99 latency, not either alone.

free(): double free detected in tcache 2. This is the key tripwire firing: the chunk being freed already carried tcache_key in its second word, so tcache_double_free_verify walked all 76 bins, found the chunk already on one of them, and aborted. It is a genuine bug in the program — a double free, or a use-after-free that got freed again — not a glibc fault. Two caveats worth knowing. First, the check is per thread: if thread A freed the chunk and thread B frees it again, B’s tcache does not contain it and the key check may not catch it (glibc’s own comment says as much). Second, a false positive is possible-but-astronomically-unlikely if user data happens to equal the random tcache_key; glibc handles that by clearing the key and retrying the free rather than aborting, which is why the verify function ends in e->key = 0; __libc_free (e);.

Cross-thread frees move memory, silently. Freeing a chunk on a different thread than allocated it is perfectly legal, but it relocates the memory’s accounting. On free, the chunk goes into the freeing thread’s tcache if it fits and there is room; only on overflow does _int_free_chunk (arena_for_chunk (p), …) send it back to the arena that actually owns it. A classic producer/consumer pipeline — one thread allocating buffers, another consuming and freeing them — therefore drains the producer’s arena into the consumer’s tcache and, once that fills, into the producer’s arena’s bins via the consumer’s lock. The producer’s arena keeps growing because it must keep calling sysmalloc, while the consumer accumulates a large warm cache. This shows up as “the allocating thread’s arena never stops growing” and is one of the few patterns where switching to jemalloc or mimalloc, which handle remote frees explicitly, changes the shape of the problem rather than just the constant factor.

Expecting M_MXFAST or fastbin statistics to do something. On glibc ≥ 2.43, mallopt(M_MXFAST, 0) returns success and changes nothing, glibc.malloc.mxfast is parsed and ignored, mallinfo().smblks and .fsmblks are permanently 0, and malloc_info emits no fast total. Tuning guides, monitoring dashboards and exploit tooling written against 2.42 or earlier will not error — they will silently measure or configure a layer that no longer exists.

Assuming the tcache is per-arena or shared. It is neither. It is per thread, in TLS, and it is not consulted by any other thread even for the same arena. Two threads bound to the same arena have completely independent caches; a chunk sitting in thread A’s tcache is unavailable to thread B even though both could reach the arena it came from. This is why MALLOC_ARENA_MAX=1 does not eliminate per-thread free-memory retention — it only eliminates the arena-level share of it.

See Also

• ptmalloc and glibc malloc — chunk layout, boundary tags, the bin taxonomy and the allocation algorithm the tcache sits on top of; that note owns the bins, this one owns the arena-as-lock and the cache in front of it
• The Buddy Allocator — the kernel-side page allocator that ultimately backs every brk and mmap an arena makes
• brk and mmap as Allocator Backends — main_arena uses brk, secondary arenas use mmap
• Allocator Trade-offs and Selection — the arena-explosion RSS problem and when to swap allocators
• Userspace Memory Allocation — the framing note for the whole section
• jemalloc · tcmalloc · mimalloc — allocators with per-CPU / arena models that handle threading differently
• Linux Userspace Runtime MOC · Linux Memory Management MOC