vault

tracemalloc and Memory Profiling

14 min read

tracemalloc and Memory Profiling

tracemalloc is CPython’s built-in memory-allocation tracer: when enabled, every block allocated through Python’s memory allocators is tagged with the Python traceback that requested it, so you can later ask “which lines of my code are holding the most memory, and how did that change between two moments?” It was added by PEP 454 and ships in the standard library (the tracemalloc module docs). Unlike sys.getsizeof (which measures one object) or external tools like Valgrind (which see only the C function PyObject_Malloc, not the Python line behind it), tracemalloc works at the allocator level inside the interpreter and attributes every byte back to Python source. It does this by installing wrapper allocators over the three Python memory domains (verified against Python/tracemalloc.c). This note is the diagnosis tool for the problem described in its sibling, Memory Fragmentation in CPython.

Why a built-in tracer exists

PEP 454’s motivation is precise about the gap it fills. Generic C memory tools fail for Python because “most memory blocks are allocated in the same C function, in PyMem_Malloc() for example” (PEP 454) — Valgrind sees only the allocator entry point, not which Python line of which Python module asked for the bytes. Pure-Python object inspectors (Heapy, Pympler) walk gc.get_objects() and call sys.getsizeof(), but they “struggle when the object type is very common like str or tuple, and it is hard to identify where these objects are instantiated” (PEP 454). The whole point of tracemalloc is provenance: not just “there are 4 million tuples” but “they were all allocated at parser.py:212.”

This makes it the natural counterpart to Memory Fragmentation in CPython: that note explains why Resident Set Size (RSS, the physical memory the OS attributes to the process) stays high; tracemalloc is how you find which allocation site is responsible for the retained blocks.

Mental Model

The mechanism is one idea: interpose on the allocator and stamp each block with a traceback. Python routes object and buffer allocations through a small set of allocator functions (The Memory Allocator Domains). tracemalloc.start() swaps those functions for wrappers that (1) call the original allocator to actually get the memory, and (2) record, in a side table keyed by the returned address, the current Python call stack truncated to a configured depth. tracemalloc.stop() swaps the originals back. Snapshots are copies of that side table; analysis is grouping and diffing the side table by filename/line/traceback.

flowchart LR
    PY["Python code:<br/>x = SomeObject()"] --> OBJM["PyObject_Malloc<br/>(OBJ domain)"]
    subgraph hook["tracemalloc wrapper (installed by start)"]
        W["tracemalloc_alloc"] --> ORIG["original allocator<br/>(pymalloc / malloc)"]
        ORIG --> PTR["block address p"]
        W --> TB["traceback_get():<br/>walk frames up to max_nframe"]
        PTR --> TBL[("traces table<br/>p → {size, traceback}")]
        TB --> TBL
    end
    OBJM --> W
    TBL --> SNAP["take_snapshot() →<br/>Snapshot.statistics / compare_to"]

    style hook fill:#eef
    style TBL fill:#ffd

Diagram: the allocation path with tracemalloc active. The insight: the wrapper sits between Python and the real allocator, so it sees every allocation with the live call stack still on the frame stack — which is exactly the information an after-the-fact heap dump has lost. The cost (and the overhead) is the traceback capture and the per-block table entry on every single allocation.

Mechanical Walk-through: how it hooks the allocators

Python has three allocator domains, each a PyMemAllocatorEx struct (a set of malloc/calloc/realloc/free function pointers plus a context pointer), and a customization API from PEP 445 — PyMem_GetAllocator(domain, &out) and PyMem_SetAllocator(domain, &in) — to read and replace them (C-API memory docs). The three domains are PYMEM_DOMAIN_RAW (the system allocator, usable without an attached thread state), PYMEM_DOMAIN_MEM (PyMem_Malloc, general buffers), and PYMEM_DOMAIN_OBJ (PyObject_Malloc, the object allocator backed by pymalloc). See The Memory Allocator Domains for the full breakdown.

tracemalloc_start (in Python/tracemalloc.c) uses exactly this API. For each of the three domains it saves the existing allocator, then installs a wrapper whose ctx points back at the saved original:

PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &allocators.raw);
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
PyMem_GetAllocator(PYMEM_DOMAIN_MEM, &allocators.mem);
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
PyMem_GetAllocator(PYMEM_DOMAIN_OBJ, &allocators.obj);
PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);

Reading these line by line: each PyMem_GetAllocator(DOMAIN, &allocators.X) copies the current allocator for that domain into tracemalloc’s saved-allocators struct, so the wrapper knows whom to delegate to. Each PyMem_SetAllocator(DOMAIN, &alloc) installs tracemalloc’s own allocator (alloc) in its place. Because all three domains are hooked, tracemalloc sees object allocations (PYMEM_DOMAIN_OBJ, the bulk of Python work), general buffers (PYMEM_DOMAIN_MEM), and raw system allocations (PYMEM_DOMAIN_RAW) alike. The domain field that appears on each Trace (0 for the object/Python domain, other values for C extensions registering their own domains via PyTraceMalloc_Track) records which domain a block came from, which is why Filter/DomainFilter can select on it.

The wrapper functions (tracemalloc_alloc, tracemalloc_realloc, tracemalloc_free, and their GIL-holding/raw variants) do the obvious two-step: call through to the original allocator stored in ctx to actually obtain or release the memory, then update the trace table. On allocation, after the underlying allocator returns a block address p, the wrapper calls traceback_get() to capture the current stack and tracemalloc_add_trace(p, size, traceback) to record it; on free, it removes p from the table; on realloc, it removes the old address and adds the new one. The wrappers also guard against re-entrancy — tracemalloc’s own bookkeeping allocates memory, and that must not be traced, or it would recurse forever.

Traceback capture and the frame-depth limit

The core data the tracer stores per block is small (Python/tracemalloc.c):

typedef struct {
    size_t size;
    traceback_t *traceback;
} trace_t;

size is the byte size of the block; traceback is a pointer into a deduplicated table of tracebacks (many allocations share the same call stack, so storing one copy and pointing at it keeps overhead bounded). The trace table itself is a _Py_hashtable_t keyed by block address.

traceback_get() walks the current Python frame stack from the most recent frame outward, recording (filename, lineno) for each frame, but it stops after tracemalloc_config.max_nframe frames. That limit is the single most important tuning knob. By default, tracemalloc.start() captures only one frame: the docs state “By default, a trace of a memory block only stores the most recent frame: the limit is 1” (tracemalloc docs). You raise it with start(nframe), where nframe ≥ 1, e.g. tracemalloc.start(25) to keep 25 frames. More frames give a more complete call path (so cumulative statistics and deep-stack attribution work), but, per the docs, “Storing more frames increases the memory and CPU overhead of the tracemalloc module.” If a traceback was truncated by the limit, Traceback.total_nframe (added in 3.9) records how many frames there actually were, so you can detect truncation.

The API in practice

The workflow is: start tracing, let the program run, take snapshots, and analyze or diff them.

import tracemalloc
 
tracemalloc.start(25)              # capture up to 25 frames per allocation
 
# ... run the code under investigation ...
snapshot1 = tracemalloc.take_snapshot()
 
# ... run more code (e.g. handle a request) ...
snapshot2 = tracemalloc.take_snapshot()
 
# Filter out noise from the import system and tracemalloc itself.
snapshot2 = snapshot2.filter_traces((
    tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
    tracemalloc.Filter(False, tracemalloc.__file__),
))
 
# What grew between the two snapshots, grouped by source line?
top = snapshot2.compare_to(snapshot1, "lineno")
for stat in top[:10]:
    print(stat)

Line by line: tracemalloc.start(25) installs the hooks and sets the frame limit to 25 (raising it from the default 1 so the diffs show a useful call path). take_snapshot() returns a Snapshot — an immutable copy of the current trace table, excluding blocks allocated before tracing started (tracemalloc cannot retroactively attribute pre-existing memory). filter_traces returns a new filtered Snapshot; each Filter(inclusive, pattern) with inclusive=False excludes matching files (here, the bootstrap importer and tracemalloc’s own allocations, which would otherwise clutter the report). compare_to(old, "lineno") returns a list of StatisticDiff objects sorted by absolute size_diff — each carries size, size_diff, count, count_diff, and the traceback — so the top of the list is the line that grew the most. Grouping keys are 'filename', 'lineno', or 'traceback' (full stack); 'traceback' cannot be combined with cumulative=True (tracemalloc docs).

A second pattern targets a single object — exactly the “which line allocated this leaker?” question:

tb = tracemalloc.get_object_traceback(some_obj)   # Traceback or None
if tb:
    for line in tb.format():
        print(line)

get_object_traceback(obj) returns the Traceback recorded when obj was allocated, or None if the object was allocated before tracing began or is not tracked. This is the feature pure-Python heap walkers cannot replicate — they can find the object but not where it was born.

To watch peak usage without snapshots:

current, peak = tracemalloc.get_traced_memory()   # bytes (current, peak)
tracemalloc.reset_peak()                           # peak := current (3.9+)
overhead = tracemalloc.get_tracemalloc_memory()    # bytes tracemalloc itself uses

get_traced_memory() returns the current and peak total of all traced blocks. reset_peak() (3.9+) sets the recorded peak down to the current value so a subsequent operation’s peak can be measured in isolation. get_tracemalloc_memory() returns how much memory the tracer’s own tables consume — the way you quantify the overhead you are paying.

You can enable tracing before main() even runs, without editing code, via the PYTHONTRACEMALLOC environment variable (PYTHONTRACEMALLOC=25) or the -X tracemalloc=25 command-line option (tracemalloc docs). This matters because blocks allocated before start() are invisible to the tracer, so for “memory was already high at startup” investigations you must enable it from the very beginning.

Overhead and how to reason about it

tracemalloc is built so that when it is off it costs essentially nothing: PEP 454 states it “attaches a traceback to the underlying layer, to memory blocks, and has no overhead when the module is not tracing memory allocations” (PEP 454). When on, every allocation pays for a frame walk (proportional to max_nframe) and a hash-table insert, and every block carries a trace_t entry plus a (deduplicated) traceback. The docs are explicit that “Storing more frames increases the memory and CPU overhead.” The dominant lever is therefore max_nframe: 1 frame is cheap, 25 frames is materially heavier. You measure the actual cost on your workload with get_tracemalloc_memory() rather than guessing.

No primary overhead figure exists (verified 2026-06-01)

Deliberately, this note states no quantitative overhead figure (no “2× slower,” no “N bytes per block”). Confirmed against the primary sources: PEP 454 and the tracemalloc docs describe overhead only qualitatively — “Storing more frames increases the memory and CPU overhead” — and give no benchmark numbers; PEP 454 explicitly recommends measuring with get_tracemalloc_memory() instead. PEP 454 also confirms the “no overhead when the module is not tracing” claim quoted above. To get a number for your workload, benchmark with and without tracing while varying max_nframe.

How it differs from the alternatives

sys.getsizeof(obj) measures the shallow size of one object in bytes — it does not follow references (a list of a million ints reports the list’s own size, not the ints’ total), it has no notion of where the object was allocated, and it gives no aggregate or time-series view. It answers “how big is this object,” whereas tracemalloc answers “what is allocating the most, and how is that changing.” They are complementary: use getsizeof for a single suspect, tracemalloc to find the suspect.

gc.get_objects() / heap walkers (Pympler, objgraph, Heapy) enumerate live objects and can build reference graphs to explain why an object is still reachable — which tracemalloc cannot do (it records allocation site, not the reference chain keeping a block alive). But they cannot tell you the allocation site, and they only see objects the cyclic GC tracks (container objects), missing buffers and raw allocations. For a true reference-cycle leak, reach for gc and a graph tool; for “what code is allocating,” reach for tracemalloc.

External / sampling tools (py-spy, Valgrind, memray) run outside the interpreter or via sampling. Sampling Profilers and py-spy can attach to a running, unmodified process with near-zero overhead — invaluable in production where you cannot afford tracemalloc’s per-allocation cost — but it samples rather than recording every allocation. Valgrind, as PEP 454 notes, sees only the C allocator entry point. tracemalloc’s niche is complete, deterministic, in-process attribution to Python source, at the price of overhead.

Failure Modes and Gotchas

  • Pre-existing memory is invisible. Anything allocated before start() has no trace; take_snapshot() silently omits it and get_object_traceback() returns None. Diagnosing “startup already used 300 MB” requires PYTHONTRACEMALLOC/-X tracemalloc so tracing is active from the first allocation.
  • stop() discards everything. Per the docs, stop() “clears all previously collected traces” — call take_snapshot() (and optionally Snapshot.dump(filename) to persist it) before stopping. clear_traces() clears traces without uninstalling the hooks.
  • The frame limit hides the real caller. With the default nframe=1, every allocation made inside a generic factory (say collections.OrderedDict.__init__) is attributed to that line, not to your code that called it. Raise nframe and group by 'traceback' to see the real path; check Traceback.total_nframe to know whether you truncated.
  • tracemalloc traces itself. Its own tables are allocated through the same allocators; the wrappers guard re-entrancy, but the tables still consume real memory that shows up in process RSS (not in get_traced_memory, which counts only traced user blocks — the tracer’s own usage is get_tracemalloc_memory()). On a memory-constrained box this can tip a process over.
  • It measures Python allocations, not RSS. get_traced_memory() reports bytes Python requested and still holds, which is not the same as RSS. Because of arena retention, RSS can be far higher than traced memory even with no leak — tracemalloc showing flat usage while RSS climbs is itself a strong signal that the problem is fragmentation or non-Python allocations, not a Python-level leak.

Production Notes

A common, genuinely useful idiom is to print the top allocation sites with the source line text, which tracemalloc does not do for you — it gives (filename, lineno), and you fetch the text yourself with linecache. This is essentially the recipe the official docs build up:

import linecache, tracemalloc
 
def display_top(snapshot, key_type="lineno", limit=10):
    snapshot = snapshot.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    top_stats = snapshot.statistics(key_type)
    for index, stat in enumerate(top_stats[:limit], 1):
        frame = stat.traceback[0]          # most recent frame: a Frame(filename, lineno)
        print(f"#{index}: {frame.filename}:{frame.lineno}: "
              f"{stat.size / 1024:.1f} KiB ({stat.count} blocks)")
        line = linecache.getline(frame.filename, frame.lineno).strip()
        if line:
            print(f"    {line}")
    other = top_stats[limit:]
    if other:
        size = sum(s.size for s in other)
        print(f"{len(other)} other: {size / 1024:.1f} KiB")
    total = sum(s.size for s in top_stats)
    print(f"Total allocated: {total / 1024:.1f} KiB")

Walking the key moves: snapshot.statistics("lineno") returns Statistic objects sorted largest-first by size, each with .size (total bytes for that line), .count (number of blocks), and .traceback. stat.traceback[0] is the most-recent Frame (a Traceback is a sequence of Frames ordered oldest-to-most-recent since 3.7, so index 0 is the deepest call); linecache.getline reads the actual source line so the report shows the code, not just a location. The trailing “N other” line and total prevent the classic mistake of staring at the top 10 while 90% of memory hides in the tail.

For long-lived services the pattern is a periodic snapshot diff: take a baseline Snapshot after warm-up, then every few minutes take another and compare_to(baseline, "lineno"); a line whose size_diff climbs monotonically across many diffs is a real leak, whereas one that oscillates is churn. Persist baselines with Snapshot.dump(path) / Snapshot.load(path) so you can diff against a snapshot taken in a previous process run — useful when the leak only manifests after hours and you do not want to hold the baseline in the leaking process’s own memory.

Because tracemalloc is in-process and per-allocation, the usual production posture is to leave it off and turn it on only when investigating, or run it with nframe=1 on a single canary instance. For always-on, low-overhead production monitoring of a running process you cannot restart, Sampling Profilers and py-spy is the better fit; tracemalloc earns its overhead during a focused investigation where complete, deterministic attribution is worth the cost.

See Also

Graph View

Backlinks