Flame Graphs and Stack Sampling

A CPU profiler does not measure where the program spends its time directly; it samples. Many times a second it interrupts the running thread, records the call stack at the interrupted instruction, and resumes. After a few seconds it has tens of thousands of stacks — a statistical picture in which a function that owns 30% of the CPU appears in roughly 30% of the samples. The hard part is two-fold: (1) capturing the stack at all — walking from the interrupted instruction back through every caller requires the kernel and the binaries to be unwindable, via frame pointers, DWARF call-frame information, the CPU’s Last Branch Records (LBR), or, for the kernel itself, the ORC unwinder; and (2) making sense of the pile — folding identical stacks and rendering width proportional to sample count, which is exactly what a flame graph (invented by Brendan Gregg) does. This note covers both halves: how perf record -g samples and unwinds stacks, and how perf script | stackcollapse-perf.pl | flamegraph.pl turns the result into the single most useful CPU-profiling visualization in systems work.

This note assumes the counting/sampling mechanics of Counting vs Sampling Mode and the hardware described in The Performance Monitoring Unit; it focuses specifically on the stack-capture and visualization layer that sits on top of sampling. The perf tool itself, the perf_event_open(2) fd model, the ring buffer, perf_event_paranoid, and sampling-rate throttling are the subject of perf Profiling Tool; the probe mechanisms that other tracers attach to are kprobes and Tracepoints, and the file-based ftrace control surface is The tracefs Filesystem.

Version pin and provenance

Kernel-source claims are read from the v6.12 tag — a maintained long-term-support (LTS) series, pinned so the note stays checkable; mainline is on 7.x as of 2026-09-04, and anything dated later than v6.12 says so explicitly. Everything labelled measured below was produced on the machine this note was written on — Fedora Linux 44, kernel 7.1.8-200.fc44.x86_64, AMD Ryzen AI MAX+ 395 (Zen 5), GCC 16.1.1, glibc from Fedora 44 — on 2026-09-04. That machine has no perf binary and no root, so the profiles below were captured by calling perf_event_open(2) with PERF_SAMPLE_CALLCHAIN directly from a short C program, mapping the ring buffer, draining it, folding the stacks, and rendering the flame graph. Everything you see is real data from a real profiler; none of it is illustrative.

Mental Model — From a Pile of Stacks to a Picture

The whole technique rests on one statistical idea: the proportion of samples in which a code path appears equals the proportion of CPU time that path consumed. If you take 99 samples per second per CPU for 30 seconds across 8 CPUs you have ~24,000 stacks. Sort and merge them, and a function that shows up in 8,000 of them was on-CPU about a third of the time. No instrumentation, no recompilation, no timing each function — just counting.

flowchart TB
  subgraph SAMPLE["Sampling loop (per CPU, ~99 Hz)"]
    OV["PMU counter overflows<br/>(or hrtimer fires)"]
    IP["Record interrupted IP"]
    UNW["Unwind the call stack<br/>(fp / dwarf / lbr / ORC)"]
    BUF["Append stack to<br/>perf ring buffer"]
  end
  OV --> IP --> UNW --> BUF
  BUF --> SCRIPT["perf script<br/>(emit text stacks)"]
  SCRIPT --> FOLD["stackcollapse-perf.pl<br/>(fold identical stacks → 'a;b;c count')"]
  FOLD --> FG["flamegraph.pl<br/>(width = count, y = depth)"]
  FG --> SVG["Interactive SVG<br/>widest frame = hottest path"]

From sample to flame graph. What it shows: the per-sample work (capture IP, unwind the stack) feeds a ring buffer; offline, perf script linearizes it, stackcollapse-perf.pl collapses identical stacks into one line each with a count, and flamegraph.pl lays them out. The insight to take: the expensive and error-prone step is unwinding — everything downstream is bookkeeping. If the stacks are broken, the flame graph is wrong no matter how pretty.

A flame graph reads in two axes. The y-axis is stack depth, counting from zero at the bottom: the bottom row is the thread’s entry point (_start, __libc_start_main, a thread function), and each row up is a frame deeper into the call chain (Gregg, flamegraphs.html). The x-axis is not time — it is “the stack profile population, sorted alphabetically” (Gregg). This is the single most misread property of the visualization. Frames are sorted left-to-right by name purely so that identical adjacent stacks merge into one wide box; the horizontal position carries no chronological meaning. The width of a frame is what matters: “the wider a frame is, the more often it was present in the stacks” — width is directly proportional to sample count and therefore to CPU time. The widest boxes are the hottest paths, and you read a flame graph by scanning for wide plateaus and following them upward to the leaf function that is actually burning the cycles.

Prose about axes is much less convincing than an actual flame graph, so here is one — a real profile of a real program, captured on the measurement machine. The workload is a deliberately shaped call tree: server_loop() alternately calls do_query() (which calls parse_row() → leaf_alloc()/leaf_hash(), and index_row() → leaf_hash()) and do_ingest() (which calls leaf_io() and leaf_alloc()). It was sampled with PERF_COUNT_SW_CPU_CLOCK in frequency mode at 9,999 Hz, with PERF_SAMPLE_CALLCHAIN set, for the 22 ms the workload ran, yielding 223 samples, which fold to 17 unique address-level stacks and, after symbolization, 5 distinct call paths. Mermaid cannot draw proportional-width nested rectangles, so this figure is the fallback: an ASCII box diagram, in which box width is exactly proportional to sample count.

 y-axis = stack depth, growing UPWARD, root at the bottom
 x-axis = SAMPLE POPULATION, sorted alphabetically  --  NOT time, NOT sequence
 ---------------------------------------------------------------------------
 [==========================================][=====]
  leaf_alloc                                  (a)
 [=================================================][====][============][===]
  do_query                                           (b)   leaf_hash     (c)
 [==========================================================================]
  server_loop
 [==========================================================================]
  main
 [==========================================================================]
  __libc_start_call_main
 [==========================================================================]
  __libc_start_main_impl
 [==========================================================================]
  _start
 [==========================================================================]
  all
 ---------------------------------------------------------------------------
 (a) leaf_hash, called via do_query   (b) leaf_alloc, called via server_loop
 (c) leaf_io                          total = 223 samples; 1 column ~ 3 samples

 folded stacks (this IS the input file, one line per unique stack):
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;do_query;leaf_alloc 130   58.3%
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;leaf_hash            41   18.4%
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;do_query;leaf_hash   20    9.0%
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;leaf_alloc           18    8.1%
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;leaf_io              14    6.3%

A real CPU flame graph, measured 2026-09-04, with its own folded-stack input printed beneath it. ASCII rather than mermaid because proportional-width nesting is the one thing mermaid genuinely cannot express. What it shows: the bottom seven rows are one solid bar because every single sample passed through _start → … → server_loop; the profile only starts branching at depth 6. leaf_alloc reached via do_query is 130 of 223 samples (58.3%) and is by far the widest box, so it is the answer. The insight to take — and this is the thing to internalise about the x-axis: leaf_hash appears twice, as box (a) at 9.0% and as the box labelled leaf_hash at 18.4%. Those are not two moments in time; they are two call paths, and they sit where they sit purely because “do_query” sorts before “leaf_hash”. Summed, leaf_hash is 27.4% of the profile — but no single box tells you that, because a flame graph is organised by path, not by function. If you find yourself reading left-to-right and thinking “first it queried, then it hashed”, you have imported an axis that is not there.

That doubling is the single most useful property of the visualization and the source of its most common misreading, so it is worth stating both ways. The useful version: a function that is hot via one caller and cold via another shows up as two boxes of different widths, and you can optimize the right caller. A flat function histogram (perf report --no-children) would report one 27.4% row for leaf_hash and leave you guessing which of its callers to fix. The misleading version: the total cost of a function is not any box’s width, it is the sum of every box with that name — which is exactly what the interactive SVG’s search feature computes and reports as a “cumulative percentage”, and why that feature exists at all.

Colors in the classic flame graph are random, chosen only “to help visually differentiate adjacent frames” (Gregg); they carry no information in the original scheme (later variants overload color — green for one language, yellow for another, or a hue gradient — but plain CPU flame graphs are random-colored).

Stack Sampling — What perf record -g Actually Does

Each sample begins the same way as any sampling-mode profile: a PMU event (typically cycles) is programmed to overflow after a count, and on overflow the kernel takes an interrupt, records the instruction pointer, and — because -g was requested — walks the call stack. The -g flag “enables call-graph (stack chain/backtrace) recording for both kernel space and user space” (perf-record(1)). In the perf_event_attr passed to perf_event_open(2), this sets the PERF_SAMPLE_CALLCHAIN bit in sample_type (perf_event.h), so each sample record carries an array of return addresses from the leaf up to the root.

The interesting question is how the kernel produces that array, and the answer differs for the kernel portion of the stack and the userspace portion. The kernel stack is always unwound in-kernel; the userspace stack is unwound using whichever of three methods you select with --call-graph.

Kernel stacks: the ORC unwinder

When the sample hits in kernel mode, the kernel must walk its own stack. Historically it did this with frame pointers; today the in-tree default on x86-64 is the ORC unwinder — “Oops Rewind Capability” — enabled by CONFIG_UNWINDER_ORC (orc-unwinder.rst).

The exact dates matter and are checkable by fetching the same files at successive tags. arch/x86/kernel/unwind_orc.c and Documentation/x86/orc-unwinder.txt both return HTTP 404 at v4.13 and HTTP 200 at v4.14, so ORC was introduced in 4.14. But arch/x86/Kconfig.debug at v4.14 still reads:

choice
	prompt "Choose kernel unwinder"
	default FRAME_POINTER_UNWINDER

and only at v4.15 does it become

choice
	prompt "Choose kernel unwinder"
	default UNWINDER_ORC if X86_64
	default UNWINDER_FRAME_POINTER if X86_32

which is still the wording at v6.12 and at v7.0. So: ORC landed in 4.14 and became the x86-64 default in 4.15 — a one-release gap that is easy to collapse and that this note previously got wrong. On the measurement machine, /boot/config-7.1.8-200.fc44.x86_64 confirms the distribution follows the default: CONFIG_UNWINDER_ORC=y, # CONFIG_UNWINDER_FRAME_POINTER is not set. ORC works from out-of-band unwind tables generated by objtool at build time: objtool emits “an array of orc_entry structs, and a parallel array of instruction addresses” into the .orc_unwind and .orc_unwind_ip sections respectively, which are sorted and turned into a fast lookup table at boot. To unwind, the kernel looks up the interrupted instruction address in .orc_unwind_ip, reads the matching orc_entry to learn how to recover the previous stack pointer and return address, and repeats.

The point of ORC is that it gets the accuracy of DWARF without DWARF’s cost. The documentation is explicit: ORC “gets rid of the complex DWARF CFI state machine and also gets rid of the tracking of unnecessary registers,” and benchmarks it “about 20x faster than an out-of-tree DWARF unwinder” — closer to 40x after optimization. Crucially, because the metadata is out-of-band, “the ORC unwinder has no effect on text size or runtime performance,” whereas compiling the kernel with frame pointers grows .text “by about 3.2%, resulting in a broad kernel-wide slowdown” (orc-unwinder.rst). This is the same problem the language runtimes solve with their own out-of-band tables — Go’s pclntab, CPython’s frame objects — discussed in Panic Propagation and Stack Unwinding and Traceback Objects and Stack Unwinding. ORC is the kernel’s answer to the identical question; see Stack Unwinding and ORC for the mechanism in detail.

Userspace stacks: frame pointers, DWARF, or LBR

For the userspace half of the stack, perf record --call-graph selects one of three unwinding strategies. The default is fp (frame pointers) (perf-record(1)).

Frame pointers (fp). With frame pointers, every function prologue saves the caller’s base pointer (%rbp on x86-64) and sets %rbp to the current frame, so each frame holds a back-link to the previous one. Unwinding is then a trivial pointer chase: read the saved %rbp, read the return address just above it, repeat. This is cheap enough to do in the interrupt handler with no per-process copying. The catch is that the code must be compiled with frame pointers: with binaries built gcc -fomit-frame-pointer “results may be unreliable” (perf-record(1)), because the compiler reuses %rbp as a general register and the back-link chain is broken. perf walks up to kernel.perf_event_max_stack frames by default (127 on the measurement machine); you can cap it with --call-graph fp,32.

The walk itself is twelve lines of kernel code, and reading them is the fastest way to understand every frame-pointer failure in this note. perf_callchain_user() in arch/x86/events/core.c, v6.12:

fp = (void __user *)regs->bp;             /* (1) start from %rbp, nothing else */
 
perf_callchain_store(entry, regs->ip);    /* (2) frame 0 is the sampled IP     */
...
while (entry->nr < entry->max_stack) {
        if (!valid_user_frame(fp, sizeof(frame)))       break;
        if (__get_user(frame.next_frame,    &fp->next_frame))    break;   /* (3) */
        if (__get_user(frame.return_address,&fp->return_address))break;
        perf_callchain_store(entry, frame.return_address);                /* (4) */
        fp = (void __user *)frame.next_frame;
}

Four things follow directly. (1) The only input is %rbp. If %rbp holds a general-purpose value rather than a frame pointer — which is what -fomit-frame-pointer makes it — the very first dereference reads garbage, valid_user_frame() rejects it, and the walk stops with one frame. (2) Frame 0 comes from regs->ip, not from the chain, so you always get the leaf function even when unwinding fails completely. (3) Every step is a __get_user() — a page-faulting read of userspace memory from NMI context; if the page is not resident the walk simply stops, which is a real and silent source of truncated stacks on a memory-pressured host. (4) The chain yields return addresses, meaning each recovered frame identifies the caller’s call site, one instruction past the call.

Point (1) and point (4) together produce a failure that is not obvious and that the measurement in the next section demonstrates: the walk starts at %rbp, which belongs to the innermost function that established a frame. If the sampled function did not establish one, its own return address is somewhere at an unknown %rsp offset, unreachable — so the immediate caller of a frameless function is skipped. The kernel source names this failure explicitly in the neighbouring uprobe special case, whose comment reads: “If we are called from uprobe handler, and we are indeed at the very entry to user function (which is normally a push %rbp instruction, under assumption of application being compiled with frame pointers), we should read return address from *regs->sp before proceeding to follow frame pointers, otherwise we’ll skip immediate caller as %rbp is not yet setup.” The kernel patches the case where the frame has not been set up yet; it cannot patch the case where the frame is never set up at all.

Here is the memory picture the loop is walking. Mermaid cannot express stack-memory layout with live pointers, so this is an ASCII/box fallback; addresses grow downward in the drawing and stack growth is toward the top, as on x86-64.

 higher addresses
   +---------------------------+
   |  ...  main's locals  ...  |
   +---------------------------+
   |  return address into      |  <-- recovered as frame N
   |  __libc_start_call_main   |
   +---------------------------+
   |  saved %rbp (caller's)    |  <-- main's rbp points HERE
   +---------------------------+  <==== struct stack_frame { next_frame; return_address; }
   |  server_loop's locals     |         the kernel reads exactly these two words
   +---------------------------+
   |  return address into main |  <-- recovered
   +---------------------------+
   |  saved %rbp               |  <-- server_loop's rbp
   +---------------------------+
   |  do_query's locals        |
   +---------------------------+
   |  return addr into         |  <-- recovered
   |  server_loop              |
   +---------------------------+
   |  saved %rbp               |  <== %rbp AT SAMPLE TIME points here
   +---------------------------+      (do_query established the last frame)
   |  return addr into         |  ****  UNREACHABLE from the %rbp chain  ****
   |  do_query  (i.e. the      |  ****  this is the SKIPPED CALLER       ****
   |  identity of parse_row)   |
   +---------------------------+
   |  leaf_alloc: NO frame,    |  <== %rsp
   |  no saved %rbp at all     |      IP comes from regs->ip, so leaf_alloc
   +---------------------------+      itself IS reported -- its caller is not
 lower addresses (stack grows this way)

What frame-pointer unwinding can and cannot reach. What it shows: the %rbp chain is a linked list whose nodes are exactly the functions that executed push %rbp; mov %rsp,%rbp. A function that skipped that prologue contributes no node, and its return address — the one that would name its caller — sits at an %rsp offset only the compiler knows. The insight to take: frame-pointer unwinding does not degrade by “losing accuracy”; it loses specific, identifiable frames, and always the same ones — the callers of frameless functions. The reconstructed stack is a valid subsequence of the real one, never a wrong one, which is why the errors are so easy to miss: nothing looks corrupt, a function has simply vanished from the middle.

The three user-space unwinding methods are compared as a tooling choice — cost per sample, perf.data growth, depth caps, availability — in the call-graph section of perf Profiling Tool; what this note owns is the mechanism above and its measured consequences below.

This compiler default is the source of years of broken profiles. GCC made -fomit-frame-pointer the default for x86-64 to free up %rbp, even though, as critics noted at the time, x86-64 already has a dozen-plus registers so “adding a 17th general purpose register isn’t going to open up a whole new world of compiler optimizations” (Gregg, return-of-the-frame-pointers). The measured cost of keeping frame pointers turned out to be tiny — Gregg’s Netflix production measurements found “the overhead of adding frame pointers to everything (libc and Java) was usually less than 1%, with one exception of 10%.” The cost of omitting them was enormous for observability: profilers hitting an unoptimized frame “[are] usually unable to walk any more frames because that data doesn’t point to the next frame,” producing flame graphs with a sea of [unknown] frames and missing application code. This is why, starting in 2023, the major distributions reversed course: Fedora was the first major distro to re-enable frame pointers (the FESCo proposal, after a 116-post debate), and Ubuntu 24.04 LTS shipped “frame pointers by default,” with Arch Linux following (Gregg). On a modern Fedora/Ubuntu box, --call-graph fp once again produces complete stacks for system libraries.

Partly resolved (2026-09-04)

Fedora: resolved. The Change proposal Changes/fno-omit-frame-pointer states “Targeted release: Fedora Linux 38”, last updated 2023-01-06, FESCo issue #2923 — quoted at length in the distro-reversal section below, along with its benchmark table.

Uncertain Ubuntu: still not confirmed against a Canonical primary source. Verify: that Ubuntu enabled frame pointers by default in 24.04 LTS. Reason: the only source consulted that names the release is Gregg's post — "Ubuntu has also announced frame pointers by default in Ubuntu 24.04 LTS" — which is strong (he states in the same article that he "worked with Canonical to have one prebuilt for Ubuntu", so he is a participant, not a bystander) but is still second-hand for a Canonical release decision. The obvious Canonical sources did not cooperate on 2026-09-04: two discourse.ubuntu.com topic URLs returned HTTP 500, another returned HTTP 404, and wiki.ubuntu.com/NobleNumbat/ReleaseNotes fetched successfully (HTTP 200, correct title) but contains zero occurrences of "frame pointer". To resolve: find the Ubuntu Foundations discourse announcement or the dpkg-buildflags default change in Noble's dpkg source package. uncertain

DWARF (dwarf). When you cannot rebuild the target with frame pointers but it ships DWARF debug info, use --call-graph dwarf. This mode uses “DWARF’s CFI — Call Frame Information” and “works better with optimized binaries” but “requires libunwind or libdw library support” (perf-record(1)). The mechanism is fundamentally different and far heavier: because DWARF unwinding needs the full register state and a chunk of the stack to interpret the CFI, perf “also records (user) stack dump when sampled. Default size of the stack dump is 8192 (bytes)” (perf-record(1)), tunable with --call-graph dwarf,4096. In the sample record this corresponds to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER (perf_event.h): the kernel copies the registers and the top N bytes of the user stack into every sample, and perf unwinds them later, in userspace, by replaying the .eh_frame/CFI program. This is accurate even for -fomit-frame-pointer code, but it is expensive — copying 8 KB per sample inflates the perf.data file dramatically and can perturb the very workload you are measuring — and it silently truncates any stack deeper than the dumped window.

LBR (lbr). The third option uses the CPU’s Last Branch Records, a small hardware ring buffer of the last N taken branches. “LBR mode doesn’t require any compiler options. It will produce call graphs from the hardware LBR registers,” but “it is only available on new Intel platforms, such as Haswell. It can only get user call chain. It doesn’t work with branch stack sampling at the same time” (perf-record(1)). The hardware logs every call/ret pair, so the chain of recent calls reconstructs a stack with no frame pointers and no DWARF and no per-sample stack copy. The fatal limitation is depth: LBR is “limited to 16 or 32 frames” (Gregg) — “most application stacks are deeper,” so LBR is “a last resort.” LBR’s real strength is elsewhere (branch profiling, and reconstructing precise IPs — see Precise Event-Based Sampling).

The Frame-Pointer Problem, Measured

The claim “omitting frame pointers breaks profiles” is repeated everywhere and demonstrated almost nowhere. It is easy to demonstrate. The same source file was compiled twice — once -O2 -g -fno-omit-frame-pointer, once -O2 -g -fomit-frame-pointer, everything else identical — and both binaries were sampled through perf_event_open(2) with PERF_SAMPLE_CALLCHAIN, at 9,999 Hz, on the same machine, minutes apart.

	-fno-omit-frame-pointer	-fomit-frame-pointer
Samples collected	223	227
Workload wall time	0.022 s	0.023 s
Unique address-level stacks	17	11
Stacks after symbolization	5 distinct paths	3 bare function names
Deepest stack	7 frames	1 frame
Samples with stack depth ≤ 2	0 (0%)	227 (100%)
Records lost / throttled	0 / 0	0 / 0

Measured frame-pointer comparison, Fedora 44 / GCC 16.1.1 / Zen 5, 2026-09-04. What it shows: the sample counts are essentially identical — the profiler worked equally well in both runs — but 100% of the -fomit-frame-pointer stacks are one frame deep. The insight to take: this is not degradation, it is total loss of the call-path dimension, and it happens without a single error message, a single dropped sample, or any visible sign that something is wrong. The profile is complete; it just has no y-axis.

Rendered as folded stacks, the difference is stark:

 -fno-omit-frame-pointer  (223 samples, 5 unique paths -- a real flame graph)
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;do_query;leaf_alloc 130
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;leaf_hash            41
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;do_query;leaf_hash   20
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;leaf_alloc           18
   _start;__libc_start_main_impl;__libc_start_call_main;main;server_loop;leaf_io              14

 -fomit-frame-pointer     (227 samples, 3 unique "paths" -- a flat histogram
                           wearing a flame graph's clothes)
   leaf_alloc 144
   leaf_hash   72
   leaf_io     11

The same program, the same profiler, the same sample rate. What it shows: the right-hand profile still correctly identifies leaf_alloc as the hottest function — the leaf is always recovered, because it comes from regs->ip and not from the chain. What it cannot tell you is that 130 of those 144 samples arrived via do_query and only 18 via do_ingest. The insight to take: a broken-stack profile is not obviously broken. It is right about the leaf and silent about everything else, which is the worst possible failure mode, because the answer it gives is the answer you would have gotten from a much cheaper tool and it looks authoritative.

The surprise: -fno-omit-frame-pointer still loses frames

The left-hand profile above is the good one, and it is still missing frames. The program’s real call tree is:

server_loop -> do_query  -> parse_row -> leaf_alloc
                         -> parse_row -> leaf_hash
                         -> index_row -> leaf_hash
            -> do_ingest -> leaf_io
                         -> leaf_alloc

but parse_row, index_row, and do_ingest appear in zero of the 223 stacks. Checking every distinct instruction address in the raw capture confirms it — no address falls inside any of those three functions. Two different mechanisms are at work, and the disassembly shows both.

The first is tail-call elision. index_row() does nothing but call leaf_hash() and return its value, so GCC turned the call into a jump:

0000000000400e40 <index_row>:
  400e40:	jmp    400d30 <leaf_hash>

A jmp pushes no return address and establishes no frame, so index_row leaves no trace on the stack whatsoever. No unwinder — not frame pointers, not DWARF, not ORC — can recover a frame that was never created. This is why the profile shows server_loop;leaf_hash (41 samples) with do_query also missing from that path: leaf_hash is frameless, so the chain starts at do_query’s frame and yields do_query’s own return address, into server_loop.

The second is the frameless-leaf skip from the previous section. parse_row() does set up a frame:

0000000000400d50 <parse_row>:
  400d50:	push   %rbp
  400d51:	mov    %rdi,%rdx
  400d54:	mov    %rsp,%rbp
  400d57:	call   400c10 <leaf_alloc>
  400d5c:	test   %rdx,%rdx        <-- the return address into parse_row

but leaf_alloc(), a genuine leaf function, does not:

0000000000400c10 <leaf_alloc>:
  400c10:	test   %rdi,%rdi        <-- no push %rbp anywhere in the prologue
  400c13:	jle    400c60 <leaf_alloc+0x50>

So when the sample lands inside leaf_alloc, %rbp still points at parse_row’s frame, whose return address is 0x400d9c — inside do_query. The unwinder therefore emits … server_loop; do_query; leaf_alloc, and parse_row is gone. Exactly as the kernel’s own comment predicted: “we’ll skip immediate caller.”

This is not a compiler bug, and GCC documents it in as many words. From gcc(1) (GCC 16.1.1, the compiler used here):

-fomit-frame-pointer — Omit the frame pointer in functions that don’t need one. […] Note that -fno-omit-frame-pointer doesn’t guarantee the frame pointer is used in all functions. Several targets always omit the frame pointer in leaf functions.

Fedora’s build flags include the companion option that is supposed to address exactly this — -mno-omit-leaf-frame-pointer. On the measurement machine, rpm --eval '%{build_cflags}' yields -O2 -g -grecord-gcc-switches -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer …. Rebuilding the test program with both flags and re-sampling produced a byte-for-byte identical prologue for leaf_alloc and a profile with the same 5 paths and the same missing parse_row.

That the extra flag changed nothing invites the obvious question — is -mno-omit-leaf-frame-pointer simply a no-op on x86-64? A three-way control experiment answers it. Two leaf functions were compiled: leaf_spill(), which allocates a 64-element double array and therefore genuinely needs stack space, and leaf_nospill(), which is a register-only loop like the leaf_alloc() in the profile above.

Compiled with	leaf_spill prologue	leaf_nospill prologue	object file
-fno-omit-frame-pointer	push %rbp	(none — no %rbp at all)	ddd5c7a5…
-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer	push %rbp	(none)	ddd5c7a5… — byte-identical
-fno-omit-frame-pointer -momit-leaf-frame-pointer	sub $0x190,%rsp	(none)	14633b49… — different

Control experiment, GCC 16.1.1 on x86-64, 2026-09-04. What it shows: rows 1 and 2 produce byte-identical object files, so -mno-omit-leaf-frame-pointer is already the default once -fno-omit-frame-pointer is given on this target — Fedora passes it for explicitness, not for effect. Row 3 proves the flag is not inert: its positive form really does strip push %rbp from a spilling leaf. And in all three rows, leaf_nospill has no frame pointer at all. The insight to take: this resolves the earlier question cleanly. The lost parse_row frame is not a missing compiler flag. It is GCC honouring its documented licence to “omit the frame pointer in functions that don’t need one” — and a leaf whose locals all live in registers genuinely does not need one, so no flag will put one there. Frame-pointer unwinding therefore has a permanent, structural blind spot at the callers of register-only leaf functions, on every distribution, however carefully it is built.

The practical takeaway is not “frame pointers are useless” — the 223-vs-227 comparison above settles that decisively — but a calibration: frame pointers buy you almost all of the call path, not all of it. Expect tail-called and trivial-leaf frames to be absent even in a well-built profile, and do not conclude that a function is uninvolved because it has no box.

The distro reversal, dated

The reason --call-graph fp works at all on a modern desktop is a policy change with a specific release attached, and the primary source is the Fedora Change proposal itself, Changes/fno-omit-frame-pointer (owners Daan De Meyer, Davide Cavalca and Andrii Nakryiko — all at Meta; last updated 2023-01-06; FESCo issue #2923):

Fedora will add -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer to the default C/C++ compilation flags […] This Change will be implemented for Fedora Linux 38 and the Change authors and FESCo will evaluate whether to retain it by Fedora Linux 40. This Change will be implemented via a %_include_frame_pointers macro to allow packages to trivially opt-out […]

So the version is Fedora Linux 38, not merely “Fedora was first”, and the opt-out mechanism is a named RPM macro rather than a per-package patch. The proposal also carries the benchmark data that made the argument, measured by rebuilding all of Fedora 37 with frame pointers and comparing:

Benchmark	Cost of keeping frame pointers
Compiling the kernel with GCC	2.4% slower
Blender rendering one frame	2% slower on their test case
openssl / botan / zstd	no significant impact
CPython (pyperformance)	1–10%, benchmark-dependent
Redis benchmarks	no significant impact

Fedora’s own frame-pointer benchmark results, from the Change proposal (fetched 2026-09-04). What it shows: the cost is real but small and highly workload-dependent, with CPython the notable outlier. The insight to take: this is the number that settled a 116-post argument, and it is the number to reach for when someone asserts frame pointers are “too expensive”. Netflix’s independent production measurement, cited by Gregg, agrees: “the overhead of adding frame pointers to everything (libc and Java) was usually less than 1%, with one exception of 10%” (Gregg, 2024). Two organizations, different fleets, the same shape of answer — a couple of percent, with a long tail on interpreters. Note also what Fedora deliberately did not do: the kernel itself is still built without frame pointers, because ORC already gives it accurate unwinding at no text-size cost.

That the change actually shipped is visible in the measurements here, in a place I did not plan for. The “after” capture used in the differential section below collected 310 samples, of which exactly one landed inside the test program’s own fprintf() call. (That one sample is excluded from the differential’s 309, since it is an artefact of the harness rather than the workload.) Its stack:

_start;__libc_start_main_impl;__libc_start_call_main;main;fprintf@@GLIBC_2.2.5;
    __vfprintf_internal;__printf_buffer;__printf_fp_l_buffer;__mpn_lshift   1

Four frames deep inside glibc, unwound in the kernel NMI handler by following %rbp. On a pre-Fedora-38 system that stack would have been [unknown] at the first glibc frame. This is the change working, observed by accident, on one sample.

Building a Flame Graph — The Canonical Pipeline

The end-to-end recipe (Gregg, flamegraphs.html) is three commands plus two Perl scripts from the FlameGraph repo:

# 1. Sample all CPUs at 99 Hz for 30s, capturing stacks.
$ perf record -F 99 -a -g -- sleep 30
#         │       │   │  └─ collect for 30 seconds (sleep is the timed dummy)
#         │       │   └──── -g: record call graphs (PERF_SAMPLE_CALLCHAIN)
#         │       └──────── -a: all CPUs, system-wide
#         └──────────────── -F 99: 99 samples/sec (99, not 100, to avoid
#                            lock-stepping with timed kernel activity)
 
# 2. Linearize the binary perf.data into one text stack per sample.
$ perf script > out.perf
 
# 3. Fold identical stacks: each unique stack becomes one line "frame;frame;... count".
$ ./stackcollapse-perf.pl out.perf > out.folded
 
# 4. Render the folded stacks to an interactive SVG.
$ ./flamegraph.pl out.folded > flame.svg

The choice of 99 Hz rather than 100 Hz is deliberate: a round 100 Hz risks sampling in lock-step with timer-driven kernel work that also runs at 100/1000 Hz, biasing the profile; an off-round frequency decorrelates the two. stackcollapse-perf.pl is where the folding happens — it reads perf script’s multi-line-per-sample output and emits exactly one line per unique stack, semicolon-separated from root to leaf, with the occurrence count appended. flamegraph.pl then assigns each frame a width proportional to its count, stacks frames by depth, sorts siblings alphabetically (so identical subtrees merge), and writes a self-contained SVG with embedded JavaScript for click-to-zoom and search-highlight.

A worked reading: suppose the folded file contains

bash;execute_command;...;malloc;_int_malloc 4200
bash;execute_command;...;malloc;tcache_get 90
bash;execute_command;...;read 310

The malloc;_int_malloc path is ~10× wider than read, so the allocator dominates this workload’s CPU — the actionable conclusion is to reduce allocation, not to optimize I/O. You would never have seen that from a flat function list, because malloc is called from dozens of places; the flame graph attributes it to the call path that actually drove it.

The three transformations are worth separating, because each one throws away something different and each one is where a specific class of bug lives.

flowchart TB
    A["perf.data / raw ring-buffer records<br/>one PERF_RECORD_SAMPLE per sample,<br/>callchain = array of return addresses"]
    A -->|"perf script"| B["text: one multi-line block per sample<br/>addresses SYMBOLIZED here<br/>against the DSOs mapped at record time"]
    B -->|"stackcollapse-perf.pl"| C["folded: one line per UNIQUE stack<br/>'root;...;leaf count'<br/>hex offsets (+0xd) stripped first"]
    C -->|"flamegraph.pl"| D["sort the folded lines LEXICALLY<br/>perl: @SortedData = sort @Data"]
    D --> E["walk sorted lines, merging any<br/>shared prefix into one wide box"]
    E --> F["assign x-width proportional to count,<br/>y-position = depth, colour at random"]
    F --> G["self-contained SVG<br/>+ embedded JS: zoom, search,<br/>cumulative-percentage readout"]

    C -.->|"two folded files"| H["difffolded.pl<br/>emits 'stack countA countB'"]
    H -.-> I["flamegraph.pl auto-detects<br/>3 columns -&gt; red/blue differential"]
    B -.->|"symbols missing"| X1["hex frames: stripped binary,<br/>no debuginfo, or JIT with no<br/>/tmp/perf-PID.map"]
    C -.->|"stacks broken"| X2["everything folds to one frame:<br/>-fomit-frame-pointer"]

The flame-graph pipeline, stage by stage. What it shows: symbolization happens in perf script, merging happens in flamegraph.pl and depends entirely on the lexical sort, and the differential path is a fork off the folded stage rather than a separate tool. The insight to take: the alphabetical x-axis is not a stylistic choice made at render time — it is the mechanism that makes merging possible. flamegraph.pl’s header states the reasoning outright: “The ordering on the x-axis has no meaning; since the data is samples, time order of events is not known. The order used sorts function names alphabetically.” Sorting brings identical prefixes adjacent; adjacency is what lets a shared prefix collapse into one box. Take the sort away and you get a flame chart with one thin sliver per sample, which is why flamegraph.pl --flamechart is documented as “sort by time, do not merge stacks” — the two behaviours are the same switch.

Two details in the folding step routinely bite. First, stackcollapse-perf.pl strips hex offsets before comparing: “If memory addresses (+0xd) are present, they are stripped, and resulting identical stacks are collapsed with their counts summed.” Without that, every distinct instruction within a function would be its own stack and nothing would ever merge. Second, perf script’s default output does not always include everything the folder needs; the script’s own header gives the escape hatch — perf script -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace | ... — and notes it “is also required for the --pid or --tid options, so that the output has both the PID and TID.”

You do not strictly need the Perl scripts at all any more. tools/perf/scripts/python/flamegraph.py has been in the kernel tree for several releases and is present at v6.12; its own usage header reads:

#     perf record -a -g -F 99 sleep 60
#     perf script report flamegraph
# Combined:
#     perf script flamegraph -a -F 99 sleep 60

It emits a d3-flame-graph-based HTML page rather than flamegraph.pl’s standalone SVG, and it is written by Andreas Gerstmayr (Red Hat) with the header crediting “Flame Graphs invented by Brendan Gregg” and “Works in tandem with d3-flame-graph by Martin Spier”. For a one-off investigation on a machine where you have perf but no internet, perf script report flamegraph is the shortest path that exists.

Four Layouts Over One Data Set: Flame, Icicle, Chart, Differential

People treat “flame graph”, “icicle graph” and “flame chart” as loose synonyms. They are not. Three of them render the same folded data with different conventions, and the fourth renders different data entirely.

flowchart TB
    F["folded stacks<br/>'a;b;c count', one line per unique stack"]
    F --> FG["FLAME GRAPH<br/>root at BOTTOM, depth grows up<br/>x = population, sorted alphabetically<br/>identical prefixes MERGED"]
    F --> IC["ICICLE GRAPH<br/>root at TOP, depth grows down<br/>x = population, sorted alphabetically<br/>identical prefixes MERGED<br/>flamegraph.pl --inverted"]
    F --> FC["FLAME CHART<br/>x = PASSAGE OF TIME<br/>stacks NOT merged<br/>flamegraph.pl --flamechart"]
    F2["a SECOND folded file"] -.-> DF
    F -.-> DF["DIFFERENTIAL FLAME GRAPH<br/>widths from profile 2<br/>colour = (2 - 1) delta<br/>red = grew, blue = shrank<br/>difffolded.pl | flamegraph.pl"]

    FG --> Q1["good for: shape of a whole profile;<br/>scanning for wide plateaus"]
    IC --> Q2["good for: very deep stacks —<br/>the root is always on screen<br/>without scrolling"]
    FC --> Q3["good for: single-threaded,<br/>time-ordered questions<br/>-- 'what ran when'"]
    DF --> Q4["good for: A/B regressions,<br/>nightly non-regression suites"]

Four renderings, distinguished by what they do to the x-axis. What it shows: flame and icicle are the same picture flipped vertically; the flame chart is the one that genuinely changes the meaning of x; the differential is the one that needs a second input file. The insight to take: the flame-vs-icicle choice is pure ergonomics, and Gregg says so — “Some people prefer it that way. […] for very deep stacks the flame graph layout (with a GUI that starts at the top) often means the initial view may be mostly empty (a few thin interrupt stacks) forcing the developer to scroll down […] I don’t have a strong opinion about this, do whichever you prefer!” Chrome DevTools and Go’s pprof web UI both default to icicle for this reason. The flame-chart choice is not ergonomics: it is a different question. Gregg is blunt that the two get conflated — “Some analysis tools have implemented flame charts and mistakingly called them flame graphs.” If a tool shows you a time axis and calls it a flame graph, it is a flame chart, and you cannot read aggregate cost off it.

Why merging and time-ordering are mutually exclusive is worth one sentence, because it explains the whole design: “Flame graphs reorder the x-axis samples alphabetically, which maximizes frame merging, and better shows the big picture of the profile. Multi-threaded applications can’t be shown sensibly by a single flame chart, whereas they can with a flame graph” (Gregg). Sampling has gaps — you never observed the transitions between samples — so a time axis over sampled data is a partial fiction anyway. Gregg’s account of the invention makes the trade explicit: he started from a time-ordered call-graph visualization, found function tracing too expensive and the dense time-ordered result unreadable, switched to sampling, and “since the function flow is no longer known (sampling has gaps) I ditched time on the x-axis and reordered samples to maximize frame merging. It worked, the final visualization was much more readable.” Flame graphs were released in December 2011; the structure, in his words, “is really an adjacency diagram with an inverted icicle layout.”

A measured differential

Red/blue differential flame graphs answer the question a single profile cannot: what changed. The algorithm is four steps (Gregg, 2014):

1. Take stack profile 1. 2. Take stack profile 2. 3. Generate a flame graph using 2. (This sets the width of all frames using profile 2.) 4. Colorize the flame graph using the “2 - 1” delta. If a frame appeared more times in 2, it is red, less times, it is blue. The saturation is relative to the delta.

flamegraph.pl implements the colour ramp in nine lines — color_scale() drives green and blue down together for a positive delta (giving red) and red and green down together for a negative one (giving blue), with 210 * (max - value) / max setting the saturation, and --negate simply flipping the sign.

To make this concrete, a one-line regression was introduced into the test program — parse_row() was changed to hash the whole row (leaf_hash(n)) instead of a quarter of it (leaf_hash(n/4)) — and both versions were profiled the same way. Before: 223 samples over 22 ms. After: 310 samples over 31 ms, of which the 309 in the workload proper are used here.

 BEFORE (223 samples)
 [=========================================][=====]
  leaf_alloc                                 (a)
 [================================================][====][===========][===]
  do_query                                          (b)   leaf_hash    (c)
 [========================================================================]
  server_loop      ... main ... _start ... all

 AFTER (309 samples) -- widths are profile 2, per the algorithm above
 [================================][====================]
  leaf_alloc                        leaf_hash   <== the regression
 [======================================================][===][========][=]
  do_query                                                (b)  (d)       c
 [========================================================================]
  server_loop      ... main ... _start ... all

 legend: (a),(d) = leaf_hash    (b) = leaf_alloc via server_loop    (c) = leaf_io

 folded diff, difffolded.pl -n semantics: normalise profile 1 to profile 2's total
 (scale factor 309/223 = 1.3857), then delta = after - normalised_before
   before  after   norm.d   share before -> after   colour  stack (leaf-most 2)
      130    144    -36.1    58.3%  ->  46.6%      BLUE    do_query;leaf_alloc
       20     91    +63.3     9.0%  ->  29.4%      RED     do_query;leaf_hash    <== regression
       18     18     -6.9     8.1%  ->   5.8%      blue    server_loop;leaf_alloc
       41     42    -14.8    18.4%  ->  13.6%      BLUE    server_loop;leaf_hash <== SAME FUNCTION
       14     14     -5.4     6.3%  ->   4.5%      blue    server_loop;leaf_io

A real A/B differential, measured 2026-09-04. What it shows: do_query;leaf_hash went from 9.0% to 29.4% of the profile and is strongly red; every other path is blue. The insight to take — and this is the reason differential flame graphs exist: look at the last two red/blue rows. leaf_hash is the same function in both, and it is red on one path and blue on the other. Its absolute count via server_loop barely moved (41 → 42), but because the profile grew, its share fell from 18.4% to 13.6%. A per-function diff would report “leaf_hash: 61 → 133 samples, up 118%” and send you to read leaf_hash, where you would find nothing wrong — the function is unchanged. The differential flame graph says something much more useful: leaf_hash got more expensive only when reached through do_query, which is exactly where the one-line change was. Regressions live in call paths, not in functions.

The -n flag used above is not optional in practice. Gregg’s own warning: “This normalizes the first profile count to match the second. If you don’t do this, and take profiles at different times of day, then all the stack counts will naturally differ due to varied load. Everything will look red if the load increased, or blue if load decreased.” The measurement above is a live example — the “after” run collected 309 samples against 223 simply because it ran 40% longer, so without normalization every single path would have been red. The companion flag -x strips hex addresses, because unresolved frames that differ only by address would otherwise register as pure additions and deletions.

There is one structural blind spot, and Gregg names it: “if code paths vanish completely in the second profile, then there’s nothing to color blue.” The differential is drawn on profile 2’s skeleton, so anything that existed only in profile 1 has no box to colour. The two mitigations are to also render the negated diff (difffolded.pl after before | flamegraph.pl --negate, whose widths come from profile 1 and whose colours show what will happen), and the “elided flame graph” used by Netflix’s internal FlameCommander, which reports “X% elided” and links to a separate graph of the vanished paths. If you automate differentials in CI, generate both directions and show them side by side.

Off-CPU Flame Graphs — The Half Most Profiles Never Show

A CPU flame graph is, by construction, blind to every microsecond a thread spends not running. A request that takes 100 ms and spends 60 ms blocked in a read(2) shows up in a CPU profile as 40 ms of work, and the CPU profile will faithfully tell you where those 40 ms went while saying nothing about the 60 ms that actually dominated the latency. For latency work this is the wrong half of the picture, and Gregg frames it exactly that way: “On-CPU performance issues can [be] solved using CPU Flame Graphs. That leaves off-CPU issues: the time spent by processes and threads when they are not running on-CPU. If this time is spent during an application request, synchronously, then it directly and proportionally affects performance” (Off-CPU Flame Graphs).

The mechanism is a different measurement entirely. Instead of sampling on a timer, you trace the scheduler: when a thread goes off-CPU, record its stack and a timestamp; when it comes back, add the elapsed time to a per-stack total. The pseudocode Gregg gives is the whole idea:

on file read/write function entry:
   start[thread_id] = timestamp
on file read/write function return:
   if !start[thread_id]  return
   delta = timestamp - start[thread_id]
   totaltime[PID, execname, user stack] += delta      # accumulate TIME, not counts
   start[thread_id] = 0

sequenceDiagram
    participant App as application thread
    participant Sched as scheduler
    participant Disk as block layer / lock / futex
    Note over App: ON-CPU: a timer samples the<br/>stack ~99 times per second.<br/>THIS is what a CPU flame graph sees.
    App->>Disk: read(2) — must block
    App->>Sched: schedule() — sched_switch OUT
    Note over Sched: capture stack + timestamp here
    Note over App,Disk: OFF-CPU: the thread is not running.<br/>NO timer sample will ever fire for it.<br/>Invisible to a CPU flame graph.
    Disk-->>Sched: I/O completes, task woken
    Sched->>App: sched_switch IN
    Note over Sched: delta = now minus timestamp<br/>add delta to this stack's total<br/>THIS is the off-CPU flame graph's unit
    Note over App: ON-CPU again

Why a CPU flame graph cannot see blocking. What it shows: timer-based sampling only fires for threads that are on a CPU, so a blocked thread contributes exactly zero samples no matter how long it blocks; off-CPU profiling instead attributes the duration of each off-CPU period to the stack captured at the moment of sched_switch. The insight to take: the two graphs have different units — a CPU flame graph’s box width is a sample count (proportional to CPU time), an off-CPU flame graph’s is nanoseconds of wall time. They are not two views of one measurement and they must not be compared box-for-box. Gregg’s “hot/cold” flame graphs combine them precisely because summing the two axes is otherwise meaningless.

Two practical points, both easy to get wrong. First, the overhead model is inverted. CPU sampling costs a fixed ~99 events per CPU per second regardless of what the machine is doing. Scheduler tracing costs one event per context switch, and Gregg’s warning is unambiguous: “These tracing approaches trace I/O events or scheduler events, which can be very frequent – millions of events per second – and although tracers may only add a tiny amount of overhead to each event, due to the event rate that overhead can add up and become significant.” This is precisely the case for in-kernel aggregation (bpftrace, In-Kernel Aggregation with BPF Maps) rather than shipping every switch to userspace.

Second, off-CPU stacks are the ones most likely to be broken, because they run through the C library’s pthread/futex code. On any distribution predating the frame-pointer reversal, those graphs were, in Gregg’s word, “mostly broken” — which is a large part of why off-CPU analysis had a reputation for being impractical.

Since v6.12 you no longer need a separate tool: perf record --off-cpu is in the tree. Its documented behaviour is worth reading closely because it explains its own limitation:

Enable off-cpu profiling with BPF. The BPF program will collect task scheduling information with (user) stacktrace and save them as sample data of a software event named “offcpu-time”. The sample period will have the time the task slept in nanoseconds. Note that BPF can collect stack traces using frame pointer (“fp”) only, as of now. So the applications built without the frame pointer might see bogus addresses.

Three facts fall out of that. The event is called offcpu-time and shows up as such in perf report. The “sample period” field carries nanoseconds slept, which is what makes the widths meaningful — a consumer that counts records instead of summing periods will produce a graph of how often things blocked rather than for how long, which is a different and usually less useful question. And BPF stack collection is frame-pointer-only: there is no --call-graph dwarf equivalent for off-CPU profiling, so on a -fomit-frame-pointer system this feature does not merely degrade, it produces nothing usable.

Failure Modes — Why a Flame Graph Lies

Broken/truncated userspace stacks. The dominant failure. With --call-graph fp on -fomit-frame-pointer binaries (the pre-2023 distro default), stacks collapse to a single [unknown] frame above the leaf — Gregg’s example shows “15% of samples … in the wrong place and missing frames” (Gregg). Off-CPU flame graphs are hit worst because their blocking paths go through glibc’s pthread/futex code, which was built without frame pointers, leaving those graphs “mostly broken.” Diagnose by looking for a wall of [unknown] frames or implausibly shallow stacks; fix by running on a frame-pointer-enabled distro, switching to --call-graph dwarf, or rebuilding the hot library with -fno-omit-frame-pointer.

Missing symbols. A flame graph full of raw hex addresses or 0x... frames means perf could not resolve addresses to function names — the binary is stripped, the debuginfo package is absent, or (for JITs like the JVM or V8) there is no symbol map. Fix: install matching -debuginfo/-dbgsym packages, or generate a /tmp/perf-<pid>.map for JITted code.

Inlining hides callers. Aggressively inlined functions vanish as distinct frames — the compiler folded them into the caller — so a wide box may actually be several logical functions. Compiling with frame pointers does not undo inlining; only DWARF with inline records (and perf’s --inline) recovers them.

Sampling bias and too few samples. A 99 Hz profile over a 2-second run has only ~200 samples per CPU; thin slivers in the resulting flame graph are statistical noise, not signal. Sample longer or at higher frequency before trusting narrow frames. Also, because sampling only sees on-CPU time, a thread blocked on I/O or a lock is invisible to a CPU flame graph — that requires an off-CPU flame graph built from scheduler tracepoints instead.

The x-axis is not time. Engineers repeatedly try to read left-to-right progression into a flame graph and conclude “first it did A, then B.” It did not; the x-axis is alphabetical. To see time-ordered execution you need a flame chart (time on the x-axis), which is a different, related visualization that deliberately does not merge stacks.

Tail calls and frameless leaves silently delete frames even when frame pointers are on. Measured above: parse_row, index_row and do_ingest appear in zero of 223 stacks from a -fno-omit-frame-pointer build. index_row was tail-called (a jmp, so no frame was ever created and no unwinder can recover it), and parse_row was skipped because its callee leaf_alloc is a frameless leaf. Diagnose by comparing the flame graph against the source’s actual call tree on a function you know is on the path; if a middle function is missing, disassemble it and its callee and look for jmp instead of call, or a prologue with no push %rbp. There is no fix for the tail-call case; the DWARF path recovers the frameless-leaf case.

The function you care about is spread over several boxes. Because a flame graph is organised by call path, a function reached three ways is three boxes, and none of them is its total cost. Measured above: leaf_hash at 9.0% and 18.4% in one profile, totalling 27.4%. Diagnose: if you are trying to answer “how expensive is function F”, a flame graph is the wrong first tool — use the SVG’s search (which reports exactly this cumulative percentage) or perf report --no-children. Use the flame graph for “which path is expensive”.

The x-axis order shifts when symbol names change. Because siblings are sorted lexically by name, renaming a function moves its box, and a version bump that changes a mangled C++ symbol can visibly rearrange a flame graph that is otherwise identical. This is harmless but routinely misread as “the profile changed shape”. A differential flame graph, which draws on one profile’s skeleton, is immune.

A wrong profile can be a correctly-captured profile. Throttling, lost ring-buffer records, and multiplexing all produce data that is complete-looking and biased. None of them is visible in the flame graph. Check the capture for PERF_RECORD_THROTTLE and PERF_RECORD_LOST before trusting a profile from an unfamiliar machine — the mechanism and the measured evidence are in perf Profiling Tool.

When a flame graph looks wrong, the diagnosis is nearly always one of a small number of things, and they are distinguishable by inspection:

flowchart TB
    S["the flame graph looks wrong"] --> Q1{"are frames<br/>raw hex / 0x... /<br/>[unknown]?"}
    Q1 -->|yes| Q2{"is the frame<br/>in a JIT'd region?"}
    Q2 -->|yes| J["no /tmp/perf-PID.map<br/>-- or the map is owned by<br/>the wrong UID, or written<br/>with a container-inner PID"]
    Q2 -->|no| D["binary stripped or<br/>debuginfo missing<br/>-- install -debuginfo / -dbgsym,<br/>or keep the unstripped build"]
    Q1 -->|no| Q3{"is nearly every<br/>stack 1-2 frames<br/>deep?"}
    Q3 -->|yes| FP["frame pointers absent<br/>-- old distro, vendored binary,<br/>or a package that opted out.<br/>Use --call-graph dwarf,<br/>or rebuild the hot library"]
    Q3 -->|no| Q4{"is a function you<br/>KNOW is on the path<br/>missing from the middle?"}
    Q4 -->|yes| TC["tail call (jmp, no frame)<br/>or frameless-leaf caller skip<br/>or inlined away.<br/>Disassemble to tell which;<br/>only inlining is recoverable<br/>via DWARF + perf --inline"]
    Q4 -->|no| Q5{"are the frames you<br/>doubt only a few<br/>samples wide?"}
    Q5 -->|yes| N["statistical noise<br/>-- sample longer or faster;<br/>a 2 s run at 99 Hz is<br/>~200 samples per CPU"]
    Q5 -->|no| Q6{"was the workload<br/>ever BLOCKED?"}
    Q6 -->|yes| OC["you are looking at the<br/>wrong half. CPU flame graphs<br/>cannot see off-CPU time.<br/>Use perf record --off-cpu"]
    Q6 -->|no| TH["check the capture itself:<br/>PERF_RECORD_THROTTLE,<br/>PERF_RECORD_LOST,<br/>event multiplexing"]

A triage order for a suspicious flame graph. What it shows: the checks are ordered by how cheap they are and how often they are the answer — symbolization first (it is visible at a glance), then stack depth, then missing middles, then statistics, then the whole-measurement questions. The insight to take: the first three branches are all capture-time problems that no amount of squinting at the SVG will fix, and all three have a tell you can see without re-running anything. Getting into the habit of checking depth-distribution before reading a profile — “what fraction of my stacks are one frame deep?” — costs one command and catches the single most common failure.

Continuous Profiling in Production

Everything above assumes you decided to profile, ran a capture, and looked at it. Continuous profiling inverts that: sample the whole fleet at a low rate all the time, store the folded stacks with timestamps and labels, and query them later. The argument for it is made most directly in Fedora’s own frame-pointer proposal, which is really a continuous-profiling proposal wearing a compiler-flags hat:

An interesting approach to avoid the above hurdles is to make sure we can do profiling of the entire system directly in production. This approach means we don’t have to recompile our software, don’t need to reproduce the scenario under which the software performs poorly, and gives us a single unified approach to gather profiling data for all the applications we’re interested in. Naturally, this approach depends on being able to profile the entire system efficiently so that there’s no noticeable impact on any running services.

That last sentence is the whole engineering problem, and it is what forces every design decision in the space:

Constraint	Why it bites	What production systems do
Per-sample cost must be ~0	you are paying it on every CPU, forever	frame-pointer unwinding in-kernel; never --call-graph dwarf, whose 8 KiB-per-sample stack copy is unaffordable at fleet scale
Symbolization is expensive	resolving addresses needs the binaries	ship unsymbolized stacks plus build-IDs; symbolize centrally, once per build, not once per sample
You cannot rebuild everything	third-party binaries, vendored libraries	this is exactly why distro-wide frame pointers matter — see the Fedora 38 change above
Data volume	folded stacks from thousands of hosts	aggregate in-kernel where possible (In-Kernel Aggregation with BPF Maps); store folded, not raw
You need to compare over time	“it got slower last Tuesday”	store profiles as time series and diff them — differential flame graphs, generated automatically

The constraints that shape continuous profilers. What it shows: every row is a consequence of “always on, everywhere” rather than “once, on my laptop”. The insight to take: the unwinding-method choice is not a preference at fleet scale, it is forced. A DWARF-based continuous profiler would copy 8 KiB of stack per sample per CPU per second forever; that is the reason the industry’s answer to -fomit-frame-pointer was to change the compiler defaults across two major distributions rather than to make profilers cleverer. The proposal’s own summary of the alternatives is worth keeping: DWARF is too slow and cannot run in-kernel, ORC “can only be used to unwind kernel stack traces; it doesn’t help us with userspace stacks”, LBR gives “only the last X calls, and not the full stack trace”, and shadow stacks are “very early days” — “if we want complete stacks with reasonably low overhead […] frame pointers are currently the best option.”

The remaining piece is a userspace equivalent of ORC: a compact, out-of-band unwind table that a kernel or BPF unwinder can walk cheaply, giving DWARF-grade accuracy without DWARF’s interpreter or its per-sample stack copy. Fedora’s 2023 proposal referred to it by its working name — “CTF Frame — An in progress RFC will add support to binutils to attach a new ctf_frame section to ELF binaries containing unwinding information. This new unwinding format claims to be more compact than eh_frame, faster to unwind, and simpler to implement an unwinder with” — and it has since been renamed SFrame.

Where that stands is checkable by existence-testing kernel paths across tags, and the answer as of 2026-09-04 is: the infrastructure has landed, the SFrame format support has not.

Path	v6.15	v6.16	v6.17	v6.18	v7.0
kernel/unwind/user.c	404	404	200	200	200
kernel/unwind/sframe.c	404	404	404	404	404
include/linux/sframe.h	404	404	404	404	404
Documentation/arch/x86/sframe.rst	404	404	404	404	404

Existence-checking the kernel’s user-space unwinder across release tags, by HTTP status on raw.githubusercontent.com, 2026-09-04. What it shows: a generic deferred user-space unwind framework first appears in v6.17; no SFrame implementation exists in the tree even at v7.0. The insight to take: the framework was built SFrame-shaped and is waiting for it. kernel/unwind/Makefile at v7.0 builds only user.o deferred.o, and include/linux/unwind_user_types.h declares the type enum with a pointed comment — “Unwind types, listed in priority order: lower numbers are attempted first if available” — and exactly one member, UNWIND_USER_TYPE_FP. Its struct unwind_user_frame { s32 cfa_off; s32 ra_off; s32 fp_off; bool use_fp; } is a canonical-frame-address record, which is what an SFrame row decodes to and what a pure frame-pointer walker would never need. So the honest 2026 status is: frame pointers remain the only user-space unwinding method the kernel itself can perform, the plumbing for a second one is merged, and SFrame is not in mainline yet. Plan on frame pointers.

Alternatives and When to Choose Them

The flame graph competes with, and complements, a few other ways of looking at a profile. perf report’s TUI call-graph tree (see perf record and perf report) shows the same data as a collapsible, percentage-annotated tree — better for precise numbers, worse for spotting the shape of the workload at a glance. A flat function histogram (perf report --no-children) is fine when you suspect one hot function and do not care about call paths; useless when a function like malloc is hot via many paths. Differential flame graphs subtract one profile from another to show what changed between two builds or two load levels — invaluable for regression hunting. Icicle graphs are flame graphs drawn top-down (root at top), the convention used by Chrome DevTools and Go’s pprof web UI (pprof and Profiling). For continuous, always-on production profiling, continuous profilers (Parca, Pyroscope, Google-Wide Profiling-style systems) sample fleet-wide and store flame graphs over time; these increasingly use eBPF to unwind in-kernel and frame-pointer or .eh_frame-based unwinders to avoid the per-sample stack copy of DWARF mode.

View	Organised by	Answers	Fails at	Where it lives
Flame graph	call path	“which path costs the most”	“what does function F cost in total”	flamegraph.pl, perf script report flamegraph
Icicle graph	call path (inverted)	same, but the root is always on screen	same	flamegraph.pl --inverted, Chrome DevTools, pprof -http
Flame chart	time	“what ran when, in what order”	multi-threaded workloads; aggregate cost	Chrome DevTools, flamegraph.pl --flamechart
Differential flame graph	call path, coloured by delta	“what changed between A and B”	paths that vanished entirely in B	difffolded.pl | flamegraph.pl
Off-CPU flame graph	call path, width = nanoseconds blocked	“where is the latency, not the CPU”	high-context-switch workloads (overhead)	perf record --off-cpu, BCC/bpftrace
Call-graph tree (perf report)	call path, as text	precise percentages; scripting	seeing the shape at a glance	perf report, perf report --no-children
Flat histogram	function	“what does function F cost in total”	attributing a shared function to its callers	perf report --no-children, most language profilers

Seven ways to look at the same samples. What it shows: the differences reduce to two questions — what is on the x-axis, and what is the unit of a box’s width. The insight to take: the first and last rows are complements, not competitors, and the mistake that wastes the most time is picking one and refusing the other. A flame graph told us leaf_alloc via do_query was 58.3% of the profile; a flat histogram told us leaf_hash was 27.4% in total. Both are true, neither is derivable from the other by looking, and a real investigation uses both within the first minute.

Production Notes

The frame-pointer saga is the defining production lesson here. Netflix re-enabled frame pointers across their fleet years before the distros did, precisely because their CPU and off-CPU flame graphs were unusable without them, and measured the cost as sub-1% in nearly all cases (Gregg). The industry-wide consequence is that, as of the 2023–2024 distro changes, perf record -F 99 -a -g “just works” on stock Fedora and Ubuntu in a way it did not for the preceding decade — a quiet but enormous improvement in Linux observability. When profiling on an older or minimal distro, assume fp is broken and reach for --call-graph dwarf (accepting the overhead) or rebuild the hot path with frame pointers. The longer-term answer, SFrame, is not in mainline as of v7.0 — the evidence and the current state of the kernel’s user-space unwinding framework are in the continuous-profiling section above.

Resolved (2026-09-04)

An earlier revision of this note claimed the in-kernel ORC unwinder had been the x86-64 default “since Linux 4.14”. That was wrong by one release, and the error is instructive. Existence-checking arch/x86/kernel/unwind_orc.c shows ORC was introduced in v4.14 (404 at v4.13, 200 at v4.14), but arch/x86/Kconfig.debug at v4.14 still reads default FRAME_POINTER_UNWINDER; the default UNWINDER_ORC if X86_64 line first appears at v4.15. Introduction and default-ness are separate events, and conflating them is a standard failure mode for “stable since” claims. The corrected account, with the Kconfig text at both tags, is in the ORC section above.

A second, more general production lesson from writing this note: you do not need perf installed, or root, to do real profiling work. The measurement machine had neither. perf_event_open(2) plus mmap(2) plus about 120 lines of C reproduced counting, sampling, call-graph capture, stack folding, and flame-graph rendering — and produced facts the tooling would have hidden, such as the exact five-counter multiplexing ceiling and the frameless-leaf frame skip. When you are debugging why a profiler is giving you a strange answer, dropping to the syscall is often faster than reading the profiler’s source, because the syscall’s failure modes are exactly the ones you are hitting and it tells you about them in errno.

See Also

• Precise Event-Based Sampling — sibling; the hardware fix (PEBS/IBS, LBR) for which instruction a sample points at, complementary to which stack
• Counting vs Sampling Mode — sibling; the counting-vs-sampling distinction this note builds on
• The Performance Monitoring Unit — sibling; the cycles counter whose overflow triggers each stack sample
• perf record and perf report — sibling; the commands that capture and display call-graph data
• Stack Unwinding and ORC — the kernel’s out-of-band unwind mechanism in detail
• Panic Propagation and Stack Unwinding — the Go runtime’s pclntab-based unwinder, the userspace analogue
• Traceback Objects and Stack Unwinding — CPython’s frame-object walk, the Python analogue
• pprof and Profiling — Go’s built-in sampling profiler and icicle/flame UI
• Sampling Profilers and py-spy — out-of-process sampling for Python, same statistical idea
• perf Profiling Tool — the tool and the syscall this note’s captures rest on: perf_event_open(2), the ring buffer, perf_event_paranoid, sample-rate throttling, and the fp/dwarf/lbr cost comparison
• kprobes · Tracepoints — the probe mechanisms an off-CPU or wakeup flame graph attaches to
• The tracefs Filesystem — the ftrace control surface, an alternative route to the same scheduler events
• bpftrace — in-kernel aggregation, the practical answer to off-CPU tracing’s event-rate problem
• In-Kernel Aggregation with BPF Maps — why continuous profilers summarize before leaving the kernel
• Linux Tracing and Observability MOC — parent map