vault

Goroutine Dumps and Stack Traces

15 min read

Goroutine Dumps and Stack Traces

A goroutine dump is a point-in-time snapshot of every goroutine in the process — its state (running, runnable, blocked on a channel, in a syscall), how long it has been in that state, and its full call stack. Where the execution tracer gives a timeline and pprof gives aggregates, a goroutine dump answers one blunt question: “right now, what is every goroutine doing, and what is each one waiting for?” It is the primary instrument for diagnosing deadlocks, goroutine leaks, and stuck servers. A dump can be triggered three ways — sending SIGQUIT to the process, calling runtime.Stack or runtime/pprof’s goroutine profile from code, or letting a panic print one — and GOTRACEBACK controls how much of it you get (runtime env vars).

Mental Model

Think of a goroutine dump as Go’s equivalent of a JVM thread dump or a debugger’s “list all threads with backtraces”, except the unit is the goroutine (millions possible) not the OS thread (a handful). The runtime can produce this cheaply because it owns the scheduler: it knows every G struct, the atomicstatus field that records the goroutine’s state, the waitreason enum that records why it is parked, and how to unwind every goroutine’s stack. Three paths reach the same printer in runtime/traceback.go:

flowchart TD
  SIG["SIGQUIT (Ctrl-\\) sent\nto the process"] --> FREEZE["freezetheworld():\npreempt all threads"]
  PANIC["unrecovered panic /\nfatal runtime error"] --> FREEZE
  FREEZE --> DUMP["runtime traceback printer\n(traceback.go)"]
  CODE["runtime.Stack(buf, true)\n/ pprof goroutine profile"] --> STW["briefly stop the world\nto walk stacks consistently"]
  STW --> DUMP
  GTB["GOTRACEBACK controls\nverbosity & whether to crash"] -.governs.-> DUMP
  DUMP --> OUT["per-goroutine: id, state,\nwait duration, call stack"]
  OUT --> STDERR["stderr (SIGQUIT / panic)"]
  OUT --> BUF["[]byte / io.Writer (code path)"]

Diagram: three triggers feed one traceback printer. SIGQUIT and a panic first “freeze the world” (preempt every thread so the snapshot is consistent and nothing moves under the printer); the programmatic path instead stops the world briefly. GOTRACEBACK is the dial on verbosity and on whether the process also crashes for a core dump. The insight: a goroutine dump is cheap, always available, needs no special build, and is usually the first thing to grab when a Go process is hung.

Mechanical Walk-through

The signal path — SIGQUIT

When a Go program receives SIGQUIT (signal 3, sent by Ctrl-\ from a terminal or kill -QUIT <pid>), the default runtime signal handler prints a full goroutine dump to standard error and then terminates the process (runtime env vars). Before printing, the runtime “freezes the world” — it preempts every other thread so no goroutine state changes while the printer walks them, which is what makes the snapshot internally consistent. (The GODEBUG=dontfreezetheworld=1 knob disables this preemption, letting goroutines keep running during the dump; it exists for debugging the runtime itself and produces a less consistent dump — leave it off in production.) This is the zero-instrumentation way to inspect a wedged production binary: you do not need pprof wired in, you do not need a debugger attached — kill -QUIT and read stderr. Exactly what gets printed, and whether the process additionally crashes hard for a core dump, is governed by GOTRACEBACK.

The panic path

An unrecovered panic, or a fatal runtime error (nil dereference, slice out-of-bounds, “all goroutines are asleep - deadlock!”, concurrent map write/iteration), routes through the same traceback machinery. The runtime prints the panic value (or fatal error: line), then the current goroutine’s stack, then — depending on GOTRACEBACK — the other goroutines, then exits with status 2 (defer/panic/recover, Go blog). An important distinction: a deadlock detected by the scheduler (“all goroutines are asleep - deadlock!”) is a fatal error, not a panic — it does not unwind through deferred functions and cannot be recovered. The same is true of concurrent map access and stack-overflow: these are fatal errors by design, because the runtime cannot guarantee a sane state to continue from.

The programmatic path

runtime.Stack(buf []byte, all bool) int formats a stack trace into buf and returns the byte count (runtime.Stack docs). If all is true, it formats every other goroutine’s stack after the current one’s — and to do that consistently it briefly stops the world while it walks them, so all=true is not free. runtime/debug.Stack() is the convenience wrapper; notably, the docs state it “calls runtime.Stack with a large enough buffer to capture the entire trace” (runtime/debug docs) — so debug.Stack() does not truncate, unlike a hand-sized runtime.Stack buffer. debug.PrintStack() writes that trace to stderr. The richest programmatic dump is the goroutine profile from runtime/pprof — pprof.Lookup("goroutine").WriteTo(w, 2) with debug level 2 produces a human-readable all-goroutine dump identical in spirit to the SIGQUIT output, and the net/http/pprof blank import exposes it live at /debug/pprof/goroutine?debug=2. There is also a low-level runtime.GoroutineProfile([]StackRecord) that fills caller-supplied records, but the docs say “most clients should use the runtime/pprof package instead” (runtime.GoroutineProfile docs). For a bare count to alert on, runtime.NumGoroutine() returns the number of goroutines that currently exist.

GOTRACEBACK — the verbosity dial

The GOTRACEBACK environment variable controls how much output a crash generates (runtime env vars):

  • none (legacy 0) — omit goroutine stack traces entirely.
  • single — the default. Print the stack of the current goroutine only, eliding functions internal to the runtime, then exit with code 2. (If there is no current goroutine, or the failure is internal to the runtime, it prints all goroutines even at single.)
  • all (legacy 1) — additionally print stacks of all user-created goroutines.
  • system (legacy 2) — like all but also shows runtime stack frames and runtime-internal goroutines.
  • crash — like system but then crashes in an OS-specific manner instead of exiting; on Unix it raises SIGABRT, so the OS produces a core dump.
  • wer — like crash but on Windows does not disable Windows Error Reporting (WER).

runtime/debug.SetTraceback(level string) can raise the verbosity at runtime, taking the same level strings — but by design, “if SetTraceback is called with a level lower than that of the environment variable, the call is ignored” (runtime/debug docs). This is a deliberate security/operability property: a library cannot suppress crash diagnostics that an operator configured the process to emit.

Reading a traceback frame by frame

Each goroutine block in a dump begins with a header line:

goroutine 42 [chan receive, 5 minutes]:

42 is the goroutine id (an internal monotonic counter, not stable across runs). chan receive is the current state. 5 minutes — present only when the goroutine has been parked long enough — is how long it has been in that state. That state-plus-duration is the single most valuable field in the whole dump: a goroutine [chan receive, 10 minutes] in a service that should respond in milliseconds is almost certainly your leak or deadlock. Common states you will read: running, runnable (ready but waiting for a P), chan receive / chan send, select, IO wait (parked in the network poller), sync.Mutex.Lock / semacquire (waiting on a runtime semaphore), sleep, GC assist wait, syscall.

Below the header comes the call stack, innermost (deepest) frame first. Each frame is two lines:

main.worker(0xc0000a4060, 0x2)
	/app/worker.go:38 +0x65

The first line is package.Function(args...); the args are the raw machine words of the arguments at the time of capture — a pointer shows as a hex address, a string shows as two words (data pointer and length), a slice as three. They are not pretty-printed source values, so read them as “is this nil / zero?” hints, not as variables. The second line is file:line +0xNN, where +0xNN is the byte offset of the program counter within the function — useful when one source line compiles to several instructions. Runtime-internal frames are elided unless GOTRACEBACK is system or higher; with system you also see frames like runtime.gopark that reveal the exact park primitive.

With GODEBUG=tracebackancestors=N the dump is extended with the stack at which each goroutine was created, up to N ancestor levels — the runtime threads creation stacks so a leaked goroutine’s dump names the line that spawned it. This is the answer to “who started this leaked goroutine,” which the leaked goroutine’s own stack cannot tell you.

Code / Usage Examples

Triggering a dump from outside the process

# Find the process, then send SIGQUIT — full dump to its stderr, then it exits:
kill -QUIT $(pgrep myserver)
 
# Capture more in the dump: all user goroutines, plus creation stacks:
GOTRACEBACK=all GODEBUG=tracebackancestors=10 ./myserver
 
# On a crash, also produce a core dump for post-mortem with delve/gdb:
GOTRACEBACK=crash ./myserver

kill -QUIT is the no-instrumentation production probe for a hung server — but it is fatal, so it is a last look before death. GOTRACEBACK=all makes the eventual crash dump show every user goroutine, not just the one that crashed — essential for deadlock analysis where the crashing goroutine is rarely the culprit. GOTRACEBACK=crash turns a panic into a SIGABRT so the OS writes a core file you can open in Delve (dlv core ./myserver core) or gdb.

Dumping all goroutines from inside the program

import (
	"os"
	"os/signal"
	"runtime"
	"syscall"
)
 
func installDumpHandler() {
	c := make(chan os.Signal, 1)
	signal.Notify(c, syscall.SIGUSR1)        // 1 pick a non-fatal signal you control
	go func() {
		for range c {
			buf := make([]byte, 1<<20)        // 2 1 MiB scratch buffer
			n := runtime.Stack(buf, true)     // 3 all=true: every goroutine
			os.Stderr.Write(buf[:n])          // 4 write exactly the bytes used
		}
	}()
}

Line 1 wires a non-fatal signal (SIGUSR1) so you can dump goroutines without killing the process — SIGQUIT would terminate it. Line 2 allocates scratch; this is the truncation trap — see Failure Modes. Line 3 runtime.Stack with all=true briefly stops the world and formats every goroutine. Line 4 writes only buf[:n] — the returned count, never the whole buffer (the tail past n is uninitialized). For a self-sizing version with no truncation risk, prefer debug.Stack() (current goroutine) or the pprof goroutine profile (all goroutines).

The richer pprof goroutine profile

import "runtime/pprof"
 
func dumpGoroutines(w io.Writer) error {
	// debug=2 -> human-readable, all goroutines, same shape as SIGQUIT output
	return pprof.Lookup("goroutine").WriteTo(w, 2)
}

With the net/http/pprof blank import this is also reachable live as GET /debug/pprof/goroutine?debug=2. debug=2 gives the full human-readable dump; debug=1 gives a compact aggregated form — stacks grouped, with a count of how many goroutines share each stack — which is far better for spotting a leak: “300000 @ 0x... all blocked at the same chan receive” is one obvious line in the aggregated view, where the debug=2 form would be 300,000 near-identical blocks you would have to scroll. debug=0 (the default for pprof tooling) emits the binary protobuf profile for go tool pprof. Reach for debug=1 first when hunting a leak; debug=2 when you need the exact stack and state of specific goroutines.

Capturing a stack inside a recover

defer func() {
	if r := recover(); r != nil {
		log.Printf("panic recovered: %v\n%s", r, debug.Stack()) // current G's stack
	}
}()

runtime/debug.Stack() returns the current goroutine’s stack as a []byte, sizing the buffer itself so it never truncates — the standard way to log a stack alongside a recovered panic so the incident is debuggable after the fact. Note this captures the stack at the point of recovery, which is inside the deferred function; the original panic site is still visible deeper in that stack because the unwinding has not discarded it yet.

Failure Modes and Common Misunderstandings

SIGQUIT kills the process. It is fatal by design. To dump goroutines on a live process without killing it, use a non-fatal signal handler calling runtime.Stack/debug.Stack (above), or the net/http/pprof endpoint. Reaching for kill -QUIT is a last look before death, not a routine probe.

The default GOTRACEBACK=single hides the culprit. With the default, a crash prints only the crashing goroutine. For deadlocks and leaks the interesting goroutines are the other ones, blocked elsewhere; you must run with GOTRACEBACK=all (or system) to see them. Many incident post-mortems are inconclusive purely because the dump was captured at single and the suspect goroutines were never printed.

Truncated runtime.Stack buffers. runtime.Stack writes only what fits in buf and never grows it. If the return value n == len(buf), the dump is silently cut off — you may be missing exactly the goroutine you need. Size the buffer for your worst-case goroutine count, or sidestep the trap entirely by using debug.Stack() or the pprof goroutine profile, both of which handle sizing themselves.

all=true stops the world. runtime.Stack(buf, true) and the goroutine profile briefly pause every goroutine to walk stacks consistently. Cheap for thousands of goroutines, noticeable for millions — do not call it in a tight loop or on a hot request path. For an always-on count to alert on, runtime.NumGoroutine() is O(1) and pause-free.

Argument values are raw, not pretty. The args in a frame line are raw machine words; a pointer shows as a hex address, a string as two words, a slice as three. They are not the rendered values a debugger would show — useful for spotting a nil or a zero, misleading if read as source-level values. For real variable inspection you need Delve on a live process or a core file.

A deadlock fatal error needs all goroutines asleep. “fatal error: all goroutines are asleep - deadlock!” fires only when every goroutine is blocked with no possibility of progress. A partial deadlock — some goroutines stuck, others spinning or serving traffic — does not trigger it; the runtime cannot detect it. You find a partial deadlock by reading a goroutine dump and seeing long-blocked states ([semacquire, 8 minutes]). See Deadlocks in Go.

You cannot recover a fatal error. Deadlock detection, concurrent map write/iteration, and stack-overflow are fatal errors, not panics; recover does not catch them and the process always exits. Only panic values unwind through defer.

Interpreting [chan receive, N minutes]. The duration is only printed once the goroutine has been parked past an internal threshold, and it reflects time in the current wait, not total goroutine age. Absence of a duration does not mean “just started” — it can mean “below the print threshold.” A growing population of goroutines all showing the same blocked state with climbing durations is the textbook leak signature.

Alternatives and When to Choose Them

  • The Go Execution Tracer — when you need how the process got into this state over time, not just the current snapshot. A dump is a photograph; the tracer is the film. The tracer also records the unblocking goroutine, which a dump cannot show.
  • mutex profiles — for aggregate “where”, and the block/mutex profiles for sampled contention. The goroutine profile is itself a pprof profile and is the bridge between the dump world and the profile world.
  • Delve — an interactive debugger; dlv can attach to a live process or open a core dump and inspect variables, which a text dump cannot. Pair GOTRACEBACK=crash (for the core) with dlv core for full post-mortem.
  • GODEBUG schedtrace — periodic scheduler summaries; good for watching a process drift toward a stuck state rather than inspecting it once.
  • runtime/metrics /sched/goroutines:goroutines — a single number to alert on a climbing goroutine count; the dump is the follow-up that explains which goroutines. As of the Go 1.26 baseline, the runtime also ships an experimental goroutine-leak profile in runtime/pprof (see Goroutine Leaks), which reuses GC reachability to flag goroutines that can never be unblocked — a more automated complement to eyeballing a dump.

Production Notes

The battle-tested production setup is: import net/http/pprof so /debug/pprof/goroutine?debug=1 (aggregated) and ?debug=2 (full) are always reachable on an internal-only port; alert on the metrics /sched/goroutines:goroutines count; and when it climbs, pull the aggregated goroutine profile — a leak shows instantly as N goroutines piled on one stack, almost always blocked at a channel operation or a sync wait. Run production binaries with GOTRACEBACK=all so that if the process crashes, the dump is actually useful; reserve GOTRACEBACK=crash for environments where you have core-dump capture and storage wired up (a core file is the whole address space and can be large). For leak hunts specifically, set GODEBUG=tracebackancestors so each leaked goroutine’s dump names the line that spawned it. Watch the cost: ?debug=2 on a process with hundreds of thousands of goroutines produces a very large response and a stop-the-world pause — prefer ?debug=1 for routine checks. The enduring value of goroutine dumps is that they need no special build and no prior instrumentation beyond an environment variable and a signal — which is exactly why they are the first thing to grab when a Go service hangs at 3 a.m.

See Also

Graph View

Backlinks