pprof and Profiling

pprof is Go’s built-in, always-available profiling system: a sampling profiler that records where a program spends a scarce resource — CPU time, allocated memory, blocking time, lock-contention time — and emits the data in a portable protocol-buffer format that the go tool pprof analyzer renders as call graphs, flame graphs, and annotated source. Every Go binary ships with the machinery; you collect a profile through one of three front doors — the runtime/pprof package (programmatic), the net/http/pprof package (over HTTP), or go test’s profiling flags — and analyze it with go tool pprof. The defining design choice is sampling: rather than instrument every event, pprof records a statistically representative subset, so profiling adds only modest, bounded overhead and is safe to run in production. This note covers the profile types, the sampling rates that govern each, the collection APIs, and how to read the output.

Mental Model

A profiler answers the question “where does my program spend X?” for some resource X. The naive approach — tracing — records every event, which is exact but expensive and produces overwhelming data. pprof instead samples: it periodically records a snapshot (a stack trace plus a value) and trusts the law of large numbers to make the aggregate accurate. A function that consumes 30% of CPU will, over enough samples, appear in ~30% of CPU samples — without the profiler ever measuring that function directly.

Each profile is therefore a bag of (stack trace, value) pairs. For a CPU profile the value is “this stack was on-CPU when the sampling timer fired”; for a heap profile it is “this stack allocated N bytes / M objects.” go tool pprof aggregates these pairs by function, by line, by call path, and presents the totals. Understanding this — that a profile is a statistical sample of stacks, not a complete trace — explains both its low cost and its blind spots (rare events may be missed; very short profiles are noisy).

flowchart TD
    subgraph Collect
      A["CPU: SIGPROF timer<br/>~100 Hz"] --> S["(stack, value) samples"]
      B["Heap: 1 sample per<br/>~512 KiB allocated"] --> S
      C["Block: 1 sample per<br/>rate ns blocked"] --> S
      D["Mutex: 1/rate of<br/>contention events"] --> S
    end
    S --> P["pprof.proto<br/>(gzipped protobuf)"]
    P --> T["go tool pprof"]
    T --> V1["top / list (text)"]
    T --> V2["web / flame graph"]
    T --> V3["diff / compare two profiles"]

Figure: the pprof pipeline. Four kinds of sampler feed a common (stack, value) representation, serialized to one protobuf format that one analyzer renders many ways. Insight: because everything reduces to sampled stacks, the same go tool pprof works for CPU, heap, block, and mutex data — and even for custom profiles.

The Profile Types

The runtime/pprof package exposes a fixed set of predefined profiles (runtime/pprof docs):

• profile (CPU) — where the program spends on-CPU time. Not a “named profile” but collected via StartCPUProfile/StopCPUProfile. While active, the runtime arms an OS interval timer that delivers SIGPROF; each delivery records the running goroutine’s stack. The default rate is 100 Hz (100 samples/second/CPU), settable via runtime.SetCPUProfileRate(hz).
• heap — a sampling of live-object allocations as of the last completed GC. Shows where memory currently in use was allocated. pprof’s default view is -inuse_space (bytes of live objects).
• allocs — the same underlying data as heap, but defaulting to -alloc_space: total bytes allocated since program start, including memory already garbage-collected. Used to find allocation churn.
• goroutine — stack traces of all current goroutines. The go-to profile for “what are my goroutines doing / why do I have 200,000 of them.”
• block — where goroutines block on synchronization primitives: channel send/receive, sync.Mutex, sync.RWMutex, sync.WaitGroup, sync.Cond, and timer channels. Disabled by default.
• mutex — lock contention: stacks where goroutines had to wait for a contended mutex (sync.Mutex, sync.RWMutex, runtime-internal locks). Disabled by default.
• threadcreate — stacks that led to the creation of new OS threads.
• goroutineleak — new and experimental in Go 1.26: reports leaked goroutines — those “blocked on some concurrency primitive … that cannot possibly become unblocked” (Go 1.26 release notes). It is enabled with GOEXPERIMENT=goroutineleakprofile at build time. See Goroutine Leaks.

Sampling Rates — the Knobs That Govern Cost and Resolution

Every profile is governed by a rate, and getting these right is the difference between a useful profile and either a useless one or a slow program.

Heap. runtime.MemProfileRate controls allocation sampling. Its default is 512 * 1024 (524,288 bytes): “the profiler aims to sample an average of one allocation per MemProfileRate bytes allocated” (runtime docs). Set it to 1 to record every allocation (exact but expensive); set it to 0 to disable heap profiling. The runtime extrapolates from the sample — if it sees one 64-byte allocation out of an expected ~8,000 at the default rate, it scales the reported total accordingly. The docs warn that tools assume the rate is constant over the program’s life, so change it once, as early as possible in main.

Block. Disabled by default; enable with runtime.SetBlockProfileRate(rate int). The semantics: “the profiler aims to sample an average of one blocking event per rate nanoseconds spent blocked” (runtime docs). rate = 1 records every blocking event; rate <= 0 turns it off. A common production setting is something like rate = 10000 (sample roughly one event per 10µs of blocking) to keep overhead low.

Mutex. Disabled by default; enable with runtime.SetMutexProfileFraction(rate int) int. Here the parameter is a fraction: “on average 1/rate events are reported” (runtime docs). rate = 0 turns it off; rate < 0 just reads the current value without changing it; the previous rate is returned. So SetMutexProfileFraction(5) samples one in five contention events.

CPU. runtime.SetCPUProfileRate(hz) sets samples-per-second; the default when you call StartCPUProfile is 100 Hz. Most code should not call this directly — use runtime/pprof or the testing flags.

Collecting Profiles — the Three Front Doors

1. runtime/pprof — Programmatic

For a standalone tool, wire profiling to command-line flags (runtime/pprof docs):

var cpuprofile = flag.String("cpuprofile", "", "write cpu profile to `file`")
var memprofile = flag.String("memprofile", "", "write memory profile to `file`")
 
func main() {
    flag.Parse()
    if *cpuprofile != "" {
        f, _ := os.Create(*cpuprofile)
        defer f.Close()
        if err := pprof.StartCPUProfile(f); err != nil {  // arms the SIGPROF timer
            log.Fatal("could not start CPU profile: ", err)
        }
        defer pprof.StopCPUProfile()                       // flushes the buffer to f
    }
    // ... real work ...
    if *memprofile != "" {
        f, _ := os.Create(*memprofile)
        defer f.Close()
        runtime.GC()                                       // get up-to-date live-object stats
        if err := pprof.Lookup("allocs").WriteTo(f, 0); err != nil {
            log.Fatal("could not write memory profile: ", err)
        }
    }
}

• pprof.StartCPUProfile(w) returns an error if profiling is already on; it begins buffering samples to w.
• defer pprof.StopCPUProfile() stops profiling and only returns after all writes complete — the deferred call guarantees a complete profile even on early return.
• runtime.GC() before a heap/allocs profile forces a collection so the profile reflects the current live set rather than a stale post-GC snapshot.
• pprof.Lookup("allocs").WriteTo(f, 0) — Lookup fetches a named *Profile; WriteTo(w, debug) serializes it. The debug argument: 0 = gzipped protobuf (what go tool pprof wants), 1 = human-readable legacy text with symbolized addresses, 2 (goroutine profile only) = stacks in panic format. WriteHeapProfile(w) is a back-compat shorthand for Lookup("heap").WriteTo(w, 0).

You can also define custom profiles with pprof.NewProfile("mypkg.connections"), then Add(value, skip) and Remove(value) to track domain-specific resources (e.g. open connections) — the same go tool pprof then analyzes them.

2. net/http/pprof — Over HTTP

Importing the package for its side effects registers handlers on the default http.ServeMux (net/http/pprof docs):

import _ "net/http/pprof"   // registers /debug/pprof/* on http.DefaultServeMux
 
func main() {
    go func() {
        // a debug-only server, kept off the public port
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // ... rest of the application ...
}

This exposes, among others: /debug/pprof/ (index), /debug/pprof/profile (CPU), /debug/pprof/heap, /debug/pprof/allocs, /debug/pprof/goroutine, /debug/pprof/block, /debug/pprof/mutex, /debug/pprof/threadcreate, /debug/pprof/trace (execution tracer), /debug/pprof/cmdline, and /debug/pprof/symbol. Two query parameters matter: seconds=N — for profile and trace it sets the collection duration; for the snapshot profiles (heap, goroutine, etc.) it instead returns a delta over that window. debug=N selects the response format (0 binary, >0 plaintext). gc=1 on the heap endpoint forces a GC first.

For a server with its own router, do not rely on the import side effect — wire the exported functions onto your mux explicitly: pprof.Index, pprof.Profile, pprof.Trace, pprof.Symbol, pprof.Cmdline, and pprof.Handler(name) for the named profiles.

3. go test — Benchmarks

The testing package collects profiles around benchmarks:

go test -run=NONE -bench=. -cpuprofile=cpu.prof -memprofile=mem.prof \
        -blockprofile=block.prof -mutexprofile=mutex.prof ./mypkg
go tool pprof cpu.prof

This is the standard way to profile a specific hot path under a controlled, repeatable load.

Analyzing with go tool pprof

go tool pprof consumes a profile file or a live HTTP endpoint:

go tool pprof cpu.prof                                         # local file
go tool pprof http://localhost:6060/debug/pprof/heap           # live heap snapshot
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30  # 30s live CPU profile

At the interactive prompt, top lists the heaviest functions (with flat = time in the function and cum = time in it and its callees); list <regexp> shows annotated source for matching functions; web opens an SVG call graph; peek <regexp> shows callers/callees. The -http=:8080 flag launches the browser UI — and as of Go 1.26 that UI defaults to the flame-graph view, with the old node-graph available under “View → Graph” or /ui/graph (Go 1.26 release notes).

A particularly powerful mode is differential profiling: go tool pprof -diff_base=old.prof new.prof shows exactly what got faster or slower between two versions — the canonical way to confirm an optimization actually worked. pprof also feeds Profile-Guided Optimization: a CPU profile collected from production can be checked into the repo as default.pgo and the compiler uses it to guide inlining.

Failure Modes and Common Misunderstandings

Too-short CPU profiles are noise. At 100 Hz a 1-second profile is only ~100 samples — far too few to be trusted. Profile for tens of seconds under representative load.

Block/mutex profiles silently empty. Both are off by default. If /debug/pprof/block shows nothing, you forgot runtime.SetBlockProfileRate. There is no error — just an empty profile.

heap vs allocs confusion. heap shows what is live now (-inuse_space) — use it for memory leaks. allocs shows total churn since start (-alloc_space) — use it for GC pressure / allocation hotspots. Pointing the wrong one at the wrong question wastes hours.

Mutex profile attributes time at Unlock. Contention is recorded against the holder’s critical section ending — stacks point at Unlock, not at the waiters. A lock held 1s while 5 goroutines wait reports ~5s of contention. This is correct but counterintuitive.

Exposing /debug/pprof/ publicly. The HTTP handlers reveal stack traces, command line, and let anyone trigger an expensive CPU profile — a denial-of-service and information-disclosure risk. Bind the pprof server to localhost, or put it behind auth on a separate port. Never register net/http/pprof on a public mux.

CPU profiling and -buildmode=c-archive/c-shared. StartCPUProfile does not work by default on Unix for code built as a C archive/shared library, because the host owns the SIGPROF handler (runtime/pprof docs).

Collecting profiles concurrently interferes. The diagnostics guide warns that profiles can perturb each other; collect one at a time.

Alternatives and When to Choose Them

• The Go Execution Tracer (go tool trace) — tracing, not sampling: it records every scheduling event, GC phase, and syscall with timestamps. Use it for latency questions (“why did this request take 50ms?”, “why is a goroutine not getting scheduled?”) that a statistical profile cannot answer. Heavier; collect for seconds, not minutes.
• Runtime Metrics Package (runtime/metrics) — aggregate counters and histograms (heap size, GC cycles, goroutine count) rather than stacks. Use for dashboards and alerting; use pprof when a metric tells you that something is wrong and you need where.
• Linux perf / macOS Instruments — OS-level profilers that also see kernel and cgo frames. Useful when the cost is below the Go boundary; pprof only sees Go stacks well.
• Continuous-profiling platforms (Pyroscope, Google Cloud Profiler, Parca, Datadog) — collect pprof profiles fleet-wide, continuously, with low overhead. They consume the same pprof.proto format — Go’s profiling output is their input.

Production Notes

The Go team states plainly that it is safe to profile in production, though the CPU profile in particular adds measurable cost; the diagnostics guide recommends profiling a process for X seconds out of every Y and rotating which process is sampled. The pprof format and tool originated at Google (the pprof blog post) and the analyzer is the open-source github.com/google/pprof — Go vendors it as go tool pprof.

Practical guidance:

• Always run a debug HTTP server with net/http/pprof in production services — but bound to localhost or an internal-only port; the cost of having it idle is nil and the value during an incident is enormous.
• Keep block and mutex profiling off by default and enable them with a low rate only when investigating a contention problem.
• For Go 1.26+, the experimental goroutineleak profile (GOEXPERIMENT=goroutineleakprofile) is described as “production-ready” and adds no overhead unless used — the Go team intends to enable it by default in 1.27 (Go 1.26 release notes). See Goroutine Leaks.
• Capture profiles before and after every performance change and diff them — an optimization unconfirmed by a -diff_base comparison is a guess.

See Also

• The Go Execution Tracer — event tracing, the latency-focused complement to sampling
• Runtime Metrics Package — aggregate runtime counters and histograms
• Goroutine Leaks — what the Go 1.26 goroutineleak profile detects
• Profile-Guided Optimization — feeds a production CPU profile back into the compiler
• GC Tuning and Observability — heap profiles drive GC diagnosis
• Go Garbage Collector — heap/allocs profiles reflect its accounting
• GMP Scheduler Model — block/CPU profiles reflect goroutine scheduling
• Go Internals MOC — §16, Observability and Debugging Tooling