vault

Goroutine Leaks

15 min read

Goroutine Leaks

A goroutine leak is a goroutine that is permanently blocked on a concurrency primitive — a channel send/receive, a mutex, a [[sync.Cond Internals|sync.Cond]], a [[sync.WaitGroup Internals|WaitGroup]] — and can never become unblocked, yet is never garbage-collected because a goroutine is itself a GC root. Each leaked goroutine permanently holds its stack (a minimum 8 KiB) and every object reachable from it; a leak is therefore an unbounded memory leak that no amount of GOGC tuning fixes. Go 1.26 (released February 2026) ships an experimental goroutineleak profile in runtime/pprof that detects a large class of these leaks by reusing the garbage collector’s reachability analysis (Go 1.26 release notes).

Mental Model

A goroutine is never freed by the GC — the runtime only reclaims a goroutine’s resources when its top-level function returns. So a goroutine that blocks forever is, by construction, immortal. The leak is not the goroutine struct itself (small) but everything it pins: its stack and the transitive closure of objects its stack and registers reference.

The Go 1.26 detector’s central insight: a blocked goroutine can only ever wake if some still-running goroutine can reach the primitive it is blocked on. If goroutine G is parked on channel C, and C is unreachable from every runnable goroutine — and from every goroutine those could in turn unblock — then nothing can ever send on C, receive from it, or close it. G is provably dead. This is exactly a reachability question, and the GC already answers reachability questions for a living. The detector is a second mark phase: instead of “is this object reachable, so keep it,” it asks “is this primitive reachable from a runnable goroutine, so the parked goroutine waiting on it could still wake.”

graph LR
    subgraph "Runnable / could-run set"
      R["runnable goroutines"]
      R --> P1["channels & locks<br/>reachable from R"]
    end
    subgraph "Blocked goroutines"
      B1["G1 blocked on chan A"]
      B2["G2 blocked on chan B"]
    end
    P1 -.->|"A is reachable"| B1
    B1 -->|"could wake → its primitives join the set"| P1
    B2 -. "chan B NOT reachable<br/>from any runnable g" .-> LEAK["G2 is LEAKED"]
    caption["Transitive closure: start from runnable goroutines, mark every primitive they reach,<br/>then every goroutine that could wake on those primitives, repeat to a fixpoint.<br/>Any goroutine left blocked on an unreached primitive is reported as leaked."]

Diagram: leak detection is a fixpoint over reachability. The insight — a leak is precisely a parked goroutine whose primitive falls outside the closure of “things a runnable goroutine can ever touch.”

Mechanical Walk-through

Why goroutines leak in the first place

The canonical leak is a producer blocked on an unbuffered channel whose consumer went away. From the Go 1.26 release notes’ own example:

func processWorkItems(ws []workItem) ([]workResult, error) {
    ch := make(chan result)
    for _, w := range ws {
        go func() {
            res, err := processWorkItem(w)
            ch <- result{res, err}      // (1) blocks until someone receives
        }()
    }
    var results []workResult
    for range len(ws) {
        r := <-ch
        if r.err != nil {
            return nil, r.err            // (2) early return — stops receiving
        }
        results = append(results, r.res)
    }
    return results, nil
}

When processWorkItem errors and the function returns early at (2), the remaining worker goroutines are still blocked at (1) trying to send on ch. The local variable ch goes out of scope, the receiver is gone, and nobody else holds a reference to the channel. Those goroutines will send “until the heat death of the universe” — they leak. This is the blocked-channel leak, the most common shape in real code.

Other shapes: a range over a channel that is never closed (the range blocks on the final receive forever); a sync.WaitGroup.Wait() where one Done() is missing; a goroutine selecting on a [[The Context Package|context]] Done() channel where the context is never cancelled; a worker reading from a request channel that the producer abandoned; a deadlocked pair (which is also a deadlock — leaks and deadlocks overlap but are not the same: a deadlock typically halts the program, a leak quietly accumulates while the program runs fine).

A subtler family is the leak under cancellation that ignores cancellation. A goroutine handed a context.Context is only protected if it actually selects on ctx.Done(). A goroutine that does result := <-slowChan with no select will block on slowChan regardless of how many times its context is cancelled — the cancellation signal exists but nothing reads it. The leak is not the absence of a context; it is the goroutine’s blocking operation not being cancellation-aware. Every potentially-blocking operation in a goroutine whose lifetime is meant to be bounded must be wrapped in a select that also watches ctx.Done(), or the context buys nothing.

The reason leaks are so costly is the transitive pinning. A leaked goroutine that captured a 1-byte closure variable does not leak 1 byte — it leaks its entire stack (8 KiB minimum, grown larger if it ever recursed deep), the closure, and every object the closure’s pointers reach. A goroutine that captured a large buffer, an *http.Request, or a database row set pins all of it permanently. Ten thousand leaked goroutines, each modest on its own, is tens of megabytes of stacks plus an unbounded heap closure — and because the memory is spread thin across many goroutines, it does not look like a classic single-site heap leak.

The 1.26 detector — GC-based reachability

The experimental profile is enabled at build time with GOEXPERIMENT=goroutineleakprofile; it is not a runtime flag. With the experiment built in, the profile is exposed two ways: the goroutineleak profile type in runtime/pprof, and the /debug/pprof/goroutineleak HTTP endpoint via net/http/pprof (Go 1.26 release notes).

When the profile is collected, the runtime runs a specialized stop-the-world mark pass — a dedicated GC cycle named goroutineLeakGC (in mgc.go), invoked by runtime/pprof’s writeGoroutineLeak. The release notes describe the criterion verbatim: “A leaked goroutine is a goroutine blocked on some concurrency primitive (channels, sync.Mutex, sync.Cond, etc) that cannot possibly become unblocked. The runtime detects leaked goroutines using the garbage collector: if a goroutine G is blocked on concurrency primitive P, and P is unreachable from any runnable goroutine or any goroutine that those could unblock, then P cannot be unblocked, so goroutine G can never wake up.” (Go 1.26 release notes). Mechanically this is a transitive closure:

  1. Seed. Partition every goroutine into a “could-run” set and a “candidate-leaked” set. The runtime does this in allGsSnapshotSortedForGC (mgcmark.go): a goroutine is put in the candidate-leaked partition only if its status is _Gwaiting and its waitreason falls in the contiguous concurrency-primitive band (the predicate internalBlocked()). Everything else — running, runnable, blocked in a syscall (_Gsyscall), waiting in the network poller (IO wait), sleeping, blocked on internal-runtime semaphores, GC workers — is seeded as a root and treated as live, because any of those can resume on its own and go on to touch a primitive.
  2. Scan the could-run goroutines’ stacks plus all globals, marking every concurrency primitive they reach (the standard GC object-graph walk).
  3. findMaybeRunnableGoroutines then promotes any candidate-leaked goroutine whose primitive got marked: if a blocked goroutine’s channel/lock is now reachable, it could wake, so move it into the could-run set and scan its stack too, marking the primitives it reaches.
  4. Repeat steps 2–3 to a fixpoint (findGoroutineLeaks returns false and resumes marking as long as new could-run goroutines keep appearing).
  5. Every goroutine still in the candidate-leaked partition is waiting on a primitive no could-run goroutine can reach. findGoroutineLeaks transitions each from _Gwaiting to the new _Gleaked status and counts it. These are the leaks reported.

The precise wait-reasons that make a goroutine a candidate (per runtime2.go) are the contiguous block: chan receive (nil chan), chan send (nil chan), select (no cases), select, chan receive, chan send, sync.Cond.Wait, sync.Mutex.Lock, sync.RWMutex.RLock, sync.RWMutex.Lock, and sync.WaitGroup.Wait. A goroutine blocked for any other reason is presumed able to make progress and is never reported — which is exactly why syscall- and netpoll-blocked goroutines are roots, not leaks.

This reuses the GC’s object-graph walk: the same pointer-following machinery that determines liveness for sweeping determines primitive-reachability for leak detection. Crucially, the runtime semaphore (which backs sync.Mutex, sync.RWMutex, and sync.WaitGroup) was retrofitted to support this — sema.go’s queue function sets s.elem to the primitive’s address with the explicit comment “Storing this pointer so that we can trace the semaphore address from the blocked goroutine when checking for goroutine leaks.” Without that stored back-pointer, the detector could not tell which primitive a sema-blocked goroutine was waiting on; with it, findGoroutineLeaks can shade (mark) the exact mutex/waitgroup address (isSyncWait case) or channel address (isChanWait case) a leaked goroutine was parked on.

A deliberate carve-out lives in findGoroutineLeaks: the main goroutine blocked on select{} (wait reason select (no cases), a classic “park forever” idiom) is not reported, even though it is technically a partial deadlock. It is still treated as leaked during the analysis — so that any primitives it holds references to do not spuriously rescue other leaked goroutines — and then quietly reverted to _Gwaiting before the profile is emitted. The new goroutine status itself is defined in runtime2.go as _Gleaked (value 10), with the scan-in-progress variant _Gscanleaked (0x100a).

The release notes credit Vlad Saioc at Uber, with the underlying theory in Saioc et al., “Effective Detection of Goroutine Leaks” (PLDI/companion 2024/2025).

Cost and limitations

The detector adds no runtime overhead unless the profile is actively collected — there is no continuous tracking; the work happens only when you ask for the profile, at which point it costs one extra stop-the-world GC cycle. The Go team states the implementation is “production-ready, and is only considered an experiment for the purposes of collecting feedback on the API, specifically the choice to make it a new profile,” with the explicit aim of enabling it by default in Go 1.27 (Go 1.26 release notes).

The honest limitation, stated verbatim in the release notes: “Because this technique builds on reachability, the runtime may fail to identify leaks caused by blocking on concurrency primitives reachable through global variables or the local variables of runnable goroutines.” If a leaked goroutine’s channel is also referenced by a package-level global, the detector conservatively treats the channel as reachable and will not flag the leak — a false negative. It never produces false positives: anything it reports has been proven, by the reachability closure, to be unable to wake. (The release-notes wording was confirmed against the live page; the seed-set behaviour above — syscall/netpoll goroutines as roots, the internalBlocked() wait-reason band, the _Gleaked status, and the select{} main-goroutine carve-out — was confirmed by reading the 1.26 runtime source directly, resolving the prior uncertainty flag on this section.)

Code Examples

Collecting the experimental profile

//go:build goexperiment.goroutineleakprofile  // requires GOEXPERIMENT build
package main
 
import (
    "os"
    "runtime/pprof"
)
 
func dumpLeaks() {
    p := pprof.Lookup("goroutineleak")   // (1) experimental profile type
    if p == nil {
        return                            // (2) nil if experiment not built in
    }
    f, _ := os.Create("leaks.pb.gz")
    defer f.Close()
    p.WriteTo(f, 0)                       // (3) debug=0 → protobuf for pprof tooling
}
  1. pprof.Lookup("goroutineleak") returns the profile only when the binary was built with GOEXPERIMENT=goroutineleakprofile; otherwise the profile is unregistered.
  2. Guarding against nil keeps the same code compilable and runnable on a non-experiment build.
  3. WriteTo(f, 0) emits the protobuf format consumed by go tool pprof; debug=1 emits a human-readable count-profile of the leaked goroutines’ stacks. Note the special case in writeGoroutineLeak: debug>=2 deliberately falls back to a full goroutine-stack dump that includes non-leaked goroutines too (the same output as the ordinary goroutine profile at debug=2), so use 0 or 1 to see only the leaked set. Either way, the call first triggers runtime_goroutineLeakGC so the leak set is freshly computed.

Build and serve:

GOEXPERIMENT=goroutineleakprofile go build -o app ./...
# with net/http/pprof imported, leaked goroutines appear at:
curl http://localhost:6060/debug/pprof/goroutineleak > leaks.pb.gz
go tool pprof -http=: leaks.pb.gz

Fixing the release-notes leak

The leak in processWorkItems is fixed by giving every worker a guaranteed receiver, or by sizing the channel so sends never block:

func processWorkItems(ws []workItem) ([]workResult, error) {
    ch := make(chan result, len(ws))     // (1) buffered: every send completes
    for _, w := range ws {
        go func() { res, err := processWorkItem(w); ch <- result{res, err} }()
    }
    var results []workResult
    var firstErr error
    for range len(ws) {
        r := <-ch                         // (2) drain ALL ws results, no early return
        if r.err != nil && firstErr == nil {
            firstErr = r.err
        } else if r.err == nil {
            results = append(results, r.res)
        }
    }
    if firstErr != nil { return nil, firstErr }
    return results, nil
}
  1. A buffer of len(ws) guarantees ch <- ... never blocks — every worker can deposit its result and return even if the consumer has stopped caring.
  2. The loop now always performs len(ws) receives, so no worker is ever left stranded mid-send. The alternative — passing a [[The Context Package|context]] and select-ing on ctx.Done() in the worker — lets workers abandon a slow send instead of always completing it.

Failure Modes and Common Misunderstandings

  • “The GC will collect a blocked goroutine.” It will not. A goroutine is a GC root. Only a return (or runtime.Goexit) frees it. A goroutine blocked forever is a permanent root pinning a permanent live set.
  • A leak is not a deadlock. A deadlock where all goroutines are blocked makes the runtime’s [[#the-runtime-deadlock-detector|checkdead]] panic with “all goroutines are asleep - deadlock!” A leak is partial: most of the program runs fine while a subset of goroutines silently accumulates. checkdead never fires for a leak.
  • The 1.26 detector has false negatives, never false positives. If the channel is reachable from a global or a runnable goroutine’s locals, the detector stays silent even if the goroutine is morally leaked. It will never wrongly accuse a wakeable goroutine.
  • runtime.NumGoroutine() monotonically climbing is the symptom. Before 1.26, the standard diagnosis was watching this counter (or /debug/pprof/goroutine) trend upward under steady load. That still works as a symptom; the leak profile tells you which goroutines and why.
  • time.After in a select loop ties up a timer resource until it fires. A classic subtlety: select { case <-time.After(d): ... } in a hot loop creates a fresh timer each iteration whose resources stay live until it fires. As of Go 1.23 (for modules declaring go 1.23.0+) unreferenced Timers and Tickers become GC-eligible immediately, even without Stop() — so this is far less dangerous than before, and the timer channel also became unbuffered in the same release (Go 1.23 release notes). Pre-1.23 code (or anything pinned via GODEBUG=asynctimerchan=1) should still use time.NewTimer and Stop() to avoid the timer lingering until expiry.

Alternatives and When They Apply

Before Go 1.26, the standard tooling was the goroutine profile (pprof.Lookup("goroutine") / /debug/pprof/goroutine?debug=2), which dumps every goroutine’s stack. You then read the dump by hand looking for many goroutines parked at the same line — a manual, heuristic process with no notion of “leaked.” The 1.26 goroutineleak profile is strictly better when available: it isolates provably leaked goroutines instead of dumping all of them.

For tests, go.uber.org/goleak is the widely-used third-party detector: called in TestMain or via defer goleak.VerifyNone(t), it snapshots goroutines and fails the test if extras survive. It is heuristic (it diffs goroutine sets and ignores a known-good allowlist) and test-time only — complementary to, not replaced by, the runtime profile, which works in production.

Structurally, the real “alternative” is not leaking: scoping every goroutine’s lifetime to a [[The Context Package|context]], using [[errgroup and Structured Concurrency|errgroup]] so a parent waits for all children, and following clear channel-ownership rules so the closer is unambiguous. Detection tools find leaks; structured-concurrency patterns prevent them.

Production Notes

In long-lived services, goroutine leaks are one of the most common causes of slow memory growth that looks like a heap leak in a heap profile but is actually goroutine-stack accumulation. A telltale: the heap profile’s inuse_space grows but no single allocation site dominates — because the leaked memory is spread across thousands of goroutine stacks and their pinned closures. Cross-checking runtime.NumGoroutine() against request rate is the quick discriminator.

The 1.26 profile is built to be safe to wire into production behind the standard net/http/pprof endpoint, since collection cost is zero until invoked — though “invoked” still means a full stop-the-world GC, so it is a deliberate diagnostic action, not something to scrape every second. A practical operational pattern: scrape /debug/pprof/goroutineleak on a modest schedule and alert when the leaked-goroutine count crosses a threshold — turning a previously invisible, slow-burn failure into an observable metric. Until it defaults on in (targeted) Go 1.27, this requires a dedicated GOEXPERIMENT build, which is the main adoption friction.

Evidence for the “production-ready” claim is in the test suite itself: the runtime ships src/runtime/testdata/testgoroutineleakprofile/goker/, a corpus of leak reproductions distilled from real historical bugs in CockroachDB, etcd, gRPC-Go, Kubernetes, Docker/Moby, Hugo, and others (files named e.g. cockroach10214.go, etcd6708.go, grpc1275.go). The detector is validated against the shapes that actually bit large Go codebases, not just toy channels — which is the strongest available signal that the reachability technique generalizes beyond textbook examples.

See Also

Graph View

Backlinks