vault

GC Tuning and Observability

12 min read

GC Tuning and Observability

Tuning Go’s garbage collector is the disciplined practice of moving one point along a single, fundamental trade-off: GC frequency. Collect more often and the program uses less memory but burns more CPU; collect less often and it uses more memory but burns less CPU (the GC guide). Go exposes exactly two knobs that move this point — GOGC and GOMEMLIMIT — and a rich set of observability tools — GODEBUG=gctrace=1, the runtime/metrics package, runtime.MemStats, CPU and heap profiles, and execution traces — to tell you whether you have a GC problem at all and whether your change helped. The cardinal rule from the guide: measure first. Tuning the GC before profiling has confirmed it is a bottleneck is guessing.

The Cost Model You Are Tuning

Every tuning decision rests on the guide’s GC cost model, which has three axioms. First, the GC consumes only two resources: physical memory and CPU time. Second, the memory cost of GC cycle N is “live heap from cycle N-1 + new heap memory” — the surviving objects plus everything allocated since. Third, the CPU cost of cycle N is a fixed per-cycle cost plus a marginal cost: “average CPU time cost per byte × live heap memory found in cycle N.” Marking cost scales with the live heap, not the total heap or the allocation rate.

The lever that connects these is how often the GC runs. Every cycle pays the fixed cost and scans the live heap; running fewer cycles amortizes both over more allocation, lowering CPU overhead — but each cycle now permits more allocation before it fires, raising peak memory. This is the whole game. GOGC and GOMEMLIMIT are simply two ways to set how much the heap may grow before the next cycle.

Mental Model

flowchart TD
    A["Is GC actually my bottleneck?"] -->|"CPU profile:\nruntime.gcBgMarkWorker,\nmallocgc, gcAssistAlloc"| B{"GC is hot?"}
    B -->|no| Z["Tune elsewhere —\nnot a GC problem"]
    B -->|yes| C{"Plenty of\nspare memory?"}
    C -->|yes| D["Raise GOGC\n(e.g. 100 → 200):\ntrade memory for CPU"]
    C -->|no| E["Set GOMEMLIMIT\nto a safe ceiling;\nkeep GOGC moderate"]
    D --> F["Re-measure with\ngctrace + profile"]
    E --> F
    F --> G["Or: cut allocations —\nescape analysis,\nsync.Pool, fewer pointers"]

Diagram: the tuning decision tree. The insight: the first branch is the most important — most “GC problems” are really allocation-rate problems, and the genuinely best fix (bottom) is usually allocating less, not retuning the collector.

GOGC — Setting the Heap Target

GOGC (default 100) sets the heap-growth target after each cycle. The guide gives the formula:

Target heap memory = Live heap + (Live heap + GC roots) × GOGC / 100

Walking the symbols: Live heap is the bytes the previous cycle found reachable. GC roots are stacks and global variables — the starting points of the mark. GOGC / 100 is the multiplier: at the default 100, the next cycle is scheduled when the heap has grown by roughly 100 % of (live heap + roots), i.e. when it has roughly doubled. The guide’s rule of thumb: “Doubling GOGC will double heap memory overheads and roughly halve GC CPU cost.”

The guide’s worked example makes this concrete. Take a 10 MiB live heap, a 100 MiB/s allocation rate, and a 100 MiB/cpu-second scan rate. At GOGC=100 the heap target is 20 MiB, the program allocates 10 MiB of new garbage per cycle, and GC overhead is ~10 %. At GOGC=50 the target drops to 15 MiB — only 5 MiB of new allocation per cycle, so cycles fire twice as often and overhead doubles to ~20 %. At GOGC=200 the target rises to 30 MiB — 20 MiB of new allocation per cycle, half as many cycles, and overhead halves to ~5 %. More memory, less CPU; the trade-off is linear and predictable.

Set it three ways: the GOGC environment variable (honored by every Go program), runtime/debug.SetGCPercent, or — to disable GC entirely — GOGC=off / SetGCPercent(-1) (only safe for short-lived batch jobs that exit before exhausting memory).

GOMEMLIMIT — The Soft Memory Limit

GOGC alone has a dangerous failure mode: it is relative. If the live heap suddenly grows tenfold, the GOGC-derived target grows with it, and memory can blow past whatever the environment allows — an out-of-memory kill. GOMEMLIMIT, added in Go 1.19, sets an absolute ceiling. The guide defines it as Sys - HeapReleased — total memory mapped from the OS minus memory already handed back. As the heap approaches the limit, the GC runs more aggressively (effectively overriding GOGC downward) to stay under it. Covered in depth in GOMEMLIMIT and the Soft Memory Limit; the essentials for tuning:

It is a soft limit — “no guarantees under all circumstances”, only “some reasonable amount of effort.” To prevent a death spiral where the program does nothing but GC trying to honor an impossible limit, the runtime caps GC CPU at roughly “50 % over a 2 × GOMAXPROCS CPU-second window.” The guide’s stark warning: “In many cases, an indefinite stall is worse than an out-of-memory condition, which tends to result in a much faster failure.” Recommended for: containers with a fixed memory budget. Recommended against: CLI tools or anything whose memory scales with unknown input. Always “leave an additional 5–10 % of headroom to account for memory sources the Go runtime is unaware of” — cgo allocations, thread stacks, mmap’d files.

The robust container recipe is GOMEMLIMIT as a safety net + a moderate GOGC: GOGC governs steady-state behavior; GOMEMLIMIT only engages during a transient spike to prevent the OOM kill.

Observability: gctrace

The fastest way to see the GC working is GODEBUG=gctrace=1, which prints one line per cycle to stderr. The format (runtime package docs):

gc 14 @5.292s 2%: 0.018+1.7+0.024 ms clock, 0.14+0.42/1.5/0.18+0.19 ms cpu, 12->13->8 MB, 16 MB goal, 8 P

Field by field: gc 14 — cycle number (monotonically increasing). @5.292s — seconds since program start. 2% — cumulative fraction of total program wall time spent in GC since startup; this is the headline number — a steadily climbing percentage means GC is an increasing share of your CPU budget. 0.018+1.7+0.024 ms clock — wall-clock time for the three sub-phases: STW sweep-termination, concurrent mark/scan, STW mark-termination; the two outer numbers are your stop-the-world pause durations, and they should be sub-millisecond. 0.14+0.42/1.5/0.18+0.19 ms cpu — CPU time across phases and worker classes. 12->13->8 MB — heap size before GC start → at GC end → live heap after sweep; the third number is the live heap that feeds the next cycle’s target. 16 MB goal — the heap target the pacer aimed for (the GOGC formula output). 8 P — number of processors. A healthy trace shows the % flat, pauses sub-millisecond, and the live figure stable; a rising % or a live figure that grows unbounded points at a leak or an undersized GOGC. GODEBUG=gcpacertrace=1 adds deep pacer internals — see GC Pacing and GOGC.

Observability: Metrics, MemStats, and Profiles

runtime/metrics is the modern, structured interface — prefer it over MemStats. Key GC metrics: /gc/cycles/total:gc-cycles, /gc/cycles/forced:gc-cycles (cycles triggered by explicit runtime.GC() calls — a non-zero forced count in steady state is a smell), /gc/pauses:seconds (a histogram of STW pause durations — far better than a single average), /gc/heap/live:bytes, /gc/heap/goal:bytes, and /memory/classes/total:bytes minus /memory/classes/heap/released:bytes (the same quantity GOMEMLIMIT bounds). Metrics are versioned and stable; sample them on a timer and export to your monitoring system.

runtime.MemStats (via runtime.ReadMemStats) is the older API — note that ReadMemStats triggers a stop-the-world, so do not poll it tightly. Useful fields: Alloc/HeapInuse (current heap), TotalAlloc (cumulative bytes ever allocated — its rate of change is your allocation rate), Sys (total from OS), HeapReleased, NumGC, GCCPUFraction (the same fraction gctrace prints), PauseNs.

CPU profiles (pprof and Profiling) tell you whether GC is the bottleneck. View with top -cum and watch three functions the guide names: runtime.gcBgMarkWorker — the background mark workers; high cumulative time means heavy marking. runtime.mallocgc — the allocation entrypoint; “more than 15 % cumulative indicates high allocation volume.” runtime.gcAssistAlloc — goroutines forced to assist the GC because they are allocating faster than the background workers can mark; “more than 5 % cumulative indicates allocation outpacing the mark phase” — a clear signal to either raise GOGC or, better, allocate less.

Heap profiles find where the allocations come from. The guide singles out the alloc_space view (total bytes allocated since start) as “most useful for GC tuning” because it correlates with the allocation rate that drives cycle frequency; inuse_space shows the live footprint instead. Execution traces (The Go Execution Tracer) place GC events on a timeline so you can see assist stalls and pause clustering; Go 1.25’s runtime/trace.FlightRecorder keeps a rolling in-memory trace so you can snapshot the seconds around a rare latency spike.

Code: Reading Metrics and Tuning Programmatically

import (
    "runtime/debug"
    "runtime/metrics"
)
 
func init() {
    debug.SetGCPercent(200)              // (1) trade memory for CPU
    debug.SetMemoryLimit(3 << 30)        // (2) 3 GiB soft ceiling
}
 
func gcStats() (cycles uint64, lastPauseHi float64) {
    samples := []metrics.Sample{
        {Name: "/gc/cycles/total:gc-cycles"}, // (3)
        {Name: "/gc/pauses:seconds"},
    }
    metrics.Read(samples)                            // (4)
    cycles = samples[0].Value.Uint64()
    h := samples[1].Value.Float64Histogram()         // (5)
    lastPauseHi = h.Buckets[len(h.Buckets)-1]
    return
}

(1) SetGCPercent(200) doubles the heap target — fewer cycles, ~half the GC CPU, ~double the heap overhead; the programmatic equivalent of GOGC=200. (2) SetMemoryLimit installs the soft ceiling in bytes; combined with a high GOGC, the limit only bites during spikes. (3) Metric names are typed strings from the runtime/metrics catalog — a stable, versioned API. (4) metrics.Read fills the samples in one call; cheaper than ReadMemStats and not a stop-the-world. (5) Pause data is a histogram, not a scalar — export percentiles, never a misleading mean.

Failure Modes and Common Misunderstandings

Tuning before measuring. The most common error. The guide’s whole methodology is profile → confirm GC is hot → then tune. Bumping GOGC because a dashboard “looks high” without a CPU profile is guessing.

GOMEMLIMIT set too tight → GC death spiral. If the limit is below what the workload genuinely needs, the GC runs constantly trying to honor it; the 50 %-CPU cap prevents a total livelock but the program still crawls. The guide’s warning bears repeating: an indefinite stall is often worse than a clean OOM. Size the limit to real memory minus 5–10 % headroom.

Calling runtime.GC() in steady state. A forced collection (visible as a rising /gc/cycles/forced and in gctrace) pre-empts the pacer and usually raises total GC CPU. Reserve explicit GC() for tests and one-off pre-benchmark cleanup.

Mistaking RSS for live heap. Resident memory includes swept-but-unreturned heap, idle spans, stacks, and runtime metadata. A high RSS with a small live figure in gctrace is the Page Allocator holding pages for reuse, not a leak. (Relevant on Go 1.26: the Green Tea Garbage Collector reportedly shifts baseline RSS — re-baseline after upgrading.)

Polling ReadMemStats in a hot loop. It stops the world. Use runtime/metrics for frequent sampling.

Tuning the GC when the real fix is allocating less. A high mallocgc or gcAssistAlloc share means the program churns memory. Raising GOGC hides the symptom; the durable fix is fewer heap allocations — see Alternatives.

Alternatives and When to Choose Them

Before tuning the collector, consider reducing the work it has to do — usually the better outcome because it cuts both CPU and memory at once. Escape Analysis (go build -gcflags=-m=3) reveals which values escape to the heap; keeping them on the stack removes them from the GC’s purview entirely. sync.Pool recycles short-lived objects so the same buffers serve many requests. Reducing pointers — index-based structures instead of pointer-linked ones — shrinks the object graph the mark phase must trace. Grouping pointer fields at the start of a struct lets the scanner stop early, since the GC stops scanning at the last pointer. On Linux, Transparent Huge Pages tuning (the guide’s /sys/kernel/mm/transparent_hugepage/ settings) gives ~10 % throughput on heaps ≥1 GiB. Reach for GOGC/GOMEMLIMIT when the allocation pattern is already lean and you are deliberately buying CPU with memory or vice versa.

Production Notes

The guide’s distilled advice for containerized services: set GOMEMLIMIT to the container memory limit minus 5–10 %, keep GOGC at a moderate value (default 100, or higher if you have headroom and want less CPU), and let the limit act purely as a spike guard. Wire runtime/metrics into your monitoring stack — alert on a rising /gc/cycles/total rate, a climbing GC CPU fraction, and high-percentile /gc/pauses — and keep gctrace available to flip on for ad-hoc diagnosis. When latency spikes are rare and hard to catch, the Go 1.25 FlightRecorder lets you snapshot the trace around the event. Above all: confirm with a CPU profile that gcBgMarkWorker/mallocgc/gcAssistAlloc are genuinely hot before touching a knob — most apparent GC problems are allocation-rate problems wearing a GC costume.

See Also

Graph View

Backlinks