vault

GODEBUG and Runtime Configuration Knobs

11 min read

GODEBUG and Runtime Configuration Knobs

Every Go binary embeds a small runtime that can be steered from the outside without recompiling. The GODEBUG environment variable is the central knob: a comma-separated list of name=value pairs that toggles internal diagnostics (gctrace, schedtrace, inittrace) and acts as Go’s official mechanism for backward compatibility — when a release changes a behavior, it ships a GODEBUG setting that restores the old one (GODEBUG compatibility doc). Alongside it sit the dedicated runtime variables GOGC, GOMEMLIMIT, GOMAXPROCS, and GOTRACEBACK. Together these knobs let an operator tune garbage collection, scheduling, and crash output, and let a developer pin behavior across a toolchain upgrade — all governed by the go line in go.mod.

Mental Model

GODEBUG plays two distinct roles that are easy to conflate:

flowchart LR
    subgraph diag["Role 1 — Diagnostics"]
        D1[gctrace=1]
        D2[schedtrace=1000]
        D3[inittrace=1]
        D4[scavtrace=1]
    end
    subgraph compat["Role 2 — Compatibility"]
        C1[panicnil=1]
        C2[asynctimerchan=1]
        C3[httpmuxgo121=0]
    end
    diag -->|emit lines to stderr<br/>no behavior change| OUT[observe]
    compat -->|restore old behavior<br/>after a release changed it| OLD[revert]
    GOMOD["go line in go.mod<br/>+ //go:debug directives"] -.->|sets defaults for| compat

Diagram: the two faces of GODEBUG. The insight is that diagnostic settings only print — they never alter program semantics — while compatibility settings deliberately change behavior back to an older release’s. The go.mod go line and //go:debug directives feed the compatibility half; the environment variable can override either.

The other mental anchor: the dedicated variables (GOGC, GOMEMLIMIT, GOMAXPROCS, GOTRACEBACK) are tuning knobs that an operator legitimately sets per-deployment, whereas GODEBUG compatibility settings are transition knobs you set to buy time and then remove.

Mechanical Walk-through

The shape of GODEBUG

GODEBUG is read once at process startup. Its value is a comma-separated list of name=value pairs: GODEBUG=gctrace=1,schedtrace=1000 (runtime env-vars doc). Unknown names are ignored by the runtime (so an old binary tolerates a setting added in a newer release). Most values are integers, often 0/1 flags; a few take other forms (fips140="on").

Diagnostic settings — observation, not mutation

These settings make the runtime print internal state. They do not change what the program computes. The important ones:

  • gctrace=1 — emit one line per garbage collection to stderr: GC number, time since start, percent of CPU in GC, the mark/sweep wall-clock and CPU time breakdown, heap sizes before→after→goal, and processor count (runtime doc). The single most useful GC diagnostic; see GC Tuning and Observability.
  • schedtrace=X — every X milliseconds, print a one-line scheduler summary. With scheddetail=1, print a verbose multi-line dump of every P, M, and G. See Scheduler Tracing and GODEBUG schedtrace.
  • inittrace=1 — one line per package init, with wall time, bytes allocated, and allocation count — used to find slow package initialization.
  • scavtrace=1 — roughly one line per GC cycle about the scavenger returning memory to the OS.
  • gcpacertrace=1 — internal state of the GC pacer; see GC Pacing and GOGC.
  • asyncpreemptoff=1 — disables signal-based asynchronous goroutine preemption; a behavior knob used to debug preemption-related GC stalls. See Goroutine Preemption.
  • cgocheck — cgocheck=0 disables the runtime checks that catch a Go pointer being passed incorrectly to C; the default cheap check is on. See cgo Performance and Pitfalls.
  • madvdontneed — controls whether the runtime returns memory to the kernel with MADV_DONTNEED (RSS drops promptly) or MADV_FREE (cheaper, RSS drops lazily under pressure). See Memory Mapping and OS Interaction.

Compatibility settings — Go’s backward-compatibility mechanism

This is the deeper role. The Go 1 compatibility promise (go1compat) says old code keeps working — but sometimes a behavior must change (a bug fix, a security fix). The reconciliation: ship the change as the new default, and ship a GODEBUG setting that restores the old behavior, maintained for at least two years (four releases) (GODEBUG doc). Examples: panicnil=1 restores pre-1.21 panic(nil) behavior; asynctimerchan=1 restores pre-1.23 buffered timer channels; httpmuxgo121=0 restores the pre-1.22 http.ServeMux matching.

How defaults are chosen — the three-source resolution

The default value of every compatibility GODEBUG setting is computed from three sources, in order of increasing precedence (GODEBUG doc):

  1. The toolchain’s own defaults — the values for the Go version that compiled the binary.
  2. The go line in go.mod — if the work module declares an older version, the toolchain amends its defaults to match that older version as closely as possible. The doc’s example: a Go 1.21 toolchain compiling a module whose go.mod says go 1.20 defaults panicnil=1, matching Go 1.20. Because this mechanism arrived in Go 1.21, modules declaring anything older than go 1.20 are treated as go 1.20.
  3. //go:debug directives in the main package — explicit per-program overrides.

Then, at run time, the GODEBUG environment variable overrides whatever was compiled in. So the full precedence chain, highest first, is: environment variable → //go:debug directive → godebug block in go.mod → go line in go.mod → toolchain default (GODEBUG doc).

The key consequence: bumping only the toolchain does not flip compatibility defaults — bumping the go line does. That is why upgrading deliberately separates “use a new compiler” from “adopt new behaviors.”

The godebug block and //go:debug directives

Since Go 1.23 the go.mod can carry a godebug (...) block, including a special default=go1.21 key that picks a baseline version independent of the language version (GODEBUG doc). Only the work module’s go.mod is consulted — dependencies’ godebug directives are ignored, so a library cannot silently change your runtime behavior. For per-program control, //go:debug name=value lines placed before the package clause of a main-package file set defaults at build time.

The dedicated tuning variables

Separate from GODEBUG:

  • GOGC — the GC target percentage. Default GOGC=100 triggers a collection when freshly-allocated heap reaches 100% of live heap. GOGC=off disables the collector. Runtime-settable via runtime/debug.SetGCPercent. See GC Pacing and GOGC.
  • GOMEMLIMIT — a soft memory limit covering heap and runtime-managed memory; format is bytes with optional B/KiB/MiB/GiB/TiB suffix (GOMEMLIMIT=512MiB). Default is effectively unlimited (math.MaxInt64). Runtime-settable via runtime/debug.SetMemoryLimit. See GOMEMLIMIT and the Soft Memory Limit.
  • GOMAXPROCS — the number of OS threads that may run Go code simultaneously. As of Go 1.25 the default is derived from logical CPUs, CPU affinity, and cgroup CPU limits; setting the env var disables automatic updates unless GODEBUG=updatemaxprocs=0. Runtime-settable via runtime.GOMAXPROCS. See GMP Scheduler Model.
  • GOTRACEBACK — how much stack output a crash prints: none, single (default — current goroutine only), all, system, crash, wer. Legacy numeric synonyms 0/1/2 map to none/all/system. Can be raised at run time with runtime/debug.SetTraceback but not lowered. On setuid/setgid binaries it is forced to none for safety (runtime doc).

Configuration / Code examples

Diagnostic GODEBUG at the command line

$ GODEBUG=gctrace=1 ./server
gc 1 @0.018s 2%: 0.012+1.2+0.008 ms clock, 0.099+0.31/1.1/0.55+0.064 ms cpu, 4->4->2 MB, 5 MB goal, 0 MB stacks, 0 MB globals, 8 P
  • gc 1 — first collection; @0.018s — 18 ms after start; 2% — cumulative fraction of CPU spent in GC.
  • 0.012+1.2+0.008 ms clock — wall-clock for STW-sweep-termination + concurrent mark/scan + STW mark-termination.
  • 4->4->2 MB — heap size at GC start → at GC end → live heap after sweep. 5 MB goal — the pacer’s target. 8 P — processors. Format per the runtime doc.
$ GODEBUG=schedtrace=1000,scheddetail=1 ./server   # scheduler dump every 1s

Pinning behavior across an upgrade — three ways

// go.mod — amend version-gated defaults without changing the language version
go 1.26.0
 
godebug (
	default=go1.25   // baseline all compat settings to 1.25 semantics
	panicnil=1       // ...but specifically restore pre-1.21 panic(nil)
)
// main.go — per-program override; must precede the package clause
//go:debug asynctimerchan=1
//go:debug httpmuxgo121=0
 
package main
# Runtime override — highest precedence, beats both of the above
$ GODEBUG=panicnil=1,asynctimerchan=1 ./server
  • The godebug block and the default key require Go 1.23+. //go:debug requires Go 1.21+ and is only valid in main packages — go vet flags misplaced directives. The environment variable always wins (GODEBUG doc).

Tuning knobs for a memory-constrained container

$ GOMEMLIMIT=900MiB GOGC=off ./server      # collect only to honor the soft limit
$ GOMAXPROCS=4 GOTRACEBACK=all ./server    # cap parallelism, verbose crash dumps
  • GOGC=off with a GOMEMLIMIT is a documented pattern: the collector stops running on the allocation-ratio schedule and runs only as needed to stay under the soft limit — see GOMEMLIMIT and the Soft Memory Limit.

Observing which compatibility settings a program uses

$ go list -f '{{.DefaultGODEBUG}}' ./cmd/server   # the compiled-in defaults

Every non-default compatibility setting also increments a runtime/metrics counter named /godebug/non-default-behavior/<name>:events, so a monitoring system can detect when a program is relying on a legacy behavior (GODEBUG doc).

Failure Modes / Common Misunderstandings

“GODEBUG=gctrace=1 slows my program / changes behavior.” Diagnostic settings only print. They add a little stderr I/O but do not alter semantics. The settings that do change behavior are the compatibility ones (panicnil, asynctimerchan, …) and the explicit behavior knobs (asyncpreemptoff, cgocheck=0).

“I upgraded the toolchain so I have the new behavior.” Not for version-gated behavior changes. Those are pinned by the go line in go.mod, not the toolchain. A go1.26 toolchain building a go 1.20 module still defaults panicnil=1. Adopting new behavior requires bumping the go line.

“GOEXPERIMENT and GODEBUG are the same thing.” They are not. GOEXPERIMENT (e.g. nogreenteagc, goroutineleakprofile from Go 1.26 Release Notes) is a build-time variable consumed when compiling the runtime itself; it selects experimental implementations. GODEBUG is read at run time by an already-built binary. Different lifecycle, different purpose.

“A library set a GODEBUG default and broke me.” It cannot. Only the work module’s go.mod (and go.work) is consulted for godebug defaults; dependency modules’ directives are ignored by design (GODEBUG doc).

“This GODEBUG setting will exist forever.” Most compatibility settings are guaranteed only for two years / four releases, then removed — tlsrsakex, tls10server, asynctimerchan, and gotypesalias are all scheduled for removal in Go 1.27 (Go 1.26 release notes). A few (http2client, http2server, netdns) are maintained indefinitely. Relying on a GODEBUG setting is a transition tactic, not a permanent fix.

“GOMEMLIMIT is a hard cap.” It is a soft limit — the runtime works hard to stay under it but will exceed it rather than let the program stall indefinitely. A true hard cap is the OS/cgroup limit, which kills the process. See GOMEMLIMIT and the Soft Memory Limit.

Alternatives and When to Choose Them

  • GODEBUG diagnostics vs. pprof / the execution tracer. gctrace/schedtrace give a cheap, always-available, log-shaped view with zero setup — ideal for a first look or for a production incident where you cannot attach a profiler. For attributing cost to code, pprof and Profiling and The Go Execution Tracer are far richer. Use GODEBUG to notice a problem, the profilers to diagnose it.
  • GODEBUG compatibility settings vs. fixing the code. A compatibility setting buys time; it does not fix anything. The intended workflow is: upgrade, hit a behavior change, set the GODEBUG to unblock, then adapt the code and remove the setting before it is dropped.
  • runtime/debug setters vs. environment variables. SetGCPercent, SetMemoryLimit, SetTraceback, GOMAXPROCS() let a program adjust knobs dynamically. Choose the env var for static per-deployment policy; choose the programmatic setter when the value must respond to runtime conditions (e.g. shrinking GOMEMLIMIT when a sidecar reports memory pressure).

Production Notes

In production the high-value GODEBUG move is keeping gctrace/schedtrace available behind a flag — they cost almost nothing and are invaluable mid-incident when a heavyweight profiler is impractical. The canonical container tuning is GOMEMLIMIT set to roughly the cgroup memory limit minus headroom, optionally with GOGC=off, so the collector paces against the real memory ceiling instead of the allocation-ratio heuristic — this is the standard fix for containers that OOM-kill under bursty load.

The most important operational discipline is treating compatibility GODEBUG settings as debt with an expiry date. The /godebug/non-default-behavior/<name>:events metrics exist precisely so a fleet can be audited: if a counter is non-zero, some code path still depends on a legacy behavior, and that dependency must be removed before the setting is dropped (Go 1.27 removes several). The go line in go.mod is the real lever — bump it deliberately, read the relevant release notes, and expect the version-gated behavior changes to take effect at exactly that bump.

See Also

Graph View

Backlinks