vault

GOMEMLIMIT and the Soft Memory Limit

13 min read

GOMEMLIMIT and the Soft Memory Limit

GOMEMLIMIT is a runtime knob, introduced in Go 1.19 (August 2022), that tells the garbage collector a soft ceiling on total runtime-managed memory. It exists because [[GC Pacing and GOGC|GOGC]] alone — a ratio of new memory to live memory — has no notion that available memory is finite, so a sudden growth in the live set can drive the heap past whatever the host can supply and the process is OOM-killed. With GOMEMLIMIT set, the pacer collects more aggressively as memory approaches the limit, ignoring GOGC if necessary, to keep the program inside its memory budget. The limit is deliberately soft: the runtime promises a “reasonable effort,” not a hard guarantee, and will breach the limit rather than thrash itself to a standstill — a deliberate design choice to avoid the GC death spiral (Go GC guide; Go 1.19 release notes).

Why GOGC Was Not Enough

GOGC sizes the next collection as a fraction of the live heap — at the default 100, the heap is allowed to roughly double between collections (see GC Pacing and GOGC). This works well as a CPU-versus-memory trade-off but has one structural blind spot, stated bluntly in the GC guide: it doesn’t take into account that available memory is finite.

Two failure shapes follow. First, transient spikes: a service with a normally small live heap occasionally does something memory-heavy — a large request, a cache warm-up. With GOGC=100, the pacer lets the heap grow to twice that spiked live set, which can exceed the container’s memory limit and trigger an OOM kill, even though the spike was brief. Second, chronic under-use: operators who want to use their memory headroom must raise GOGC high, but a fixed high GOGC over-allocates during normal operation and still overshoots during spikes. There was no way to say “use memory freely, but never exceed this many bytes.” The historical workaround was the memory ballast — a large, never-touched byte slice allocated at startup purely to inflate the live heap and thus the GOGC-derived goal. Ballasts are fragile and waste address space; replacing them was an explicit motivation for GOMEMLIMIT (proposal #48409).

What the Limit Counts

GOMEMLIMIT bounds all memory managed by the Go runtime — the live heap, plus runtime-internal structures (goroutine stacks, the allocator’s metadata and free spans, GC bookkeeping). Per the Go 1.19 release notes, it excludes memory the Go runtime does not manage: the mapped binary image itself, memory allocated by C code via cgo, and memory the operating system holds on the program’s behalf outside the runtime.

The GC guide gives a precise definition in terms of runtime.MemStats: the quantity bounded is

Sys − HeapReleased

— total memory obtained from the OS minus heap memory already returned to the OS. Equivalently, via the runtime/metrics package: /memory/classes/total:bytes − /memory/classes/heap/released:bytes. The exclusion of HeapReleased matters: pages the runtime has handed back to the OS no longer count against the limit even if they remain in the process’s virtual address space.

Mental Model

flowchart TD
    Alloc["Program allocates"] --> Pacer["Pacer computes heap goal"]
    Pacer --> Min{"Two candidate goals"}
    Min -->|"GOGC goal:<br/>live + (live+stk+gl)×GOGC/100"| G1["GOGC-derived goal"]
    Min -->|"GOMEMLIMIT goal:<br/>limit − non-heap memory"| G2["Limit-derived goal"]
    G1 --> Pick["Effective heap goal =<br/>min(GOGC goal, limit goal)"]
    G2 --> Pick
    Pick --> Near{"Near the limit?"}
    Near -->|"yes"| Cap["GC CPU capped at 50%<br/>over a 2×GOMAXPROCS window —<br/>breach the limit rather than thrash"]
    Near -->|"no"| Normal["Normal GOGC-paced collection"]

The diagram shows GOMEMLIMIT as a second, competing heap goal. The pacer computes the usual GOGC-derived goal and a limit-derived goal (the memory limit minus everything that is not heap), and uses whichever is smaller. As the live heap rises toward the limit, the limit-derived goal wins and collections become more frequent. The key safety insight is the bottom branch: when collecting harder still cannot keep up, the runtime does not collect without bound — it caps total GC CPU at 50% and lets memory exceed the soft limit, because a 2× slowdown is survivable but an infinite GC loop is not.

Mechanical Walk-through — How the Soft Limit Steers the Pacer

GOMEMLIMIT does not introduce a separate collector. It feeds the existing pacer (see GC Pacing and GOGC) a second heap goal and tells it to honor the tighter of the two.

Each cycle the pacer already computes the GOGC-derived goal: live_heap + (live_heap + stacks + globals) × GOGC/100. With a memory limit set, it also computes a limit-derived goal: take GOMEMLIMIT, subtract the memory that is not collectible heap — goroutine stacks, allocator free spans and metadata, GC structures — and the remainder is the largest the heap may grow to without the total exceeding the limit. The pacer then sets the effective heap goal to min(GOGC goal, limit goal).

In ordinary operation, when the live heap is small relative to the limit, the GOGC goal is the smaller one and the limit has no effect — the program is paced exactly as GOGC dictates. As the live heap grows, the limit-derived goal shrinks (more of the budget is consumed by live data and non-heap overhead) and at some crossover point it becomes the smaller goal. From there the pacer triggers collections to hold the heap near the limit-derived goal, which means collections become more frequent and start earlier than GOGC would ask. This is also why the limit is honored even with GOGC=off: with the GOGC trigger disabled, the limit-derived goal is the only goal, and the program will “always make maximal use of [its] memory limit” (Go 1.19 release notes) — useful for a program that wants to fill a fixed budget and collect only when it must.

The non-heap accounting and the limit goal are computed inside gcControllerState in mgcpacer.go; the limit value is stored as memoryLimit and folded into the goal computation in gcControllerState.commit.

The Death Spiral and the 50% CPU Cap

The danger of any hard memory limit on a tracing collector is the GC death spiral, also called thrashing. Suppose the live heap is genuinely close to the limit — there is simply that much live data. The limit-derived goal is then barely above the current heap. The pacer triggers a collection almost immediately; the collection runs, recovers little (most of the heap is live); the heap is again at the goal almost at once; another collection triggers. The program now spends nearly all its CPU collecting and makes almost no forward progress. The GC guide names this precisely: the program fails to make reasonable progress due to constant GC cycles — and notes it is “particularly dangerous because it effectively stalls the program,” a worse outcome than a clean OOM.

Go’s defense is to make the limit soft and bound the cost of honoring it. The runtime limits total GC CPU utilization to 50% — excluding idle time, per the Go 1.19 release notes — measured over a sliding window of roughly 2 × GOMAXPROCS CPU-seconds (the window figure is from the GC guide). When this limiter last engaged is exposed by the runtime/metrics series /gc/limiter/last-enabled:gc-cycle, which the 1.19 notes introduced specifically to surface this “exceptional” case. If holding the heap under the limit would require more than half the CPU, the runtime instead lets the heap exceed the soft limit and keeps running the program. The reasoning, from the GC guide: in the worst case — a memory limit accidentally set far too low — the program slows by at most 2×, because the GC can never seize more than half the CPU. A bounded 2× slowdown is a survivable misconfiguration; an unbounded death spiral is not. The “soft” in “soft memory limit” is exactly this: the runtime makes a reasonable effort and will breach the limit rather than thrash itself to death.

Code and Configuration Examples

Setting the limit from the environment. GOMEMLIMIT accepts a byte count with optional unit suffixes (B, KiB, MiB, GiB, TiB — powers of 1024):

GOMEMLIMIT=2GiB ./myserver       # soft cap at 2 GiB of runtime memory
GOMEMLIMIT=900MiB ./myserver     # e.g. 90% of a 1 GiB container limit
GOMEMLIMIT=off ./myserver        # the default — no limit

Setting it at runtime via runtime/debug:

package main
 
import "runtime/debug"
 
func main() {
	// SetMemoryLimit takes a byte count and returns the PREVIOUS limit.
	// math.MaxInt64 is the sentinel meaning "no limit".
	prev := debug.SetMemoryLimit(2 << 30) // 2 GiB
	_ = prev
 
	// Passing -1 only QUERIES the current limit without changing it.
	current := debug.SetMemoryLimit(-1)
	_ = current
 
	// Common production pattern: keep GOGC at default, add a ceiling.
	debug.SetGCPercent(100)
	debug.SetMemoryLimit(2 << 30)
}

Line-by-line: SetMemoryLimit(2 << 30) sets a 2 GiB soft limit and returns the previous limit. The runtime/debug doc comment states the conventions verbatim: the initial setting is math.MaxInt64 unless GOMEMLIMIT is set, and “math.MaxInt64 is the canonical value for disabling the limit” (so a no-limit program returns MaxInt64 from the call). SetMemoryLimit(-1) is the query form — per the same doc comment, “a negative input does not adjust the limit, and allows for retrieval of the currently set memory limit” (runtime/debug/garbage.go). The unit suffixes the GOMEMLIMIT environment variable accepts (B, KiB, MiB, GiB, TiB) are powers of two per the IEC 80000-13 standard — KiB is 2^10, MiB is 2^20, and so on. The final pair shows the recommended combination: leave GOGC at its default to pace normal operation and add GOMEMLIMIT as the safety ceiling — the two cooperate rather than conflict. Note the symmetry with SetGCPercent: its doc comment warns that the GOGC percentage “may be effectively reduced in order to maintain a memory limit,” and that a negative percentage “effectively disables garbage collection, unless the memory limit is reached” — the same soft-limit-always-wins rule from the other side of the knob.

Deriving the limit from a Kubernetes container’s memory limit, leaving headroom (the GC guide recommends 5–10% for non-heap and unmanaged memory):

env:
  - name: GOMEMLIMIT
    valueFrom:
      resourceFieldRef:
        resource: limits.memory   # the pod's memory limit, in bytes
        divisor: "1"

Then either consume that value directly, or scale it down ~10% in code with debug.SetMemoryLimit before serving traffic. The downward-API value is the hard container limit; GOMEMLIMIT should sit below it so the soft cap engages before the kernel OOM-killer does.

Failure Modes and Common Misunderstandings

“GOMEMLIMIT is a hard cap — the process can’t exceed it.” False, and dangerously so. It is soft. Under genuine memory pressure the runtime will breach it (capping GC CPU at 50%) rather than thrash. It reduces OOM-kill risk; it does not eliminate it.

Setting the limit too low causes thrashing. If the limit is below — or barely above — the genuine live heap, the program enters or skirts the death spiral and runs at up to 2× slowdown. The GC guide’s rule: do not set GOMEMLIMIT when program memory is proportional to unbounded input, or when already near the environment’s memory limit.

It does not bound cgo, mmap’d files, or the binary image. Only runtime-managed memory counts. A program that leaks memory through C code can OOM with GOMEMLIMIT set and never approaching it.

It is not a substitute for fixing leaks. A real leak grows the live heap monotonically; GOMEMLIMIT just turns “OOM in an hour” into “thrash in an hour.” Diagnose with heap profiles.

“With GOMEMLIMIT set I should also disable GOGC.” Usually not. GOGC=off plus GOMEMLIMIT makes the program always fill its budget and collect only at the ceiling — fine for a cache-like workload, but for most services it means a permanently high resident set and collections that always run near the limit. The default recommendation is GOGC on and GOMEMLIMIT set.

Small limits are unreliable. The Go 1.19 notes warn that limits “on the order of tens of megabytes or less” are less likely to be respected, because OS scheduling latency and other external factors dominate at that scale.

Alternatives and When to Choose Them

GOMEMLIMIT’s sibling knob is [[GC Pacing and GOGC|GOGC]], and the two are complementary, not alternatives — the standard setup uses both. The pre-1.19 memory ballast achieved a similar effect by inflating the live heap to raise the GOGC goal; GOMEMLIMIT makes ballasts obsolete and the Go team recommends removing them. External enforcement — a cgroup memory limit plus the kernel OOM-killer — is the backstop, not a substitute: it kills the process rather than steering the collector. Manual runtime.GC() can cap memory at known phase boundaries in a batch program but does nothing for a continuously running server. Use GOMEMLIMIT when the program runs in a controlled environment with a known memory budget — a container with a fixed memory limit is the canonical case. Avoid it for CLI tools, desktop apps, and any program whose memory footprint is dictated by unbounded user input.

Production Notes

GOMEMLIMIT has become standard practice for Go services in containers. The recommended recipe: set GOMEMLIMIT to 90–95% of the container memory limit (5–10% headroom for non-heap and unmanaged memory), keep GOGC at its default, and let the soft limit engage only during spikes. Real-world reports — e.g. the Weaviate vector-database team’s writeup (Weaviate blog) — describe GOMEMLIMIT eliminating OOM kills for memory-heavy workloads that previously needed ballasts or aggressive GOGC tuning. The signals to watch in production: rising runtime.gcAssistAlloc and runtime.gcBgMarkWorker CPU time as the program approaches the limit (the pacer collecting harder), and — the warning sign — GC CPU pinned at the 50% cap with the heap above GOMEMLIMIT, which means the limit is too tight for the genuine live set and the program is in or near the death spiral. The runtime/metrics series /gc/limiter/last-enabled:gc-cycle and /memory/classes/total:bytes expose this directly — see GC Tuning and Observability.

Green Tea GC and the Limit — No Change to the Contract

As of Go 1.26 (February 2026) the default collector is Green Tea (opt out with GOEXPERIMENT=nogreenteagc at build time, expected to be removed in Go 1.27) (Go 1.26 release notes; Green Tea blog). Green Tea is a redesign of the scan kernel — it marks page-by-page in a cache-friendly, vectorizable order rather than chasing pointers object-by-object across the heap — and it leaves the pacer, the GOGC/GOMEMLIMIT goal arithmetic, the min-of-two-goals rule, and the 50% GC-CPU thrash cap untouched. The min(GOGC goal, limit goal) mechanism, the memoryLimit atomic.Int64 field, the memoryLimitHeapGoal, and the death-spiral defense all live in gcControllerState exactly as described above. The practical relevance for this note is second-order but real: by cutting GC overhead 10–40% in GC-heavy programs (and ~10% more from AVX-512 scanning on Intel Ice Lake / AMD Zen 4 and newer), Green Tea lets the runtime hold the heap near a given GOMEMLIMIT for less CPU, which widens the margin before the 50% cap engages and the soft limit is breached. The tuning advice is unchanged; Green Tea simply buys headroom.

See Also

Graph View

Backlinks