vault

Stop-the-World Pauses

11 min read

Stop-the-World Pauses

A stop-the-world (STW) pause is an interval during which the Go runtime halts every user goroutine so it can do something that requires a globally consistent, frozen view of the program — most prominently, the bracketing steps of a garbage collection. Go’s collector is mostly concurrent, so STW is not where the GC does its work; it is the two short transitions into and out of concurrent marking. Years of runtime engineering have driven these pauses from the hundreds of milliseconds of Go 1.4 down to the sub-millisecond, typically tens-of-microseconds range — the single biggest of those wins being the hybrid write barrier of Go 1.8, which eliminated the end-of-cycle stack rescan STW (Hudson, ISMM keynote; rescan-elimination proposal).

Why Stop the World at All

A concurrent garbage collector spends almost all of its effort running alongside the program — see Tricolor Mark and Sweep and GC Assists and Mark Workers. But a few operations are intrinsically global: they need a moment when no goroutine is mutating state, so the runtime can flip a phase, scan roots, or take a consistent measurement without a write racing it. To get that moment, the runtime stops the world: it preempts every running goroutine and parks every P (processor) until the operation completes, then starts the world again.

STW is not unique to GC. The runtime also stops the world for runtime.GOMAXPROCS changes, for some profiling and tracing operations, for runtime.ReadMemStats (historically), and for heap dumps. But the GC transitions are the STW pauses that matter most for latency-sensitive services, because they recur on every collection cycle.

The Two GC Stop-the-World Phases

A Go GC cycle has, per the phase comments in mgc.go, the following shape. Two of the phases are STW; the long-running work between and around them is concurrent.

1. Sweep Termination (STW). The world stops. The runtime finishes any sweeping left over from the previous cycle that has not yet been done lazily, and prepares to mark: it transitions the GC phase, and enables the write barrier and mark assists. This pause exists because marking cannot begin while a stale sweep is in flight, and the write-barrier flag must flip atomically with respect to all goroutines.

2. Mark phase (concurrent). The world restarts. With the write barrier on, background mark workers and allocation-driven assists trace the heap from the roots, shading objects gray then black. The program runs the whole time. This is where the GC actually spends its effort, and it is not STW.

3. Mark Termination (STW). The world stops a second time. The runtime verifies marking is complete, disables the mark workers and assists, performs housekeeping, and flushes per-P allocator caches (mcaches). It then transitions out of the mark phase. This pause exists because declaring “marking is done” requires a frozen instant in which no goroutine can still be producing gray work.

4. Sweep phase (concurrent). The world restarts. The GC phase is _GCoff, the write barrier is off, and reclamation of white (unreachable) objects proceeds lazily and concurrently — typically a span is swept just before it is next handed out for allocation. New allocations are colored white.

So a cycle is STW → concurrent mark → STW → concurrent sweep. The two STW intervals are the only times the program is fully halted, and modern Go keeps each one short — the design target from the rescan-elimination proposal was worst-case STW under 50µs, and production services routinely see sub-millisecond pauses (Hudson, ISMM keynote).

Mental Model

gantt
    title One GC cycle — STW phases are the thin bars
    dateFormat X
    axisFormat %s
    section GC
    Sweep termination (STW)   :crit, a, 0, 1
    Concurrent mark           :active, b, 1, 9
    Mark termination (STW)    :crit, c, 9, 10
    Concurrent sweep          :active, d, 10, 18
    section Program
    Halted                    :crit, 0, 1
    Running (with assists)     :active, 1, 9
    Halted                     :crit, 9, 10
    Running                    :active, 10, 18

The diagram shows one GC cycle on a timeline. The two red crit bars — sweep termination and mark termination — are the stop-the-world pauses; they are deliberately thin because the runtime works hard to keep them in the tens-of-microseconds range. The long active bars are the concurrent mark and sweep, during which the program keeps running. The insight: STW cost in modern Go is not proportional to heap size — it is a small, roughly fixed transition cost, because all the O(heap) work was moved into the concurrent bars.

Mechanical Walk-through — How the Runtime Actually Stops the World

Stopping the world is implemented by stopTheWorld / stopTheWorldWithSema in proc.go. The mechanism is cooperative preemption layered over asynchronous preemption:

  1. The goroutine initiating the stop acquires the global scheduler lock and sets a flag (sched.stopwait, with sched.gcwaiting raised) indicating the world should stop.
  2. It walks every P and tries to flip it out of running state. A P that is idle or in a syscall is claimed immediately. A P running a goroutine must be preempted.
  3. Preemption uses two mechanisms (see Goroutine Preemption). The cooperative path sets a preempt flag the goroutine notices at the next function-call safe point. Since Go 1.14, asynchronous preemption can also interrupt a goroutine that is not hitting safe points — the runtime sends the thread a signal (SIGURG on Unix) and the signal handler parks the goroutine at a safe instruction (Go 1.14 release notes). Before 1.14, a tight loop with no calls could delay an STW indefinitely — a real latency bug that asynchronous preemption fixed.
  4. The initiator spins/waits until stopwait reaches zero — every P parked. At that point the world is stopped: no user goroutine is executing on any P.
  5. The GC transition work runs.
  6. startTheWorld reverses it: Ps are made runnable again, parked threads woken, the scheduler lock released, and goroutines resume.

The reason modern STW is cheap is that the only work done inside the stopped window is the phase flip and a bounded amount of housekeeping. The expensive, heap-proportional work — scanning the heap graph, scanning goroutine stacks, sweeping — was moved out of STW over successive releases:

  • Go 1.5 (2015) introduced concurrent marking, moving the bulk of marking out of STW and cutting pauses from ~300–400 ms to ~30–40 ms.
  • Go 1.6 (2016) removed O(heap) operations from the STW phases, reaching ~4–5 ms.
  • Go 1.7 (2016) continued eliminating O(heap-size) STW work, reaching sub-millisecond on large heaps.
  • Go 1.8 (2017) delivered the decisive change: the hybrid write barrier eliminated the stack rescan. Before 1.8, mark termination had to re-scan every goroutine’s stack under STW, because stacks were “permagrey” — and with thousands of goroutines that rescan took 10s–100s of milliseconds. The hybrid barrier lets a stack be scanned once, concurrently, and stay black, so mark termination no longer touches stacks. This is why mark-termination STW became roughly heap-size-independent (rescan-elimination proposal).
  • Go 1.14 (2020) added asynchronous preemption, removing the tail latency of STW caused by uncooperative loop-bound goroutines.

Observing and Measuring STW Pauses

GODEBUG=gctrace=1 prints, per cycle, a line whose first timing triple is the STW and concurrent costs:

gc 12 @0.481s 2%: 0.025+0.83+0.028 ms clock, 0.20+0.15/1.6/0.5+0.22 ms cpu, ...

In the clock triple 0.025+0.83+0.028 ms, the first number (0.025 ms = 25 µs) is the sweep-termination STW, the middle is concurrent mark time, and the third (0.028 ms = 28 µs) is the mark-termination STW. The two outer numbers are the actual stop-the-world durations — here, tens of microseconds each.

Programmatically, runtime.ReadMemStats exposes PauseNs (a ring buffer of recent total pause times) and PauseTotalNs; the runtime/metrics package exposes /gc/pauses:seconds as a histogram. For latency analysis, the execution tracer (runtime/trace) shows each STW interval directly on the timeline, attributed to its cause.

package main
 
import (
	"runtime"
	"fmt"
)
 
func main() {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)              // historically itself an STW; cheap now
	fmt.Println("GC cycles:", m.NumGC)
	fmt.Println("total STW pause ns:", m.PauseTotalNs)
	// m.PauseNs is a 256-entry circular buffer of recent pause durations.
	last := m.PauseNs[(m.NumGC+255)%256]
	fmt.Println("most recent pause ns:", last)
}

Line-by-line: ReadMemStats fills m with a snapshot of memory statistics. m.NumGC is the count of completed cycles. m.PauseTotalNs is cumulative STW time across the process’s life. m.PauseNs is a 256-slot circular buffer; indexing it by (NumGC+255)%256 retrieves the most recent cycle’s pause. These are total pause numbers (both STW phases summed); for the per-phase split use gctrace or the tracer.

Failure Modes and Common Misunderstandings

“The GC stops the world to do its work.” This is the single most common misconception and it describes Go’s pre-1.5 collector, not the current one. Modern Go marks and sweeps concurrently; STW is only the two brief transitions. The heavy lifting is concurrent.

“STW pause time grows with heap size.” Largely false since Go 1.8. The O(heap) work — heap scan, stack scan, sweep — is concurrent. STW is a roughly fixed transition cost. A 100 GiB heap and a 100 MiB heap have STW pauses in the same ballpark.

Long pauses from delayed preemption (pre-1.14). If a goroutine ran a tight loop with no function calls, the runtime could not preempt it cooperatively, so the entire STW stalled waiting for that one goroutine — observed as multi-millisecond or worse pauses. Asynchronous preemption (Go 1.14) fixed this. On a pre-1.14 toolchain it is still a live hazard.

Confusing STW with GC CPU cost or with assist latency. A program can have negligible STW pauses yet still feel slow under GC because of assists — allocating goroutines being throttled to do marking work. Assist latency is not STW; it does not show in PauseNs. Diagnose assists with CPU profiles (runtime.gcAssistAlloc), not pause metrics.

Non-GC STW. runtime.GOMAXPROCS, heap dumps, and some tracing operations also stop the world. A latency spike correlated with none of the GC metrics may be one of these.

Alternatives and When They Apply

Within Go there is no “turn off STW” — the two transitions are intrinsic to a concurrent tracing collector. The relevant tuning levers act on STW frequency, not its per-pause cost: raising [[GC Pacing and GOGC|GOGC]] makes cycles rarer, so fewer STW pauses occur per unit time, at the cost of a larger heap; [[GOMEMLIMIT and the Soft Memory Limit|GOMEMLIMIT]] under pressure makes them more frequent. Across language runtimes, the design space is wider: the JVM’s ZGC and Shenandoah, and the Go-adjacent research collectors, push toward fully pauseless or sub-millisecond-guaranteed collection using load/read barriers and concurrent relocation; Go instead bet on keeping the collector non-moving and accepting two tiny transition pauses, which sidesteps the read-barrier tax and the cgo pinning problems a moving collector creates (see Go Garbage Collector). For Go, the practical answer to “STW is hurting me” is almost never “switch collectors” — it is to allocate less so cycles are rarer (see Escape Analysis, sync.Pool Internals).

Production Notes

For the vast majority of Go services, GC STW pauses are no longer a latency concern — sub-millisecond, often 10–50 µs, and heap-size-independent. The 99.9th-percentile latency tail in a modern Go service is far more often dominated by assist throttling, scheduler preemption latency, lock contention, or syscalls than by STW itself. When STW does show up as a problem, the usual causes are: running a pre-1.14 toolchain with an uncooperative loop blocking preemption; an unexpectedly high collection rate (too-low GOGC or a too-tight GOMEMLIMIT) multiplying the number of pauses; or a non-GC STW source such as periodic GOMAXPROCS churn. The diagnostic path is GODEBUG=gctrace=1 to see per-phase STW durations, then the execution tracer to place each pause on the timeline and confirm whether it is GC or something else — see GC Tuning and Observability.

See Also

Graph View

Backlinks