vault

Goroutine Preemption

12 min read

Goroutine Preemption

Preemption is how the Go runtime takes back a CPU from a goroutine that has not voluntarily yielded. Without it, a single goroutine running a tight loop with no function calls could monopolise an operating-system thread forever, stalling the scheduler and — far worse — indefinitely delaying a garbage-collection stop-the-world that must observe every goroutine. Go has two preemption mechanisms working together: cooperative (synchronous) preemption, which has existed since the early days and works by poisoning a goroutine’s stack guard so its next function call traps into the runtime; and asynchronous (signal-based) preemption, added in Go 1.14, which suspends a running thread with an operating-system signal and can preempt a goroutine at almost any instruction, including inside a loop that never calls a function (src/runtime/preempt.go; proposal #24543).

Mental Model

A goroutine can only be safely paused at a safe-point — an instruction where the runtime has complete, precise information about which CPU registers and stack slots hold pointers, so the garbage collector can scan the goroutine and the scheduler can deschedule it with minimal saved state. The preempt.go comment enumerates three categories of safe-point: blocked safe-points (the goroutine is descheduled, blocked on a channel/mutex, or in a syscall — its state is already minimal); synchronous safe-points (a running goroutine actively checks for a preemption request, which the compiler arranges to happen at function prologues); and asynchronous safe-points (any instruction in user code where a conservative register-and-stack scan can still find all the pointer roots).

flowchart TD
    R["Requester: scheduler / sysmon / GC<br/>calls preemptone(P) or suspendG(g)"]
    R --> S["Set gp.preempt = true<br/>Set gp.stackguard0 = stackPreempt"]
    S --> C{"Goroutine reaches<br/>a function prologue<br/>(within ~10ms)?"}
    C -->|yes| SYNC["Synchronous path:<br/>prologue check fails -> morestack -> newstack<br/>sees stackPreempt -> gopreempt_m / preemptPark"]
    C -->|"no — tight loop,<br/>no calls"| ASYNC["Asynchronous path"]
    ASYNC --> SIG["preemptM: signalM(mp, SIGURG)"]
    SIG --> H["Signal handler: doSigPreempt"]
    H --> ISP{"isAsyncSafePoint(gp, pc)?"}
    ISP -->|yes| INJ["pushCall: rewrite signal context<br/>so thread 'just called' asyncPreempt"]
    ISP -->|no| RETRY["Acknowledge; retry later"]
    INJ --> AP["asyncPreempt (assembly):<br/>spill ALL registers -> asyncPreempt2 -> scheduler"]
    SYNC --> SCHED[Scheduler reschedules the P]
    AP --> SCHED

Figure: the two preemption paths. The crucial insight is that they share the same request (gp.preempt plus the poisoned stackguard0) but differ in delivery: the synchronous path waits passively for the goroutine to hit a prologue; the asynchronous path actively forces the issue with a signal when the goroutine refuses to cooperate.

Why Preemption Exists: The Tight-Loop Problem

Before Go 1.14, preemption was purely cooperative. The only places a goroutine could be preempted were function prologues (and a handful of explicit checks). This is fine for the vast majority of code — most goroutines call functions constantly — but it has a catastrophic failure mode. Consider:

func spin() {
	for {            // no function call inside this loop
	}
}

A goroutine running spin reaches a prologue exactly once (entering spin) and then never again. It can never be preempted. The proposal that introduced asynchronous preemption, #24543 “non-cooperative goroutine preemption”, describes the consequences bluntly: delayed preemption causes “mysterious system-wide latency issues… and sometimes complete freezes.” The reason it is system-wide and not merely a stuck goroutine is the garbage collector. Go’s GC must perform a brief stop-the-world to start and finish a cycle, and STW means every goroutine must reach a safe-point. One uncooperative goroutine holds the entire STW hostage: the GC cannot proceed, mark-assist pressure builds, and every other goroutine in the program stalls. A single tight loop could freeze a whole service.

The Go 1.14 release notes record the fix in one sentence: “Goroutines are now asynchronously preemptible. As a result, loops without function calls no longer potentially deadlock the scheduler or significantly delay garbage collection” (Go 1.14 release notes).

Cooperative (Synchronous) Preemption

Making the request

The runtime requests preemption of a goroutine by doing two things, visible throughout proc.go and preempt.go: it sets gp.preempt = true, and it sets gp.stackguard0 = stackPreempt. The constant stackPreempt is a deliberately enormous sentinel (0xfffffade on 32-bit lanes) — a value the stack pointer can never legitimately be below. Poisoning the guard this way guarantees the next prologue stack-bound check will fail.

The prologue trap

When the goroutine next enters a function, its compiler-generated prologue compares the stack pointer against gp.stackguard0, sees it “below” the poisoned guard, and — exactly as for a genuine stack-growth event — calls morestack, which calls newstack (stack.go). newstack reads the guard, sees the value is stackPreempt, and recognises this is a preemption, not an overflow. It then asks canPreemptM: is it currently safe to stop this M? If the M holds locks, is in the middle of malloc, or has m.preemptoff set, preemption is unsafe — newstack restores the real guard and lets the goroutine run on, leaving gp.preempt set so the next prologue retries. If preemption is safe, newstack diverts:

  • gopreempt_m — the goroutine yields its P and goes back on a run queue, exactly as if it had called runtime.Gosched(). Used for time-slice preemption.
  • preemptPark — the goroutine is stopped in _Gpreempted status so the garbage collector can scan it. Used when the GC needs the world stopped. The goroutine is later readied by whoever ran suspendG.

This is “free” in the steady state: a prologue check is one comparison and a not-taken branch. The cost is paid only when the guard is actually poisoned.

Loop preemption — the partial fix before signals

A purely-prologue scheme still cannot touch a loop with no calls. Before async preemption, the compiler experimented with inserting explicit preemption checks at the back-edge of loops, enabled by GOEXPERIMENT=preemptibleloops. This “loop preemption” worked but cost a check on every iteration of every loop and was never made the default. Asynchronous preemption (Go 1.14) superseded it — signals give loop preemption “for free” with no per-iteration cost, so the compiler’s loop-back-edge insertion is no longer used by default. The PreemptibleLoops field still appears in internal/goexperiment/flags.go at go1.26.3 as a now-dormant experiment toggle, but preempt.go carries no loop-back-edge machinery — preemption is driven entirely by the synchronous-prologue and signal paths described here (preempt.go at go1.26.3).

Asynchronous (Signal-Based) Preemption — Go 1.14

Sending the signal

When cooperative preemption is not making progress — the background monitor sysmon notices a P has run the same goroutine for longer than forcePreemptNS (10 ms), see System Calls and the Scheduler and Sysmon System Monitor — the runtime escalates to a signal. preemptone calls preemptM(mp), which on Unix executes signalM(mp, sigPreempt) (signal_unix.go).

The chosen signal is SIGURG, and the signal_unix.go comment gives a careful four-point rationale: the signal must (1) be passed through by debuggers by default, (2) not be used internally by libc in mixed Go/C binaries, (3) be one “that can happen spuriously without consequences,” and (4) work on platforms lacking real-time signals (macOS). SIGURG — urgent out-of-band socket data — fits: “out-of-band data is basically unused… even if it is, the application has to be ready for spurious SIGURG.” preemptM also coalesces requests: it uses mp.signalPending as a flag and sends the signal only if mp.signalPending.CompareAndSwap(0, 1), so a storm of preemption requests against one M produces at most one in-flight signal — this was a real live-lock fix for Darwin (issue #37741).

Handling the signal

The kernel delivers SIGURG to some thread in the process; Go’s signal handler runs and, for sigPreempt, calls doSigPreempt(gp, ctxt). The handler does not preempt from within itself — the preempt.go comment explains why: running the scheduler from a signal handler “would consume an M for every preempted G, and the scheduler itself is not designed to run from a signal handler, as it tends to allocate memory and start threads.”

Instead, doSigPreempt performs a clever trick. It first checks wantAsyncPreempt(gp) (does this goroutine actually still want preempting?) and then isAsyncSafePoint(gp, pc, sp, lr) — is the exact instruction the signal interrupted a place where a conservative scan can find all pointer roots? If yes, it calls ctxt.pushCall(asyncPreempt, newpc): it rewrites the interrupted thread’s saved register context so that, when the signal handler returns and the kernel resumes the thread, the thread appears to have just executed a call instruction to asyncPreempt. The original PC is preserved as the return address.

asyncPreempt

asyncPreempt is hand-written assembly. Because it was injected at an arbitrary instruction, every register may hold live data, so asyncPreempt spills the entire register file to the goroutine’s stack, then calls asyncPreempt2, which enters the scheduler. When the goroutine is later resumed, the registers are restored and the original PC re-executes — the goroutine never knows it was interrupted. The injected frame is a special one the runtime’s stack walker recognises, so the conservative scan knows the registers it must treat as potential pointers.

isAsyncSafePoint is what makes this sound. Not every instruction is safe: you cannot preempt inside the runtime’s own delicate sequences, inside an unsafe-point the compiler marked, while the M is in a state where a conservative scan would be wrong, or when there is insufficient stack space to spill registers. When isAsyncSafePoint returns false, doSigPreempt simply acknowledges the signal (gp.m.preemptGen.Add(1)) and the request is retried later.

Code: Demonstrating the Difference

package main
 
import (
	"fmt"
	"runtime"
	"time"
)
 
func main() {
	runtime.GOMAXPROCS(1) // one P: the spinner and main must share one OS thread
	go func() {
		for {            // tight loop, no function calls -> only async preemption can stop it
		}
	}()
	time.Sleep(10 * time.Millisecond) // give the spinner the P
	fmt.Println("main still runs — async preemption took the P back")
}

On Go 1.14 and later this program prints and exits: sysmon detects the spinner has held the P past forcePreemptNS, sends SIGURG, and the asynchronous path forces the spinner off the thread so main can run. On Go 1.13 and earlier the same program deadlocks — with GOMAXPROCS(1) the spinner never reaches a prologue, never yields, and main never wakes. That behavioural difference is the single clearest demonstration of what Go 1.14 changed.

# Disable async preemption to reproduce the pre-1.14 behaviour:
GODEBUG=asyncpreemptoff=1 ./prog   # the program above will now hang under GOMAXPROCS(1)

GODEBUG=asyncpreemptoff=1 turns off signal-based preemption at runtime (the handler checks debug.asyncpreemptoff). It exists mainly to diagnose whether a bug is caused by async preemption itself — useful when chasing signal-related races in cgo-heavy programs.

Failure Modes and Common Misunderstandings

“Async preemption made signals appear that break my syscalls.” True and documented. The Go 1.14 notes warn: “programs built with Go 1.14 will receive more signals… programs that use packages like syscall… will see more slow system calls fail with EINTR.” Because SIGURG can interrupt a blocking syscall, code calling raw syscalls (or golang.org/x/sys/unix) must now handle EINTR by retrying. The standard library already does; hand-rolled syscall code may not. This is a genuine, real-world breakage that surfaced when 1.14 shipped.

“Async preemption can stop my goroutine anywhere, so my critical section is unsafe.” No. Preemption only deschedules a goroutine; it does not run other code “inside” your function. A mutex-protected section stays protected — the goroutine simply pauses and resumes later still holding the lock. The thing async preemption breaks is the assumption that a goroutine runs uninterrupted, which only matters for code that was (incorrectly) relying on never yielding the CPU.

“Disabling async preemption is a safe optimisation.” It is not an optimisation — async preemption has near-zero steady-state cost (no signal is sent unless sysmon escalates). Disabling it reintroduces the tight-loop freeze. The only legitimate use of asyncpreemptoff=1 is diagnostic.

“The preempt flag means the goroutine stops immediately.” No. Setting gp.preempt is a request. Cooperative preemption acts at the next safe prologue; even async preemption is bounded by signal-delivery latency and the isAsyncSafePoint check, which may defer it. Preemption is prompt (sub-forcePreemptNS-plus-a-bit) but never instantaneous.

Alternatives and Design Trade-offs

The space of preemption designs: OS-thread preemption (every thread is a kernel thread, the kernel preempts on a timer) is simple and always works but caps you at thousands of units of concurrency — it is the model Go explicitly rejects. Pure cooperative preemption (early Go, classic coroutine libraries, Lua) has zero forced-interrupt machinery but the tight-loop pathology. Pure signal preemption would avoid prologue checks but cannot be the only mechanism — signals are coarse and isAsyncSafePoint rejects many instructions, so you still want the cheap, precise synchronous path for the common case. Go’s actual design — synchronous as the default, asynchronous as the escalation — is the considered combination: the prologue check is so cheap it is always worth having, and signals are reserved for the rare goroutine that genuinely refuses to cooperate. The proposal explicitly weighed this and chose the hybrid for exactly this reason.

Production Notes

In practice preemption is invisible and needs no tuning. It surfaces in two places. First, scheduler latency: the execution tracer (The Go Execution Tracer) shows goroutine preemption events; a goroutine repeatedly preempted at the 10 ms forcePreemptNS slice is CPU-bound and competing for Ps. Second, the EINTR interaction: when porting old code or chasing a “syscall fails intermittently after upgrading to 1.14+” bug, the cause is almost always an unhandled EINTR from a SIGURG interrupting a slow syscall — the fix is a retry loop, not disabling preemption. For platforms where async preemption is unsupported (preempt.go and the 1.14 notes list js/wasm, plan9/*, and historically some arm variants), the tight-loop freeze can still occur, so loop-heavy code targeting those platforms should call runtime.Gosched() periodically as a manual yield.

See Also

Graph View

Backlinks