vault

System Calls and the Scheduler

16 min read

System Calls and the Scheduler

A goroutine that makes a blocking operating-system call presents the scheduler with a problem: the underlying OS thread (the M) is stuck inside the kernel and cannot run Go code, yet a P — the scarce permit that grants the right to run Go code — must not be wasted sitting idle behind it. Go’s solution is the pair entersyscall/exitsyscall: before a syscall the runtime moves the goroutine to the _Gsyscall state while leaving the P loosely attached to the M, so that if the syscall blocks for long, the background monitor sysmon can take the P away and hand it off to another thread, keeping GOMAXPROCS-worth of parallelism alive. On return, the goroutine tries to re-acquire a P; if none is free it parks until one is. This is why a Go program with 8 Ps can have thousands of goroutines blocked in read() and still keep all 8 cores busy (src/runtime/proc.go).

One subtlety worth stating up front, because older write-ups (and older Go releases) get it wrong: in current Go (verified against Go 1.26.3) the P does not change to a special “syscall” status while the M is in the kernel. The P remains _Prunning; the goroutine’s _Gsyscall state is the sole marker that the P is “in a syscall.” The runtime’s runtime2.go is explicit — the former _Psyscall constant is now named _Psyscall_unused and documented as “a now-defunct state … A P is identified as ‘in a system call’ by looking at the goroutine’s state” (runtime2.go).

Mental Model

Recall the G-M-P model (GMP Scheduler Model): a G is a goroutine, an M is an OS thread, a P is a “processor” — one of exactly GOMAXPROCS permits, and an M needs a P to execute Go code. The HACKING.md comment is precise: an M “can have an associated P to execute Go code, however it can be blocked or in a syscall w/o an associated P.” A syscall is exactly the case where the M leaves Go code. The runtime therefore distinguishes two situations and treats them differently:

flowchart TD
    G["Goroutine calls a syscall"] --> E["entersyscall:<br/>G -> _Gsyscall<br/>P stays _Prunning, M keeps it (loosely)<br/>save m.syscalltick = p.syscalltick"]
    E --> SC["M blocks in the kernel"]
    SC --> FAST{"Syscall returns fast<br/>(before sysmon notices)?"}
    FAST -->|yes| EX1["exitsyscall fast path:<br/>M still holds its P (m.p != nil)<br/>-> reacquire same P, keep running"]
    FAST -->|"no — sysmon's retake()<br/>sees G in _Gsyscall too long"| RT["sysmon retake:<br/>pin thread, thread.takeP(), P released<br/>handoffp -> start/wake another M"]
    RT --> EX2["exitsyscall slow path:<br/>M's P was taken (m.p == nil)<br/>-> try to grab an idle P (prefer oldp)<br/>-> else G -> _Grunnable, M parks"]
    G2["Known-blocking syscall<br/>(entersyscallblock)"] --> HB["Immediately handoffp(releasep())<br/>before the _Gsyscall switch"]

Figure: the two ways a P is freed during a syscall. The insight: for a fast syscall the runtime gambles that it will return before sysmon intervenes, so it does the cheap thing (keep the P loosely attached, P still _Prunning); for a known-blocking syscall it skips the gamble and hands the P off up front. sysmon is the safety net that retakes the P when the gamble loses. Note that nowhere does the P enter a distinct “syscall” status — the goroutine’s _Gsyscall state and the syscalltick counter are what the runtime keys off of.

Why This Is Hard

If Go simply let an M block in the kernel while holding a P, then GOMAXPROCS blocking syscalls would consume all the Ps and the program would have zero parallelism even on a many-core machine — every other goroutine would starve. The naive fix, “spawn a fresh M for every syscall,” thrashes: thread creation is expensive and you would end up with thousands of threads. Go’s design threads the needle: it keeps the common, fast syscall cheap (no thread creation, no P handoff in the typical case) while guaranteeing that a slow syscall cannot strand a P. The two halves are entersyscall/exitsyscall (run by the goroutine) and retake (run by sysmon, the background monitor).

Mechanical Walk-through

entersyscall — handing the M into the kernel

When Go’s syscall package (or a cgo call) is about to invoke a real OS call, it calls entersyscall, which delegates to reentersyscall (proc.go). The function is marked //go:nosplit and its comment states the binding constraint: “Entersyscall cannot split the stack: the save must make g->sched refer to the caller’s stack segment, because entersyscall is going to return immediately after.” Nothing it calls may grow the stack either, “because we do not know which of the uintptr arguments are really pointers (back into the stack)” — a moving stack (Goroutine Stacks) during an active syscall would silently corrupt pointer arguments the kernel still holds. To enforce this, reentersyscall sets gp.throwsplit = true and poisons gp.stackguard0 = stackPreempt so any accidental stack-growth attempt crashes loudly.

reentersyscall then:

  1. Increments gp.m.locks to disable preemption — the comment says “during this function g is in Gsyscall status, but can have inconsistent g->sched, do not let GC observe it.”
  2. Records gp.m.syscalltick = gp.m.p.ptr().syscalltick — copying the P’s tick counter so that, on the way out, a mismatch reveals the P was stolen (“Copy the syscalltick over so we can identify if the P got stolen later”).
  3. Saves the P into gp.m.oldp so exitsyscall can prefer to re-acquire the same P for cache locality.
  4. Saves gp.syscallsp, gp.syscallpc, gp.syscallbp so a traceback or the GC can still walk the stack of a goroutine sitting in a syscall, and double-checks they fall within the stack bounds.
  5. If a stop-the-world is already pending (sched.gcwaiting), it short-circuits to the equivalent of entersyscallblock via entersyscallHandleGCWait, giving away the P immediately rather than making the STW wait on this syscall.
  6. Transitions the goroutine: a fast atomic CAS _Grunning → _Gsyscall, falling back to casgstatus only if that CAS fails or the goroutine is inside a synctest “bubble.” The comment marks the danger: “As soon as we switch to _Gsyscall, we are in danger of losing our P. We must not touch it after this point.”
  7. Finally, if sysmon is in deep sleep (sched.sysmonwait), it wakes it via entersyscallWakeSysmon so the monitor resumes its retake duties promptly, then drops gp.m.locks.

Crucially, in the fast path entersyscall does not release the P. The M and P stay loosely associated, and — verified against Go 1.26.3 — the P’s status stays _Prunning; only the goroutine moves to _Gsyscall. This is the gamble: most syscalls (a non-blocking read on ready data, a clock_gettime) return in microseconds, and for those the cheapest thing is to do nothing — keep the P, and reclaim it instantly on return.

exitsyscall — the goroutine comes back

On return, exitsyscall (proc.go) tries to resume cheaply:

  • Fast path. It optimistically transitions _Gsyscall -> _Grunning (again a fast CAS, falling back to casgstatus), clears gp.m.oldp, then checks the single decisive condition: does the M still hold a P, i.e. is gp.m.p.ptr() != nil? If yes, the P was never taken — the goroutine just keeps running on the same P, increments pp.syscalltick, restores its real stack guard, clears throwsplit, and returns. No locks, no scheduler entry. This is the overwhelmingly common case. (The saved m.syscalltick vs pp.syscalltick comparison still appears here, but in Go 1.26 it only chooses which trace events to emit — modelling a “lost P” for the tracer when dropm trashed the tick — not whether to take the fast path.)
  • Slow path. If the M’s P is gone (gp.m.p.ptr() == nil — sysmon took it), exitsyscall calls exitsyscallTryGetP(oldp) on the system stack to grab an idle P, preferring the goroutine’s original oldp for cache locality. If it gets one, it installs it (acquirepNoTrace) and resumes. If no P is free, the goroutine cannot run: it falls through to exitsyscallNoP, transitions to _Grunnable, is placed on a run queue, and the M parks. The goroutine will be picked up later by whatever M next finds it.

entersyscallblock — skipping the gamble

Some syscalls are known to block — a blocking accept, a read on an empty pipe with no deadline, anything the runtime calls deliberately to wait. For these the runtime calls entersyscallblock instead. The difference is decisive: entersyscallblock hands the P off immediately, before the goroutine actually enters _Gsyscall. It bumps gp.m.p.ptr().syscalltick++ itself (so a concurrent sysmon already sees a fresh tick), marks the M as giving up its P with addGSyscallNoP, and its comment lays out the binding ordering — “the order here must be (1) trace, (2) handoff, (3) _Gsyscall switch” — running handoffp(releasep()) on the system stack. By releasing the P up front, entersyscallblock ensures a known-slow syscall never even briefly strands a P; the cost (a handoffp call, possibly starting an M) is paid eagerly because it would have to be paid anyway.

handoffp — giving the P to someone else

handoffp (proc.go) decides what to do with a P that has been freed from a syscall. Its comment states the invariant tying it to work-stealing: “handoffp must start an M in any situation where findRunnable would return a G to run on pp.” The logic, in order: if the P has local run-queue work or global-queue work (!runqempty(pp) || !sched.runq.empty()), or trace work, or GC work, immediately startm(pp, false, false) — wake or create an M to use the P. If there is no work but no spinning/idle M exists to discover future work, start a spinning M anyway (startm(pp, true, false)). If the GC wants the world stopped, park the P in _Pgcstop. There is one more wrinkle: if this is the last running P and nobody is polling the network, it starts an M to keep the poller alive. Otherwise the P genuinely has nothing to do: it goes onto the idle-P list (pidleput), and if it owns a soon-to-fire timer, wakeNetPoller is nudged. The point is that a freed P is never silently lost — it is always either matched to work or explicitly idled.

sysmon and retake — the safety net

The fast path’s gamble loses when a syscall the runtime thought was fast turns out to block for milliseconds. The recovery is sysmon, the runtime’s P-less background monitor thread (see Sysmon System Monitor), which periodically calls retake(now) (proc.go).

retake iterates allp, but immediately skips any P that is not _Prunning — if pp == nil || atomic.Load(&pp.status) != _Prunning { continue }. This is the crux of the modern model: a P with a goroutine in a syscall is still _Prunning (it is owned by the syscalling M), so retake does not filter on a separate syscall status — there is no _Psyscall to filter on. For each running P, retake does two things off a single pass.

Time-slice preemption. Each P carries a sysmontick recording its schedtick. If schedtick is unchanged since the last observation, the same goroutine (or a runnext chain sharing one slice) has run uninterrupted; once that exceeds forcePreemptNS — const forcePreemptNS = 10 * 1000 * 1000, i.e. 10 ms (verified in Go 1.26.3 proc.go) — retake calls preemptone(pp) (see Goroutine Preemption). It also sets a local sysretake = true flag, because as the comment notes, “If pp is in a syscall, preemptone doesn’t work … so we need to take the P ourselves.”

Stealing a P stuck in a syscall. After the preemption check, retake drops allpLock, calls incidlelocked(-1) (to keep the deadlock detector honest while it borrows the P), and then attempts thread, ok := setBlockOnExitSyscall(pp). This is the modern mechanism, confirmed in Go 1.26.3: setBlockOnExitSyscall performs intentionally racy reads, and on success acquires the goroutine’s scan bit, which pins the syscalling thread so it cannot advance out of exitsyscall while the P is being inspected. If the P is not in a syscall (!ok), retake jumps to done and moves on. If it is, retake reads pp.syscalltick: if the tick has advanced since the last observation (!sysretake && int64(pd.syscalltick) != syst), this is a different, fresh syscall — record it, call thread.resume(), and wait at least one more sysmon cycle before acting. This is what the comment means by “Retake the P if it’s there for more than 1 sysmon tick (at least 20us)”: a P must be seen in the same syscall across two consecutive observations before it is stolen, and since sysmon’s minimum poll is 20 µs, the floor on the delay is ~20 µs.

If the same syscall has now persisted across observations, retake applies one restraint: if runqempty(pp) && sched.nmspinning.Load()+sched.npidle.Load() > 0 && pd.syscallwhen+10*1000*1000 > now { thread.resume(); goto done } — i.e. if the P has no local work, there is already a spinning or idle P to absorb future work, and the syscall is younger than 10 ms, leave it alone to avoid pointless thread churn. Otherwise it commits the steal: thread.takeP() detaches the P from the blocked thread, thread.resume() lets the thread continue (it will discover in exitsyscall that gp.m.p is now nil and take the slow path), and handoffp(pp) gives the freed P to a fresh or woken M. The eventual-retake clause exists because a P held in a long syscall keeps npidle != gomaxprocs, which would otherwise prevent sysmon from entering deep sleep.

Code: How a Blocking Read Looks

package main
 
import (
	"fmt"
	"os"
)
 
func main() {
	buf := make([]byte, 64)
	// os.Stdin.Read -> internal/poll.FD.Read -> syscall.Read.
	// For a regular file the syscall package wraps the call in
	// entersyscall()/exitsyscall(): the M may keep its P (fast path) or,
	// if the read blocks long enough, sysmon hands the P to another M.
	n, err := os.Stdin.Read(buf)
	fmt.Printf("read %d bytes, err=%v\n", n, err)
}

For a regular file (or a device that the poller cannot handle), Read ends in a true blocking syscall wrapped by entersyscall/exitsyscall exactly as described above — and if the read stalls, sysmon retakes the P. For a socket or pipe, the path is different: internal/poll first attempts a non-blocking syscall, and if it would block, the goroutine parks on the Network Poller instead of holding an M in a blocking syscall at all. That distinction — file syscalls go through entersyscall, network I/O goes through the poller — is the single most important thing to internalise about Go’s I/O model.

# Watch P/M/syscall transitions:
GODEBUG=schedtrace=1000,scheddetail=1 ./prog
# scheddetail prints each P's numeric status and syscalltick, and each M/G state.
# A goroutine in a syscall shows G status _Gsyscall (3) while its P stays
# _Prunning (1) -- the P is NOT in a distinct syscall status. A handoffp shows
# up as an extra M appearing to run the freed P, plus a bump in the P's
# syscalltick when the original M finally returns through exitsyscall.

Failure Modes and Common Misunderstandings

Thread explosion from many blocking syscalls. If a program has thousands of goroutines all making genuinely blocking file syscalls (not socket I/O — see below), the runtime must back each blocked M with a thread, because the P gets handed off and a new M is started to use it. The result is thousands of OS threads. runtime/debug.SetMaxThreads caps this with a hard crash. The real fix is to bound concurrency (a worker pool, context cancellation) or to use file I/O sparingly. This is the most common production surprise: “my Go server spawned 5,000 threads” almost always means many concurrent blocking file/DNS/cgo calls.

“All I/O goes through the poller.” False. Only file descriptors the OS poller can watch — sockets, pipes, some character devices — use the Network Poller and park the goroutine without an M. Regular file reads/writes on Linux are not pollable in the classic epoll sense and go through blocking entersyscall. So heavy file I/O does cause M growth, while heavy socket I/O does not.

“cgo calls are syscalls.” Mechanically similar but distinct. A cgo call also goes through entersyscall-like machinery (the comment in reentersyscall notes cgo callbacks use it) because, like a syscall, it leaves Go’s control. A long-running C function therefore also triggers retake and P handoff — which is why a cgo call into slow C code costs an M and is a classic source of thread growth. See cgo Performance and Pitfalls.

“A syscall preempts my goroutine.” A blocking syscall is itself a blocked safe-point — the goroutine is descheduled and the GC can scan it freely. But the goroutine inside the syscall cannot respond to a SIGURG async-preemption request, because it is not running Go code. That is exactly why retake exists for syscalls: as the retake comment says, “preemptone doesn’t work” on a P in a syscall, so sysmon must take the P itself.

Alternatives and Design Trade-offs

The competing designs: one M per syscall, always (simple, but thread-thrashing); all syscalls non-blocking + an event loop (Node.js, Rust async — no thread growth, but every blocking primitive must be re-implemented non-blockingly, and CPU-bound work still needs a thread pool); block the worker and accept reduced parallelism (early cooperative-threading libraries). Go’s hybrid — don’t release the P for fast syscalls, do release it for known-blocking ones, and let sysmon retake it for the ones that fooled you — keeps the fast path genuinely free while bounding the worst case. The price is that file/cgo blocking really does spawn threads; the benefit is that programmers write ordinary blocking-style code and the runtime makes it scale. For network I/O specifically, Go does use the event-loop approach via the Network Poller, getting the best of both: blocking-style source code, event-loop-style scalability.

Production Notes

The metrics that matter: runtime.NumGoroutine() and the OS thread count (visible as /proc/<pid>/status Threads: on Linux, or runtime/pprof’s threadcreate profile). A thread count far above GOMAXPROCS signals many goroutines in concurrent blocking syscalls or cgo calls — investigate with the threadcreate profile and the execution tracer’s syscall events. GODEBUG=schedtrace=…,scheddetail=1 shows Ps in syscall state and is the direct way to confirm syscall-induced handoffs. The Go 1.14 async-preemption change (Go 1.14 release notes) also made syscalls more likely to fail with EINTR because SIGURG can interrupt them — robust syscall code must retry on EINTR. The standard advice: prefer socket/pipe I/O (poller-backed, no thread growth) over file I/O on hot paths; bound concurrency around any blocking-file or cgo work; and treat a runaway thread count as a syscall-concurrency problem, not a goroutine-count problem.

See Also

Graph View

Backlinks