vault

GC Assists and Mark Workers

15 min read

GC Assists and Mark Workers

During Go’s concurrent mark phase, the actual work of tracing the heap graph — moving objects from gray to black — is performed by two distinct categories of worker. Mark workers are dedicated background goroutines the scheduler runs, sized to consume about 25% of the CPU so the program keeps the other 75%. GC assists are a throttling mechanism: when the program allocates faster than the mark workers can keep up, the runtime forces the allocating goroutine itself to perform a proportional amount of marking work before its allocation is granted. Mark workers are the steady-state engine of marking; assists are the back-pressure valve that guarantees marking finishes before the heap blows past its goal (Go GC guide; GC pacer redesign).

The Marking Problem They Solve

Go’s collector marks concurrently (see Tricolor Mark and Sweep): while the program runs, the collector traces every reachable object. The mark phase is a race — marking must finish before the program’s continued allocation drives the heap past the heap goal the pacer set. Two things determine the winner: how fast the collector marks, and how fast the program allocates.

If the runtime only ever marked using spare background goroutines, a fast-allocating program could outrun the collector — the heap would sail past its goal and, with [[GOMEMLIMIT and the Soft Memory Limit|GOMEMLIMIT]] set, past the memory limit. If the runtime instead always stopped the program to mark, it would not be a concurrent collector at all. The design answer is two cooperating mechanisms: background mark workers that do the bulk of marking on a CPU budget, and GC assists that make a runaway allocator pay for its own allocation in marking work — coupling allocation rate to marking rate so the program physically cannot outrun the collector.

Mark Workers — The 25% Background Engine

When the mark phase begins, the scheduler arranges for background mark worker goroutines (gcBgMarkWorker, one per P) to run such that, averaged across all Ps, GC marking consumes roughly 25% of GOMAXPROCS. This 25% target is the GC’s CPU budget — the program keeps ~75%. The figure is a named constant in the pacer, gcBackgroundUtilization = 0.25 (mgcpacer.go). (The historical target was 50%; it was lowered to 25% — see Hudson, ISMM keynote — and the Go 1.18 pacer redesign settled on 25% precisely so that, in the steady state, “the target CPU utilization can completely exclude GC assists.”)

25% of an arbitrary GOMAXPROCS is usually not an integer number of Ps — 25% of 6 Ps is 1.5. The runtime achieves a fractional average by running mark workers in three modes, decided per P by the pacer (findRunnableGCWorker), with the worker counts computed once per cycle in gcControllerState.startCycle:

  • Dedicated mode (gcMarkWorkerDedicatedMode, drained via gcDrainMarkWorkerDedicated). The worker takes its P entirely for the duration of the mark phase — that P does nothing but marking. The number of dedicated workers is not a simple floor of 0.25 × GOMAXPROCS as is often stated; the pacer rounds to the nearest integer and then corrects if rounding overshot the budget. Concretely, startCycle computes totalUtilizationGoal := GOMAXPROCS × 0.25, then dedicatedMarkWorkersNeeded := int64(totalUtilizationGoal + 0.5) — round-half-up. It then checks the utilization error, utilError := dedicated/totalUtilizationGoal − 1: if rounding put the result more than maxUtilError = 30% off the goal, and there are too many dedicated workers, it decrements dedicatedMarkWorkersNeeded by one and lets a fractional worker make up the difference. Worked example for 6 Ps: totalUtilizationGoal = 1.5; round-half-up gives int64(2.0) = 2; utilError = 2/1.5 − 1 ≈ +0.33 > 0.30, so it decrements to 1 dedicated worker and enables a fractional worker for the leftover 0.5-of-a-P. The code comment notes this >30%-off case “happens for GOMAXPROCS<=3 or GOMAXPROCS=6.” For, say, 8 Ps the goal is 2.0, rounding gives 2, utilError = 0, so 2 dedicated workers and no fractional worker.
  • Fractional mode (gcMarkWorkerFractionalMode, drained via gcDrainMarkWorkerFractional). When the rounding above left a remainder, one P runs a worker that marks for a measured fraction of its time and runs program goroutines the rest. The pacer sets fractionalUtilizationGoal = (totalUtilizationGoal − dedicatedMarkWorkersNeeded) / GOMAXPROCS (for the 6-P case, (1.5 − 1)/6 ≈ 0.083), and schedules a fractional worker only on a P whose accumulated fractional mark time is behind that per-P goal — so over the cycle the fractional worker contributes exactly the leftover sub-P share. When the rounding came out clean (fractionalUtilizationGoal == 0), there is no fractional worker at all.
  • Idle mode (gcMarkWorkerIdleMode, drained via gcDrainMarkWorkerIdle). When a P would otherwise sit idle — the program is not saturating GOMAXPROCS — the scheduler hands it a mark worker that drains GC work using the otherwise-wasted CPU. Idle workers are pure opportunism: they let a mostly-idle program finish its GC faster than the 25% budget would allow, at zero cost to the program. The GC guide notes that in a largely idle application you will see runtime.gcBgMarkWorker (running gcDrainMarkWorkerIdle) prominently in CPU profiles.

All three are thin wrappers over a single gcDrain routine — distinguished by drain flags (gcDrainUntilPreempt, gcDrainFractional, gcDrainIdle, gcDrainFlushBgCredit) and existing, per their source comment, “to better account mark time in profiles” (mgcmark.go). All three drain GC work from the same shared work queues (work buffers, or workbufs) of gray objects, scanning each gray object’s pointers and shading its white referents — the tricolor mechanic. The dedicated/fractional split is purely about how much CPU marking is allowed to take; idle mode is bonus throughput.

Mental Model

flowchart TD
    Goal["Pacer sets heap goal + assist ratio"]
    Goal --> WorkQ["Shared GC work queues<br/>(gray objects / workbufs)"]
    subgraph Workers["Background mark workers — ~25% of GOMAXPROCS"]
        Ded["Dedicated worker<br/>(whole P; round(0.25×GOMAXPROCS),<br/>corrected if >30% off goal)"]
        Frac["Fractional worker<br/>(covers the remainder)"]
        Idle["Idle worker<br/>(uses otherwise-idle Ps)"]
    end
    Workers --> WorkQ
    subgraph Mutator["User goroutine"]
        A["allocates N bytes"] --> Debt["assist debt += N × assistRatio"]
        Debt --> Check{"debt > 0?"}
        Check -->|"yes"| Assist["gcAssistAlloc:<br/>do marking work to repay debt,<br/>goroutine blocked until repaid"]
        Check -->|"no"| Proceed["allocation proceeds"]
    end
    Assist --> WorkQ

The diagram shows the two worker categories feeding one shared work queue. Background mark workers (top) drain the queue continuously on the GC’s 25% CPU budget. A user goroutine (bottom) that allocates accrues assist debt proportional to bytes allocated times the pacer’s assist ratio; if it is in debt it must itself drain marking work — gcAssistAlloc — before its allocation is granted. The insight: assists make allocation and marking the same currency. A goroutine cannot allocate faster than it is willing to mark, so the program is physically incapable of outrunning the collector.

Mechanical Walk-through — How GC Assists Work

Mark workers handle the steady state. Assists handle the case where the program allocates faster than the 25% budget can mark — they are the pacer’s enforcement arm. The mechanism is a credit/debt system implemented in mgcmark.go (gcAssistAlloc) and accounted in gcControllerState.

The assist ratio. The pacer continuously maintains an assist ratio — bytes of marking work that should be performed per byte allocated — such that if every allocation pays its proportional assist, marking finishes exactly as the heap reaches its goal. The ratio is revised mid-cycle (gcControllerState.revise) as the collector discovers how much scan work the heap actually contains, so it tracks reality rather than the start-of-cycle estimate.

Incurring debt. When a goroutine allocates, the allocator charges its per-G assist debt: debt += bytesAllocated × assistRatio. The debt is the amount of marking work this goroutine now owes.

Repaying debt. Before the allocation is allowed to complete, the runtime checks the goroutine’s debt. If it is in debt, gcAssistAlloc runs: the allocating goroutine itself is put to work draining gray objects from the GC work queues, scanning and shading, until it has performed enough scan work to bring its debt back to (or below) zero. Only then does the allocation proceed. A fast-allocating goroutine therefore spends an increasing share of its time marking — it is throttled in proportion to how much it is straining the collector.

Assist credit. A goroutine can also bank credit. Background mark workers contribute scan work to a global pool, and a goroutine can also do marking speculatively — building assist credit it can later spend to satisfy debt without blocking. This smooths the system: a goroutine that did extra marking earlier need not stall on its next allocation. The accounting is a global “background scan credit” pool plus per-G local credit.

The white-allocation rule. Objects allocated during the mark phase are colored black (or at least not white) immediately, so freshly allocated memory is never mistakenly swept in the cycle it was born in — but it also is not scanned for free, hence the debt.

The consequence is an elegant invariant: the program’s allocation rate and the collector’s marking rate are mechanically coupled. The harder the program allocates, the more of its own CPU it is forced to spend marking, until the two rates balance. The program cannot outrun the collector, because the act of outrunning it is exactly what triggers the throttle.

Code Example — Seeing Assists in a Profile

There is no API to “do an assist” — assists are transparent. You observe them. A CPU profile of an allocation-heavy program under GC pressure shows the assist path:

(pprof) top
flat  flat%   cum   cum%
...           38.1%  runtime.gcAssistAlloc
...           24.7%  runtime.gcDrainMarkWorkerDedicated
...            6.2%  runtime.gcDrainMarkWorkerIdle

Reading it: runtime.gcDrainMarkWorkerDedicated (~25%) is the healthy background mark engine — expected, that is the GC’s budget. runtime.gcAssistAlloc at 38% is the alarm: more than a third of CPU time is allocating goroutines being forced to mark because the program out-allocates the collector. The GC guide’s rule of thumb: cumulative gcAssistAlloc above 5% means the application is “out-pacing the GC with respect to how fast it’s allocating.” gcDrainMarkWorkerIdle appearing means the program also has idle Ps the GC is opportunistically using.

A GODEBUG=gctrace=1 line exposes the same split in its CPU triple:

gc 9 @1.20s 6%: 0.03+5.4+0.02 ms clock, 0.12+2.1/3.8/0.4+0.08 ms cpu, ...

In the cpu triple 0.12+2.1/3.8/0.4+0.08 ms, the middle group 2.1/3.8/0.4 is assist time / (dedicated + fractional) mark-worker time / idle mark-worker time for the concurrent mark phase — the three /-separated GC-CPU components bracketed by the two +-joined STW figures (mark-start 0.12 and mark-termination 0.08). The exact order is fixed in the runtime’s trace formatter, which prints, in sequence, stwprocs × (tMark − tSweepTerm), assistTime, dedicatedMarkTime + fractionalMarkTime, idleMarkTime, and stwprocs × (tEnd − tMarkTerm), joining the middle three with / (mgc.go). Note the grouping: fractional time is summed with dedicated time, not with idle — a common misreading. A large first number (2.1) relative to the second (3.8) means assists are carrying an outsized share of marking — the program is allocating too fast for its GOGC setting.

Failure Modes and Common Misunderstandings

High assist time is the canonical “GC is slow” symptom. It is not STW, and it does not appear in PauseNs (see Stop-the-World Pauses). It shows as latency on allocation — a goroutine that wanted to allocate is instead marking. P99 latency rises with no STW spike. Diagnose with CPU profiles, not pause metrics.

Assists are a feature, not a bug. Without them, a fast allocator would overshoot the heap goal — or, with [[GOMEMLIMIT and the Soft Memory Limit|GOMEMLIMIT]], the memory limit — and risk OOM. Assists are correctness under load: they guarantee marking completes. The remedy for high assist time is not to “disable assists” (impossible) but to allocate less or raise GOGC.

“Mark workers always use exactly 25%.” They target 25% on average. Idle workers can push GC CPU above 25% when the program leaves Ps idle (free CPU, so harmless). Under [[GOMEMLIMIT and the Soft Memory Limit|GOMEMLIMIT]] pressure the total GC CPU cap rises to 50%. The 25% is the steady-state design point for dedicated+fractional workers, not a hard ceiling.

Assist credit can cause a goroutine to do “too much” marking. A goroutine that allocates a huge object can incur large debt at once and be forced into a long assist; banked credit and the smoothing in the pacer redesign (assists paced against the extrapolated ratio, not the hard goal) mitigate but do not eliminate occasional long assists on outsized allocations.

A well-paced program shows near-zero assists. The Go 1.18 pacer redesign explicitly aimed to eliminate assists in the steady state — its stated goal that “the target CPU utilization can completely exclude GC assists in the steady-state” and that “steady-state mark assist drops to zero if not allocating too heavily” (pacer redesign). Sustained assist time means the program is not in steady state — the allocation rate genuinely exceeds what the 25% budget can mark.

Green Tea GC — Same Workers and Assists, a New Scan Kernel

As of Go 1.26 (February 2026), the Green Tea collector is the default garbage collector, with an opt-out of GOEXPERIMENT=nogreenteagc at build time (the opt-out is expected to be removed in Go 1.27) (Go 1.26 release notes; Green Tea blog). It is important to be precise about what Green Tea does and does not change for this note. Green Tea redesigns the scan kernel — the inner loop that the mark workers and assists execute — but it does not change the worker model, the 25% CPU budget, the pacer, the assist credit/debt mechanism, or [[GOMEMLIMIT and the Soft Memory Limit|GOMEMLIMIT]] behavior described above. There are still background mark workers running on the 25% budget and still mutator assists charging allocation in marking work; what changed is how a unit of marking work is performed.

The classic tricolor marker traces the heap object-by-object in a depth-first, stack-like order, “jumping all over memory doing tiny bits of work in each place” — terrible for CPU caches and hard to vectorize. Green Tea instead works page-by-page: “Instead of scanning objects we scan whole pages” and “instead of tracking objects on our work list, we track whole pages,” in a first-in-first-out (queue-like) order, letting “seen” objects accumulate on a page before scanning so each page is processed in fewer, longer, left-to-right passes (Green Tea blog). On x86 CPUs with AVX-512 (Intel Ice Lake / AMD Zen 4 and newer) it uses vector instructions — notably VGF2P8AFFINEQB — to scan an entire page’s worth of mark metadata in a handful of straight-line instructions. The net effect quoted by the Go team is a 10–40% reduction in GC overhead in GC-heavy programs, plus a further ~10% from vectorization on newer amd64 hardware. For the purposes of this note: the symptoms, profiles, and tuning levers (high gcAssistAlloc → allocate less or raise GOGC) are unchanged; Green Tea simply makes each marking second go further, which reduces the pressure that triggers assists rather than altering the assist machinery itself.

Alternatives and When They Apply

There is no Go knob that turns assists or mark workers on or off — they are intrinsic to the concurrent collector. The levers are indirect. Raising [[GC Pacing and GOGC|GOGC]] gives the collector more runway (a higher heap goal), so the mark phase has more time, the assist ratio drops, and assists fade — at the cost of a larger heap. Lowering the allocation rate is the more fundamental fix: keep short-lived objects on the stack (see Escape Analysis), reuse buffers with sync.Pool Internals, and avoid pointer-dense structures that inflate scan work. Adding CPU cores raises GOMAXPROCS and thus the absolute size of the 25% budget. Under a memory limit, when even maximal assisting cannot keep the heap down, the runtime caps GC CPU at 50% and lets the limit be breached rather than thrash — assists are bounded, not unbounded. Across runtimes, the JVM’s concurrent collectors use analogous mutator-assist concepts (e.g. G1’s and Shenandoah’s pacing of allocating threads); the idea of charging allocation in marking work is not unique to Go, but Go’s tight coupling to the pacer’s assist ratio is a clean realization of it.

Production Notes

In a healthy Go service the CPU profile shows gcBgMarkWorker / gcDrainMarkWorkerDedicated near 25% during collections and gcAssistAlloc near zero — the pacer’s steady-state target. The reliable production signal of GC trouble is rising gcAssistAlloc: it means allocation has outrun the collector, and it manifests as allocation-latency tail growth, not STW spikes. Real incident playbooks for Go services treat sustained assist time as the trigger to either raise GOGC (if memory headroom exists) or hunt the allocation hotspot with a heap/alloc profile. The idle-worker mode is a quiet performance win that needs no tuning — a service with spare capacity finishes its collections faster for free. The interaction to remember: [[GOMEMLIMIT and the Soft Memory Limit|GOMEMLIMIT]] under pressure deliberately intensifies assists (collecting harder to stay under the limit), so a too-tight memory limit shows up as high assist CPU — the same symptom as a too-low GOGC, and worth distinguishing via runtime/metrics (/gc/heap/goal:bytes vs the memory limit). See GC Tuning and Observability.

See Also

Graph View

Backlinks