vault

Profile-Guided Optimization

17 min read

Profile-Guided Optimization

Profile-Guided Optimization (PGO), also called Feedback-Directed Optimization (FDO), is a compiler technique that feeds “information (a profile) from representative runs of the application back into to the compiler for the next build of the application, which uses that information to make more informed optimization decisions” (Go PGO user guide). Without a profile, the Go compiler can reason only about the source code of a package; with a CPU profile collected from production, it learns which functions are hot and which indirect calls land on which concrete callees, and uses that to make better decisions. PGO shipped as a preview in Go 1.20 and became “ready for general production use” in Go 1.21, where -pgo=auto also became the default (Go 1.21 PGO blog post). In Go 1.21 it delivered “between 2% and 7% CPU usage improvements”; as of Go 1.22 the documented range is “around 2-14%” across a representative set of programs (Go PGO docs).

Mental Model

The ordinary Go compiler is static: it sees a function, estimates its cost with a fixed-budget heuristic, and decides — once, for all callers — whether to inline it. It has no idea whether a given branch runs a billion times or never. PGO breaks that limitation by handing the compiler a dynamic artifact: a CPU profile, the same pprof-format file you would open in go tool pprof. The profile is a statistical sample of the program’s call stacks gathered while it served real traffic. From it the compiler builds a weighted call graph — every edge annotated with how many samples flowed across it — and a per-callsite breakdown of which concrete callees showed up behind interface variables and function-value variables.

The right way to think about PGO is the profile is an input to the build, exactly like a source file. The recommended workflow makes this literal: you commit a file named default.pgo into your main package directory, and from then on every go build of that program is profile-guided, reproducibly, for anyone who clones the repo (PGO docs — “Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds”).

The mechanism is a special case of AutoFDO — the sampling-profiler-driven branch of feedback-directed optimization pioneered for GCC at Google. The contrast is instrumentation-based PGO: a special build that inserts a counter at every branch, runs it to gather exact counts, and rebuilds. Instrumentation is exact but requires a separate, slower binary and a representative workload run offline. Sampling is statistically noisier but uses the profiler that is already in every Go binary and can be collected from live production. Go deliberately chose only the sampling route.

flowchart LR
    A[Build v1<br/>no PGO] --> B[Deploy to production]
    B --> C["Collect 30s CPU profiles<br/>(net/http/pprof or<br/>runtime/pprof) from<br/>many instances"]
    C --> M["go tool pprof -proto<br/>merge profiles"]
    M --> D["Commit as<br/>main pkg/default.pgo"]
    D --> E["go build v2<br/>-pgo=auto detects it"]
    E --> F["Compiler builds weighted<br/>call graph; hot-callsite<br/>inlining + devirtualization"]
    F --> G[Deploy v2]
    G -.->|collect again| C

Figure: the PGO feedback loop. The key insight is that it is a cycle — each release’s profile feeds the next build, and Go’s PGO is deliberately designed to be robust to the inevitable skew between the profiled version and the version being built, so the loop converges rather than oscillating.

Mechanical Walk-through

Profile format and ingestion. PGO consumes a standard CPU pprof profile — the kind produced by runtime/pprof or exposed by net/http/pprof at /debug/pprof/profile (PGO docs). The compiler does not need an instrumented build; it uses the sampling profiler already built into every Go binary. The profile must satisfy concrete structural requirements that the user guide spells out: it must contain a sample index of type/unit samples/count or cpu/nanoseconds; it must be symbolized (the Function.name field set); it must contain stack frames for inlined functions — “If inlined functions are omitted, Go will not be able to maintain iterative stability”; and crucially each Function record must carry Function.start_line, “the line number of the start of the function. i.e., the line containing the func keyword” (PGO docs). Profiles produced by the Go runtime satisfy all of this automatically — the requirement matters only if you transform profiles with external tooling.

The IR call graph. When go build runs with a profile, the toolchain parses it once. The cmd/compile/internal/pgoir package then associates the profile with the intermediate representation of the package being compiled, building an IRGraph: “a call graph with nodes pointing to IRs of functions and edges carrying weights and callsite information” (pgoir/irgraph.go). Nodes for indirect calls may have a nil AST when the callee is in another package and not visible.

Line-offset matching, not exact match. Between collecting the profile and building, source drifts — you fixed a bug, added a line. PGO does not require an exact match: it identifies a callsite by its line offset within the enclosing function (the call on the 5th line of foo), not by absolute line number. The implementation comment in irgraph.go is explicit that it works in “binary-visible relative line number” space precisely because “pprof profiles generated by the runtime always contain line numbers as adjusted by //line directives.” The consequence, per the user guide: changes outside a hot function, or moving a function to another file in the same package, do not break matching (the compiler ignores filenames); changes within a hot function, renaming it, or moving it to another package may break matching because they shift line offsets or the symbol name. When matching fails, “this is a graceful degradation. A single function failing to match may lose out on optimization opportunities, but overall PGO benefit is usually spread across many functions” (PGO docs).

Program-wide scope. A critical mechanical fact: “PGO in Go applies to the entire program. All packages are rebuilt to consider potential profile-guided optimizations, including standard library packages” and dependency packages (PGO docs). This is why enabling PGO invalidates the build cache for the whole dependency graph the first time — the profile is a new input to every package, so every package’s cache key changes. It also means “the unique way your application uses a dependency impacts the optimizations applied to that dependency.”

Optimization 1 — hot-callsite inlining. The compiler’s normal inliner works against a fixed cost budget: inlineMaxBudget = 80 “hairiness” units; a function whose estimated cost exceeds 80 is not inlined (inline/inl.go). PGO raises this dramatically at hot callsites. The mechanism, traced in inl.go: PGOInlinePrologue walks the profile’s edges sorted by weight and computes a cumulative distribution function — it marks as “hot” the smallest set of callsites whose summed weight reaches the top inlineCDFHotCallSiteThresholdPercent, which is 99. So a callsite is “hot” if it is in the heaviest 99% of total edge weight. For any callee reached by a hot edge, the inline budget is raised from 80 to inlineHotMaxBudget = 2000 — a 25× increase. Functions normally far too large to inline get inlined where the profile says it matters.

The Go 1.21 blog walks a concrete case: mdurl.Parse() exceeded the default budget, but the profile flagged it as hot, so it was inlined. Inlining then unlocked a downstream win — once Parse was spliced into normalizeLink, escape analysis could see that the local URL value did not escape the merged caller, so it moved from the heap to the stack. The blog reports the result precisely: inlining mdurl.Parse eliminated 4,974,135 allocations (100% of that site’s allocations) and inlining mdurl.(*URL).String eliminated 4,249,044 more, cascading into reduced GC pressure.

Optimization 2 — conditional devirtualization (interface and function-value calls). An indirect call — an interface method call (r.Read(b) where r is an io.Reader) or a function-value call (fn()) — cannot be inlined because the compiler does not know the target. If the profile shows one concrete callee dominates a given indirect callsite, PGO rewrites it into a type-checked (or pointer-checked) fast path. The package doc-comment in devirtualize/pgo.go shows both forms exactly. For an interface call:

// Source
func foo(i Iface) { i.Foo() }
 
// After PGO devirtualization
func foo(i Iface) {
    if c, ok := i.(Concrete); ok {
        c.Foo()        // direct call — now inlinable
    } else {
        i.Foo()        // fallback indirect call preserves correctness
    }
}

For a function-value call the check compares program counters via internal/abi.FuncPCABIInternal:

func foo(fn func()) {
    if internal/abi.FuncPCABIInternal(fn) == internal/abi.FuncPCABIInternal(Concrete) {
        Concrete()     // direct call
    } else {
        fn()           // fallback
    }
}

“The primary benefit of this transformation is enabling inlining of the direct call” (devirtualize/pgo.go). The fallback branch matters: “a profile is not a guarantee that this will always be the case” (PGO blog) — the else keeps the program correct when the callee is something else. This is why it is conditional (or speculative) devirtualization. Go 1.22 made this materially better: “the compiler now interleaves devirtualization and inlining” so a devirtualized direct call is then offered to the inliner in the same pass, and “PGO builds can now devirtualize a higher proportion of calls than previously possible” (Go 1.22 release notes).

Optimization 3 — hot-block alignment (amd64/386). Since Go 1.23, “the compiler will use information from PGO to align certain hot blocks in loops. This improves performance an additional 1-1.5% at a cost of an additional 0.1% text and binary size” (Go 1.23 release notes). Aligning the entry of a hot loop body to a cache-line / instruction-fetch boundary reduces front-end stalls. It is implemented only on 386 and amd64 “because it has not shown an improvement on other platforms,” and can be turned off with -gcflags=-d=alignhot=0.

These are the PGO-driven optimizations confirmed for Go 1.21–1.26: hot-callsite inlining (1.21), interleaved devirtualization of interface and function-value calls (1.21, improved 1.22), and PGO-guided hot-block alignment (1.23). The Go 1.24, 1.25, and 1.26 release-note Compiler sections add no further PGO-specific optimizations, though the user guide notes “We expect performance gains to generally increase over time as additional optimizations take advantage of PGO.”

Configuration and Usage

The mechanism is controlled by one flag, -pgo, passed to go build:

# 1. Build and deploy an initial (non-PGO) binary.
go build -o markdown ./...
 
# 2. Collect a 30-second CPU profile from the running production service.
curl -o cpu.pprof "http://localhost:8080/debug/pprof/profile?seconds=30"
 
# 3. Place the profile where the toolchain looks for it: the main package dir.
mv cpu.pprof default.pgo
 
# 4. Rebuild — PGO is now on automatically.
go build -o markdown ./...
 
# Verify PGO was applied:
go version -m markdown   # prints  build  -pgo=/abs/path/default.pgo

Line-by-line: step 2 uses the net/http/pprof endpoint — importing _ "net/http/pprof" registers /debug/pprof/ handlers; the seconds=30 query parameter sets the sampling window. Step 3 is the convention that does the work: go build “will detect default.pgo files automatically and enable PGO” when the file sits in the main package directory. Step 5 confirms it — go version -m dumps build settings, and the -pgo line is the receipt; the Go 1.21 blog shows exactly this output: build -pgo=/tmp/pgo121/default.pgo.

The flag’s explicit forms:

go build -pgo=auto              # default since Go 1.21: find default.pgo
go build -pgo=off               # disable PGO entirely
go build -pgo=/path/to/p.pprof  # use a specific profile, ignoring default.pgo

“Before Go 1.21, the default is -pgo=off” — in the 1.20 preview you had to opt in explicitly (PGO docs). A path passed to -pgo “applies to all main packages,” so a multi-binary repository typically needs a separate go build invocation per binary.

Merging profiles. A production service runs on many machines; one profile from one instance is narrow. Merge several with pprof:

go tool pprof -proto a.pprof b.pprof c.pprof > merged.pprof

The -proto flag writes the merged result in protobuf profile format, so the output is itself a valid default.pgo. One subtlety the user guide flags: “This merge is effectively a straightforward sum of samples in the input, regardless of wall duration.” If profiles have different durations, the longer one is silently weighted more — so collect every profile for the same window (e.g. all 30s) before merging.

Failure Modes and Common Misunderstandings

“My microbenchmark shows no PGO gain.” Expected. “Microbenchmarks are usually bad candidates for PGO profiling, as they only exercise a small part of the application, which yields small gains when applied to the whole program” (PGO docs). PGO optimizes the whole-program hot set; a microbenchmark profile teaches the compiler about a sliver.

Unrepresentative profiles do nothing — but should not regress. “Using an unrepresentative profile is likely to result in a binary with little to no improvement in production.” A natural fear is that a bad profile makes the program slower; the user guide answers directly: “It should not. While a profile that is not representative of production behavior will result in optimizations in cold parts of the application, it should not make hot parts of the application slower.” The downside of a bad profile is wasted optimization budget, not regression — but the official advice remains blunt: “collecting profiles directly from the production environment is recommended.”

New code is invisible to the first build. “When adding new code or enabling new code paths with a flag flip, that code will not be present in the profile on the first build, and thus won’t receive PGO optimizations until a new profile reflecting the new code is collected” (PGO docs). PGO always optimizes last release’s hot set; new code catches up one cycle later.

Build-time and binary-size cost. The first PGO build “requires a rebuild of every package in the dependency graph,” because the profile is a new input invalidating every cache entry; subsequent incremental builds with the same profile are cached normally. Early on this was severe — “Previously, large builds could see 100%+ build time increase from enabling PGO” — but Go 1.23 reduced it to “single digit percentages” (Go 1.23 release notes). Binary size also grows slightly: “PGO can result in slightly larger binaries due to additional function inlining” — inlining duplicates code at each callsite.

The iterative-stability worry. If release N is built with release N−1’s profile, and is itself PGO-optimized, will feeding its profile into N+1 cause drift or oscillation? Go’s PGO is explicitly designed against this — it is “generally robust to skew between the profiled version of an application and the version building with the profile, as well as to building with profiles collected from already-optimized binaries.” The user guide notes “The Go compiler takes a conservative approach to PGO optimizations, which we believe prevents significant variance.” This is also why the profile-format requirement to include inlined-function frames exists: without them “Go will not be able to maintain iterative stability.” The loop converges; it does not spiral.

Wrong workload mix in a multi-tenant service. A service that serves several distinct workloads has several hot sets. The guide offers three options: build a separate binary per workload; profile only “the most important” workload; or merge profiles across workloads weighted by their production footprint. Picking one workload’s profile silently de-optimizes the others.

Alternatives and When to Choose Them

PGO is one lever among the compiler’s optimizations and is complementary to, not a replacement for, the static ones — Escape Analysis, Function Inlining, Bounds Check Elimination, Devirtualization. Those run on every build with no profile; PGO sharpens inlining and devirtualization specifically at hot callsites and adds hot-block alignment. Note that Go 1.26 also improved static devirtualization independently of PGO via better concrete-type analysis (go126ImprovedConcreteTypeAnalysis in devirtualize/devirtualize.go) — so an interface call whose concrete type is provable statically gets devirtualized with no profile at all; PGO covers the cases where the type is only probable.

Compared to other toolchains: GCC and Clang/LLVM offer both instrumentation-based PGO and sampling-based AutoFDO. Go deliberately implemented only sampling, reusing the built-in CPU profiler so there is no separate instrumented binary to build and deploy. The trade-off is statistical noise versus operational simplicity — collect from prod with the same pprof you already use — which Go’s “just works” philosophy treats as decisive. A heavier alternative is post-link optimization such as LLVM BOLT, which rewrites the linked binary using profile data (basic-block reordering, function splitting) — Cloudflare explicitly names exploring BOLT as future work on top of PGO. There is also a separate, preview-stage GOEXPERIMENT=newinliner (tracked in issue #61502) — a call-site-scoring inliner that, as of Go 1.26, is still a non-default experiment, not the PGO mechanism itself.

When not to bother: a CPU-bound program where one function already dominates is better served by hand-optimizing that function; a program that is I/O-bound or GC-bound will see little from PGO since its CPU profile is dominated by waiting or garbage collection — tune GC Pacing and GOGC or the GMP Scheduler Model instead.

Production Notes

The canonical first figure is from the Go 1.21 blog: a commonmark Markdown-to-HTML service, profiled under load and rebuilt with default.pgo, ran ~3.8% faster (Load-12 374.5µs → 360.2µs, p=0.000, n=40), with differential profiling over 300k requests confirming ~3% less total CPU.

Two well-documented production deployments corroborate the 2–5% band. Cloudflare applied PGO to wshim, a telemetry push gateway on every edge server consuming over 3,000 cores globally (Cloudflare engineering blog). Their workflow is a good template: query Thanos for the top CPU consumers, use fetch-pprof to pull 30-second profiles from ~1,000 instances, merge with go tool pprof -proto, commit the profile, and add -pgo to the Makefile. The measured saving was ~71 CPU cores on a 7-day average (3,067.83 → 2,996.78 cores, a 3.5% reduction) “without any code changes.” Their hard-won lesson: “selecting exactly which servers to profile is paramount” — they deliberately profiled datacenters under heavy peak load rather than sampling randomly, and used PromQL’s offset operator to control for time-of-day confounders when measuring the win. DoltHub tested PGO on the Dolt database under its SQL benchmark suite and saw “about a 5% decrease in read latency and a 1.5% decrease in write latency”; they subsequently shipped PGO builds by default from v1.32.4 onward, automating profile collection through GitHub Actions (DoltHub blog).

Operational guidance distilled from the docs and these write-ups: (1) commit default.pgo to version control so builds are reproducible and every developer’s local build matches CI; (2) refresh the profile periodically — a profile that lags the code by many releases optimizes an increasingly stale hot set; (3) merge profiles from multiple production instances and multiple time windows to capture the real traffic mix, deliberately choosing high-load instances; (4) keep all profiles the same duration before merging, since the merge sums samples; (5) treat the profile like a dependency — review changes to it, because a bad profile silently produces a less-optimized binary with no compile error; (6) accept that the first build with a new profile rebuilds the whole dependency graph, but Go 1.23+ keeps the overhead in single-digit percentages.

See Also

Graph View

Backlinks