vault

Burn-Rate Alerting

16 min read

Burn-Rate Alerting

Burn-rate alerting is the practice of paging on how fast a service is consuming its error budget rather than on a raw error count, a static resource threshold, or a bare comparison against the SLO target. The burn rate is a single dimensionless number that says how many times faster than “sustainable” the budget is draining: a burn rate of 1 exactly exhausts the whole budget over the SLO’s compliance window, a burn rate of 2 exhausts it in half that time, and a burn rate of 1,000 empties a 30-day budget in about 43 minutes (SRE Workbook, ch. 5, “Alerting on SLOs”). Framing alerts this way ties every page directly to user-visible impact measured against a deliberate reliability target, and it lets a single alert scale its urgency to the severity of the incident: the worse the outage, the faster the budget burns, the sooner it fires. This note develops the burn-rate concept and the arithmetic that connects it to time-to-exhaustion and alert thresholds; the production-grade final recipe built on top of it is Multi-Window Multi-Burn-Rate Alerts.

Burn-rate alerting is the fourth of six successive refinements the Site Reliability Engineering Workbook (Google, 2018) walks through in Chapter 5 while searching for an alerting scheme that is simultaneously precise, sensitive, quick to detect, and quick to reset. Understanding why the earlier three attempts fail is the fastest route to understanding what burn rate buys you, so this note traces the whole progression and stops at the doorstep of the final answer.

Mental Model

The core idea is to convert the error budget — a static pool of allowable failures — into a flow problem. You have a fixed budget for the compliance window (say 0.1% of all requests over 30 days). At any moment errors are draining that pool at some rate. The burn rate is that drain rate expressed as a multiple of the “even” rate that would empty the pool exactly at the window’s end.

flowchart LR
    subgraph budget["Error budget for the 30-day window"]
        POOL[(0.1% of all<br/>requests may fail)]
    end
    ERR["Current error rate<br/>(observed over a lookback window)"] -->|"drains"| POOL
    ERR --> CALC{"burn rate =<br/>error rate ÷ budget rate"}
    CALC -->|"= 1 → lasts 30 days"| SLOW["sustainable:<br/>no alert"]
    CALC -->|"= 14.4 → lasts ~50 h"| FAST["fast burn:<br/>page"]
    CALC -->|"= 1000 → lasts 43 min"| CRIT["catastrophic:<br/>page immediately"]

What it shows and the insight to take: the error budget is a tank; observed errors are the outflow; burn rate is the outflow measured against the “drain evenly over the whole window” rate. A burn rate of 1 is the break-even flow — spend it all, exactly on schedule. Anything above 1 is over-spending; the number is the acceleration factor. The single insight that makes burn-rate alerting work is that this ratio is self-scaling: you do not need one alert for “small problem” and another for “the site is down” — a bigger outage simply produces a bigger burn rate on the same metric, so one threshold on burn rate detects a total outage in seconds and a mild degradation over hours, automatically.

What Burn Rate Is — The Arithmetic

Start from the error budget itself. If the SLO is a success-ratio target — for instance 99.9% of valid requests must succeed over a 30-day window — then the budget is the complement, 1 − SLO = 0.1%. Over a four-week period serving 3 million requests, that is a budget of 3,000 permitted errors (SRE Workbook, “Implementing SLOs”). The budget is a rate at which the SLO may be missed (SRE Book, “Service Level Objectives”).

Burn rate is defined as how fast, relative to the SLO, the service is consuming that budget (SRE Workbook, ch. 5). Concretely, for a success-ratio SLO:

burn rate  =  observed error rate  ÷  error budget  =  observed error rate ÷ (1 − SLO)

Walking the symbols: the observed error rate is the fraction of valid events that failed, measured over some recent lookback window (10 minutes, 1 hour, whatever the alert uses). The error budget (1 − SLO) is the fraction you are allowed to fail. Dividing one by the other normalizes the observed failure against your allowance. For a 99.9% SLO the budget is 0.001, so an observed error rate of 0.1% (0.001) gives burn rate 0.001 / 0.001 = 1; an observed rate of 1% gives 0.01 / 0.001 = 10; a full outage at 100% errors gives 1.0 / 0.001 = 1000.

The reason burn rate 1 is the meaningful pivot is that it is the rate that would consume the entire budget over exactly the SLO’s compliance window. That gives the second key relationship — time to exhaustion:

time to exhaust the budget  =  SLO compliance window  ÷  burn rate

For a 30-day window this yields the canonical table from the Workbook:

Burn rateError rate (99.9% SLO)Time to exhaust the 30-day budget
10.1%30 days
20.2%15 days
101%3 days
1,000100%~43 minutes

(30 days ÷ 1,000 = 0.03 days ≈ 43 minutes.) Read the table as: at burn rate 10 you would blow a whole month’s budget in three days if it continued; at burn rate 1,000 (everything failing) you have less than an hour before the budget is gone.

The third relationship is the one you actually design alerts around — how much budget a given burn rate consumes over a given alerting window:

budget consumed  =  burn rate  ×  (alerting window ÷ SLO compliance window)

This is where the famous constant 14.4 comes from. Suppose you want an alert to fire when a 1-hour window has burned 2% of a 30-day budget. Solve for the burn rate: budget consumed = 0.02, window ratio = 1h / 720h = 1/720, so burn rate = 0.02 ÷ (1/720) = 0.02 × 720 = 14.4. The same arithmetic gives the other standard tiers: consuming 5% over 6 hours needs burn rate 0.05 × (720/6) = 6; consuming 10% over 3 days needs burn rate 0.10 × (720/72) = 1. These three pairs — 14.4×/1h, 6×/6h, 1×/3d — recur throughout the Workbook’s recommendation and in Multi-Window Multi-Burn-Rate Alerts.

Why Alert on Burn Rate and Not Raw Errors or Thresholds

Burn-rate alerting is a specific answer to a general question the SRE tradition insists on: page on user-visible symptoms measured against a target, never on internal causes or arbitrary numbers (Symptom-Based versus Cause-Based Alerting, and Rob Ewaschuk’s alerting philosophy in the SRE Book, “Monitoring Distributed Systems”). Three properties make burn rate the right symptom to alert on:

  • It is normalized to the thing you actually promised. A raw error count of “500 errors/minute” means nothing without knowing traffic and target. Burn rate folds both the SLO and current traffic into one number, so the same threshold is correct for a service doing 10 requests/second and one doing 100,000. This is the deeper contrast developed in SLO-Based Alerting versus Threshold Alerting: “CPU > 80%” or “errors > 500” are cause- or magnitude-based guesses; “burning budget 14× too fast” is impact measured against a commitment.
  • It self-scales with severity. Because burn rate rises with the error rate, one alert threshold covers everything from a slow leak to a total outage — a hard outage produces a burn rate near 1,000 and trips even a very short window almost instantly, while a subtle 0.3% regression trips only a longer, more patient window. You do not maintain separate “warning” and “critical” thresholds on the same metric; the math does the scaling.
  • It bounds worst-case budget loss, which is the quantity you care about. Designing the alert around “how much of the budget could burn before I am guaranteed to be paged” is exactly the SRE question. A threshold alert can let you quietly bleed the whole budget below its trip point; a burn-rate alert is defined in terms of budget spent, so its recall (its guarantee of catching real budget threats) is expressible and tunable.

The Six-Iteration Journey to a Good Alert

Chapter 5 of the Workbook does not present burn rate as an axiom; it derives it by fixing the flaws of simpler schemes one at a time, judged against four qualities every alert has:

  • Precision — of the events that fire the alert, the fraction that were genuinely worth paging on. Low precision means false alarms and alert fatigue.
  • Recall — of the events that genuinely threatened the budget, the fraction the alert caught. Low recall means missed outages.
  • Detection time — how long after the problem starts the alert fires. Long detection burns more budget before anyone responds.
  • Reset time — how long the alert keeps firing after the problem has stopped. Long reset means a resolved incident keeps paging.

Iteration 1 — alert when the recent error rate crosses the SLO threshold. Take a short window (10 minutes) and page when the error ratio is at or above the budget, e.g. rate10m > 0.001. Detection is superb — a total outage trips it in well under a second of real error data — and recall is good, but precision is terrible. A brief spike that consumes a trivial slice of the monthly budget still pages you; the Workbook notes such a rule could fire up to 144 times a day while the service is comfortably inside its SLO. Paging on a momentary threshold crossing has no notion of how much budget is actually at stake.

Iteration 2 — lengthen the window. Widen the lookback to 36 hours so the alert only fires on sustained badness (specifically, errors consuming ~5% of the 30-day budget). Precision improves sharply because only real, prolonged problems clear the bar, and detection is still acceptable (a total outage trips it in about 2 minutes 10 seconds). But reset time is now dreadful: a long averaging window keeps the alert lit for many hours after the incident is over, because the window still contains the bad data. You also pay in data-storage cost to keep 36 hours of high-resolution history. A resolved outage that keeps paging is its own reliability hazard — responders learn to ignore the pager.

Iteration 3 — add a “for” duration. Keep a short 1-minute window but require the condition to hold continuously for an hour (for: 1h) before firing. This is the classic Prometheus pattern. It improves precision, but the Workbook rejects it for two reasons. Recall collapses: the required duration does not scale with severity, so a 100%-error total outage and a mild 0.2% degradation both must wait the full hour, meaning you lose an hour of budget on a catastrophe. And it is fragile: any momentary dip below the threshold — even one scrape — resets the duration timer to zero, so a flapping incident may never accumulate a continuous hour and never fire at all. Detection time is poor by construction. The Workbook explicitly does not recommend this approach.

Iteration 4 — alert on burn rate. This is the pivot this note is about. Instead of comparing the error rate to the raw budget, compare it to a multiple of the budget chosen so that firing corresponds to a meaningful fraction of budget spent. A 1-hour window with a 5%-budget target is a burn-rate-36 alert: rate1h > 36 × 0.001. Precision is good (you only fire when a real, budget-threatening burn is underway), detection time is good, and — crucially — reset time is good (about 58 minutes, bounded by the 1-hour window, versus 36 hours in iteration 2). The remaining weakness is recall: a single fixed burn-rate threshold has a blind spot. A burn rate of 35× would exhaust the budget in roughly 20.5 hours — a genuine emergency — yet never trips a ”> 36×” alert. One threshold cannot be both sensitive to slow burns and specific about fast ones.

Iteration 5 — multiple burn-rate alerts. Fix the blind spot with a ladder of burn-rate thresholds, each on its own window, routed to different urgencies. The Workbook’s recommended tiers for a 99.9% SLO are:

Budget consumedLong windowBurn rateAction
2%1 hour14.4Page
5%6 hours6Page
10%3 days1Ticket

A fast, severe burn trips the 14.4×/1h page; a moderate sustained burn trips the 6×/6h page; a slow leak that will still eventually blow the budget files a low-urgency ticket rather than waking someone. This gives good precision and good recall. Two problems remain. First, longer windows still carry long reset times (the 3-day window keeps a ticket-level alert warm for days). Second, when a bad incident is underway, the fast and slow alerts fire together, so you need alert suppression / inhibition to avoid duplicate notifications — a lower-severity alert should be silenced while a higher-severity one for the same cause is active.

Iteration 6 — multiwindow, multi-burn-rate. The final refinement pairs each long window with a short one and requires both to breach before firing, which slashes reset time while keeping the recall of the multi-tier ladder. That is the Workbook’s most-recommended scheme and is developed in full — parameter table, PromQL, firing/reset dynamics, and trade-offs — in Multi-Window Multi-Burn-Rate Alerts.

Prometheus Rules — Concrete Form

Burn-rate alerts are built on a recording rule that pre-computes the SLI as a ratio, so the alerting expressions stay cheap and readable:

record: job:slo_errors_per_request:ratio_rate1h
expr: |
  sum(rate(slo_errors[1h])) by (job)
    /
  sum(rate(slo_requests[1h])) by (job)

Line by line: sum(rate(slo_errors[1h])) by (job) is the per-second rate of failing events averaged over the trailing hour, summed per job; dividing by the same construction over slo_requests yields the error ratio — good-vs-total in the valid-events SLI form. The result is stored as a new time series named for the metric and its window, one recording rule per window you intend to alert on (ratio_rate5m, ratio_rate1h, ratio_rate6h, etc.).

The single-burn-rate alert (iteration 4/5) is then a one-line comparison against the budget times the target burn rate:

- alert: ErrorBudgetFastBurn
  expr: job:slo_errors_per_request:ratio_rate1h{job="myjob"} > (14.4 * 0.001)
  labels:
    severity: page
  annotations:
    summary: "Burning 30-day error budget >14.4x (would exhaust in ~50h)"

Reading the expression: 0.001 is the error budget for a 99.9% SLO, 14.4 is the target burn rate (2% of budget over 1 hour, from the arithmetic above), and the alert fires whenever the trailing-hour error ratio exceeds that product. Note that the same numeric threshold 0.0144 is correct regardless of the service’s traffic volume — that traffic-independence is precisely what makes burn rate the right quantity to alert on.

Failure Modes and Common Misunderstandings

  • Confusing burn rate with error rate. Burn rate is error rate divided by the budget. For a 99.9% SLO they differ by a factor of 1,000. For a 99% SLO (budget 0.01) burn rate 1 is a 1% error rate. The threshold budget × burn_rate bakes the SLO in; copying a threshold across services with different SLOs silently changes what it means.
  • Choosing windows without checking traffic. Burn rate is a statistic over a window; on a low-traffic service a very short window sees only a handful of requests, so one failure swings the ratio wildly and a fast-burn alert becomes noisy. Low-traffic SLOs need longer windows, artificially generated traffic, or a shift to aggregating over larger units — the Workbook devotes a whole section to the low-traffic problem.
  • Assuming a single threshold has good recall. As iteration 4 shows, one burn-rate threshold has a slow-burn blind spot (the 35× counterexample). A production alert needs at least the multi-tier ladder of iteration 5, and preferably the multiwindow scheme.
  • Ignoring reset time. A long window with no short-window companion keeps paging long after recovery. This is not cosmetic: an alert that stays lit through the postmortem trains responders to silence the pager, which is how real future pages get missed.
  • Forgetting the compliance-window cap on some platforms. Google Cloud Monitoring’s built-in burn-rate SLO alerts cannot use a compliance period longer than 24 hours (Google Cloud burn-rate alerting docs), which constrains the longest slow-burn tier if you rely on the managed feature rather than raw PromQL.

Uncertain

The precise per-approach detection/reset figures (e.g. “2 minutes 10 seconds” for iteration 2, “58 minutes” reset for iteration 4, “144 alerts per day” for iteration 1) are as stated in the SRE Workbook Chapter 5 worked example for a 99.9% SLO over a 30-day window and depend on that exact scenario. Reason: they are scenario-specific outputs, not universal constants. To resolve: recompute for your own SLO and window using detection ≈ window × (threshold error rate ÷ actual error rate) and reset ≈ window length. #uncertain

Alternatives and When to Choose Them

  • Static threshold / cause-based alerts (CPU, memory, queue depth). Useful as diagnostic signals and for capacity trends, but they should feed dashboards and tickets, not the pager — see SLO-Based Alerting versus Threshold Alerting and Symptom-Based versus Cause-Based Alerting. Their failing is exactly what burn rate fixes: they are not tied to user impact or a commitment.
  • Raw error-budget-remaining alerts (“budget < 10%”). These tell you the tank is nearly empty but not how fast it is draining, so they detect the emergency late. Burn rate is the derivative and gives earlier, severity-scaled warning. A remaining-budget alert is a reasonable low-urgency backstop alongside burn-rate pages.
  • Duration-based (for:) alerts. Fine for genuinely binary conditions (“this node has been unreachable for 5 minutes”) but poor for SLOs because, as iteration 3 shows, the duration does not scale with severity and flapping resets it.
  • Multiwindow multi-burn-rate. The recommended production choice; this note’s concept is the foundation it stands on. Go to Multi-Window Multi-Burn-Rate Alerts for the full recipe.

See Also

Graph View

Backlinks