vault

Alert Fatigue and Actionability

16 min read

Alert Fatigue and Actionability

Alert fatigue is the desensitization that sets in when an on-call engineer receives more pages than they can meaningfully act on — the state where alerts get skimmed, second-guessed, silenced, or ignored, including the one page that was real. Its antidote is a single, uncompromising bar on what may page a human: every page must be actionable, novel, and require intelligence — an action a human must take now that the system could not take itself (Google SRE Book, Monitoring Distributed Systems). This note argues the counterintuitive core claim of SRE alerting: a noisy alerting system is not merely annoying, it is a reliability risk, because attention is a finite resource and every non-actionable page spends some of it. The discipline — pruning, deduplicating, symptom-alerting, and auditing noise — exists to protect the scarce human at the other end of the pager (Ewaschuk, My Philosophy on Alerting).

This note is the practice of keeping the pager quiet and trustworthy. It pairs with Symptom-Based versus Cause-Based Alerting (which decides what to alert on), On-Call Rotation Design (which designs the human system that receives the alerts), and Runbooks and Playbooks (which handle the actionable-but-routine residue). The mechanism of an alerting pipeline — routing, grouping, deduplication engines — lives in Alerting System Design; here we teach why the bar is set where it is and how to hold the line.

Mental Model: Attention Is the Constrained Resource

The wrong mental model treats alerts as free — “more monitoring is more safety, so alert on everything.” The right model treats the on-call human’s attention as a depletable budget, spent by every interruption regardless of whether it mattered. Rob Ewaschuk, who wrote Google’s foundational alerting essay, put the human cost plainly: he can react to a pager with genuine urgency only a few times a day before fatigue sets in (Ewaschuk). Past that, the response degrades from investigate to acknowledge and hope.

flowchart TD
    subgraph NOISE["Noisy alerting (the failure mode)"]
        A1["Many alerts fire<br/>most non-actionable"] --> A2["On-call skims,<br/>second-guesses, mutes"]
        A2 --> A3["Desensitization<br/>(the 'crying wolf' effect)"]
        A3 --> A4["A REAL page is<br/>masked by the noise"]
        A3 --> A5["Burnout · attrition<br/>· sleep loss"]
        A4 --> A6["Missed / delayed<br/>incident → worse outage"]
        A5 --> A6
    end
    subgraph GOOD["Actionable alerting (the goal)"]
        B1["Every page is<br/>actionable + novel"] --> B2["On-call trusts<br/>the pager"]
        B2 --> B3["Fast, considered<br/>response"]
        B3 --> B4["Reliability + a<br/>sustainable human"]
    end

What it shows and the insight to take: the two loops share the same physiology. The left loop is a reliability failure disguised as a nuisance — its terminal node is a missed real page, not merely a tired engineer. The lever that flips the system from the left loop to the right is not a better pager app; it is fewer, better alerts. Reducing noise is a reliability intervention, ranked equal to fixing the underlying bugs.

The Actionability Bar, Stated Precisely

Google’s monitoring philosophy states the bar as three conjoined tests every candidate page must pass (Google SRE Book, Monitoring Distributed Systems):

  1. Actionable. There is an action a human must take, and the system cannot take it itself. Merely observing “this fired again” is not an action. The SRE Workbook sharpens this: a human should be expected to do something the moment the page arrives (SRE Workbook, On-Call).
  2. Requires intelligence. If the correct response is a rote, scriptable sequence — restart the process, clear the queue — then a script should do it, not a person. In the SRE formulation, a page that merits only a robotic response should not be a page. Automate it or ticket it; do not wake someone to run a known recipe.
  3. Novel. A page should concern a problem not seen before, or at least not one already being handled. The same symptom firing for the fourth time in a week is a signal that the system is broken (an unfixed bug, a missing auto-remediation), not that a human needs re-notifying.

The third test is where fatigue and actionability meet: a page can be individually actionable yet collectively fatiguing if it recurs. The SRE book’s own Bigtable case study is the canonical example — a service so alert-heavy that the team spent its energy triaging noise and, in doing so, missed the problems actually hurting users; the fix was to accept a temporarily lower reliability target so engineers could stop firefighting symptoms and repair root causes (Google SRE Book). That trade — deliberately spending error budget to buy down alert load — is the clearest proof that noise and unreliability are the same problem viewed from two angles.

The three legitimate outputs of a monitoring system

Ewaschuk and the SRE book agree there are exactly three valid destinations for a monitoring signal, and choosing correctly is most of the battle (Ewaschuk; Google SRE Book):

  • Pages — a human must act now. Urgent, actionable, real. This is the scarce channel; ration it ruthlessly.
  • Tickets — a human must act, but not this instant. Sub-critical work that needs timely attention drives a queue, not a pager. Having this channel is what makes a quiet on-call possible — without a ticket path, everything that “matters” defaults to paging.
  • Logging / dashboards — no action expected; recorded for later diagnosis and for the cause-level context you attach to a symptom page.

A fourth de-facto channel — email alerts — is explicitly condemned. The SRE book observes email alerts are rarely read or acted on: they are neither urgent enough to page nor tracked enough to be a ticket, so they accumulate unread and train the team to ignore an inbox (Google SRE Book). If something is worth alerting, page or ticket it; if it isn’t, log it.

Mechanical Walk-Through: Why Noise Corrodes Reliability

The causal chain from noise to a worse outage is physiological, not merely attitudinal, and the SRE book grounds it in the neuroscience of stress (Google SRE Book, Being On-Call). Humans respond to incidents in two cognitive modes: an intuitive, automatic, rapid mode and a rational, deliberate, focused mode. Good incident response wants the deliberate mode — form a hypothesis, check it, mitigate. But stress hormones (cortisol, corticotropin-releasing hormone) bias the brain toward the fast, heuristic mode and impair the deliberate one. Under a barrage of pages, the on-call engineer is pushed into shortcut thinking exactly when careful thinking is most needed.

The concrete failure this produces is confirmation bias: when a familiar alert fires for the fourth time this week, it is tempting to reflexively pin it on last time’s cause and apply last time’s fix — an unconsidered response the SRE book calls potentially disastrous, because this occurrence may be a different problem wearing the same symptom (Google SRE Book, Being On-Call). Noise thus does double harm: it depletes attention and it manufactures the false familiarity that makes the depleted responder guess wrong.

At the extreme this becomes operational overload, a named condition in SRE with recognized remedies: misconfigured monitoring and excessive alerts disrupt productivity, and the fatigue they induce causes serious alerts to be treated with less attention than they warrant (Google SRE Book, Being On-Call). The prescribed interventions are aggressive — temporarily loan an experienced SRE to an overloaded team for breathing room; set hard numeric ceilings (daily tickets < 5, paging events per shift < 2); drive toward a 1:1 alert-to-incident ratio by killing duplicate and fan-out alerts; and, in the extreme, hand the pager back to the developers until the service meets reliability standards. Every one of these treats alert volume as a first-class operational hazard.

The human-factors evidence: alarm fatigue in medicine

SRE’s claim that noise kills is not folklore — the strongest empirical corroboration comes from a different high-stakes domain that measured the effect directly. In 2013 the Joint Commission (the US hospital-accreditation body) issued Sentinel Event Alert 50 on medical-device alarm safety. Its findings map onto the pager one-for-one (Joint Commission SEA 50, April 2013; full text PDF):

  • An estimated 85% to 99% of alarm signals require no clinical intervention — an actionability rate that would be considered catastrophic for a pager.
  • Clinicians consequently become desensitized to alarms and, drowning in signals, may turn down, silence, or mis-set them — the exact “mute and hope” behavior seen on saturated on-call rotations.
  • The database recorded 98 alarm-related sentinel events over ~3.5 years, 80 of them fatal, plus permanent loss of function and extended stays.

The lesson transfers cleanly: when the overwhelming majority of alerts are non-actionable, the humans stop attending to all of them, and the failure surfaces precisely when a rare real signal is masked by the routine noise (NCBI, Making Healthcare Safer III, Alarm Fatigue). A pager that cries wolf 90% of the time is not 90% wasteful and 10% useful — it is actively dangerous, because it has trained its reader to disbelieve it.

Configuration and Practice: The Noise Audit and Alert Tuning

Holding the actionability bar is an ongoing engineering activity, not a one-time config. Three practices carry the load.

1. The per-alert review questions

Before any rule is allowed to page, and periodically for every existing paging rule, run it through the SRE book’s qualifying questions (Google SRE Book, Monitoring Distributed Systems):

  • Does it detect an otherwise-undetected condition that is urgent, actionable, and actively or imminently user-visible?
  • Will I ever be able to ignore this alert knowingly? When, and how do I suppress it then?
  • Does it definitely indicate users are being harmed — or can it fire when they are fine?
  • Can I take an action? Is that action urgent, or can it wait for business hours (i.e. should this be a ticket)?
  • Are other people paged for the same thing, making this redundant?

Any “no” to actionability or user-impact demotes the rule from page to ticket or log. Ewaschuk’s blunt heuristic: err on the side of removing noisy alerts, because over-monitoring is a harder problem to fix than under-monitoring — a missed problem is recoverable, a team that ignores its pager is not (Ewaschuk).

2. Alert on symptoms, and let one page cover many causes

The single most effective noise-reduction move is architectural: alert on the symptom the user feels, not the dozens of causes that could produce it (developed fully in Symptom-Based versus Cause-Based Alerting). One SLO-based symptom alert — “the fraction of requests served successfully under 300 ms has dropped below the objective” — replaces a swarm of cause alerts (disk full, replica down, cache cold, dependency slow) that would each fire independently for the same incident. Cause detail is attached to the page as context and lives on dashboards; it does not page on its own. This is also why Burn-Rate Alerting and Multi-Window Multi-Burn-Rate Alerts are the modern default: they page on the rate the error budget is burning, collapsing many threshold crossings into one high-signal alert tuned for precision and recall.

3. Deduplication, grouping, and the noise audit

Even good rules generate multiple notifications per incident — a fan-out across replicas, or the same failure seen by three dependent services. The alerting pipeline must deduplicate (collapse identical firings), group (bundle related alerts into one notification), and suppress/inhibit (silence downstream alerts when a known upstream cause is already firing) so that one problem produces one page. These are properties of the routing layer — see Alerting System Design for the mechanism — but tuning them is the on-call team’s responsibility, measured against that 1:1 alert-to-incident target.

The noise audit operationalizes all of this. On a regular cadence (many teams do it weekly, at on-call handoff), the team reviews every page from the period and classifies it: actionable or not, novel or repeat, correctly-routed or should-have-been-a-ticket. Repeat offenders get an auto-remediation or a code fix; mis-routed alerts get demoted; flapping alerts get a minimum-duration guard. Google’s tooling encodes the timing discipline directly — an alert condition must hold for a minimum window (e.g. remain true for two minutes) before it fires, so momentary blips and metric-collection gaps do not page (Google SRE Book, Practical Alerting). New paging rules are also expected to run in a test/silent mode for at least a week before being armed, so their real-world firing behavior is known before they can wake anyone (SRE Workbook, On-Call).

A worked Prometheus example

groups:
- name: slo-symptom-alerts
  rules:
  # SYMPTOM alert: user-visible availability burn, not a cause.
  - alert: HighErrorBudgetBurn
    # Multi-window burn rate: fast burn confirmed over a short AND a long window.
    expr: |
      (
        job:slo_errors:ratio_rate5m{job="api"}  > (14.4 * 0.001)
        and
        job:slo_errors:ratio_rate1h{job="api"}  > (14.4 * 0.001)
      )
    for: 2m                     # minimum-duration guard: ignore sub-2m blips (anti-flap)
    labels:
      severity: page            # the SCARCE channel — reserved for actionable + novel
    annotations:
      summary: "API burning error budget 14.4x too fast (page-worthy)"
      runbook: "https://runbooks.example.com/api-slo-burn"   # actionable: link the response

Line-by-line, the load-bearing choices: the expr alerts on a success-ratio symptom (slo_errors:ratio), not on CPU or a dead replica, so one rule covers every cause of user pain. It requires the fast burn to be confirmed on both a 5-minute and a 1-hour window (the multi-window recipe) so a transient spike cannot page — this is the precision/recall tuning that kills false positives. for: 2m adds the minimum-duration guard against flapping. severity: page is a deliberate assertion that this is actionable and novel enough for the scarce channel; a slow, lower-multiplier burn would instead carry severity: ticket. The runbook annotation makes the page actionable on arrival — the responder is handed the entry point to the response, not left to start from zero at 3 a.m. (See Runbooks and Playbooks.)

Common Misunderstandings and Failure Modes

  • “More alerts = safer.” The most damaging myth. Beyond the point where the on-call human can attend to every page, additional alerts reduce safety by masking real ones. Coverage is not measured in alert count; it is measured in whether the user-visible symptoms are caught, which a handful of good symptom alerts achieve.
  • “An actionable page is automatically a good page.” Not if the action is rote (should be automated) or the page is the fourth identical one this week (the underlying bug should be fixed). Actionability is necessary, not sufficient; novel and requires-intelligence are the other two legs.
  • “We’ll tune it later.” Alert debt compounds like technical debt. A rotation that inherits 200 stale alerts trains new on-callers to ignore the pager from day one. The noise audit must be continuous, and killing an alert is a legitimate, celebrated outcome — not a coverage regression to be feared.
  • “Route the noise to email so it’s not a page.” Email alerts are a graveyard: rarely read, never tracked, and they normalize ignoring notifications. Demote to a ticket (tracked, actioned) or a log (queryable, silent) — never to a channel that trains people to skim.
  • “Silencing an alert during an incident is cheating.” It is the opposite — a mature alerting system inhibits known-downstream alerts when an upstream cause is already paging, so the responder sees one problem, not fifty symptoms of it. Suppression is a feature.
  • Flapping and metric gaps. Without a minimum-duration guard, a metric that briefly crosses a threshold — or a scrape that momentarily fails — pages spuriously. The for: clause and minimum-collection windows exist precisely to convert these into non-events (Google SRE Book, Practical Alerting).

Alternatives and When to Choose Them

The actionability bar is near-universal, but the routing of the non-actionable residue varies:

  • Threshold alerting on resource causes (CPU > 80%, disk > 90%) is the traditional model and still has a place — but as tickets and dashboards, not pages. It answers “is a resource trending toward exhaustion,” which is capacity work for business hours, not a 3 a.m. wake-up. Paging on raw thresholds is the single most common source of alert fatigue; see SLO-Based Alerting versus Threshold Alerting.
  • SLO / burn-rate alerting is the modern default for the page channel because it is symptom-based and rate-based, giving the best precision/recall trade — see Burn-Rate Alerting.
  • Anomaly-detection / ML alerting promises to learn “normal” and page on deviations. In practice it is prone to low precision (novel-but-benign patterns page) unless heavily tuned, and its black-box nature makes an alert hard to make actionable — if you cannot say what a human should do, it fails the bar. Use it as a dashboard signal that feeds investigation, rarely as a direct pager.

Production Notes

The industry telemetry backs the human cost. PagerDuty’s operations research reported that a majority of practitioners were working 10+ extra hours a week and a large share expected burnout to become a problem — with poor on-call and incident load a named driver (PagerDuty, Global Developer and ITOps Report). Alert fatigue is thus not only a reliability risk but a retention risk: the engineers who understand a system best are the ones the noisy pager drives out, which is itself a slow-motion reliability regression. This closes the loop with On-Call Rotation Design — the sustainability of the rotation and the quietness of the pager are the same problem measured on two axes.

A practical maturity marker: teams that have internalized this treat deleting an alert as a win worth announcing, publish their pages-per-shift as a tracked metric with a hard ceiling (the SRE guidance: no more than two incidents per 12-hour shift; daily tickets under five), and run the noise audit as a fixed agenda item at handoff. When those habits are in place, the pager becomes trustworthy — and a trustworthy pager is the whole point, because the only alert that matters is the one the on-call engineer believes.

See Also

Graph View

Backlinks