vault

Alerting System Design

32 min read

Alerting System Design

An alerting system receives alert events from monitoring sources (metrics, logs, traces, synthetics, application errors), routes them through deduplication and grouping, evaluates schedules and escalation policies to find the on-call human, delivers a notification (push, SMS, voice, email, chat), waits for acknowledgement, and escalates if no response. After resolution, it supports post-incident review. The canonical industrial deployments are PagerDuty (founded 2009, the de facto SaaS for incident response, https://www.pagerduty.com/), Atlassian Opsgenie (acquired by Atlassian 2018), Splunk On-Call (formerly VictorOps), Prometheus Alertmanager (the open-source alert routing layer that sits behind Prometheus, https://prometheus.io/docs/alerting/latest/alertmanager/), Grafana OnCall, and the alerting subsystems built into observability vendors (Datadog Alerts, New Relic Alerts). The interview problem is interesting because alerting is the operational interface between machines and humans — it must be reliable in ways most systems aren’t (a missed alert can cost millions), it must be respectful of humans’ attention (alert fatigue is a documented health hazard for on-call engineers, per Beyer et al. 2018 §5), and it must work especially when the rest of your infrastructure is failing, because that’s exactly when alerts are needed most. Almost every production failure mode of an alerting system is a variant of “we missed the page” or “we paged someone for nothing at 3am” — and both are organization-killers.

1. Why This System Exists

Monitoring observes; alerting interrupts. Without alerting, an outage at 03:00 is detected only when the next human happens to glance at a dashboard or when a customer complains. With alerting, the right person is woken up within seconds of the threshold crossing. The cost-of-failure asymmetry is massive: a single missed page during a production outage commonly costs hundreds of thousands of dollars per minute (transaction loss + reputational damage). The cost of a spurious page is smaller per-event (a tired engineer, an awakened spouse) but accumulates in chronic alert fatigue that ends careers and reduces overall reliability (Beyer et al. 2018 Ch. 5; Allspaw 2017 https://snafucatchers.github.io/).

The interview lens is that alerting touches every reliability and human-factors concern at once. Reliability: how do you reliably page when your data center is down? Human factors: how do you avoid waking up six engineers for one root cause? Schedule arithmetic: how do you handle on-call rotations across time zones, holidays, and team transitions? Integration: how do you handle the long tail of notification channels and their varying delivery guarantees?

2. Requirements

2.1 Functional Requirements

  1. Receive alert events from many sources: metrics evaluators (Prometheus, Datadog), log-pattern matchers, trace-error detectors, synthetic monitoring (HTTP/DNS uptime probes), application error trackers (Sentry, Bugsnag), webhook ingest from arbitrary external systems.
  2. Deduplicate. Identical alerts arriving within a short window become one notification. Identity is defined by an alert fingerprint — typically a hash of (alert_name, set_of_critical_labels).
  3. Group. Related alerts (same root cause, e.g., “every host in cluster A is unreachable”) collapse into one notification. Configurable grouping keys.
  4. Inhibit. When a higher-level alert fires (e.g., “datacenter-east is down”), suppress lower-level alerts that are symptoms of the same root cause (“service-X-in-east is down”).
  5. Route. Apply a routing tree of label matchers → escalation policy → on-call schedule → notification channel.
  6. Notify. Deliver via push (mobile app), SMS, voice call, email, chat (Slack, Teams), incident channel auto-creation (Slack channel, Zoom bridge), and arbitrary webhooks.
  7. Acknowledge. Receiver clicks “ack” → the alert is silenced for that incident. Without ack within timeout, escalate.
  8. Escalate. If no ack within N minutes: page primary’s secondary; then escalation tiers up to manager, director, VP for severe incidents.
  9. Silence / mute. Operators can silence alerts during planned maintenance windows or known-noisy periods.
  10. Resolve. Either operator marks resolved, or the underlying alert event clears (the metric crosses back below threshold). Resolution closes the incident.
  11. Post-incident review — collect timeline of who was paged when, what was acknowledged when, what was escalated, attach to the incident retrospective.
  12. Multiple severities. page (wakes someone), ticket (creates a ticket but does not page), info (logged only). Different severities use different channels and escalation policies.

2.2 Non-Functional Requirements

  1. Reliability of delivery. Every page that should fire, must fire. The system aspires to “five nines for paging” (99.999% — about 5 minutes/year of allowed missed-page time). To hit this, the alerting system must be more reliable than the systems it monitors and must not share fate with them. This is an aspirational system-level target reached only by stacking redundancies — no single vendor delivers it (see §14 for why, with the Twilio SLA and the 2025 PagerDuty outage as evidence).
  2. Latency. Alert event received → page delivered: < 30 seconds at p99. Faster is better; the SRE community considers anything > 1 minute to be a serious lag.
  3. Capacity. Bursty input — major incidents trigger thousands of correlated alerts in seconds. The pipeline must absorb spikes without dropping.
  4. Multi-tenancy. A SaaS alerting system handles thousands of customers’ alerts in shared infrastructure; one customer’s burst must not delay another’s page.
  5. Sane on-call experience. Aim for fewer than 2 pages per on-call shift (12 or 24 hours); pages per shift is a leading indicator of team health (Beyer et al. 2018 §11.5).
  6. Auditability. Every page, every escalation, every silence — recorded with timestamps and actor IDs.

3. Capacity Estimation

A SaaS alerting system serving 10,000 customer organizations with average 1,000 alerts/day (some have 100, some have 50,000):

fleet alert events/day = 10,000 × 1,000 = 10^7 events/day
average events/sec     = 10^7 / 86,400  ≈ 116/sec
peak (10× burst)       ≈ 1,160/sec     -- during major-incident pile-on

Pages-per-second is much lower than alert events/sec because of dedup + grouping + inhibition: typical compression ratio 10–100×.

peak pages/sec  ≈ 100/sec
peak SMS/sec    ≈ 50  (subset)
peak voice/sec  ≈ 10
peak push/sec   ≈ 100

These are modest absolute numbers — the system is not a high-throughput problem; it is a high-reliability problem. The constraint is that the 100 pages/sec must be served with five-nines reliability during the same incident that may be taking down half the customer’s infrastructure.

3.1 Storage

Per alert event: ~2 KB (alert name, labels, timestamps, status changes). At 10⁷/day → 20 GB/day raw, 6 TB/year. Negligible.

The expensive storage is the audit trail — every routing decision, every notification attempt, every state transition — perhaps 10× the alert event size, so ~60 TB/year. Still manageable in a single moderately-sized OLAP database (ClickHouse, BigQuery).

3.2 Bandwidth

Trivial in bytes. The real constraint is carrier delivery throughput on SMS and voice. In the US, SMS sending is governed by A2P 10DLC (application-to-person, 10-digit long-code) rules: a per-number throughput that used to be a flat ~1 message/sec is now set by the sender’s Brand Trust Score and campaign type, scaling well above 1 MPS for well-registered brands (Twilio: Message throughput and Trust Scores for A2P 10DLC). The point for an alerting system is that this throughput is per sending number, not per-system — so a system pushing 50 SMS/sec spreads them across multiple numbers (and ideally multiple providers; see §9.6), staying well within aggregate commercial capacity. The same per-number/per-provider ceiling is exactly why a major-incident page-storm can hit a carrier rate limit precisely when you can least afford it.

4. API Design

4.1 Alert Ingest API

The Prometheus Alertmanager API (https://prometheus.io/docs/alerting/latest/clients/) is the most-imitated:

POST /api/v2/alerts
[
  {
    "labels": {
      "alertname": "HighRequestLatency",
      "service":   "checkout",
      "severity":  "page",
      "region":    "us-east-1"
    },
    "annotations": {
      "summary":     "checkout p99 latency > 1s",
      "description": "p99 over the last 10 minutes is 2.3s; SLO is 1s",
      "runbook":     "https://wiki/runbooks/high-checkout-latency"
    },
    "startsAt": "2026-05-08T14:23:11.123Z",
    "endsAt":   "0001-01-01T00:00:00Z",      // zero = ongoing
    "generatorURL": "https://prometheus/graph?..."
  }
]

The same alert sent again with the same labels is deduplicated — Alertmanager updates the existing alert rather than creating a new one. When the underlying condition clears, the source either: (a) Stops sending the alert (it auto-resolves after resolve_timeout). (b) Sends an explicit resolution: endsAt: <past timestamp>.

PagerDuty’s Events API v2 (https://developer.pagerduty.com/docs/events-api-v2/overview/) is similar in shape:

POST https://events.pagerduty.com/v2/enqueue
{
  "routing_key": "<integration key>",
  "event_action": "trigger" | "acknowledge" | "resolve",
  "dedup_key": "checkout/p99/us-east-1",
  "payload": {
    "summary": "...", "severity": "critical", "source": "...",
    "custom_details": { ... }
  }
}

The dedup_key is the deduplication identity — sender-supplied. event_action: trigger opens or updates an incident; resolve closes it.

4.2 Schedule and Policy API

POST /schedules
{
  "name": "checkout-primary",
  "time_zone": "America/New_York",
  "rotation": {
    "type": "weekly",
    "users": ["alice", "bob", "carol"],
    "start": "2026-01-06T09:00:00",      // first Monday of the year
    "duration_seconds": 604800            // weekly handoff
  },
  "overrides": [
    { "user": "alice", "start": "...", "end": "...", "reason": "vacation" }
  ]
}

POST /escalation_policies
{
  "name": "checkout-page",
  "rules": [
    { "schedules": ["checkout-primary"], "escalation_delay_seconds": 300 },
    { "schedules": ["checkout-secondary"], "escalation_delay_seconds": 300 },
    { "users": ["manager"], "escalation_delay_seconds": 600 }
  ],
  "repeat": false
}

The schedule resolves “who is on call right now?” given the current time, the rotation rules, and any overrides. The escalation policy is a list of “page these users; if not acked in N seconds, advance to the next rule.”

4.3 Routing Tree API

# Alertmanager-style routing tree (https://prometheus.io/docs/alerting/latest/configuration/#route)
route:
  receiver: default
  group_by: ['alertname', 'cluster', 'service']
  group_wait:      30s    # wait this long before firing first notification (collect related alerts)
  group_interval:  5m     # send updates every this often if alerts in group changed
  repeat_interval: 4h     # remind every 4h if still firing
  routes:
    - match: { severity: page }
      receiver: pagerduty-primary
    - match: { severity: ticket }
      receiver: jira
    - match_re: { service: ^db-.* }
      receiver: pagerduty-database-team

The route tree is a tree of label-matcher → child-route, with the matched leaf determining the receiver. The group_wait (typically 30 s) is the first notable design choice: when the first alert in a new group arrives, wait before sending — give related alerts a chance to arrive and group. Without this wait, a major incident emits a flood of single-alert pages instead of one consolidated page.

5. Data Model

5.1 Alert

struct Alert {
    fingerprint:   u64                  // hash(alertname + critical_labels)
    labels:        map[string]string    // alertname, service, region, severity, ...
    annotations:   map[string]string    // human-readable text, runbook URL
    starts_at:     time
    ends_at:       time                 // zero/inf if ongoing
    generator_url: string               // link back to the source (graph, log search)
    state:         enum { pending, firing, resolved }
    received_at:   time
    last_updated:  time
}

The fingerprint is the deduplication identity. Two alerts with the same fingerprint within repeat_interval are merged.

5.2 Incident (PagerDuty’s higher-level abstraction)

struct Incident {
    id:                uuid
    title:             string
    urgency:           enum { high, low }
    status:            enum { triggered, acknowledged, resolved }
    created_at:        time
    resolved_at:       time?
    service_id:        uuid                         // which service's incident
    escalation_policy: uuid
    current_assignee:  uuid                          // current on-call paged
    assignments:       list[Assignment]              // history
    log_entries:       list[Event]                   // every state change
    related_alerts:    list[uuid]                    // de-duped underlying alerts
}

PagerDuty’s mental model is incident-first: alerts are the underlying signals; an incident is the human-actionable bundle. Multiple alerts (same dedup_key, or grouped by rules) collapse into one incident.

5.3 On-Call Schedule

struct Schedule {
    id:           uuid
    timezone:     string                       // IANA tz; "America/New_York"
    layers:       list[ScheduleLayer]          // priority-ordered
}

struct ScheduleLayer {
    rotation_type:    enum { daily, weekly, custom }
    users:            list[uuid]
    handoff_time:     LocalTime                // e.g., "09:00"
    handoff_day:      enum { mon, tue, ... }
    duration_seconds: int
    start:            time
    restrictions:     list[TimeRange]?         // "only on-call M-F 9-5"
}

struct Override {
    schedule_id:  uuid
    user_id:      uuid
    start:        time
    end:          time
    reason:       string
}

Resolving “who is on-call at time T?” walks layers in priority order: each layer is “does some user cover T given the rotation rules and overrides?” — the highest-priority layer that covers T wins.

The TIME-ZONE handling is the most underestimated complexity. A weekly rotation handing off “Monday 9am Pacific” must compute correctly across daylight-saving boundaries, leap-seconds (rarely), and operators’ local clocks. Use IANA tz database; do not roll your own.

5.4 Escalation Policy

struct EscalationPolicy {
    id:     uuid
    rules:  list[EscalationRule]   // applied in order
    repeat: bool                   // restart from rule 1 after exhausting?
}

struct EscalationRule {
    targets:                  list[Target]   // schedules or specific users
    escalation_delay_seconds: int            // how long to wait before next rule
}

The state machine: trigger → notify all targets in rule[0] → wait escalation_delay_seconds → if not acked, notify rule[1] targets → etc.

6. High-Level Architecture

flowchart TB
    subgraph Sources[Alert Sources]
        Prom[Prometheus<br/>Datadog metrics]
        Logs[Log Pattern Matcher]
        Sentry[Sentry / Bugsnag]
        Synth[Synthetic Probes]
        Webhook[Webhook ingest]
    end
    Sources --> Ingest[Ingest API<br/>HTTP POST events]
    Ingest --> Q[Durable Queue<br/>Kafka / SQS]
    Q --> Pipeline[Alert Pipeline]
    Pipeline --> Dedup[Deduplicator<br/>fingerprint + window]
    Dedup --> Group[Grouper<br/>group_by + group_wait]
    Group --> Inhib[Inhibitor<br/>suppression rules]
    Inhib --> Route[Router<br/>routing tree match]
    Route --> ResolveOC[On-Call Resolver<br/>schedule + escalation]
    ResolveOC --> Notify[Notification Dispatcher]
    Notify --> Push[Push: APNS / FCM]
    Notify --> SMS[SMS: Twilio / Bandwidth]
    Notify --> Voice[Voice: Twilio]
    Notify --> Email[Email: SES / SendGrid]
    Notify --> Chat[Chat: Slack / Teams]
    User[On-Call User] -->|Ack via app/SMS reply| AckAPI[Acknowledgement API]
    AckAPI --> Pipeline
    Notify --> EscTimer[Escalation Timer]
    EscTimer -.no ack in N min.-> Notify
    Pipeline --> Audit[(Audit Log<br/>ClickHouse)]
    Pipeline --> State[(State Store<br/>PostgreSQL or DynamoDB)]
    DMS[Dead-Man's Switch<br/>independent monitor] --> Notify

What this diagram shows. Alerts flow left to right through a multi-stage pipeline. Sources push events into the Ingest API, which writes them to a durable queue so a brief processing-tier outage doesn’t drop alerts. The alert pipeline runs sequentially: deduplication (collapse identical alerts), grouping (collect related alerts together with a small wait), inhibition (suppress symptoms of higher-level firing alerts), routing (match labels to receiver via a routing tree), on-call resolution (compute the live on-call person from schedules and escalation policies), and notification dispatch to the appropriate channel(s). The escalation timer kicks in if no acknowledgement arrives — a delayed action in the pipeline that re-runs notification with the next escalation rule’s targets. The state store holds in-flight incidents and the audit log for review. The dead-man’s switch at the bottom is an independent monitor: if the alerting system itself stops emitting heartbeats, another (separately-deployed, separately-owned) monitor pages a fallback channel — solving the meta-problem of “how do you alert when alerting is down?”

The durable queue between ingest and pipeline is load-bearing: an alerting system without one drops alerts during processing-tier deploys, and “we lost a page during a routine deploy” is a career-ending production failure.

7. Request Flow

7.1 Alert Trigger to Page Delivered

sequenceDiagram
    participant P as Prometheus
    participant I as Ingest
    participant Q as Queue
    participant DG as Dedup + Group + Inhibit + Route
    participant OC as On-Call Resolver
    participant N as Notification Dispatcher
    participant T as Twilio
    participant U as User

    P->>I: POST /alerts {alertname=HighLatency, service=checkout, severity=page}
    I->>Q: enqueue
    I-->>P: 202 Accepted
    Q->>DG: consume
    DG->>DG: fingerprint = hash(alertname + service)
    DG->>DG: existing? merge : new alert created
    DG->>DG: wait group_wait=30s for siblings
    Note over DG: 30s later
    DG->>DG: build group; check inhibitions
    DG->>DG: route match → severity=page → receiver=pagerduty-primary
    DG->>OC: resolve(receiver=pagerduty-primary, time=now)
    OC-->>DG: user=alice, channels=[push, sms, voice@5min]
    DG->>N: notify(alice, channels, payload)
    N->>T: send SMS
    T-->>U: ring + SMS
    par parallel push
        N->>U: APNS push
    end
    Note over N,U: timer set for 5 min
    U->>I: POST /ack (incident_id)
    I->>DG: ack received
    DG->>N: cancel pending escalations

The 30-second group_wait is what keeps a major incident from sending 100 individual SMS to the on-call. Instead, the first 30 s aggregates the group; the on-call gets one message that says “100 hosts down in cluster A” rather than 100 individual messages.

7.2 Escalation

sequenceDiagram
    participant N as Notification Dispatcher
    participant Timer as Escalation Timer
    participant OC as On-Call Resolver
    participant U1 as Alice (primary)
    participant U2 as Bob (secondary)
    participant U3 as Manager

    N->>U1: page alice (rule[0], delay=5min)
    N->>Timer: schedule check at +5 min
    Note over U1: alice is asleep / phone dead
    Timer-->>N: tick — no ack
    N->>OC: resolve rule[1]
    OC-->>N: bob (secondary)
    N->>U2: page bob (rule[1])
    N->>Timer: schedule check at +5 min
    Note over U2: bob acks
    U2->>N: ack
    N->>Timer: cancel escalation

Standard escalation tiers in production:

Rule 0 (5 min):  primary on-call
Rule 1 (5 min):  secondary on-call
Rule 2 (10 min): team manager
Rule 3 (10 min): engineering director
Rule 4:          VP / CTO + auto-resolve to high-severity incident channel

The numbers are tuned to “the worst case is 30 minutes from alert to executive notice,” which is the threshold beyond which most major-customer SLAs require credit.

8. Deep Dive

8.1 Deduplication Fingerprinting

A naïve fingerprint is hash(all_labels). This is too specific — every label change creates a new alert, so a transient label flip (“was the request from us-east-1a or 1b?”) opens a new incident.

The fix: explicitly designate which labels are fingerprint labels (always part of the dedup key) and which are informational (kept in the alert, but not part of the key). Prometheus Alertmanager declares this via group_by for grouping (not deduplication directly, but adjacent — see the spec at https://prometheus.io/docs/alerting/latest/configuration/#route).

Common fingerprint labels: alertname, service, cluster (sometimes region, often not pod/host because these flap).

8.2 Grouping and group_wait

When the first alert in a new group arrives, the system waits group_wait (typically 30 s) before firing. The wait is the difference between “100 individual pages” and “1 page that says 100 things are broken.”

But the wait introduces latency: the first alert takes 30 extra seconds to reach the human. For most kinds of incident this is invisible (the human can’t act in 30 s anyway), but for hyper-critical things you may want group_wait: 0.

The follow-on parameters:

  • group_interval (~5 min) — how often to send updates to an existing group as new alerts join it. Prevents per-second update spam.
  • repeat_interval (~4 hours) — how often to re-page the on-call if the alert is still firing and unacknowledged. Long enough to not be noise; short enough to catch “ack’d then forgot.”

8.3 Inhibition

Inhibition rules suppress one alert when another is firing. The classic use case:

inhibit_rules:
- source_match: { alertname: DatacenterDown, datacenter: us-east-1 }
  target_match: { datacenter: us-east-1 }
  equal: [datacenter]

When DatacenterDown{datacenter=us-east-1} is firing, suppress all alerts with datacenter=us-east-1. Without this, a datacenter outage emits 10,000 individual symptom alerts when the operator already knows the root cause.

Inhibition is the difference between an alerting system the operators trust and one they mute permanently because it cries wolf.

8.4 The On-Call Schedule Resolver

The most algorithmically-interesting subcomponent. Given a time T and a schedule, return the user(s) on call. Algorithm:

function resolve(schedule, time T):
    # walk schedule layers in priority order
    for layer in schedule.layers:
        # find the rotation cell at time T
        cell = compute_rotation_cell(layer, T)
        user = layer.users[cell.index]
        # apply overrides (from override table)
        for ov in overrides_for(layer, T):
            if ov.start <= T < ov.end:
                user = ov.user
                break
        # apply restrictions (only on-call M-F 9-5?)
        if not within_restrictions(layer, T):
            continue
        return user
    return null  # no coverage — page the team default

The compute_rotation_cell function encapsulates the time-zone math:

function compute_rotation_cell(layer, T):
    # convert T to layer's local time-zone using IANA tz
    local_T = T.astimezone(layer.timezone)
    # offset from rotation start
    elapsed = local_T - layer.start
    cell_index = (elapsed.total_seconds() // layer.duration_seconds) % len(layer.users)
    return cell_index

The bug-prone parts are: DST transitions (a “weekly Monday 9am” handoff jumps an hour twice a year), leap years, and rotation-during-handoff (if the rotation handoff is mid-page, who handles the active page? — convention is the triggering-time user, even if their shift just ended).

8.5 Notification Channel Reliability Ranking

Different channels have different delivery guarantees. From most to least reliable for waking someone up at 03:00:

  1. Repeating voice call. Auto-redial until answered. Ringtone bypasses Do Not Disturb on iOS/Android with proper carrier features (PagerDuty’s app uses a custom ringtone that can override silent). Most reliable; most expensive ($0.01–0.02/call); most painful for the recipient.
  2. Mobile push via dedicated app. PagerDuty / Opsgenie apps register as critical-alert notification (iOS critical alerts, Android high-priority FCM) which bypass Do Not Disturb. Reliable when the phone has connectivity and battery; fails when the device is offline or app is killed.
  3. SMS. Mostly reliable but carrier-dependent; international SMS can fail or be delayed minutes. Some carriers strip URLs in shortcodes. Twilio publishes per-country delivery rates.
  4. Email. Slow (gateway delays of minutes), often filtered by spam. Useful for ticket-severity, useless for paging.
  5. Slack / Teams chat. Useful for incident-channel announcements; not reliable as primary page channel because chat apps respect DND.

Production deployments stack channels: simultaneous SMS + push + voice-after-5-min; if all fail, escalate. The cost (a few cents per page) is negligible compared to the cost of a missed page.

8.6 Burn-Rate vs Threshold Alerting

The classical pattern is error_rate > 10/sec → page. This has two failure modes:

  • Too sensitive: a 30-second blip exceeds 10/sec briefly; pager fires; engineer wakes up to find everything green.
  • Too insensitive: raising the threshold to 20/sec misses real outages that linger at 15/sec.

Google’s SRE Workbook (Beyer et al. 2018, “Alerting on SLOs”) advocates multi-window, multi-burn-rate alerting. The recommended starting values for a 99.9% SLO over a 30-day period are given in the chapter’s Table 5-8:

# Fast burn (catches sudden, severe outages) — Page
burn_rate_1h  > 14.4  AND  burn_rate_5m  > 14.4
  → page  (consumes 2% of the 30-day budget in 1h)

# Slower burn (catches significant degradation) — Page
burn_rate_6h  > 6     AND  burn_rate_30m > 6
  → page  (consumes 5% of the 30-day budget in 6h)

# Slow burn (gradual erosion) — Ticket, not page
burn_rate_3d  > 1     AND  burn_rate_6h  > 1
  → ticket (consumes 10% of the budget over 3 days)

Each alert pairs a long window (the headline condition — has the budget burned fast over the last hour / 6 hours / 3 days?) with a short window (a recency check — is it still burning right now, over the last 5 / 30 / 6h?), and both must be true. The short window suppresses transient blips (a spike that has already recovered fails the short-window check) and lets the alert resolve quickly once the burn stops, instead of lingering for the full long window.

The math, walked symbol by symbol. Let the SLO be 99.9% over a 30-day period, so the error budget is 0.1% of total requests over those 30 days. Define burn rate B as the multiple of the budget-exhausting error rate: a constant 0.1% error rate over the whole 30 days is exactly B = 1 (it spends precisely 100% of the budget in 30 days). The fraction of the total monthly budget consumed by burning at rate B for a window of length w is:

budget_consumed = B × (w / 30 days)

For the fast-burn page: B = 14.4 and w = 1 hour, so 14.4 × (1h / 30d) = 14.4 × (1 / 720) ≈ 0.02 = 2%. Burning at 14.4× for a single hour spends 2% of the entire month’s error budget — large enough to be worth waking someone, small enough that brief noise won’t trip it. (Note: 1 hour is 1/720 of a 30-day month, not 1/672 of 28 days — an earlier draft of this note used 28 days and quoted “≈2.1% / 5% of budget,” both of which are wrong; the workbook uses a 30-day period and the table values are 2%, 5%, and 10% for the three tiers.) The 14.4 itself comes from inverting that relation: to make “1 hour of burn = 2% of a 30-day budget,” you need B = 0.02 × 720 = 14.4.

The implementation: PromQL recording rules compute burn_rate_5m, burn_rate_1h, burn_rate_6h, etc. as error_ratio_over_window / (1 - SLO); alert rules combine the long- and short-window expressions with AND. See Metrics and Monitoring System Design for the recording-rule mechanics.

8.7 The Meta-Problem: How Do You Alert When Alerting Is Down?

The single hardest design constraint. Approaches in increasing order of investment:

Dead-man’s switch. The alerting system periodically emits a heartbeat to an independent external service (a separate alerting system on different cloud, or a service like Healthchecks.io, https://healthchecks.io/). The external service expects a heartbeat every N seconds; if it doesn’t arrive, it pages someone (typically via a different telephony provider). This catches “Alertmanager process crashed” but not “Datadog account expired.”

Redundant pair. Run two or more Alertmanagers configured identically. The key design point — and a favorite interview probe — is how they avoid double-paging without a consensus protocol. Alertmanager’s HA mode (Prometheus Alertmanager HA docs) forms a peer-to-peer mesh using HashiCorp’s memberlist gossip library (the same SWIM-based library behind Consul and Serf) for membership, failure detection, and replication of the notification log (which notifications have already been sent). Critically, it deliberately does not use Raft or Paxos: the design is AP (available + partition-tolerant), not CP, and it favors duplicate notifications over missed ones. Coordination is by a position-based staggered wait: each instance sorts the cluster peers deterministically and computes its own position i; before dispatching a notification it waits i × peer_timeout (the timeout defaults on the order of 15 s per position). The position-0 instance sends immediately; lower-priority instances wait, and during their wait they consult the gossiped notification log — if a peer already sent the notification, they skip it. The result: in the happy path exactly one instance pages; if the position-0 instance has crashed or is partitioned, a higher-positioned instance’s timer fires and it pages, so a page is never lost. The cost is occasional duplicate pages during a partition — an acceptable trade for an alerting system, where a missed page is catastrophic and a duplicate is merely annoying. This is the canonical worked example of choosing at-least-once + fail-open over exactly-once for a reliability-critical pipeline.

Multi-vendor. Critical alerts route to both PagerDuty and Opsgenie simultaneously. If PagerDuty is down (it has been, for hours), Opsgenie still pages. Doubles cost; halves single-vendor risk. Used by financial-services and other ultra-reliability shops.

Independent infrastructure. Run the alerting system on a different cloud, region, account from the systems it monitors. A datacenter outage in your primary region must not affect your alerting in your secondary region.

Synthetic alert tests. Periodically intentionally fire a known-noop alert and verify it pages. If the synthetic stops working, you know the pipeline is broken before a real alert arrives.

9. Scaling Strategy

9.1 Stage 1 — Single Alertmanager

Single Prometheus Alertmanager instance. Hundreds of alerts/day. One escalation policy.

9.2 Stage 2 — Alertmanager HA Pair

Two (or more) Alertmanagers in a memberlist gossip mesh that replicates the notification log; the position-based staggered-send protocol (detailed in §8.7) means both run hot but downstream normally sees one page, with a higher-positioned peer covering for a failed position-0 instance. Survives single-instance failure without losing a page, at the cost of rare duplicates during a partition.

9.3 Stage 3 — Independent Routing Service

Past Alertmanager’s per-instance limits, build a custom routing service. Stateless workers consuming from Kafka, with state stored in PostgreSQL. Horizontal scaling on the worker tier.

9.4 Stage 4 — Multi-Tenant SaaS Topology

For PagerDuty-scale: per-customer tenant isolation, per-tenant rate limits and ingestion quotas, regional deployments to minimize cross-region latency. Per-customer state shards (e.g., by hash of customer_id).

9.5 Stage 5 — Cross-Region Failover

Two complete deployments in two regions. Each handles its assigned customers; on regional failure, traffic fails over (DNS-level, with TTL ~30s for fast failover). State is replicated cross-region asynchronously; brief data-loss window acceptable for the auditing trail (not for active pages, which are stateless once delivered).

9.6 What Breaks First

In practice: (1) telephony provider rate limits (Twilio, Bandwidth) during major incidents — the system must spread page volume across multiple providers; (2) escalation timer storage at restart — losing in-flight escalation state is bad; persist in a queryable durable store; (3) schedule resolver under thundering herd of “who is on-call now?” queries; cache aggressively; (4) routing tree complexity — a 1000-rule routing tree gets slow if naively iterated, requires compiled label-matcher trie.

10. Real-World Examples

PagerDuty (https://www.pagerduty.com/, founded 2009). The market leader. Its public engineering blog confirms a Kafka-backed, decoupled async event pipeline, with a distributed task scheduler built on Apache Kafka (task queuing/partitioning), Cassandra (durable task persistence), and Akka (concurrency structuring) — tasks are persisted to Cassandra so they cannot be lost, while near-term scheduled tasks are also held in memory for low-latency firing (PagerDuty Engineering: Distributed Task Scheduling). Integration with Twilio for SMS/voice and direct push for the PagerDuty mobile app; the “Acknowledge / Resolve” buttons on the mobile app and the SMS-reply-/ack pattern are the de facto standard ack mechanism. The PagerDuty mobile app uses iOS Critical Alerts (an Apple-granted entitlement, available since iOS 12, requiring explicit per-user opt-in) to bypass Focus/silent modes for high-urgency pages (PagerDuty mobile-app settings).

A real “the alerting system itself went down” post-mortem. On 28 August 2025 PagerDuty suffered two cascading incidents (roughly 03:53–10:10 UTC and 16:38–20:24 UTC), with a peak ~95% of API events rejected for a 38-minute window (PagerDuty: August 28 Kafka Outages). The root cause is a textbook lesson for this note: a new API-usage-tracking feature instantiated a fresh Kafka producer on every API request instead of reusing one — at peak ~84× the normal rate of new producers (~4.2 million extra producers/hour). The per-producer metadata exhausted JVM heap on the Kafka brokers, triggering garbage-collection thrash and heap exhaustion that cascaded across the cluster and stalled downstream processing. Engineers first doubled the broker heap and did rolling restarts; the fix that held was rolling back the offending feature. This incident is the empirical answer to the §14 “five nines for paging” question — even the market leader’s pipeline can be down for hours, which is why dead-man’s switches and multi-vendor routing exist.

Atlassian Opsgenie (https://www.atlassian.com/software/opsgenie). Founded 2012, acquired by Atlassian 2018. Similar architecture; deeper integration with Jira and Statuspage. As of 2025 Opsgenie is being wound down: Atlassian stopped new Opsgenie purchases on 4 June 2025, its alerting/on-call features have been folded into Jira Service Management (and Compass), and Opsgenie access ends with all unmigrated data deleted on 5 April 2027 (Atlassian: Migrate from Opsgenie). For interview purposes Opsgenie is now best framed as “the on-call engine inside Jira Service Management,” not a standalone product.

Splunk On-Call (formerly VictorOps, acquired 2018). Splunk-integrated alerting with custom collaboration features (a chat pane for on-call response).

Prometheus Alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/). The open-source workhorse. Single-binary, gossip-clustered for HA, configured via YAML. Receives alerts from Prometheus servers (or any HTTP source); handles routing/grouping/inhibition; dispatches to email/PagerDuty/Slack/webhook receivers. Most “we run our own alerting” deployments are some Alertmanager + PagerDuty hybrid.

Grafana OnCall (https://grafana.com/oss/oncall/). 2021, open source. Grafana’s answer to the PagerDuty-Alertmanager spectrum: full schedules, escalation policies, mobile push — open source, integrated with Grafana dashboards.

Google’s SRE Workbook alerting chapter (Beyer et al. 2018, Ch. 5, https://sre.google/workbook/alerting-on-slos/). The intellectual foundation for modern burn-rate alerting. Most modern alerting practice traces to its prescriptions.

11. Tradeoffs

Design ChoiceOption AOption BWhen A winsWhen B wins
Alert philosophyThreshold-basedBurn-rate against SLOSimple ops, no SLO disciplineMature SRE, want noise reduction
Grouping waitgroup_wait: 0 (fire immediately)group_wait: 30s (collect siblings)Hyper-critical, can’t waitMost cases — reduces flood
Receiver fan-outSingle receiverMulti-channel (SMS + push + voice)Reliable channel existsWorst-case reliability mandate
HA strategySingle Alertmanager + dead-man’s switchRedundant pair gossip-clusteredSmall deploymentHigh-reliability mandate
Vendor strategySingle vendor (PagerDuty)Multi-vendor (PD + Opsgenie)Simpler ops, smaller costCatastrophe-class mandate
Schedule formatBuilt-in rotationsiCal importStandard rotationsComplex, externally-defined schedules
Escalation durationShort (5 min between tiers)Long (15 min)Critical incidents, awake fastReduce wake-ups; tolerate slower response
Source of truthSelf-hosted AlertmanagerSaaS PagerDutyCost control; data-sovereigntyOperational simplicity; mature mobile app

12. Pitfalls

  1. Alert fatigue. The catastrophic failure mode. When the on-call gets 50 pages per shift, they start ignoring all of them, and the one real page gets missed. Treat pages-per-shift as a leading indicator (Beyer et al. 2018 Ch. 11.5); set an explicit budget (target < 2 pages/shift); systematically clean up noisy alerts every retrospective.

  2. Pages firing during planned maintenance. Operators forget to silence alerts before a deploy; the deploy triggers transient errors; the on-call gets paged for the operators’ own change. Solve with maintenance windows (silence by service + time range) and change correlation (auto-silence if a deploy event from the CI system arrived in the last N minutes).

  3. The acknowledged-but-forgotten incident. Alice acks at 03:05 to silence the page; then falls back asleep without fixing. The alert is silenced for repeat_interval (4 hours) and the issue persists. Solve with ack timeouts: if the alert is still firing N minutes after ack and unacked progress, re-page.

  4. Cascading inhibition turning into “everything is suppressed”. A poorly-written inhibition rule (alertname=* matching all targets when datacenter_down fires) suppresses everything including unrelated alerts. Test inhibition rules with synthetic firings; review them in code review.

  5. Time-zone bugs in schedules. A weekly handoff at 9am Pacific double-fires during DST spring-forward (handoff at 9am STARTS, then the “second 9am” hour also triggers a handoff). Use IANA tz; test across DST boundaries; freeze handoff times in UTC if the team can stomach the calendar inconsistency.

  6. Missing the page because the on-call’s phone died. No mitigation in the alerting system itself; the only fix is redundancy in humans — escalation to secondary after 5 min if no ack.

  7. Routing rule conflicts. Two routes match the same alert; ambiguous which receiver wins. Most routing engines pick “first match wins” — make sure that’s documented and the rule order is reviewed.

  8. Notification provider outages. Twilio has had multi-hour SMS outages. The alerting system must have a secondary notification provider configured and auto-failover.

  9. Webhook receivers that swallow errors. A custom webhook receiver returns 200 OK but does nothing (a bug in the receiver). Pages “succeed” but no human is notified. Mitigate with downstream confirmation: the human must ack within N minutes or escalate, even on “successful” delivery.

  10. The alerting system itself causing alerts. Restart of Alertmanager fires “Alertmanager restarted” alerts that propagate through the pipeline, possibly creating a cycle. Mark internal alerts with a special label that excludes them from normal routing.

  11. Forgetting to test silences. A silence is created for service=foo, but its label match has a typo; nothing is actually silenced; the pager continues. Alertmanager’s amtool silence query can list active silences and the alerts they affect; review periodically.

  12. PII in alert annotations. A developer writes a runbook with customer.email = {{ $labels.email }} in the alert annotation; the email leaks to the on-call and into Slack. Strip PII at the source; review alert annotations for sensitive data.

13. Common Interview Variants

  • “How do you reliably page someone when your alerting system is down?” Discuss dead-man’s switch (independent heartbeat to external service), redundant pair, multi-vendor, independent infrastructure. Synthesize: there is no single solution; you stack mitigations.
  • “Implement a rotation schedule resolver.” Walk through the algorithm in §8.4; cover overrides, restrictions, time zones, DST.
  • “Design the deduplication system.” Fingerprint hashing; sliding-window dedup table (last seen time per fingerprint); resolution mechanics (when does an alert clear? resolve_timeout vs explicit endsAt).
  • “How do you implement burn-rate alerting?” Recording rules in PromQL computing per-window error rates; alert rules combining short and long windows; the math behind the 14.4× factor.
  • “How do you avoid waking up four people for one incident?” Inhibition rules suppressing symptoms; grouping by service+cluster collapses related alerts; per-team routing so unrelated alerts go to the right team only.
  • “Build the on-call mobile app’s notification path.” APNs/FCM critical-alert registration; bypassing Do Not Disturb; ack via a one-tap action that hits the alerting API; recovery on reconnect (replay missed pages).
  • “Implement maintenance windows.” Time-bounded silences with label matchers; integration with CI/CD so deploys auto-create silences; guard against silences that outlive the deploy window.
  • “How do you handle a noisy alert that pages at 03:00 every night for the same false positive?” Track pages by alertname; surface the worst offenders in a “noise leaderboard”; require an action item from the on-call retrospective for any alert paging > N times per week.

14. Open Questions / Uncertain

Three claims from earlier drafts were re-verified against primary sources in this pass and the flags resolved:

  • PagerDuty’s architecture is now cited to PagerDuty’s own engineering blog (Kafka + Cassandra + Akka task scheduler; see §10), not inferred. The August 2025 Kafka outage post-mortem corroborates the Kafka-centric pipeline in fine detail.
  • Burn-rate thresholds (14.4× / 6× / 1×) are confirmed against Google’s SRE Workbook “Alerting on SLOs” chapter, Table 5-8, for a 99.9% SLO over a 30-day period, with the three tiers consuming 2% / 5% / 10% of the monthly budget. An earlier draft incorrectly used a 28-day window and “5% per page”; §8.6 is corrected.
  • Opsgenie’s status is confirmed: winding down into Jira Service Management, end-of-life 5 April 2027 (see §10).

What remains genuinely uncertain:

Uncertain

Verify: deeper PagerDuty/Opsgenie internals beyond what their engineering blogs disclose — exact tenant-sharding keys, per-region replication topology, and current escalation-state datastore. Reason: these specifics are proprietary and not published; the public blog confirms the Kafka/Cassandra/Akka building blocks but not the full topology. To resolve: only authoritative confirmation would be a current PagerDuty architecture talk or a deeper engineering post. uncertain

On “five nines for paging,” the verification strengthens the skepticism rather than removing it: Twilio’s published delivery SLA is 99.95% — already two orders of magnitude short of five nines on its own — and PagerDuty itself was down for the better part of two windows totalling ~10 hours on 28 August 2025 (§10). Five-nines paging is therefore not achievable from any single vendor or pipeline; the practical figure comes only from stacking independent mitigations (dead-man’s switch + redundant Alertmanagers + multi-vendor routing + independent infrastructure, per §8.7). Treat “five nines for paging” as an aspirational system-level target met by redundancy, never a property of one component.

15. See Also

Graph View

Backlinks