vault

Chaos Engineering Principles

15 min read

Chaos Engineering Principles

Chaos engineering is the discipline of experimenting on a system in order to build confidence in its capability to withstand turbulent conditions in production (principlesofchaos.org). The premise is uncomfortable but exact: a distributed system is too complex to reason about in the abstract, so even when every individual service is behaving correctly, the interactions between them produce emergent failures no one predicted — retry storms, cascading overloads, mis-tuned fallbacks. You do not actually know your system is resilient until you have made it fail on purpose, watched what happened, and found the weaknesses before your users do. Chaos engineering formalizes that into a scientific method: define what “healthy” looks like as a measurable steady state, hypothesize it will survive a realistic disturbance, inject the disturbance, and see whether the hypothesis holds. Crucially it is not “randomly breaking things in production” — it is controlled experimentation with a deliberately minimized blast radius, so that the information you gain costs as little user harm as possible.

This is an operational-practice note about the philosophy and principles. The concrete tooling and exercises — Chaos Monkey and the Simian Army, fault-injection platforms, scheduled GameDays, the chaos maturity model — live in its sibling Fault Injection and GameDays. The resilience patterns that chaos experiments test (circuit breakers, bulkheads, retries with jitter) live in System Architectures MOC; the failure math it exercises lives in Retry Storms and Cascading Failures. Here we teach why testing resilience beats assuming it, and the four (plus one) principles that make the testing rigorous rather than reckless.

Mental Model — Resilience Is a Hypothesis Until You Test It

The mental shift is to treat every belief about your system’s resilience as an unverified hypothesis. “We can lose an instance and users won’t notice.” “If the recommendations service is down, we fall back to a static list.” “Our retries have jitter so they won’t stampede.” Each of these is a claim about behavior under conditions you have probably never actually created. Chaos engineering is the practice of converting those claims into experiments and running them, because the alternative — assuming the resilience mechanisms work — is how organizations discover during a real outage that the failover was never wired up or the fallback path itself throws.

flowchart LR
    SS["1. Define STEADY STATE<br/>a measurable output:<br/>e.g. video-plays/sec,<br/>error rate, p99 latency"] --> HYP["2. HYPOTHESIZE<br/>steady state persists in<br/>control AND experiment<br/>groups"]
    HYP --> VARY["3. INJECT real-world<br/>variables into experiment<br/>group: kill instance,<br/>add latency, fail dependency"]
    VARY --> COMPARE{"4. Compare experiment<br/>vs control steady state"}
    COMPARE -->|"no difference"| CONF["Confidence gained;<br/>widen blast radius,<br/>automate, repeat"]
    COMPARE -->|"difference — hypothesis<br/>DISPROVEN"| WEAK["Weakness found:<br/>fix it BEFORE it<br/>becomes an outage"]
    WEAK -.-> SS

What it shows and the insight to take: this is the scientific method with an experiment group and a control group, applied to production. Notice the goal is to try to disprove the hypothesis — a difference in steady state between the group you disturbed and the group you left alone is not a failure of the experiment, it is its success: you found a weakness on your own terms, cheaply, instead of during a 2 a.m. incident. And notice steady state is a measurable output (throughput, errors, latency) — chaos engineering verifies that the system works, not how it works internally (principlesofchaos.org).

Why Test Resilience Instead of Assuming It

The motivating observation, stated plainly on the Principles site, is that in a distributed system the interactions between correctly-functioning services can still produce unpredictable outcomes (principlesofchaos.org). The specific systemic weaknesses it names are exactly the failure modes cataloged elsewhere in this vault: improper fallback settings when a service is unavailable; retry storms from badly-tuned timeouts; a downstream dependency collapsing under too much redirected traffic; and cascading failures from a single point of failure — see Retry Storms and Cascading Failures and Timeouts Deadlines and Deadline Propagation. These are not bugs in any one service; they are properties of the system, invisible to unit and integration tests that exercise components in isolation.

This is the sharp line between chaos engineering and ordinary testing. A test verifies a known condition: given input X, assert output Y — it checks assertions you already knew to write. Chaos engineering explores for unknown conditions — it is a search for the weaknesses you did not know you had (the “unknown-unknowns,” the same distinction drawn in Observability versus Monitoring). Testing tells you the properties you thought to check are intact; chaos tells you which properties you never thought to check are broken. Microsoft’s Well-Architected guidance makes the same point operationally: without deliberate testing against failure scenarios you cannot know whether your resiliency patterns actually work or whether the workload recovers within its targets (Azure WAF, reliability testing).

The Principles, Walked Through

The Principles of Chaos site lists four “advanced principles” that a mature chaos practice embodies, plus the overriding safety constraint of minimizing blast radius. Each is a discipline that separates rigorous chaos engineering from reckless breakage.

1. Build a Hypothesis around Steady-State Behavior

Start by defining a steady state — a measurable output that represents the system operating normally. The canonical Netflix example is video plays started per second (labelled SPS): a business-level signal of customer engagement that, under normal conditions, follows a smooth, predictable curve (Netflix, Chaos Engineering Upgraded). Good steady-state metrics are outputs the user cares about — throughput, error rate, latency percentiles — not internal attributes like CPU or queue depth (principlesofchaos.org). The hypothesis is then: “this steady-state metric will remain within its normal band in both the group I disturb and the group I leave alone.” Framing it as steady-state-that-should-persist is what makes the experiment falsifiable and what ties chaos directly to your SLOs — a chaos experiment that pushes an SLI out of band is, by definition, a resilience gap worth a fix.

2. Vary Real-World Events

The variables you inject should mirror events that actually happen in production: hardware failures like a server dying, software failures like a malformed response, and non-failure events like a traffic spike or a scaling operation (principlesofchaos.org). Prioritize by impact or frequency — inject the failures that are either most likely or most damaging first, because those are where verified resilience pays off most. This is the principle that keeps chaos relevant: injecting a fault that could never occur teaches you nothing useful, while injecting the region outage you actually fear teaches you whether your regional failover works. Azure’s guidance operationalizes this as driving experiments from a failure-mode analysis and prioritizing by impact and likelihood (Azure WAF).

3. Run Experiments in Production

This is the principle that unsettles people, and it is the one that matters most. Systems behave differently across environments — traffic patterns, data volumes, real dependency latencies, and cache states in production simply do not exist in staging. A resilience property verified in staging is verified about staging, not about the system your users hit. To gain confidence about production, you must experiment against production (principlesofchaos.org). This does not mean recklessly — it means you build the safeguards (blast-radius controls, abort conditions, the ability to stop and roll back) that let you learn from the real system without harming it. Where production experimentation is genuinely too risky, run as close to it as possible; but recognize that every step away from production is a step away from the truth. (Pragmatically, teams start in non-production to shake out the obvious, then graduate to production once they have safeguards — Azure WAF.)

4. Automate Experiments to Run Continuously

Running experiments by hand is unsustainable and, worse, one-shot — a resilience property verified once decays as the system changes underneath it. So automate both the orchestration (injecting the fault) and the analysis (comparing steady states) and run experiments continuously (principlesofchaos.org). This is what turns chaos from a heroic quarterly event into a regression suite for resilience: every architectural change is a chance for a previously-verified property to silently break, and continuous chaos catches that drift. Azure frames it identically — run experiments on a regular cadence to catch regressions, validate new dependencies, and confirm recent changes did not introduce weaknesses (Azure WAF).

5. Minimize Blast Radius

The overriding safety obligation. It is, in the site’s own framing, the responsibility and obligation of the chaos engineer to ensure the fallout from experiments is minimized and contained (principlesofchaos.org). Experiments are permitted to cause short-term negative impact — that is the price of information — but they must never risk a full outage. In practice this means: target the smallest population that still yields signal, set explicit abort conditions that stop the experiment automatically if the steady state degrades past a threshold, and always have a fast way to reverse (which is why clean rollback is a precondition for chaos). Netflix’s automated approach routes a small percentage of traffic — described as adding, for example, 30 ms of latency to a fraction of requests — into an experiment group and compares it against a matched control, so a bad result harms a bounded slice and is detected fast (Netflix, Chaos Engineering Upgraded). Azure’s version: contain blast radius, target components you can recover quickly, and stop the experiment if it exceeds scope (Azure WAF).

The Control-vs-Experiment Structure in Practice

The principles come together in the experimental design Netflix uses. Rather than disturb the whole system, you create two matched groups: a control that continues under normal conditions and an experiment group into which you inject the fault, both drawn from real production traffic (Netflix, Chaos Engineering Upgraded). You then compare the steady-state metric across the two. This design is powerful for two reasons. First, it isolates the effect of the injected fault from the ordinary noise of production — if control and experiment diverge, the fault caused it; if both dip together, that was just the daily traffic curve, not your experiment. Second, it bounds the blast radius mechanically: only the experiment group is ever at risk, so the maximum harm is capped by how small you make that group. This is the same statistical structure as a canary — a small population exposed and compared against a control — which is why canarying and chaos are close cousins.

A Worked Experiment, End to End

To make the principles concrete, consider a team that believes its checkout service degrades gracefully when the recommendations service is unavailable — falling back to a generic product list rather than failing the page. That belief is a hypothesis; here is how chaos engineering tests it.

  1. Define steady state. The team picks checkout completions per minute and checkout p99 latency as the steady-state metrics — user-visible outputs, not internal counters. Under normal load these sit in a known band (say, 480–520 completions/min, p99 under 800 ms).
  2. Form the hypothesis. “When the recommendations dependency is unavailable, checkout completions and p99 latency in the experiment group stay within their normal band, indistinguishable from the control.” Note this is written to be disproven.
  3. Scope the blast radius. Rather than blackhole recommendations for everyone, the team routes 1% of checkout traffic into an experiment group and injects the fault only there, with an abort condition: if experiment-group completions fall more than 10% below control for 60 seconds, the injection stops automatically.
  4. Inject the variable in production. Using a fault-injection tool, the team makes calls from the experiment group’s checkout instances to recommendations time out — a real-world event (a dependency being down), against real production traffic.
  5. Compare and decide. They watch the two groups. If completions and latency track together, the fallback works and confidence is earned — they can widen the experiment and eventually automate it as a recurring regression check. If the experiment group’s completions crater while the control holds steady, the hypothesis is disproven: the “graceful” fallback is not graceful — perhaps it retries the dead dependency with no budget (a retry storm) or the fallback path itself throws. Either way they found the weakness on 1% of traffic, at a time of their choosing, with an engineer watching — not at 2 a.m. when recommendations really dies.

The value is starkest in the failure case. The alternative to running this experiment is assuming the fallback works — and that assumption is only ever tested for the first time during the real outage, at 100% blast radius, with no one expecting it. The experiment converts a latent, unbounded risk into a bounded, observed one. That trade — a little controlled harm now to prevent a lot of uncontrolled harm later — is the entire economic argument for chaos engineering, and it is why Gremlin likens the practice to a vaccine: a small, managed dose of the pathogen to build immunity to the real thing (gremlin.com).

A Brief History

Chaos engineering grew out of Netflix’s migration from its own data centers to Amazon Web Services around 2010–2011. Moving onto commodity cloud instances that could vanish at any moment, Netflix engineers (among them Greg Orzell, with Nora Jones and Casey Rosenthal later central to the discipline) built Chaos Monkey — a tool that randomly terminates production instances during business hours to force engineers to build services resilient to instance loss (Wikipedia, chaos engineering; Chaos Monkey docs). The philosophy inverted the usual instinct: rather than protect the pampered instance, assume instances die constantly and make that the normal, well-exercised case. Netflix open-sourced Chaos Monkey in 2012, and it grew into the Simian Army — Chaos Gorilla (drops an entire AWS Availability Zone) and Chaos Kong (drops a whole AWS Region) escalating the blast radius to test zonal and regional failover (Wikipedia). The generalized discipline — with its steady-state, hypothesis-driven framing — and the Principles of Chaos manifesto followed, and the practice spread industry-wide, spawning commercial “failure-as-a-service” platforms like Gremlin (gremlin.com) and managed offerings such as Azure Chaos Studio and AWS Fault Injection Service. Precursors existed — Amazon’s “GameDay” exercises (Jesse Robbins, ~2003) and Google’s Disaster Recovery Testing (DiRT, Kripa Krishnan, ~2006) — reflecting the same conviction from the operations side: you do not trust a recovery path you have not exercised (see The Wheel of Misfortune and Incident Preparedness).

Uncertain

Verify: exact founding year of the principlesofchaos.org manifesto and the first coining of the term “chaos engineering” (as distinct from the Chaos Monkey tool). Reason: sources disagree — the site itself shows a “last updated” of March 2019, Wikipedia associates a 2020 date with the O’Reilly Chaos Engineering book (Rosenthal & Jones), and secondary sources variously cite 2015–2017 for the manifesto’s origin. The tool (Chaos Monkey, 2010–2011; open-sourced 2012) is well-attested; the manifesto’s precise date is not. To resolve: check the site’s version history / the O’Reilly book’s front matter for the manifesto’s original publication date. #uncertain

Common Misunderstandings

  • “Chaos engineering is randomly breaking production.” No — it is controlled experimentation with a hypothesis, a bounded blast radius, and abort conditions. The randomness in Chaos Monkey is a tactic for surfacing untested cases, not the definition. Uncontrolled breakage is negligence, not chaos engineering.
  • “You need a fancy platform first.” No — you need a measurable steady state and a hypothesis first. The tool is secondary; a team that cannot state its steady state is not ready to inject faults, however good its tooling.
  • “Do it once and you’re covered.” No — resilience decays as the system changes; the fourth principle (automate, run continuously) exists precisely because a property verified last quarter may be broken by this week’s deploy.
  • “Staging is good enough.” Staging verifies staging. The third principle insists on production because production is where the real traffic, data, and dependency behavior live — and those are exactly what emergent failures depend on.
  • “It replaces good design / testing.” No — it complements them. Chaos assumes you have already built resilience patterns (Circuit Breaker Pattern, Fallback and Graceful Degradation Pattern, Retry with Backoff Pattern); its job is to verify those patterns actually work under real conditions.

Preconditions — Do Not Start Here

Chaos engineering is a maturity practice, not a starting point. Before injecting faults you need: observability good enough to measure the steady state and detect divergence quickly (The Three Pillars of Observability, The Four Golden Signals); a fast, reliable rollback / abort so you can stop an experiment that goes wrong (Rollback and Forward-Fix); and resilience mechanisms already in place to be worth testing. Running chaos against a system with no monitoring and no rollback is not an experiment — it is just an outage you scheduled. Azure states the prerequisite bluntly: make sure mitigation strategies are in place before running experiments, and put safeguards in place so you can stop experiments and roll back quickly (Azure WAF).

See Also

Graph View

Backlinks