vault

Load Testing Types and Practice

16 min read

Load Testing Types and Practice

Load testing is the operational discipline of deliberately driving synthetic (or replayed) traffic at a system in order to answer a question you cannot answer by reasoning alone: how does this thing actually behave as demand rises — where does latency start to climb, where does throughput stop growing, and where does it break? It is the empirical half of capacity planning — forecasting tells you how much demand to expect, load testing tells you whether the system can serve it and what it does when it cannot. The field has settled on a small taxonomy of test shapes, each isolating a different failure question: a load test validates behavior at expected peak, a stress test pushes past that peak to find the breaking point, a soak (endurance) test holds moderate load for hours to surface slow leaks, a spike test slams the system with a sudden surge, and a breakpoint test ramps continuously to locate the exact capacity ceiling (the “knee”) where the response-time curve turns vertical (k6 test-types guide). The SRE Book frames the imperative bluntly: you must test a component to its breaking point and beyond, because the way a system transitions from healthy to failing is itself a design property you need to have observed before a real incident forces you to observe it (SRE Book ch. 22).

This note is the SRE-practice framing. The mechanisms it consumes live elsewhere in the vault: the queueing math behind the knee is in Utilization Targets and the Latency Knee and The Universal Scalability Law; the overload responses a load test provokes (rejecting or degrading requests) are Load Shedding and Graceful Degradation and Autoscaling in Practice; the Kubernetes-specific capacity mechanics are Capacity Planning on Kubernetes. Here the concern is the practice: which test to run, in what order, against what environment, and — most important — how to keep the test honest so its results are load-bearing rather than reassuring theater.

Mental Model: Test Shapes as Questions

flowchart TD
    Q["What do I need to know<br/>about behavior under demand?"]
    Q --> L["Does it meet SLOs<br/>at expected peak?"]
    Q --> S["Where does it break,<br/>and how gracefully?"]
    Q --> E["Does it stay healthy<br/>over hours/days?"]
    Q --> P["Can it survive a<br/>sudden surge and recover?"]
    Q --> B["What is the exact<br/>capacity ceiling?"]

    L --> LT["Load / average-load test<br/>hold at expected VUs"]
    S --> ST["Stress test<br/>ramp above average, hold"]
    E --> SO["Soak / endurance test<br/>average load, many hours"]
    P --> SP["Spike test<br/>near-instant jump, then drop"]
    B --> BP["Breakpoint test<br/>ramp forever until failure"]

    LT -. "reveals" .-> R1["latency/error at peak"]
    ST -. "reveals" .-> R2["bottleneck, error onset"]
    SO -. "reveals" .-> R3["memory leaks, resource drift"]
    SP -. "reveals" .-> R4["recovery, state corruption"]
    BP -. "reveals" .-> R5["the knee / max QPS"]

What it shows and the insight to take: the five shapes are not five tools — they are five questions, each answered by a different load-versus-time profile. The single most useful thing to internalize is that the test shape and the failure question are the same choice. If you want to know whether you meet your SLO on Black Friday, you hold expected-peak load and watch the golden signals; if you want to know what happens when Black Friday is 3× bigger than forecast, no amount of at-peak testing will tell you — you must ramp past peak and watch the failure mode. Running the wrong shape answers the wrong question, confidently.

The Load-vs-Throughput Curve: Where the “Knee” Lives

Every capacity test is ultimately probing the same underlying curve, so it is worth stating it precisely before walking the individual tests. As offered load rises, a healthy server’s throughput (useful work completed per second, sometimes called goodput) climbs roughly linearly — each additional unit of demand is served. Meanwhile latency stays low and flat, because there is spare capacity and requests are not queueing behind each other. This continues until the system approaches a resource limit (CPU, a thread pool, a connection pool, a lock, a downstream dependency). Past that point the behavior changes character: throughput flattens (it cannot exceed what the bottleneck resource permits) while latency rises sharply — requests now spend most of their time waiting in queue rather than being served. Push further and throughput can actually collapse below its peak, as the machinery of overload (context switching, garbage collection, retry traffic, health-check failures) consumes capacity that would otherwise do work (SRE Book ch. 22).

The knee is the inflection where latency stops being flat and turns steeply upward. It is not a single magic number — it is a region — but it is the most important region on the curve, because it is the boundary between “adding load costs a little latency” and “adding load costs unbounded latency.” Queueing theory explains why the knee exists and why it is so sharp: for an M/M/1-style queue, mean waiting time scales as roughly 1 / (1 − ρ) where ρ is utilization, so as ρ → 1 latency goes to infinity — the derivation and its consequences for provisioning targets live in Utilization Targets and the Latency Knee and connect to Little’s Law (L = λW, mean concurrency = arrival rate × mean latency), the same law Netflix uses to derive concurrency limits automatically (Netflix, Performance Under Load). The practical takeaway for a load tester: the goal of most capacity tests is to locate the knee relative to your expected load, so you know how much headroom you actually have — not the theoretical headroom the spec sheet promises.

The Test Types, Walked Through

Load test (average-load / expected-peak)

The baseline test. You drive the system at the traffic level you expect — either average steady-state or forecast peak — and hold it long enough (k6’s guide suggests roughly 5–60 minutes at the plateau) to see steady-state behavior rather than transient warm-up (k6 average-load). The question is narrow and important: at the load we actually anticipate, do we meet our latency and error SLOs, and how much resource does that consume? A load test is not trying to break anything; it is establishing the reference point every other test is measured against. It should be preceded by a smoke test — a tiny run (a handful of virtual users for seconds to minutes) whose only job is to confirm the test script itself works and the system is not already broken, so you do not waste a 60-minute run on a typo (k6 smoke test).

Stress test (above-peak, held)

A stress test asks how does the system behave when load exceeds the average? You ramp — deliberately with a longer ramp-up than a spike, to let the system respond realistically — up to an above-average level, hold it on a plateau, then ramp down. k6’s canonical example ramps to 200 virtual users over 10 minutes, holds 30 minutes, and ramps down over 5 (k6 stress testing). There is no universal “how much above average” — the guidance is explicitly that the overshoot might be a few percent or several orders of magnitude, chosen to match a realistic risk (a sale, a viral event, a failover that doubles a surviving cluster’s traffic). Crucially, the k6 guide insists you run stress tests only after average-load tests: you need the healthy baseline first, both to isolate what the extra load specifically breaks and to avoid burning resources chasing a problem that already existed at normal load. What a stress test reveals is the character of degradation — which resource saturates first, at what point error rates climb, and whether the service degrades smoothly or falls off a cliff.

Soak / endurance test

Some defects are invisible at any instant and only emerge over time. A soak test holds a moderate (typically average) load for a long duration — k6 lists 3, 4, 8, 12, 24, and 48–72 hours as representative — specifically to surface slow-accumulating problems: memory leaks, file-descriptor or connection-pool leaks, disk filling with logs, cache eviction pathologies, database connection churn, and gradual response-time drift (k6 soak testing). The load level is deliberately unremarkable — the point is not intensity but duration. The expected outcome is a flat line: resource utilization and latency should stay stable across the whole soak. A rising memory curve over 24 hours at constant load is the signature of a leak that would take days or weeks to crash the service in production, which is exactly the kind of failure that pages someone at 3 a.m. for reasons no one can immediately explain. The k6 guidance is emphatic that a soak test is worthless without backend monitoring — you must be watching RAM, CPU, network, disk, and application metrics for the whole run, because the failure is in the trend, not in any single request’s result.

Spike test

A spike test validates survival and recovery under a sudden, short, massive surge — the near-vertical jump with little or no ramp-up, a brief (or absent) plateau, then a rapid drop, mimicking a flash sale, a ticket on-sale, a viral link, or a thundering-herd reconnect (k6 spike testing). Two things distinguish it from a stress test: the shape (instantaneous vs. gradual ramp) and the question (survival and recovery vs. steady-state degradation). Errors during the spike are expected and normal — the interesting questions are whether the system recovers to normal operation once the spike passes, or whether it stays wedged (a retry storm or overload death-spiral that outlives the traffic), and whether any state was corrupted in the chaos. A spike test also exercises the reaction time of autoscaling: a system that scales up in 90 seconds will still eat 90 seconds of overload during an instantaneous spike, and the spike test is where you find out whether that gap is survivable.

Breakpoint test

Where a stress test picks an above-average level and holds it, a breakpoint test refuses to pick a level: it ramps load continuously, with no plateau, until the system breaks — then you read off the load at which it broke (k6 breakpoint testing). This is the direct way to find the knee and the maximum sustainable throughput. k6’s advice is mechanically specific: use the ramping-arrival-rate executor, which keeps pushing the arrival rate up even as the system slows down — as opposed to a VU-based executor, where a slowing system naturally throttles its own request rate and hides the true ceiling. You watch for the failure signature (response times crossing a threshold, timeouts, HTTP errors, or resource saturation) and abort at that point, either manually or with an abortOnFail threshold. A critical pitfall the guide flags: do not run a breakpoint test against an auto-scaling environment, because the scaler will keep adding capacity and obscure the true per-unit limit — you are then measuring your cloud budget, not your software’s ceiling.

The Ordering Discipline

The k6 guides repeat one rule across every page, and it is the load-testing equivalent of “walk before you run”: each test type assumes the previous ones passed. Smoke before load, load before stress, stress before spike/soak/breakpoint. The logic is not bureaucratic — it is about attribution. If you run a breakpoint test and it fails at low load, you cannot tell whether you found a real capacity ceiling or merely re-discovered a bug that was already present at average load. By establishing that the system is healthy at each lower intensity first, any failure at a higher intensity is attributable to that intensity. Skipping the ladder produces results you cannot interpret.

Environment: Staging vs. Production

Where you run the test is as consequential as which test you run, and it is a genuine trade-off with no free answer.

Staging / pre-production is safe — you can push a system to collapse without hurting a single real user — and it is the only responsible place to run destructive breakpoint and stress tests early. Its fatal weakness is fidelity: a staging environment almost never has production’s data volume, cache state, dependency topology, network conditions, or hardware, so its knee sits at a different load than production’s. A test that passes in staging can still fail in production because the staging database had a thousand rows and production has a billion, so a query that was memory-resident in staging hits disk in production.

Production load testing — running synthetic load against the real system, ideally with real traffic patterns — trades safety for realism. The SRE Book explicitly recommends testing with real traffic patterns where possible, because caching behavior differs dramatically between a gradual organic ramp and an impulse of synthetic requests, and between real user-behavior distributions and a uniform synthetic one (SRE Book ch. 22). Production testing must be bounded by safety margins: load shedding armed, a kill switch ready, blast radius limited to a fraction of capacity, and observability watching for real-user impact. The SRE Book goes further and describes testing at the cluster level — deliberately reducing task counts beyond expected patterns, losing whole clusters, and blackholing backends — to verify the system degrades rather than cascades. This overlaps with chaos engineering: the boundary is that a load test asks “how much can it take?” while chaos asks “what happens when a specific thing fails?” — but both are ultimately production experiments with a steady-state hypothesis and a controlled blast radius.

Pitfalls: How Load Tests Lie

A load test’s danger is not that it fails but that it passes dishonestly — giving you confidence the production system will not honor. The recurring failure modes:

  • Unrealistic traffic mix. Hammering a single cheap endpoint (a health check, a static page) at high QPS produces a beautiful number that says nothing about the expensive, contended paths real users hit. The traffic distribution must mirror production — same endpoint mix, same payload sizes, same think-time between requests — or the knee you measure is the knee of the wrong workload. The SRE Book’s preference for replaying real traffic patterns is precisely to avoid this.
  • Warm caches / unrealistic data. If the test replays the same small set of requests, everything is served from a warm cache and you measure cache throughput, not system throughput. Production sees a long tail of cold keys that miss cache and hit the database. A realistic test must exercise a realistic cardinality of keys, or it flatters the system by hiding the cache-miss path — the same class of mistake that makes a soak test’s stable memory line meaningless if the working set never rotates.
  • Warm-up masking cold-start. Conversely, measuring only steady-state hides cold-start costs — JIT warm-up, connection-pool establishment, lazy initialization — that dominate the first seconds after a deploy or a scale-up event. This is exactly what a spike test is designed to expose, which is why the shapes are complementary rather than redundant.
  • The client is the bottleneck. If your load generator cannot generate the load — because it is single-node, thread-starved, or its own network is saturated — you measure the generator’s ceiling and mistake it for the system’s. The SRE Book also warns about connection load specifically: the cost of establishing and health-checking connections can dominate, so a test that reuses a handful of connections misses the connection-churn cost a real client fleet imposes (SRE Book ch. 21).
  • Auto-scaling hides the ceiling. As noted, a breakpoint test against an elastic environment measures the scaler, not the software. Pin the capacity to find the true per-instance limit, then test the scaler’s reaction separately.
  • Ignoring recovery. A test that only measures behavior during overload misses the most operationally important question: does the system recover when load drops, or does it stay wedged? The SRE Book stresses measuring how much load reduction is required to stabilize — sometimes recovery needs traffic dropped to a small fraction of normal because restarted servers immediately meet a retry flood. A spike test with a proper post-spike observation window is where this shows up.

Uncertain

Verify: the specific k6 duration/VU figures cited (e.g. average-load 5–60 min, stress 10/30/5-min stages, soak 3–72 h, the 200-VU example). Reason: these are illustrative defaults from the k6 test-type guides as of 2026-07, and Grafana revises these docs; they are conventions, not universal standards, and other tools (Gatling, Locust, JMeter) frame the taxonomy slightly differently. To resolve: re-fetch the k6 guides at time of use and treat the numbers as representative shapes, not prescriptions. #uncertain

Interpreting Results: What “Good” Looks Like

A load test produces numbers; turning them into a capacity decision is the actual skill. Anchor the interpretation to your SLO, not to raw maxima. The maximum QPS a breakpoint test survives is not your usable capacity — usable capacity is the load at which you still meet your latency SLO with headroom to spare, which sits comfortably below the knee (this is why utilization targets are typically 60–70%, not 95%). From a stress or breakpoint test you want three numbers: the load at which latency crosses your SLO threshold (your effective ceiling), the load at which errors begin (the hard ceiling), and the ratio of that ceiling to your expected peak (your headroom multiple). A soak test wants a slope: is any resource trending upward at constant load? A spike test wants a recovery time: how long after the spike until latency returns to baseline, and did anything break permanently? Feed these back into forecasting to decide provisioning, and into load-shedding configuration to decide where the system should start rejecting work — because a load test’s most valuable output is knowing where to place the tripwire before real traffic finds it for you.

See Also

Graph View

Backlinks