AB Testing Platform System Design

An A/B testing platform (also called online controlled experimentation or experimentation platform) is the infrastructure that lets a product team ship two or more variants of a feature, randomly assign users to variants in a way that is deterministic and reproducible, measure downstream behavioral and business metrics, and compute whether the difference between variants is statistically significant — all at scales of millions of users and hundreds of concurrent experiments. It is the operational embodiment of Ron Kohavi’s “Trustworthy Online Controlled Experiments” (Kohavi, Tang & Xu 2020, the canonical textbook cited above) and the in-house systems at Microsoft Bing, Google, Facebook, LinkedIn, Airbnb, Netflix, Uber. The hosted-product version is Optimizely, LaunchDarkly Experiment, Statsig, Split, Eppo. The deceptive simplicity of the headline operation — flip a coin per user, count clicks per side, compare — masks the engineering and statistics depth: the bucketing must be deterministic by user so a returning user always sees the same variant; the bucketing must be statistically random so the variants’ user populations are unbiased; exposure events are recorded only when the user enters the experiment, never preemptively; multiple-comparisons correction (Bonferroni / Benjamini-Hochberg) prevents false positives across many simultaneous experiments; interaction effects between concurrent experiments require randomization layers (Tang et al. KDD 2010); sequential testing / peeking inflates false-positive rates unless using always-valid p-values (Johari et al.); CUPED (Deng et al. WSDM 2013) reduces metric variance by adjusting for pre-experiment behavior — sometimes by 50%, equivalent to needing half the sample size; and guardrail metrics (latency, errors, churn) must not regress while the headline metric is improving. The interview tests whether the candidate can hold the engineering side (low-latency variant assignment, exposure logging, metric pipeline) and the statistics side (significance computation, peeking, multiple comparisons, CUPED) coherently. Most candidates know one or the other; staff-level candidates know both and can talk about how they interact (e.g., why the metric pipeline must capture exposure timing precisely so CUPED can adjust correctly).

1. Why This System Appears in Interviews

A/B testing is the closest thing to “design a complete data product” in modern interviews. The reasons:

• The functional surface is small (assign user to variant, log exposure, measure metric, compute stats), but every element involves nontrivial design.
• The statistics are testable. Senior interviewers ask about peeking, multiple comparisons, sample-ratio mismatch — knowing the right answer matters.
• The infrastructure is testable. Variant assignment must be sub-10ms because it’s on the hot path of every page load; the metric pipeline is a streaming aggregation; the storage backend interacts with Real Time Analytics System Design and Click Tracking System Design.
• It exercises tradeoffs across consistency models, hash functions, multi-tenancy, and ML-adjacent statistics — so the interviewer can probe the candidate’s depth in many directions.

The right answer is a layered architecture: edge variant-assignment service → exposure logging → event metric pipeline → daily aggregation → significance computation → experiment registry / dashboard.

2. Requirements

2.1 Functional Requirements

1. Variant assignment. Given (experiment_id, user_id), return one of {control, treatment_1, treatment_2, ...} deterministically (same input → same output) and uniformly randomly (across users, the distribution is the configured allocation, e.g., 50/50, 33/33/34, 90/10).
2. Targeting rules. An experiment may target only a subset: “US users on iOS, signed up after 2024-01-01.” Users outside the target are not exposed.
3. Exposure logging. Record an exposure event the first time a user encounters the experiment (i.e., enters a code path that calls getVariant). Never log exposure for users who don’t enter.
4. Metric event ingestion. Downstream events (“user clicked Buy,” “user signed up,” “purchase amount = $42”) flow into the platform; metric values per (user, experiment, variant) are computed.
5. Significance computation. Per (experiment, metric), compute the difference between control and treatment and a p-value (or Bayesian probability of being better). Provides the “ship/no-ship” recommendation.
6. Multi-armed experiments. A/B/C/D with arbitrary number of arms; the platform handles N-way comparisons.
7. Holdout groups. Some users globally never participate in any experiment — used to measure long-term effect of all experiments combined.
8. Mutually-exclusive groups. Two experiments that interact must be put in mutually-exclusive randomization layers — a user is in at most one of them.
9. Experiment registry / dashboards. UI for product managers to define experiments, monitor progress, see results.

2.2 Non-Functional Requirements

1. Variant assignment latency. p99 < 10ms on every call. The assignment is on the critical path of page rendering.
2. Concurrent experiments. A large product runs 100–10,000 concurrent experiments. The platform must serve assignments for all of them in one call.
3. Determinism. Same user always sees the same variant. Tested by re-reading the assignment and confirming.
4. Statistical randomness. The bucketing function must produce a uniform distribution across users; statistical tests should not detect systematic bias.
5. Sample-ratio integrity. If the configured allocation is 50/50 but the realized allocation is 51/49, something is broken — must alert.
6. Metric freshness. Daily aggregations available within 1 hour of day end. Real-time dashboards within 5 minutes of event arrival.

Concrete published anchors for these numbers: Microsoft’s Bing reported running over 200 concurrent experiments on any given day and finishing on the order of hundreds of experiment treatments per week (Kohavi et al., “Online Controlled Experiments at Large Scale,” KDD 2013), and Fabijan, Gupchup, Kohavi et al. explicitly reason about “a product running ten thousand experiments in a year” (Fabijan et al., “Diagnosing Sample Ratio Mismatch,” KDD 2019). So the design target of hundreds-to-thousands of concurrent experiments, tens evaluated per page render is grounded, even though any single company’s exact live count is operational and fluctuates.

3. Capacity Estimation

3.1 Variant Assignment QPS

DAU                                          = 200M
page renders per DAU per day                = 30
experiments evaluated per render            = 50  (concurrent experiments)
total assignment lookups per day            = 200M × 30 × 50 = 3 × 10^11
average QPS                                 ≈ 3 × 10^11 / 86,400 ≈ 3.5 × 10^6/sec
peak (5×)                                   ≈ 1.7 × 10^7/sec   (~17M/sec)

17 million assignment lookups per second at peak. This is the hot path; cached at edge, computed locally on the application server, or both.

3.2 Exposure Event QPS

Exposure is logged only the first time per (user, experiment) per session (or per day, depending on policy):

unique (user, experiment, day) tuples       ≈ 200M × 50 = 10^10 / day
exposure event QPS average                  ≈ 10^10 / 86,400 ≈ 1.2 × 10^5 events/sec

Lower than total assignments by orders of magnitude (most assignments are repeat lookups for the same user).

3.3 Metric Event QPS

Metric events are downstream user actions (clicks, purchases, etc.):

metric events per DAU per day               = 100  (clicks, page views, purchases)
metric event QPS                            ≈ 200M × 100 / 86,400 ≈ 2.3 × 10^5 events/sec
peak                                        ≈ 1.2 × 10^6 events/sec

About 1M events/sec at peak — overlapping with the Click Tracking System Design / Real Time Analytics System Design event pipelines.

3.4 Storage

daily exposure events                       = 10^10
size per event                              = 200 bytes
daily exposure storage (raw)                ≈ 2 TB/day
metric events                               = 2 × 10^10/day × 200 bytes = 4 TB/day
combined raw storage                        ≈ 6 TB/day
30-day retention                            ≈ 180 TB
3× replication                              ≈ 540 TB

After aggregation (daily roll-up per experiment per variant per metric), storage drops by 1000× — small. Raw events are needed for ad-hoc analysis but can be tiered to cheap cold storage after the experiment ends.

4. API Design

4.1 Get Variant

GET /api/v1/variant?experiment_id=exp_42&user_id=u123&context=...
→ 200 OK
{
  "variant":  "treatment",
  "config":   {"button_color": "blue", "headline": "New copy"},
  "experiment_id": "exp_42",
  "exposure_logged": true
}

The context field carries targeting attributes (country, app version, device type) so the assignment service can evaluate targeting rules.

In practice, clients fetch all relevant experiments in one call (getAllVariants(user_id, context)), reducing per-page latency to one round-trip.

4.2 Log Metric Event

POST /api/v1/metric_event
{
  "user_id":     "u123",
  "event_name":  "purchase",
  "value":       29.99,
  "timestamp":   "2026-05-08T14:23:11Z",
  "properties":  {"sku": "abc"}
}
→ 202 Accepted

Metric events are not labeled with experiment IDs — the platform joins them downstream against the user’s exposure record.

4.3 Define Experiment (admin)

POST /api/v1/experiments
{
  "name":            "Pricing page redesign",
  "experiment_id":   "exp_42",
  "variants": [
    {"name": "control",   "allocation": 0.5, "config": {"version": "old"}},
    {"name": "treatment", "allocation": 0.5, "config": {"version": "new"}}
  ],
  "targeting": {
    "include": [{"property": "country", "op": "in", "value": ["US"]}]
  },
  "primary_metric":   "purchase_revenue_per_user",
  "guardrail_metrics": ["latency_ms_p95", "error_rate"],
  "randomization_layer": "checkout_flow"
}

4.4 Get Experiment Results

GET /api/v1/experiments/exp_42/results
→ 200 OK
{
  "as_of": "2026-05-08T00:00:00Z",
  "variant_results": [
    {
      "variant":    "control",
      "users":      1234567,
      "metric":     "purchase_revenue_per_user",
      "mean":       3.42,
      "std_error":  0.01
    },
    {
      "variant":    "treatment",
      "users":      1232890,
      "metric":     "purchase_revenue_per_user",
      "mean":       3.51,
      "std_error":  0.01,
      "lift_pct":   2.6,
      "p_value":    0.003,
      "significant": true
    }
  ],
  "sample_ratio_mismatch": false,
  "guardrail_violations": []
}

5. Data Model

5.1 Experiment Registry

Table:   experiments
PK:      experiment_id
Cols:
  name, status, owner, created_at, started_at, ended_at
  variants               JSON  (variant configs and allocations)
  targeting              JSON  (rules)
  randomization_layer    STRING
  primary_metric         STRING
  guardrail_metrics      JSON
  hash_salt              STRING  (used for variant assignment hash)

5.2 Exposure Events

Table:    exposure_events
Partition: (user_id, day)
Cols:
  user_id, experiment_id, variant, exposed_at_ts

Stored in the same event store as Real Time Analytics System Design events, sharded by user_id so per-user joins (for funnel-like analyses) are fast.

5.3 Metric Aggregations

Table:    metric_aggregations
PK:       (experiment_id, variant, metric_name, day)
Cols:
  user_count           BIGINT
  metric_sum           DOUBLE
  metric_sum_squared   DOUBLE     (for std_error computation)
  metric_value_hll     BLOB       (HLL of users with metric > 0)

Pre-aggregated daily; the variance is computed from (sum_squared - sum²/N) / (N-1). HLL of distinct active users for distinct-count metrics.

5.4 Randomization Layers

Table:    layers
PK:       layer_id
Cols:
  name (e.g., "checkout_flow")
  description
  member_experiments    LIST<experiment_id>

Experiments in the same layer are mutually exclusive; the layer’s hashing namespace ensures a user is assigned to at most one experiment within the layer.

6. High-Level Architecture

flowchart TB
    Client[App / Backend Client] -->|getVariant| Edge[Edge Assignment Service]
    Edge -->|cache| EdgeCache[Edge Cache<br/>variant configs + targeting rules]
    Edge -->|deterministic hash| BucketCalc[Bucket Calculator<br/>hash(salt, exp_id, user_id)]
    Edge --> ExpReg[Experiment Registry]
    Edge --> Client

    Client -->|exposure event| LogAPI[Exposure Log API]
    LogAPI --> Kafka[(Kafka Events Stream)]

    Client -->|metric events| MetricAPI[Metric Event API]
    MetricAPI --> Kafka

    Kafka --> JoinSvc[Join Service<br/>exposure × metric per user]
    JoinSvc --> AggSvc[Daily Aggregator]
    AggSvc --> AggStore[(Aggregations Store)]

    AggStore --> StatsSvc[Significance Engine]
    StatsSvc -->|p-values, lift, CIs| ResultsCache[(Results Cache)]
    ResultsCache --> Dashboard[Experiment Dashboard]

    AggSvc --> SRMCheck[Sample-Ratio Mismatch Detector]
    SRMCheck --> Alerts[Alerts]
    AggSvc --> GuardCheck[Guardrail Monitor]
    GuardCheck --> Alerts

    ExpReg --> Edge
    AdminUI[Admin UI] --> ExpReg

What this diagram shows. The architecture has three principal flows. (1) Assignment hot path (top): the edge service receives getVariant calls, looks up the experiment config from the registry (cached locally for sub-millisecond lookups), evaluates targeting rules, and computes the variant via deterministic hashing — all without I/O beyond the local cache. (2) Event ingestion path (middle): exposure events (when the user is first assigned) and metric events (downstream user actions) flow into a unified Kafka stream. The join service correlates exposure with metric per user, producing per-(experiment, variant, user, metric) records. The daily aggregator collapses these into per-(experiment, variant, day, metric) summary statistics. (3) Statistics and monitoring path (bottom): the significance engine consumes the aggregations and computes p-values, lifts, confidence intervals; results are cached for the dashboard. In parallel, the SRM detector and guardrail monitor watch for assignment bugs and metric regressions, raising alerts immediately. The experiment registry is the source of truth for experiment configs; the admin UI writes here, and the edge service reads. The crucial design property is that the assignment is performed locally without round-trips to the metric or aggregation pipelines — this is what gives sub-10ms latency. The metric pipeline runs entirely off the critical path; experiment results are eventually consistent with the user actions that drove them.

7. Request Flow

7.1 Get Variant

sequenceDiagram
    participant App
    participant E as Edge Service
    participant C as Edge Cache
    participant K as Kafka

    App->>E: getVariant(exp_42, u123, context)
    E->>C: lookup experiment config
    C-->>E: {variants, allocation, targeting, salt}
    E->>E: check targeting rules
    alt user qualifies
        E->>E: bucket = hash(salt, exp_42, u123) mod 100
        E->>E: variant = lookup(allocation, bucket)
        Note over E: first-time exposure?<br/>local LRU cache check
        alt new exposure
            E->>K: publish ExposureEvent
        end
        E-->>App: {variant, config, exposed=true}
    else user doesn't qualify
        E-->>App: {variant: "not_in_experiment"}
    end

The hash + lookup is microseconds. The exposure publish is async and doesn’t block the response.

7.2 Daily Aggregation

sequenceDiagram
    participant K as Kafka
    participant J as Join Service
    participant A as Aggregator
    participant S as Stats Engine
    participant R as Results Cache

    K->>J: stream {ExposureEvent, MetricEvent}
    J->>J: per user: collect exposures + metrics
    J->>A: per (user, experiment, variant): metric values
    A->>A: nightly: aggregate per day per (exp, var, metric)
    A->>A: compute count, sum, sum_squared, HLL
    A->>S: triggers significance computation
    S->>S: per metric: t-test (treatment - control), p-value, CI
    S->>S: apply CUPED variance reduction
    S->>S: multiple-comparisons correction
    S->>R: cache results

The whole pipeline runs once per day per experiment; intermediate results may also be computed hourly for real-time dashboards.

8. Deep Dive

8.1 Deterministic Variant Assignment via Hashing

The assignment must be:

• Deterministic. Same (experiment_id, user_id) → same variant on every call.
• Statistically random. Across the user population, the bucket distribution is uniform (50/50 for an even allocation, etc.).
• Independent across experiments. A user assigned to “treatment” in experiment A should be assigned to control or treatment in experiment B with probability matching B’s allocation, regardless of A.

The standard implementation:

def get_variant(experiment_id, user_id, allocation, salt):
    # salt is per-experiment, prevents one experiment's bucketing from
    # leaking signal into another's
    h = hashlib.md5(f"{salt}:{experiment_id}:{user_id}".encode()).hexdigest()
    bucket = int(h, 16) % 100        # 100 buckets
    cumulative = 0
    for variant_name, fraction in allocation.items():
        cumulative += fraction * 100
        if bucket < cumulative:
            return variant_name
    return list(allocation.keys())[-1]   # fallback

Why MD5 (or BLAKE2b, etc.) and not a fast non-cryptographic hash? The hash needs strong avalanche properties — flipping one bit of the input must produce uniformly distributed output. MurmurHash3 satisfies this and is faster than MD5; both are used in production. Avoid Python’s built-in hash() (randomized per process; not deterministic across servers).

Why include salt and experiment_id in the hash? Without salt, two experiments using the same hash would produce correlated assignments — a user in treatment for A is more likely to be in treatment for B, biasing both. Per-experiment salt decouples assignments. This is the same logic as universal hash families (Hash Function Design).

Why 100 buckets? Convention; allows allocation in 1% increments. Some platforms use 1000 buckets for 0.1% precision (useful for very small treatment allocations like 0.1% canary rollouts).

8.2 Targeting Rules

Targeting filters which users are eligible for the experiment. Eligibility check happens before bucket-and-assign:

def is_eligible(targeting, context):
    for rule in targeting.include:
        if not eval_rule(rule, context):
            return False
    for rule in targeting.exclude:
        if eval_rule(rule, context):
            return False
    return True

Common targeting attributes: country (context.country == "US"), app version (>= 4.5.0), platform (iOS / Android / web), signup cohort, premium / free tier, geographic region. A user not eligible for the experiment receives a not_in_experiment response and is not exposed (no exposure event logged).

A subtle but important point: eligibility evaluation must be deterministic and stable. If a user’s country changes (they travel) and they become ineligible, what happens? Best practice: use stable attributes (signup_country, not current location) for targeting, or accept that some users will switch in/out and document the analysis caveat.

8.3 Exposure Logging — When and How

Exposure is the moment the user first encounters the experiment. Logging exposure correctly is critical because it defines the analysis cohort.

Wrong patterns:

• Log on every getVariant call. Spams the logs; same user produces 50 exposure events per page load. Wasteful; doesn’t bias analysis but wastes pipeline capacity.
• Log when the experiment starts, for every targeted user. Logs exposures for users who never enter the experiment’s code path. Biases results — those users behave like control because they are control (they never saw the treatment).

Right pattern: log exposure lazily, the first time the application calls getVariant for that user × experiment. This means the user actually entered the code path that uses the variant. Implementation:

exposure_cache = LRUCache(maxsize=1_000_000)  # per server instance
 
def get_variant(exp_id, user_id, ...):
    variant = compute_variant(exp_id, user_id, ...)
    cache_key = (user_id, exp_id)
    if cache_key not in exposure_cache:
        exposure_cache[cache_key] = True
        kafka.publish_async("exposures", {
            "user_id": user_id, "exp_id": exp_id,
            "variant": variant, "ts": now()
        })
    return variant

The LRU cache prevents repeat exposures for the same user × experiment from a single server. Across servers, some duplicate exposures will occur (the cache is per-server); the metric pipeline must dedupe on (user_id, experiment_id, day).

8.4 Significance Computation — Frequentist t-Test

The default test for “is the treatment metric different from control” is a two-sample t-test:

$t=\frac{{\bar{X}}_T-{\bar{X}}_C}{\sqrt{s_T^2/n_T+s_C^2/n_C}}$

where:

• X̄_T, X̄_C are the sample means (e.g., revenue-per-user) in treatment and control.
• s_T², s_C² are the sample variances.
• n_T, n_C are the user counts.
• The denominator is the standard error of the difference of means.

The t-statistic is converted to a p-value — the probability of observing this difference (or larger) if the null hypothesis (no difference) were true. Reject the null at p < 0.05 (the conventional threshold).

The textbook formula is appropriate for metrics that are approximately normally distributed; for revenue-per-user (highly skewed) the delta method or bootstrap is more accurate. Most platforms cap revenue at the 99th percentile to reduce skew (Kohavi-Tang-Xu 2020 §17 cited above).

Statistical power. Before launching the experiment, compute the minimum detectable effect (MDE) — the smallest lift the experiment can detect at desired α=0.05 and β=0.2 (80% power). MDE depends on metric variance and sample size: smaller variance or larger sample → smaller MDE. Experiments without enough power to detect the expected lift will under-detect — a common waste of effort.

8.5 The Peeking Problem and Always-Valid p-Values

The frequentist t-test gives a valid p-value if computed once, at a pre-determined sample size. If the analyst peeks at the p-value daily and stops the experiment when it crosses 0.05, the realized false-positive rate is much higher than 0.05 — random walks of the p-value cross 0.05 at some point with probability much greater than 0.05 if you check repeatedly.

Optimizely’s research (Johari, Pekelis & Walsh — authors who were advisors to / employed by Optimizely when the work was done, per the paper’s own disclosure) introduced always-valid p-values via the mixture Sequential Probability Ratio Test (mSPRT), a family first introduced by Robbins (1970): the test stops the first time a mixture (over possible treatment-effect sizes) of the likelihood ratio of the alternative to the null crosses a threshold. The paper states the problem starkly — under naive continuous monitoring, “very high false positive probabilities are obtained — well in excess of the nominal α,” whereas an always-valid p-value remains valid at any stopping time, so “continuous monitoring does not inflate” the error rate (Johari, Pekelis & Walsh, “Always Valid Inference,” arXiv 1512.04922). The trade-off is that the always-valid p-value is more conservative (larger, harder to reject) than the fixed-horizon p-value at a single pre-committed decision point — you pay in either power or expected run-time for the freedom to peek. The methodology was deployed in Optimizely’s commercial platform across hundreds of thousands of experiments.

Modern platforms (Statsig, Eppo) use always-valid p-values by default; legacy platforms expose both and warn users about peeking with fixed-horizon p-values.

8.6 Multiple-Comparisons Correction

Running 100 experiments simultaneously, each with α=0.05, gives expected false-positive count of 5 — even if no experiment has any real effect. Multiple-comparisons correction is required:

8.6.1 Bonferroni (Bonferroni 1936 cited above)

For k comparisons, use significance threshold α/k. Strict — controls family-wise error rate (FWER) at α. Excessively conservative for many comparisons; reduces power dramatically.

8.6.2 Benjamini-Hochberg (1995 cited above)

Controls false discovery rate (FDR) — the expected fraction of rejected hypotheses that are false positives. Less conservative than Bonferroni; preferred in modern practice. Procedure: sort p-values ascending; reject the largest i such that p_(i) ≤ (i/k)·α.

8.6.3 Per-Metric vs Per-Experiment Correction

Within one experiment with multiple metrics (primary + 5 guardrails), use per-experiment correction. Across experiments, more debatable — some platforms apply BH within each team’s experiments only.

8.7 Sample-Ratio Mismatch (SRM)

The configured allocation is 50/50, but the realized split is 51,234 control / 49,766 treatment — a meaningful 1.4% difference. This is a sample-ratio mismatch: the assignment is broken. Sources:

• A bug in the variant assignment code (e.g., one variant returns null and the user falls into a default).
• Differential drop-off (one variant has slower load time, more users abort before exposure is logged).
• The targeting rule depends on a property that is correlated with the variant.
• A logging pipeline failure that drops one variant’s events more than the other’s.

Detection: a chi-squared goodness-of-fit test of observed vs expected split. A small absolute deviation can be wildly significant at scale — Fabijan et al. give the example that a 50.2/49.8 split (821,588 vs 815,482 users) has under a 1-in-500,000 chance of arising from a true 50/50 randomization (Fabijan, Gupchup, Kohavi et al., “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments,” KDD 2019). Practitioners therefore use a strict threshold (commonly p < 0.0005 or p < 0.001) so that everyday statistical noise does not trip the alarm; below it the experiment is invalid and must not be analyzed until the SRM is resolved.

SRM is not a rare edge case — it is a routine data-quality problem. The same paper reports that approximately 6% of experiments at Microsoft exhibit an SRM, and Chen et al. at LinkedIn found roughly 10% of triggered analyses have one; a product running ten thousand experiments a year can expect many. Production platforms therefore run SRM checks automatically on every experiment and block analysis on the ones that fail.

8.8 CUPED — Variance Reduction (Deng et al. WSDM 2013)

CUPED (Controlled-experiment Using Pre-Experiment Data) reduces metric variance by adjusting for the user’s pre-experiment behavior. The intuition: a user who already engaged a lot before the experiment will engage a lot during the experiment regardless of variant; subtracting that baseline variance reveals the treatment effect more clearly.

Formally, define the adjusted metric:

$Y_{\text{adj}}=Y-\theta \cdot (X-\bar{X})$

where:

• Y is the post-experiment metric (e.g., revenue during experiment week).
• X is the pre-experiment metric (e.g., revenue during week before).
• X̄ is the pre-experiment population mean (constant).
• θ = Cov(Y, X) / Var(X) is the regression slope.

Choosing the optimal θ = Cov(Y, X)/Var(X) (which is exactly the ordinary-least-squares slope of regressing centered Y on centered X) yields, per the paper’s Equation (5), Var(Y_adj) = Var(Y) · (1 - ρ²), where ρ = corr(Y, X) is the correlation between the post- and pre-experiment metric (Deng, Xu, Kohavi & Walker, “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data,” WSDM 2013). So variance is cut by a factor of ρ². As a worked example: if pre-experiment behavior correlates with experiment behavior at ρ = 0.6 (plausible for engagement/revenue metrics), variance shrinks by 1 - 0.6² = 1 - 0.36 = 64%, and because the required sample size for fixed power scales with variance, the experiment needs roughly 64% fewer users for the same statistical power.

The paper reports that on Bing’s platform CUPED delivered a variance reduction of about 50%, which the authors describe as “equivalent to doubling our traffic or halving the sample size” — making it one of the highest-leverage statistical interventions in experimentation. Two conditions make it work and are easy to get wrong: the covariate must come from the pre-experiment period (the paper’s key observation is that E[X_treatment] − E[X_control] = 0 before any treatment is applied, because randomization has not yet been confounded by the treatment effect — this is what keeps the adjusted estimator unbiased), and it must be a metric unaffected by the treatment. The Deng et al. paper also details extensions: multiple covariates (where ρ² generalizes to the regression R²), non-linear adjustment via E[Y|X], and partially-missing pre-experiment data.

8.9 Holdout Groups

A holdout is a small population (typically 1–10%) that never participates in any experiment. Its metric history is the “would-have-happened-without-any-experiments” baseline. Used for:

• Long-term accumulation effect. Each individual experiment may show small lift; over a year of compounded experiments, the lift should be visible. The holdout vs full-population comparison measures it.
• Detecting regression. If experiments degrade something subtle (e.g., long-term retention) that isn’t measured per-experiment, the holdout’s superiority over full-population reveals it.

Holdouts are sticky — once a user is in the holdout, they stay in it (typically for a quarter or year, then re-randomized). Implementation: a top-level randomization layer with the holdout as one of its “experiments.”

8.10 Randomization Layers and Interaction Effects

Two experiments may interact: experiment A changes the homepage; experiment B changes the checkout page. A user might see treatment_A AND treatment_B, control_A AND treatment_B, etc. — four cells, each with potentially different lift.

If the experiments are independent (no shared user-level mechanism), interaction is benign — the marginal effect of each is approximately additive. If they interact (e.g., A’s headline copy mentions a feature that B is testing), naive analysis double-counts.

Tang et al.’s Overlapping Experiment Infrastructure (Google, KDD 2010 cited above) introduced randomization layers: each layer has its own hashing namespace; experiments in the same layer are mutually exclusive (a user is in at most one experiment per layer); experiments in different layers are independent. Product teams put potentially-interacting experiments in the same layer; non-interacting ones in different layers (gaining the ability to run them concurrently).

The implementation: every experiment has a layer_id; the variant-assignment hash includes the layer_id, ensuring that a user’s bucket assignment within one layer is statistically independent of their bucket assignment in others.

8.11 Guardrail Metrics

Beyond the primary metric (e.g., conversion rate), experiments should monitor guardrails — metrics that should not regress regardless of the experiment’s intent. Typical guardrails:

• Latency (p95 page load time). A change that improves conversion 5% by adding a heavy ML inference may regress latency 200ms — net negative.
• Error rate (5xx, JS errors, app crashes). Regressing reliability is rarely worth a metric lift.
• Revenue per visit (when primary is, e.g., engagement). Engagement up but revenue down is a bad trade.
• Retention 7-day, 28-day. Short-term lift that destroys long-term retention is a Pyrrhic victory.

The platform displays guardrail status alongside primary results; product reviews block on guardrail violations. This is one of the most important cultural / process levers a platform can encode.

9. Scaling Strategy

9.1 What Breaks First

1. Variant assignment latency. Solved by local computation — the edge service has the experiment configs cached locally and computes the hash without I/O.
2. Experiment registry update propagation. When a PM tweaks an experiment, the change must reach all edge instances within seconds. Solved by config push (e.g., gRPC streaming from registry to edges) or short TTL polling.
3. Exposure event volume. 1.2 × 10⁵/sec * 1KB = 120 MB/sec ingest. Standard Kafka cluster handles easily.
4. Daily aggregation cost. Per-experiment, per-day, per-metric aggregations across 10¹⁰ events. Spark or Flink batch job; runs in minutes.
5. Significance recomputation on every experiment view. Solved by caching results and recomputing on aggregation refresh, not on dashboard read.

9.2 Sharding

• Experiment configs: small (thousands of experiments × kilobytes each); replicated to every edge node.
• Exposure events: by user_id (joins with metric events on user_id).
• Metric aggregations: by (experiment_id, day) — naturally distributes by experiment.

9.3 Geo-Distribution

Edge variant-assignment service runs at every region; experiment config is replicated globally. Exposure / metric events flow into per-region Kafka, replicate to a central analytics region for global aggregation.

10. Real-World Example

Microsoft ExP (Bing’s experimentation platform, Kohavi et al. 2009 cited above; expanded in Kohavi-Tang-Xu 2020 textbook). The canonical reference architecture: variant assignment via deterministic hashing, randomization layers, CUPED variance reduction, multiple-comparison correction, SRM detection, peeking-aware always-valid statistics. ExP runs many thousands of experiments concurrently across Bing, Office, Windows, Azure.

Google’s Overlapping Experiment Infrastructure (Tang et al. KDD 2010 cited above). Introduced the randomization-layer concept. Allows Google to run hundreds of concurrent experiments on Search Ads, Image Search, etc., without interaction effects between non-interacting experiments. The same layer-based design has since been adopted across the industry.

Facebook PlanOut (Bakshy, Eckles & Bernstein WWW 2014 cited above). A specification language for experiment definitions; the experiment is a parameterized PlanOut script that returns variant assignments. Compiles down to deterministic hashing. Open-sourced.

LinkedIn ExP (Xu et al. KDD 2015 cited above). Discusses challenges specific to social networks: network effects (users in one variant influence users in the other), measurement aggregation across friend graphs.

Optimizely Experiment (Johari et al. always-valid p-values cited above). Hosted product; the always-valid p-value research is published.

Statsig, Eppo, LaunchDarkly Experiment, Split are the modern hosted products competing in this space; each layers different features (Statsig: pulse / scorecard / metric library; Eppo: multi-touchpoint; LaunchDarkly: feature-flag-first; Split: feature-flag-first with experimentation overlay).

Airbnb’s experimentation reports (publicly published) show how mature in-house platforms operate; Netflix has talked about its experimentation culture in talks though architecture is less public.

Uncertain

Verify: the internal implementation specifics of in-house platforms (Microsoft ExP, Google’s overlapping-experiment system, Meta) — exact SRM thresholds, exact CUPED covariate choices, current live experiment counts. Reason: the methods (deterministic-hash assignment, randomization layers, CUPED, mSPRT, automatic SRM checks) are well-documented in the cited primary papers, but the operational tuning is not fully published and changes over time. To resolve: these are mostly unknowable from outside; treat the published papers as the architecture-of-record and any “company X does exactly Y today” claim as dated. uncertain

11. Tradeoffs

Design Choice	Option A	Option B	When A wins	When B wins
Statistics	Frequentist (t-test)	Bayesian	Long history, conventional	Decision-theoretic, intuitive
Significance	Fixed-horizon p-value	Always-valid (mSPRT)	Strict, single look	Sequential, allow peeking
Multiple comparisons	Bonferroni (FWER)	Benjamini-Hochberg (FDR)	Small set of critical tests	Large set, want power
Variance reduction	None	CUPED	Simple, no covariate	Pre-experiment data available
Randomization	Independent per experiment	Layer-based	No interaction risk	Many concurrent experiments
Variant assignment	Server-side	Client-side cached	Tight control	Offline-capable apps
Exposure logging	On every getVariant	First call only (LRU)	Conservative	Matches analysis intent
Holdouts	None	Long-term holdout layer	Maximize sample	Need long-term measurement
Metric pipeline	Daily batch	Streaming SQL real-time	Cost-bounded	Need real-time results

12. Pitfalls

1. Non-deterministic assignment. Using Python’s randomized hash() or a session-based RNG produces different variants for the same user on different requests. Always use a stable hash with explicit salt.

2. Logging exposure for users who never see the treatment. Biases the analysis (the unexposed users are mistakenly classified as treatment but behave like control). Always log exposure lazily on first getVariant call.

3. Peeking at p-values without using always-valid statistics. False-positive rate balloons. Either commit to a fixed sample size and look once, or use mSPRT.

4. Forgetting to apply multiple-comparisons correction. With 50 metrics × 100 experiments = 5000 hypothesis tests, ~250 will appear significant by chance alone. Always correct.

5. Interpreting an SRM-failed experiment as if it were valid. The bucketing is broken; the analysis is meaningless. Halt and fix.

6. Targeting rules that depend on a non-stable attribute. A user’s bucket should not change as their country field updates. Use stable cohort attributes.

7. Ignoring guardrails. Shipping a +5% conversion experiment that adds 200ms latency degrades platform quality cumulatively.

8. CUPED with a covariate that’s affected by the treatment. CUPED assumes the covariate is fixed at experiment start. Using a post-experiment value as covariate leaks treatment information into the baseline; biases the estimate.

9. Ignoring novelty effects. A new design appears better in week 1 because users react to “new”; fades by week 4. Run experiments long enough (typically 2 weeks minimum) to outlast novelty.

10. Test-of-tests anti-pattern. Running an experiment to test the experiment infrastructure (A/A test, where both variants are control). A/A tests should produce p-values uniformly distributed in [0, 1] — if they cluster, the platform is broken.

11. Missing the consent / privacy layer. Variant assignment and exposure logging may be PII-adjacent in some jurisdictions. Hash user IDs, document data flow, support data deletion.

12. Long-running experiment that mutates over time. The treatment changes (e.g., new copy added in week 3); the analysis assumes a static treatment. Either freeze the treatment for the experiment’s duration, or split the analysis into phases.

13. Common Interview Variants and Follow-Ups

• “Add support for multi-arm bandit (MAB) instead of fixed-allocation A/B.” Replace static allocation with Thompson sampling or UCB; allocation shifts toward the apparently-winning arm over time. Reduces regret at the cost of harder statistical analysis.
• “Design for offline / mobile clients.” Cache the variant assignments on the device; sync new configs in the background; fall back to a default variant if the assignment service is unreachable.
• “Handle the network-effect problem (one user’s variant affects another).” Cluster-based randomization (assign by community, not user); switchback experiments (alternate the entire population over time).
• “Add a ‘switchback’ design for marketplace experiments.” Used by Uber/DoorDash where treating one rider differently affects driver supply across the marketplace.
• “Compute the experiment result on a streaming dashboard.” Use a real-time aggregation engine like ClickHouse or Druid; significance is recomputed every minute.
• “Add support for fractional allocation in 0.01% increments (small canary rollouts).” Increase bucket count from 100 to 10,000.
• “Build an experiment scorecard for monthly product reviews.” Aggregated view across all experiments shipped in the period; total lift attributed; guardrail violations.
• “Detect ‘gaming’ — an analyst running 50 variants of the same experiment until one shows significance.” Pre-register the primary metric in the experiment registry; lock the metric set before launch.

14. Open Questions / Uncertain

There is no single “winning” inference framework, but the landscape is clearer than “unsettled.” The newer hosted platforms have largely converged on frequentist analysis with sequential (always-valid) testing as the default, while offering Bayesian as an opt-in: Statsig runs frequentist two-sided z-tests by default with always-valid sequential analysis available, and exposes a Bayesian mode (Statsig docs); Eppo supports frequentist, sequential, and Bayesian, with frequentist as its primary centralized workflow (Eppo, “Frequentist vs. Bayesian vs. Sequential”). Bayesian-first platforms exist (notably GrowthBook). So the genuinely debated part is narrower than framework choice: it is whether to surface Bayesian posteriors or frequentist always-valid p-values as the primary decision number — a UX and decision-theory preference, not a correctness question.

Uncertain

Verify: whether long-term holdout groups are net-worth the statistical power they cost individual experiments. Reason: this is a genuine, unresolved practice debate — some mature teams maintain holdouts religiously to measure cumulative long-term effect, others have removed them to reclaim sample size — and no primary source settles it because the answer depends on each org’s metric horizon and experiment volume. To resolve: it is org-specific; decide based on whether you have long-term metrics (retention, LTV) that per-experiment analysis cannot capture. uncertain

• Causal-inference techniques (synthetic controls, difference-in-differences) for situations where randomization is impossible (regulatory, ethical) — how does an experimentation platform integrate them?
• The right framework for long-term metric measurement: most experiments measure 1–4 week outcomes; the lifetime value impact is unmeasured. Bandit + reward shaping is one direction.
• Adaptive metric variance estimation under non-stationarity — many production metrics drift seasonally; static-baseline CUPED can over- or under-correct.

15. See Also

• Major System Designs MOC
• SWE Interview Preparation MOC
• Real Time Analytics System Design — sibling system; metric events flow through both pipelines, often the same backend
• Click Tracking System Design — sibling event-pipeline; conversion events feed both
• Distributed Counter System Design — for per-(experiment, variant, metric) exposure and outcome counts
• Top K Trending System Design — sibling for heavy-hitter metric analysis
• Recommendation Engine System Design — every recommendation system runs many concurrent A/B tests
• Fraud Detection Pipeline System Design — to filter spurious metric events
• Consistent Hashing — the assignment-hashing primitive
• Distributed Log System Design — Kafka, the durable transport for exposure and metric events
• Bloom Filter — for the exposure-cache LRU’s negative lookup
• HyperLogLog — for distinct-user counts in metrics
• Hash Function Design — universal hashing properties needed for variant assignment
• ML Model Serving System Design — frequently the system being A/B tested