vault

Ad Serving System Design

37 min read

Ad Serving System Design

An ad serving system decides — within roughly one hundred milliseconds of a user requesting a web page or opening an app — which advertisement among potentially millions of eligible candidate creatives to render in each available ad slot, charges the advertiser the right amount, accounts for the impression so the advertiser’s daily budget is not overspent, logs the event for billing and machine-learning training, and (if the user clicks) attributes the conversion back to the ad. This is the system that funds Alphabet/Google’s ~USD 264.6 billion in 2024 advertising revenue and Meta’s ~USD 160.6 billion (both per FY2024 reporting; Statista/Q4 2024 earnings), the entire programmatic ecosystem (The Trade Desk, DV360, Magnite, AppNexus/Xandr), and effectively the modern Internet’s free-content economic model. The interview value is enormous because the system unifies four hard sub-problems into one tight latency budget: (1) information retrieval at scale (find the few thousand eligible ads from hundreds of millions of campaigns); (2) machine learning inference (predict click-through rate, conversion rate, and downstream value for each candidate in single-digit milliseconds); (3) mechanism design (run a sealed-bid auction that incentivizes truthful or near-truthful bidding while remaining computationally trivial); and (4) financial accounting under partial failure (every impression and click must reconcile to billable events with at-most-once or exactly-once semantics, despite all the upstream services being eventually consistent). Ad serving is therefore where game theory, information retrieval, low-latency systems, and financial accounting all meet under a hard real-time deadline.

1. Why This System Appears in Interviews

Ad serving compresses a half-dozen distinct distributed-systems and machine-learning topics into one problem that has to answer in 100 ms. Interviewers use it to test:

  • Whether the candidate understands that the auction mechanism (Generalized Second-Price, Vickrey-Clarke-Groves, or first-price) is a design choice with economic consequences, not an implementation detail. Picking the wrong mechanism produces strategic bidding behaviour that destroys revenue or fairness.
  • Whether the candidate sees the multi-stage retrieve-then-rank pipeline familiar from Recommendation Engine System Design but adapted to bidder economics — the candidate generation step here is constrained by targeting predicates (geography, demographics, interests, retargeting lists) rather than purely by collaborative-filtering similarity.
  • Whether the candidate can articulate latency budget decomposition — every millisecond of response time costs measurable revenue because real-time-bidding partners drop bids that arrive after their own auction’s deadline.
  • Whether the candidate understands budget pacing — naively spending an advertiser’s daily budget at line rate exhausts it by 09:00 local time, missing the high-converting evening; this requires a control-loop architecture analogous to a Token Bucket limiter.
  • Whether the candidate appreciates that billing accuracy is regulated and audited; a 0.5 % over-counting of clicks is a class-action-grade scandal in an industry where Procter & Gamble or Unilever spends nine-figure sums.

Ad serving is therefore a “tour” interview question — a single problem that exercises retrieval, ranking, ML serving, mechanism design, financial systems, and stream processing simultaneously.

2. Requirements

2.1 Functional Requirements

  1. Ad decision. Given an incoming ad request describing the user (cookie/device ID, geolocation, demographics if known), the publisher (the site/app), the slot (size, format, position, video vs banner vs native), and context (the URL or app screen, page keywords, time-of-day), return the chosen ad creative within the latency budget.
  2. Auction. Across the eligible candidate ads, run an auction that selects a winner (or the top k winners, for multi-slot pages such as a Google search results page with three sponsored slots) and computes the price each advertiser pays.
  3. Creative rendering. Serve the winning ad’s image, video, or HTML5 creative — typically from a CDN for low latency.
  4. Impression logging. Record an impression event (the ad was actually rendered/viewable) for billing and machine-learning training.
  5. Click and conversion tracking. Record click events (user clicked the ad) and conversion events (user later purchased, signed up, etc.). See Click Tracking System Design and AB Testing Platform System Design for measurement infrastructure.
  6. Targeting. Advertisers express targeting predicates: demographics (“women aged 25–34 in California”), interests (“interested in hiking”), retargeting lists (“users who visited example.com in the last 30 days”), lookalike audiences (“users similar to my converting customers”), keywords (for search ads), contextual (“placement: pages mentioning ‘mortgage’”).
  7. Budget pacing. Each advertiser specifies a daily budget (USD 1000/day) and optionally a schedule (only run between 09:00 and 17:00 local time). The system must spend the budget evenly across the schedule, not exhaust it at midnight.
  8. Frequency capping. Don’t show the same ad to the same user more than N times per day (typical: 3–5 times) — over-frequency annoys users and reduces incremental conversion lift.
  9. Brand safety. Don’t run a luxury-watch ad next to violent news content; advertisers specify category exclusions.
  10. Reporting. Advertisers see real-time dashboards: spend so far today, impressions delivered, clicks, conversions, average cost-per-click (CPC).
  11. Policy enforcement. Disallowed categories (gambling in jurisdictions where it’s illegal, certain medical claims) must be filtered.
  12. Fraud detection. Detect bot traffic, click farms, and ad-stacking (publishers stacking invisible ads to inflate impressions). Forward reference Fraud Detection Pipeline System Design.

2.2 Non-Functional Requirements

  1. End-to-end latency. From ad request arrival to creative rendered in browser: < 100 ms p99, with the auction itself in < 50 ms p99 and the embedded ML inference within < 20 ms p99. For programmatic real-time bidding (RTB) where one buyer is just a participant, OpenRTB specifies a hard deadline (usually 100 ms or 120 ms) after which the buyer’s bid is dropped.
  2. Throughput. Google search ads alone handle on the order of 100,000 search-ad auctions per second at peak. Display advertising (banner, video) handled by exchanges runs at 10+ million bid requests per second industry-wide; an individual demand-side-platform (DSP) like The Trade Desk handles millions of QPS during peak.
  3. Availability. 99.99 % at minimum on the decision path; an ad-serving outage during the Super Bowl is a nine-figure revenue event.
  4. Billing accuracy. Impressions and clicks billed to advertisers must match logged events with very high precision — the industry’s discrepancy tolerance is typically below 1 % between buyer-side and seller-side counts.
  5. Fraud resistance. Sophisticated adversaries continuously generate fake clicks and impressions; the system must score events for likely fraud and discount or rebate fraudulent activity.
  6. Privacy / regulation. Compliance with GDPR (EU), CCPA (California), the deprecation of third-party cookies, Apple’s App Tracking Transparency (ATT), and similar regimes constrains what data can be used in targeting.

3. Capacity Estimation

3.1 Auction Throughput

Take Google search as the concrete benchmark. Assume:

google searches per day               ≈ 8.5 × 10^9      (publicly cited, ~2020)
fraction of searches with ads          ≈ 0.30
ad-bearing searches per day            ≈ 2.55 × 10^9
seconds per day                        = 86,400
average ad-auction QPS                 ≈ 2.55 × 10^9 / 86,400 ≈ 30,000 auctions/s
peak-to-average ratio                  ≈ 3
peak ad-auction QPS                    ≈ 90,000 auctions/s

About 90,000 auctions per second at peak for one product. Display advertising adds at least an order of magnitude on top, because display ads run on every page view of every ad-supported website.

3.2 Candidate Count per Auction

For a given query, how many ads are eligible to bid?

total advertiser campaigns                ≈ 10^7
average campaigns matching one query       ≈ 10^3 to 10^4   (after targeting filters)
candidates that survive into ranking      ≈ 10^2            (top-K from retrieval)
slots actually shown                      ≈ 1 to 4

Retrieval narrows ~10^7 → ~10^3 in milliseconds via inverted indexes keyed on targeting attributes (Inverted Index); ranking (CTR prediction + bid) further narrows to top-K; the auction picks the winner(s).

3.3 ML Inference Cost

For each of the ~100 ranking-stage candidates, we need a CTR prediction. With a 20 ms ML budget total and 100 candidates:

budget per candidate                     = 20 ms / 100 = 200 µs

200 microseconds per inference is infeasible for a deep neural network on CPU; this drives major architectural choices:

  • Batch all 100 candidates into a single forward pass on a GPU or specialized accelerator.
  • Aggressively cache the user-side embedding (it does not depend on the candidate ad) and only run the per-candidate part of the model.
  • Use a “two-tower” or “Embedded MoE” architecture where the user tower runs once and produces a vector that is combined with each candidate’s pre-computed vector (a dot product is sub-microsecond).

3.4 Storage

Aggregating logs:

events per day (impressions+clicks+conv) ≈ 10^11    (industry-wide programmatic)
event size                                ≈ 1 KB
raw daily log volume                      ≈ 100 TB/day
                                          ≈ 36 PB/year

Compressed (typical 5–10× compression on columnar Parquet or ORC) brings it to ~10 PB/year per major operator. This is the input to billing reconciliation, training data construction, and reporting dashboards.

3.5 Revenue Sensitivity to Latency

Multiple industry studies have measured revenue lift per ms of latency reduction in the single-digit basis points. The reason is twofold: (1) faster ad rendering means more impressions complete before the user navigates away (“viewability”); (2) in real-time-bidding scenarios, slower bids miss the deadline and the buyer doesn’t even get to participate. This is why ad-serving teams obsess about every microsecond — it directly maps to revenue.

4. API Design

4.1 Ad Request (Publisher → Ad Server)

In the OpenRTB protocol (the IAB Tech Lab standard for programmatic display), an ad request is a JSON object posted to the ad exchange:

POST /openrtb/2.6/auction
{
  "id": "bid-req-7c3a-...",
  "imp": [{
    "id": "1",
    "banner": { "w": 300, "h": 250 },
    "tagid": "homepage_top",
    "bidfloor": 0.50,
    "bidfloorcur": "USD"
  }],
  "site": {
    "id": "publisher_abc",
    "domain": "example.com",
    "page": "https://example.com/article/12345"
  },
  "user": {
    "id": "user_xyz_hashed",
    "geo": { "country": "USA", "region": "CA" }
  },
  "device": {
    "ua": "Mozilla/5.0 ...",
    "ip": "192.0.2.1",
    "devicetype": 4
  },
  "tmax": 100
}

tmax is the deadline in milliseconds; the buyer must respond before that or be dropped from the auction. bidfloor is the publisher’s reserve price.

4.2 Bid Response (Ad Server → Publisher/Exchange)

HTTP/1.1 200 OK
{
  "id": "bid-req-7c3a-...",
  "seatbid": [{
    "bid": [{
      "id": "bid-resp-9f2-...",
      "impid": "1",
      "price": 1.42,
      "adm": "<html>...</html>",
      "adomain": ["coolwidget.com"],
      "crid": "creative_42",
      "w": 300, "h": 250
    }]
  }],
  "cur": "USD"
}

price is the bid in USD; adm is the rendered HTML markup or a URL pointing to the creative. The exchange runs the auction across all returned bids and notifies the winner.

4.3 Impression Beacon

When the creative renders in the user’s browser, it loads a tiny tracking pixel:

GET /imp?bid_id=...&advertiser_id=...&impression_token=...&ts=...

This GET — which returns a 1×1 transparent GIF or a 204 No Content — is the trustworthy impression event. The mere fact that the ad was delivered in the bid response doesn’t mean it was rendered (the user could have navigated away); the beacon is how we know the impression actually occurred and is therefore billable.

4.4 Click Tracking

The clickable region of the ad is wrapped in a redirect URL:

<a href="https://adserver.example.com/click?bid_id=...&advertiser_id=...&token=..."
   rel="noopener noreferrer">
   <img src="https://cdn.example.com/creatives/abc.jpg">
</a>

When clicked, the click handler logs the event and 302-redirects to the advertiser’s landing page. See Click Tracking System Design for the click-pipeline deep dive.

5. Data Model

5.1 Campaigns and Creatives

Table:        campaigns
PK:           campaign_id
Columns:      advertiser_id, daily_budget_usd, total_budget_usd,
              start_date, end_date,
              targeting_predicates (JSON: geo, demo, interests, retargeting list ids),
              brand_safety_categories,
              schedule (hours-of-day mask),
              objective (clicks | conversions | viewability | reach),
              max_bid_usd,
              status (active | paused | exhausted)

Table:        creatives
PK:           creative_id
Columns:      campaign_id, format (banner | video | native),
              dimensions (w x h),
              asset_url (CDN URL of image/video file),
              click_url (the advertiser's landing page),
              policy_review_status (approved | rejected | pending)

5.2 Targeting Indexes (Inverted Indexes)

To answer “which campaigns target country=USA AND interest=hiking AND device=mobile” in milliseconds, build an inverted index per targeting dimension (Inverted Index):

Index: country
  USA      → {campaign_1, campaign_5, campaign_42, ...}
  Canada   → {campaign_2, campaign_8, ...}

Index: interest
  hiking   → {campaign_1, campaign_19, ...}
  mortgage → {campaign_3, campaign_7, ...}

Index: device
  mobile   → {campaign_1, campaign_2, ...}
  desktop  → {campaign_5, ...}

Retrieval intersects the posting lists for the targeting predicates that match the request. Bloom filters (Bloom Filter) accelerate exclusion checks (frequency caps, brand-safety exclusions) before paying the cost of a full posting-list intersection.

5.3 Per-User State

Table:        user_profile           (in a key-value store: Redis / DynamoDB / Bigtable)
PK:           user_id (hashed cookie or device ID)
Columns:      demographic_segments (LIST<segment_id>),
              interest_segments (LIST<segment_id>),
              retargeting_list_ids (LIST<list_id>),
              recent_impressions (LIST<(creative_id, ts)>)   -- for frequency capping
              recent_clicks      (LIST<(creative_id, ts)>)
              precomputed_user_embedding (BYTES, 64 floats)  -- for the CTR model

Frequency capping is a hot read on every ad request, so the recent_impressions list is kept small (last day, capped at 100 entries) and stored in-memory.

5.4 Logs and Counters

Impressions, clicks, and conversions flow into a Kafka stream and are aggregated into per-campaign counters that feed budget pacing, reporting, and CTR-model training data.

5.5 ML Model Artifacts

Table:        models
PK:           (model_name, version)
Columns:      framework (TF | PyTorch | XGBoost),
              artifact_url (object-store URL),
              training_dataset_id,
              metrics (offline AUC, log-loss),
              status (production | shadow | retired)

See ML Model Serving System Design for how model versions get loaded and routed in serving.

6. High-Level Architecture

flowchart TB
    User[User Browser/App] -->|page load| Pub[Publisher Page]
    Pub -->|ad request| Edge[Ad Edge Service]
    Edge --> Cand[Candidate Retrieval]
    Cand -->|inverted-index lookups| Idx[(Targeting Index<br/>per-attribute postings)]
    Cand -->|fetch user state| UProf[(User Profile KV<br/>Redis/DynamoDB)]
    Cand --> Rank[Ranking / CTR Prediction]
    Rank -->|batch inference| MLServ[ML Model Server<br/>TF Serving / Triton]
    MLServ -->|feature lookup| FStore[(Feature Store<br/>online tier)]
    Rank --> Auc[Auction Service<br/>GSP/VCG]
    Auc -->|consult budget| Pacer[Budget Pacer<br/>token-bucket per campaign]
    Auc --> WinDecision{Winner}
    WinDecision -->|creative URL| Pub
    Pub -->|fetch creative| CDN[CDN Edge]
    CDN -->|render| User
    User -->|impression beacon| ImpLog[Impression Logger]
    User -->|click beacon| ClickLog[Click Logger]
    ImpLog --> Kafka[Kafka Event Stream]
    ClickLog --> Kafka
    Kafka --> Bill[Billing Aggregator]
    Kafka --> Train[Training Data Builder]
    Kafka --> Frd[Fraud Scoring]
    Bill --> CampaignStats[(Campaign Stats<br/>per-day spend)]
    Train --> WH[Data Warehouse<br/>training tables]
    Frd -->|invalidations| Bill

What this diagram shows. The architecture is divided into a synchronous decision path (top) and an asynchronous reconciliation path (bottom). The synchronous path runs on every ad request and must complete inside the 100 ms budget: it retrieves candidate ads via targeting indexes, ranks them by bid × pCTR using the ML model server, applies the auction mechanism, consults the budget pacer to check whether the candidate’s campaign can still afford to bid, and returns the winning creative. The asynchronous path absorbs the 10⁵-per-second event firehose: impression and click beacons land on a Kafka log, where multiple downstream consumers pick them up — the billing aggregator updates per-campaign daily spend (which feeds back into pacing), the training-data builder joins impressions with clicks (positive labels) and unclicked impressions (negative labels) to construct CTR training tables, and the fraud scorer identifies suspect events whose billing is later reversed. The key insight is that the system runs three loops at very different timescales: the per-request synchronous loop (~50 ms), the budget-feedback loop (~seconds to minutes), and the model-retraining loop (~hours to days). The architecture is essentially a Lambda architecture (real-time + batch) wrapped around a synchronous decision service.

7. Request Flow / Sequence

sequenceDiagram
    participant U as User Browser
    participant P as Publisher
    participant E as Ad Edge
    participant CR as Candidate Retrieval
    participant FS as Feature Store
    participant ML as ML Server
    participant A as Auction
    participant BP as Budget Pacer
    participant CDN as CDN
    participant K as Kafka

    U->>P: GET /article (page load)
    P->>E: Ad Request (slot, user_id, context)
    E->>CR: retrieve(user, context)
    CR->>CR: intersect targeting indexes
    CR->>FS: lookup user features
    FS-->>CR: user embedding + segments
    CR-->>E: ~100 candidate ads
    E->>ML: batch predict CTR(user, [ad1...adN])
    ML-->>E: pCTR for each candidate
    E->>A: rank by bid × pCTR
    A->>BP: check budget(top-K candidates)
    BP-->>A: allowed / disallowed (per campaign)
    A->>A: apply GSP or VCG pricing
    A-->>E: winner + price
    E-->>P: bid response (creative URL)
    P-->>U: page with ad slot
    U->>CDN: GET creative.jpg
    CDN-->>U: creative bytes
    U->>E: GET /imp?token=... (impression beacon)
    E->>K: emit ImpressionEvent
    Note over U,E: user later clicks ...
    U->>E: GET /click?token=...
    E->>K: emit ClickEvent
    E-->>U: 302 redirect → advertiser landing page

The sequence reveals the strict latency ordering: candidate retrieval (~10 ms) → ML inference (~20 ms) → auction (~5 ms) → return bid response (~5 ms) leaves about 60 ms for network round-trips between publisher → exchange → buyer → exchange → publisher → user. Anything slower than this leaks into the next step’s budget. The impression beacon in the lower half is independent of the bid response — it is what makes the impression billable, and it can arrive seconds later (or never, if the user navigated away before the ad rendered).

8. Deep Dive

8.1 The Auction — GSP, VCG, and First-Price

The auction is the system’s economic heart, and choosing among GSP, VCG, and first-price has profound consequences on bidder behaviour and platform revenue. This subsection walks through the mechanics, the strategic incentives, and why Google has run Generalized Second-Price (GSP) for AdWords for two decades while other platforms (notably Facebook circa 2010s) have at times used Vickrey-Clarke-Groves (VCG).

8.1.1 Single-Slot Vickrey Auction (the textbook starting point)

Vickrey’s 1961 paper proved that in a sealed-bid second-price auction for a single item — winner pays the second-highest bid, not their own — bidding truthfully (your bid equals your true valuation) is a dominant strategy: it is the best response regardless of what other bidders do. This is the most beautiful result in auction theory: by decoupling what you pay from what you bid, the auctioneer removes any incentive to shade your bid.

For a single slot with a single ad, this works directly. But ad pages typically have multiple slots (a search results page might have three ad positions); there are k slots, each with a different click-through rate that decays with position. This is no longer a single-item auction.

8.1.2 Generalized Second-Price (GSP)

Google designed GSP for multi-slot ad auctions. The mechanism (Edelman-Ostrovsky-Schwarz 2007 American Economic Review formalizes it; Google has used variants since 2002):

  1. Each advertiser submits a per-click bid b_i (USD per click).

  2. The system computes for each ad an ad rank r_i = b_i × pCTR_i × QualityAdjustments where pCTR_i is the predicted click-through rate.

  3. Sort ads by r_i descending. The top-K slots are awarded to the top-K ads.

  4. Pricing: ad in slot j pays the price needed to beat the next-rank ad — specifically, the minimum bid that would have kept their r_j above the next ad’s rank r_{j+1}:

    price_j = (r_{j+1} / pCTR_j) + ε

    Plus a small ε so the top ad pays just enough to be in slot j rather than slot j+1.

  5. Charging only happens on click — the second-price aspect is per-click.

price paid per click by ad i = (r_{i+1} / pCTR_i) + 0.01

where:

  • r_{i+1} is the ad rank of the ad immediately below ad i in the sorted list.
  • pCTR_i is ad i’s predicted click-through rate (between 0 and 1; typically 0.001 to 0.05 for display ads).
  • The + 0.01 is the minimum increment (one cent in USD); Google uses different increments for different markets.
  • r_i = b_i × pCTR_i (in the simplified version without quality adjustments) is the ad rank.

The intuition: ad i pays just enough that its rank r_i exceeds r_{i+1} — i.e., it pays “the price of beating the next ad.”

Crucial caveat: GSP is NOT truthful. Edelman-Ostrovsky-Schwarz (2007) proved that GSP does not have a dominant-strategy truthful equilibrium for k > 1 slots. Bidders have incentive to shade their bids strategically — sometimes overbidding for higher slots, sometimes underbidding to land in a cheaper slot. The paper showed there exists a “locally envy-free equilibrium” (a Nash equilibrium with desirable properties) where strategic bidding is stable, but achieving truthfulness requires a different mechanism.

So why use GSP? Two reasons:

  1. Simplicity and explainability. Advertisers understand “second-price-ish” intuitively. Sales teams can explain bidding strategy.
  2. Empirical revenue. GSP has been empirically very profitable for Google for two decades; switching mechanisms is risky when revenue exceeds USD 200 billion annually.

8.1.3 Vickrey-Clarke-Groves (VCG)

VCG is the truthful generalization to multi-item allocation. The pricing rule for each winner is the externality they impose on the other bidders — the loss in welfare to other bidders caused by the winner taking the slot.

For multi-slot ad auctions, VCG charges advertiser i the sum, over each lower-ranked slot j > i, of:

( pCTR_at_slot_{j-1} - pCTR_at_slot_j ) × b_at_slot_j

That is: ad i’s presence pushes the ad in slot j down to slot j+1 (lower CTR), so ad i’s externality on that other ad is the lost CTR times that ad’s bid.

VCG is provably truthful (dominant-strategy) — bidding your true valuation maximizes your expected utility. Facebook switched from a GSP-style auction to a Vickrey-Clarke-Groves auction around 2009, with the explicit reasoning that VCG is genuinely incentive-compatible while GSP is not, so advertisers can bid their true value and let the system handle the rest. Facebook’s own advertiser documentation has long described its auction as selecting the ad with the highest “total value” = advertiser bid × estimated action rates + ad quality, with the winner charged the minimum needed to win — a VCG-style externality price rather than the bidder’s own bid. Multiple independent analyses (e.g., Cornell INFO 2040 course write-ups, 2017–2022) consistently characterize the Meta auction as VCG/total-value; the company has not publicly announced a move away from it, though the exact production pricing rule is proprietary and surely augmented by learning-based components.

Why hasn’t Google switched? Because for an existing GSP advertiser base, a switch from GSP to VCG would change the prices they pay — some up, some down — and any change risks an exodus of advertisers who feel cheated. Edelman & Ostrovsky’s later work (e.g., “Internet auctions: GSP and VCG perform similarly under realistic conditions”) suggested empirical revenue impact is small in practice, but the migration risk is large.

8.1.4 First-Price (the surprising 2018+ trend)

Programmatic display advertising moved from second-price to first-price auctions roughly during 2017–2019, driven by the rise of “header bidding” (publishers running pre-auctions before the main exchange auction; second-price logic compounded across multiple layers becomes incoherent). The most consequential single move was Google Ad Manager’s switch to a unified first-price auction, announced in 2019 and rolled out between May and August 2019 (Google Ad Manager blog 2019). In a first-price auction, the winner pays exactly their bid. This is not truthful — bidders should shade their bids — and DSPs (demand-side platforms) had to invest heavily in bid-shading models to avoid overpaying.

The Trade Desk’s blog and other DSP engineering posts document the transition. The lesson: mechanism design in production is path-dependent and pragmatically chosen, not always theoretically optimal.

8.2 Click-Through-Rate Prediction

The pCTR model is the bridge between raw bidding (advertiser-stated USD-per-click willingness) and expected revenue per impression (bid × pCTR). A 1 % improvement in pCTR prediction directly maps to ~1 % more revenue, so pCTR-prediction is one of the highest-leverage ML problems in industry.

8.2.1 Historical Arc

  • 2000s — Logistic Regression with hand-engineered features. Google’s “Ad Click Prediction: A View from the Trenches” (McMahan et al. KDD 2013) describes a logistic regression with billions of features (one-hot user-times-advertiser cross features, etc.) trained with FTRL-Proximal optimization. This was the workhorse for a decade.
  • 2014 — Gradient Boosted Decision Trees + Logistic Regression. Facebook’s “Practical Lessons from Predicting Clicks on Ads” (He et al. ADKDD 2014) used GBDTs to generate features that fed into logistic regression. Hybrid model.
  • 2016 — Wide & Deep. Cheng et al. (Google) 2016 combined a wide linear part (memorization of cross-features) with a deep neural network part (generalization). This became the foundational deep architecture for ad ranking. See Wide & Deep for the detailed note.
  • 2019 — DLRM. Naumov et al. (Meta) 2019 introduced Deep Learning Recommendation Model, the deep architecture for sparse-categorical features (every user, ad, context becomes an embedding) plus dense numerical features fed into MLPs. See DLRM.
  • Post-2019 — Continual evolution: DCN-V2, BST-style transformers over user-history, multi-task heads predicting CTR, conversion rate, viewability simultaneously, etc.

8.2.2 Feature Engineering

Features fall into roughly four buckets:

  1. User features. Demographic segments, interest segments, recent behaviour (last N clicks/impressions), user embedding (from a separately trained two-tower model).
  2. Ad features. Advertiser, vertical, ad embedding (from creative content), historical CTR of this ad over various time windows.
  3. Context features. Page URL, page topic embedding, time of day, day of week, device, geo, weather (yes, weather influences ad performance — coffee ads outperform in cold weather).
  4. Cross features. User-times-advertiser interactions (“this user × this advertiser” historical CTR), user-times-page-topic, etc. The cross-features are typically what require the “wide” part of Wide & Deep.

8.2.3 Real-Time Features

Real-time features are the hardest engineering challenge. “How many ads has this user seen in the last 5 minutes?” requires a streaming aggregation joined into the request path. The Feature Store System Design note covers the online/offline parity required to keep training and serving consistent. A common architecture:

  • A streaming pipeline (Flink/Kafka Streams) maintains 1-minute, 5-minute, 1-hour aggregates per user, written to the online feature store (Redis/Aerospike).
  • The training pipeline reads the same aggregates’ historical values from the offline feature store (Parquet on S3) using point-in-time-correct joins.

8.3 Real-Time Bidding (RTB) and Programmatic Display

For display ads served via exchanges (not Google’s own search ads), the protocol is OpenRTB (IAB Tech Lab spec). The flow:

  1. User loads a publisher’s page.
  2. The publisher’s ad server sends a bid request to a supply-side platform (SSP) which forwards to multiple ad exchanges.
  3. Each exchange forwards the request to demand-side platforms (DSPs) — companies like The Trade Desk, DV360, or AppNexus that bid on behalf of advertisers.
  4. Each DSP has 100 ms (the typical tmax) to decide whether to bid and at what price.
  5. The exchange runs the auction across the returned bids and picks a winner.
  6. The winning DSP’s creative renders.

This all happens in less than 200 ms total — the user perceives the page loading. The DSP’s 100 ms is brutally tight: receive the request, do the candidate retrieval, run the ML inference, decide a bid price, return the response.

The DSP economics:

  • The DSP bids on behalf of advertisers; each advertiser has a campaign budget.
  • The DSP charges a margin (typically 10–20 % of media spend) to advertisers for its services.
  • The DSP’s ML models predict not just CTR but also the eventual conversion value, so they bid the user’s expected lifetime value to the advertiser.

OpenRTB 2.6 (released April 2022 by the IAB Tech Lab; spec) defines extensions for video ads, native ads, DOOH (digital out-of-home), audio ads (Spotify/Pandora), and especially CTV (Connected TV) — its headline feature is structured ad-pod support for streaming-TV inventory.

8.4 Budget Pacing

A naive approach: every time a campaign wins an auction, charge the campaign the auction price; when the daily budget hits zero, stop bidding.

This catastrophically over-spends in the morning if you process bid requests at ~10⁶ QPS:

  • A campaign with USD 1000 daily budget at average CPM (cost per thousand impressions) of USD 5 wins ~200,000 impressions/day.
  • 200,000 impressions/day = ~2.3/sec average, but peak bid-request rates may produce hundreds of wins per second early in the day.
  • The budget exhausts at 09:30 instead of distributing across 24 hours.

The fix is rate-limited bidding via a Token Bucket-style controller per campaign:

bucket_capacity = daily_budget_usd
tokens_per_second = daily_budget_usd / 86400  # USD per second
# refill: every second add tokens_per_second
# on auction win: deduct (price * pCTR) from bucket
# if bucket empty: drop bid

This produces an even spend rate. Production implementations are more sophisticated:

  • Adaptive pacing. Adjust the rate based on observed win-rate and inventory availability (if there’s a 2-hour lull at 03:00 with little inventory, save budget for the morning surge).
  • Hour-of-day weighting. Some advertisers prefer evening conversion peaks; the pacer can weight token-refill higher during high-converting hours.
  • Probabilistic throttling. Instead of hard-rate-limit, accept each bid request with probability p chosen so that expected spend matches the pacing target. Smoother behaviour than hard limits.

PID controllers borrowed from control theory are a common implementation choice; the budget pacer is essentially a feedback loop where the setpoint is “spend rate that exhausts budget at end-of-day” and the manipulated variable is the bidder’s participation probability.

8.5 Frequency Capping and Brand Safety

Frequency capping (“don’t show this user this ad more than 3 times in 24 hours”) is enforced at retrieval. The user’s recent_impressions list is consulted; campaigns whose creative has been impressed too recently are filtered.

This requires a fast lookup — typically a Redis sorted set per user, capped at the top-100 most recent impressions, with a TTL of 24 hours. Bloom filters can serve as a faster pre-check (Bloom Filter) to avoid expensive lookups for users with no recent impressions.

Brand safety is similar: the publisher’s page-content classifier produces a category (politics-extreme, drugs, violent, etc.), and campaigns with category-exclusion lists filter out incompatible inventory.

8.6 Click and Conversion Attribution

When the user clicks the ad, a click event lands on Kafka. When the user later converts (purchases, signs up, downloads the app), a conversion event lands on Kafka — typically via the advertiser’s pixel on their thank-you page or via an SDK in their app.

Attribution joins the conversion back to the click: which ad click “caused” this conversion? Models:

  • Last-click attribution — the most recent click within a window (typically 7 to 30 days) gets full credit. Simple, dominant historically.
  • First-click attribution.
  • Multi-touch attribution (MTA) — distribute credit across all clicks the user had in the conversion window. Approaches range from simple (linear, time-decay) to ML-based (Shapley values, attribution models).
  • Data-driven attribution — Google’s term for an ML-trained attribution model that learns each touchpoint’s contribution.

The architectural challenge: conversions arrive days or weeks after the click (delayed reward), so the attribution pipeline must store per-user click history and join asynchronously. This is also the labelling pipeline for the conversion-rate-prediction model — see Fraud Detection Pipeline System Design for the analogous delayed-label problem.

8.7 Fraud and Invalid Traffic

Estimates of invalid traffic (IVT) in display advertising have ranged from 5% to 30% depending on inventory quality and measurement methodology; recent (2024–2025) third-party benchmarks cluster around the high-teens to low-twenties percent globally, with mobile-app and connected-TV inventory often the worst (Pixalate Q2 2025 / Fraudlogix 2025). Sources of fraud:

  • Click farms — paid humans or bots clicking on competitors’ ads to drain budgets, or clicking on a publisher’s own ads to inflate revenue.
  • Bot traffic — non-human traffic generating impressions and clicks.
  • Ad stacking — publishers placing multiple ads in the same slot, only one visible.
  • Pixel stuffing — placing 1×1 pixel iframes off-screen.
  • Domain spoofing — bid requests claiming to come from premium publishers but actually originating elsewhere.

Detection is a Fraud Detection Pipeline System Design in itself: ML models score every event for fraud-likelihood, and identified fraudulent clicks/impressions are excluded from billing (or refunded). Industry initiatives (Ads.txt, Sellers.json, SupplyChain Object) standardize trust signals so buyers can verify the supply path.

9. Scaling Strategy

9.1 What Breaks First

  1. Auction-path latency. The 100 ms budget is the tightest constraint. Adding any synchronous network hop (e.g., a database lookup that wasn’t pre-cached) is fatal. Mitigation: aggressive in-process caching, pre-loading of campaign data, and batched ML inference.

  2. ML-inference throughput. At 100 candidates × 100K QPS = 10M predictions/sec, even GPU-accelerated inference becomes the bottleneck. Mitigations:

    • Two-tower architecture so the user-side runs once per request (not per candidate).
    • Quantization (int8) for 2–4× throughput.
    • Embedding caching (the per-ad part is stable across requests).
    • See ML Model Serving System Design.

  3. User-profile lookup throughput. Every ad request reads the user’s profile. This is a 10⁵-QPS hot KV-store path. Mitigations: aggressive in-memory caching at the edge, consistently-hashed sharding of the user-profile store, profile pre-fetching during page-load anticipation.

  4. Budget-pacing race conditions. Two simultaneous wins for the same campaign racing on budget deduction can over-spend. Solved with atomic increment in Redis or a per-campaign single-writer pattern.

  5. Log ingestion volume. 10¹¹ events/day is petabyte-scale daily. Mitigations: Kafka cluster sized appropriately, columnar offline storage (Parquet), aggressive aggregation (most queries operate on per-minute roll-ups, not raw events).

9.2 Geographic Distribution

Ad serving is fundamentally regional — the 100 ms budget cannot include trans-Pacific round-trips. Each region (US-East, US-West, EU, APAC) runs a full ad-serving stack: candidate retrieval, ML serving, budget pacer, log ingestion. Cross-region replication concerns:

  • Campaign metadata is replicated globally (every region needs to know about every campaign).
  • User profiles are typically pinned to the user’s home region with cache replication.
  • Budget state is the trickiest: a campaign’s budget is global, but spending happens regionally. Common pattern: each region maintains a local budget allowance refreshed from a central pacer; over-spend reconciliation is done batch.

9.3 Scaling Past Single-Box Limits

A single ranking server can hold ~10⁵ pre-computed ad embeddings; campaigns in the millions exceed this. Sharding the index by candidate-set partition is standard — each ranking shard handles a fraction of the candidate space; the retrieval layer fans out and merges top-K. This is the same pattern as Distributed Search System Design.

10. Real-World Example

Google AdWords / Google Ads. The largest single deployment. Run on Google’s internal Borg/Kubernetes-style infrastructure with custom ad-serving servers (“the ad mixer”). The auction is GSP, with quality-adjusted ad rank (bid × pCTR × QualityScore) — a famously consequential design choice introduced by Google around 2002 that rewards advertisers for high-quality landing pages and creatives. Google does not publish detailed ad-system architecture, but ex-Googler talks (e.g., Hal Varian, Google’s chief economist, has given several public talks on AdWords mechanism design) and academic papers (Edelman-Ostrovsky-Schwarz 2007 American Economic Review formalizes GSP) cover the principles.

Meta Ads. Documented in their engineering blog. Auction historically VCG-style. Heavy use of the Wide & Deep / DLRM family of CTR models. Meta’s ads infrastructure runs on their internal Ent/Tao graph stores plus custom inference accelerators (originally Big Basin GPUs, now various ASICs).

The Trade Desk (DSP). A major demand-side platform participating in programmatic exchanges. Their engineering blog covers their migration to first-price auctions (2018), their bid-shading models, and their Koa AI bidding optimization. They operate at multi-million-QPS bid-request rates worldwide.

Magnite / PubMatic / Google Ad Manager (SSPs). Supply-side platforms representing publishers; they run header bidding, multi-exchange waterfalls, and price floor optimization.

Open-Source Reference. RTB-heavy implementations are mostly proprietary, but the OpenRTB spec is public; reference implementations like Pre-bid.js (header-bidding library) and various open-source DSPs offer learning material.

Uncertain

Verify: the exact production internals of Google Ads (current ML model architecture, data-warehouse stack, the precise GSP variant with all quality and anti-abuse adjustments) and the current Meta auction pricing rule. Reason: these are proprietary and not disclosed; published material describes principles, not the running systems. To resolve: only authoritative disclosure would settle them. (What is verified: Meta switched from GSP to a VCG/“total value” auction around 2009 and still publicly describes a total-value mechanism — see §8.1.3; the textbook GSP and VCG mechanics are settled in the cited papers.) uncertain

11. Tradeoffs

Design ChoiceOption AOption BWhen A winsWhen B wins
Auction mechanismGSPVCGExisting advertiser base; explainability mattersNew platform; truthful bidding is desired
Auction mechanismSecond-priceFirst-priceSingle auction layer (no header bidding)Multiple auction layers (programmatic display post-2018)
pCTR modelLogistic regression + FTRLDeep neural net (Wide & Deep / DLRM)Tight infra budget; small advertiser poolLarge scale, plenty of features and data
Candidate retrievalInverted index intersectionANN over candidate-ad embeddingsTargeting predicates dominateSemantic similarity matters (native ads, contextual)
Budget pacerHard rate limit (token bucket)Probabilistic throttlingPredictable behaviourSmoother spend, less bursty
Frequency capping storeRedis per-user listBloom filter pre-check + KVModest user baseMassive user base; check is hot
PricingCPM (cost per thousand impressions)CPC (cost per click)Brand campaigns, viewabilityDirect response, performance
PricingCPCCPA (cost per acquisition)Mid-funnel campaignsLower-funnel; advertiser trusts platform attribution
Real-time featuresStreaming aggregation (Flink)Periodic batch refresh (1-hour)High-cardinality real-time signalsCoarse-grained signals tolerable
Inference batchingPer-requestBatched across nearby requestsLatency-critical, large modelThroughput-critical, small model

12. Pitfalls

  1. Forgetting that an “impression” in the bid response is not the same as a billable impression. The bid response means we won the auction; the impression beacon means the ad was actually rendered to the user. Bill on beacons, not on bid wins. Many naive systems over-bill or get sued.

  2. Running the ML model on every candidate without batching. 100 candidates × 1 ms each is 100 ms — gone. Batch all candidates into one forward pass.

  3. Treating GSP as truthful. Edelman-Ostrovsky-Schwarz proved it is not. Advertiser bidding tools incorporate strategic shading; if your platform’s documentation suggests “just bid your true value”, you are misleading advertisers.

  4. Not pacing budgets, then over-spending in the morning. The classic. Token-bucket-style pacing per campaign is non-negotiable.

  5. Ignoring frequency caps until users complain. Showing the same insurance ad 20 times in an hour is the fastest way to ruin a brand campaign and produce hate mail.

  6. Letting fraud through to billing without scoring. The platform pays the cost (refunds to advertisers, sometimes lawsuits). Fraud scoring before the billing aggregator is essential.

  7. Forgetting click attribution windows. A 30-day click window means a click in January can attribute a conversion in early February. The system must retain enough click history to attribute correctly; reporting “this campaign drove zero conversions” right after launch is misleading without the window.

  8. Mixing different currencies without explicit conversion. Advertisers across regions bid in local currency; budget aggregation requires explicit FX with snapshot rates.

  9. Computing bid × pCTR ranking without reserve prices. Publishers set a reserve price (bidfloor in OpenRTB); ads below reserve must be excluded even if they have the highest rank. Forgetting this serves under-priced ads in violation of the publisher’s wishes.

  10. Stale model deployments at scale. A buggy model deployment that produces zero clicks for an hour costs millions. Robust ML serving with shadow traffic, canary, and rapid rollback is essential. See ML Model Serving System Design.

  11. Cookie blindness — but mind the 2024–2025 reversal. Designing a 2026 system that depends solely on third-party cookies is fragile, but the precise status matters and is widely misremembered. Apple’s App Tracking Transparency (ATT, 2021) really did gut mobile identifier-based tracking on iOS (apps must get explicit opt-in for the IDFA, and most users decline). EU privacy regulation (GDPR/ePrivacy) genuinely constrains cookie use. But Chrome did not deprecate third-party cookies. Google reversed its long-promised deprecation: in July 2024 it announced it would not unilaterally phase out third-party cookies, in April 2025 it dropped even the planned standalone cookie-choice prompt, and in October 2025 it wound down a large part of the Privacy Sandbox program (Google Apr 2025). So as of 2026 third-party cookies remain live in Chrome (the dominant browser), while being blocked by default in Safari and Firefox. The defensible design stance is therefore not “the cookie is dead” but “do not bet the architecture on third-party cookies”: support first-party identifiers, hashed emails (e.g., Unified ID 2.0), contextual targeting, and the surviving Privacy Sandbox APIs as parallel signals.

  12. Synchronous budget check on the hot path without local caching. If every auction queries a central budget service synchronously, that service becomes the bottleneck. Local budget allowances refreshed periodically scale better.

  13. Mixing brand safety classification into the inline path. Page classification can take seconds (vision/NLP models on the article body). It must be precomputed at crawl time and looked up by URL hash, not run inline.

  14. Not handling tied bids deterministically. Two ads with identical ad-rank produce non-deterministic ordering, leading to A/B-test contamination and reproducibility nightmares. Use a stable tiebreaker (campaign ID hash).

13. Common Interview Variants and Follow-Ups

  • “Now add video ads.” Video creatives are larger; serving from CDN is the only viable path. Quartile completion events (impressions, 25 %, 50 %, 75 %, 100 %) become billable events. VAST/VPAID protocols define the video-ad protocol.

  • “Add native ads.” Native ads match the publisher’s content style. Templates per publisher, dynamic creative composition (the headline + image come from the advertiser, the styling from the publisher).

  • “Build a contextual targeting system without cookies.” Contextual classification of pages (NLP on article body) → match advertiser’s contextual targeting criteria. Brand-safety subsystem doubles as targeting.

  • “Add lookalike audience modeling.” Given a “seed list” of an advertiser’s converting customers, find the larger audience that resembles them. Train a binary classifier with seed = positive class, random users = negative class; predict on the broader user base; threshold for the lookalike audience. This is essentially a retrieval-and-ranking problem similar to Recommendation Engine System Design.

  • “Implement retargeting (showing ads to users who visited the advertiser’s site).” The advertiser’s pixel fires on their site, recording visitor user IDs into a “retargeting list” stored as a list of user-ids. Targeting predicates include “user is in retargeting list X”. TTL per list (typically 30 days).

  • “How do you measure incremental lift from ads?” Run B tests where the treatment group sees ads and the control group sees PSAs (public service announcements) or empty placements. Compare conversion rates. This is the gold-standard ad-effectiveness measurement and is one of the largest applications of online experimentation.

  • “Now make the system GDPR-compliant.” User consent gating before any tracking; user-id deletion on request (right to be forgotten); data residency (EU user data stays in EU); a documented data-processing agreement with publishers.

  • “How do you handle a major advertiser doubling their budget overnight?” The pacer must adapt without flooding the auction with bids. Smooth ramp-up over hours/days; alarm on aggressive new spend.

  • “Build the bidding engine for a DSP.” That’s a slightly different problem: the DSP doesn’t run the auction (the exchange does), but must decide whether and how much to bid in <100 ms based on its predicted value of the impression. Bid shading models are central.

  • “How does ad serving change for connected TV (Roku, Smart TVs)?” Higher-CPM inventory; longer creative durations (15s, 30s); SSAI (server-side ad insertion) instead of client-side rendering; DAR (device-to-ad-response) measurement instead of pixel beacons.

14. Open Questions / Uncertain

The post-cookie identity question got messier, not cleaner. The framing of “which post-third-party-cookie standard wins” partly dissolved in 2024–2025: Google reversed course and kept third-party cookies in Chrome (July 2024), dropped the planned cookie-choice prompt (April 2025), and retired much of the Privacy Sandbox program (October 2025), narrowing its focus to a smaller set of APIs (Google Apr 2025). So Privacy Sandbox (Topics API, Protected Audience API) is no longer on a glide path to becoming the replacement; Unified ID 2.0 (industry consortium, hashed-email-based) and contextual targeting persist alongside surviving cookies. The genuinely open question is now which blend of signals the ecosystem settles on, not which single standard “wins” — and that remains unresolved as of 2026.

Uncertain

Verify: the relative production performance of Wide & Deep vs DLRM vs newer transformer-based CTR models. Reason: no apples-to-apples public benchmark at industrial scale exists; each operator publishes wins on its own data and traffic. To resolve: a controlled public benchmark, which is unlikely to appear. uncertain

Uncertain

Verify: the exact GSP variant Google Ads runs today. Reason: the production mechanism layers quality scoring, ad-rank thresholds for slot eligibility, automated bidding (Smart Bidding), and anti-abuse adjustments on top of textbook GSP, and these are not disclosed. To resolve: authoritative Google disclosure. (The textbook GSP mechanics and the quality-adjusted ad-rank principle are settled — see §8.1.2 and §10.) uncertain

Invalid-traffic rates remain a moving, methodology-dependent target. Recent third-party measurements put global invalid traffic (IVT) in the high-teens-to-low-twenties percent range — Fraudlogix reported ~20.6% across 105.7 billion 2025 impressions, and Pixalate’s Q2 2025 benchmarks found ~18.6% of desktop/mobile web, ~28.5% of mobile-app, and ~17.9% of connected-TV traffic invalid (Pixalate benchmarks). These exceed the old “post-mitigation single-digit” figures and sit toward the upper end of the historical 5–30% band, but they vary enormously by channel and inventory quality, and the detection/evasion arms race keeps the true figure unstable. Treat any single percentage as as-of and channel-specific.

15. See Also

Graph View

Backlinks