ML Model Serving System Design

An ML model serving system is the runtime layer that turns a trained model artifact (a TensorFlow SavedModel, a PyTorch TorchScript file, an ONNX graph, an XGBoost JSON dump, or a Hugging Face Transformers checkpoint) into a low-latency online prediction service that the rest of the company’s products can call as if it were a regular RPC. It must answer per-request inferences in single-digit to low-double-digit milliseconds for ranking and CTR-prediction workloads, tens to hundreds of milliseconds for image and video models, and hundreds of milliseconds to seconds per token for large language models — across hundreds of distinct models, multiple frameworks, and a mix of CPU, GPU, and specialized accelerator hardware. The reference implementations are TensorFlow Serving (Olston et al. NeurIPS 2017, Google), NVIDIA Triton Inference Server (NVIDIA’s universal multi-framework server), KServe (Kubernetes-native, formerly KFServing), Seldon Core (Kubernetes-native commercial), BentoML (Python-friendly), and for LLMs specifically vLLM (Kwon et al. SOSP 2023, paged attention), HuggingFace Text Generation Inference (TGI), and Orca (Yu et al. OSDI 2022). The interview value is that model serving compresses several distinct hard problems into one platform: it has to abstract over ML frameworks (TF, PyTorch, ONNX), batch inference dynamically to amortize hardware launch cost across concurrent requests (the dominant performance technique for GPUs), manage GPU memory and concurrency to keep expensive accelerators highly utilized, support multiple model versions simultaneously for A/B testing and canary rollouts, integrate with feature stores for input lookup, and monitor for prediction drift and feature drift as a first-class observability signal. The recent rise of LLMs (2023+) has added a new dimension: continuous batching and KV-cache management are now first-class concerns because token-level autoregressive decoding has a very different performance profile from feed-forward dense-model inference.

1. Why This System Appears in Interviews

Model serving is the operational endpoint of an ML system; without it, models trained in notebooks are useless. Interviewers use it to test:

Whether the candidate understands dynamic batching — that a GPU launching one inference at a time is wildly under-utilized, but batching a few dozen requests amortizes the launch overhead across them at the cost of a small batching delay; this latency-vs-throughput tradeoff is the core operational lever.
Whether the candidate distinguishes CPU inference (cheap, fine for small models, no batching needed) from GPU inference (expensive, demands batching, high throughput per dollar at scale) and knows when each applies.
Whether the candidate handles multiple model versions simultaneously — canary deployments (1 % of traffic to v2), shadow traffic (run v2 in parallel without returning its predictions), A/B routing — all required for safe model rollouts.
Whether the candidate appreciates inference optimization — quantization (int8/fp16), kernel fusion (TensorRT, OpenVINO, XLA), graph rewriting (ONNX Runtime optimizations), pruning, distillation — and knows the order-of-magnitude impact of each.
Whether the candidate addresses operational concerns beyond latency: warm-up (the first request after a model is loaded is slow), drift monitoring, autoscaling, memory accounting on shared accelerators, model registry integration.
Whether the candidate understands LLM-specific challenges — paged-attention KV-cache management, continuous batching (per-token rather than per-request), long-context memory pressure — which dominate the modern serving landscape.

This system pairs with Feature Store System Design (the input substrate), Recommendation Engine System Design (heavy model-serving consumer), Ad Serving System Design (latency-critical model-serving consumer), and AB Testing Platform System Design (which canary and shadow rollouts depend on).

2. Requirements

2.1 Functional Requirements

Predict. Accept a request containing input features (or a request-context that triggers feature lookup), run the loaded model, and return a prediction within the latency budget.
Multi-framework support. Load and serve models trained in TensorFlow, PyTorch, scikit-learn, XGBoost, and exported via ONNX. Different frameworks need different runtimes.
Multi-model support. A single serving fleet often hosts many models — recommender ranking, click-through-rate predictor, fraud scorer, search-relevance — each with different SLAs.
Versioning. Multiple versions of the same logical model coexist. Routing rules (canary 1 % to v2, the rest to v1) must be configurable at runtime without redeployment.
Batch and online modes. Online: per-request inference with strict latency. Batch: scoring a million records overnight for analytics; latency-irrelevant, throughput-critical.
Dynamic batching. Concurrent online requests should be auto-batched to amortize GPU launch overhead.
Warm-up. The first request after a model is loaded is slow (just-in-time compilation, kernel autotuning, cache warming). Run a synthetic warm-up at load time so user requests don’t hit cold paths.
Health and metrics. Expose health checks, latency histograms, batch-size distributions, GPU utilization, error rates, prediction-distribution histograms.
Model loading / unloading. Hot-swap a new model version without restarting the serving process; gracefully drain in-flight requests on unload.
Authentication and authorization. Some models are restricted to specific callers; the serving layer authenticates and authorizes.

2.2 Non-Functional Requirements

Latency targets (per workload class):
- Ranking / CTR models: p99 < 50 ms (often < 20 ms for ad ranking).
- Image classification: p99 < 200 ms.
- Object detection / segmentation: p99 < 500 ms.
- LLM token generation: time-to-first-token p99 < 1 s, then inter-token latency < 50 ms.
Throughput. Per replica: 10⁵ QPS for small models on CPU; 10⁴ QPS for medium models on GPU; ~10–100 sequences/s for LLMs.
GPU utilization. Target ≥ 70 % at peak load — under-utilization is wasted budget on expensive hardware.
Availability. 99.99 % minimum on the inference path.
Cost efficiency. GPU hours per million predictions is a board-level metric for AI-heavy companies. Quantization and batching can move this by 2–10×.
Multi-tenancy isolation. A misbehaving model (memory leak, OOM, infinite loop) must not take down other tenants on the same hardware.

3. Capacity Estimation

3.1 GPU Throughput as the Constraint

A single A100 GPU can sustain roughly:

throughput per A100, dense ranker model (~10M params)        ≈ 50,000 inferences/s (batched)
throughput per A100, small CV model (ResNet-50)              ≈ 5,000-10,000 images/s (batched)
throughput per A100, large CV model (ViT-Large)              ≈ 500-1,000 images/s
throughput per A100, 7B-parameter LLM (continuous batching)  ≈ 50-200 generations/s

These numbers vary heavily with batch size, sequence length, and quantization. The order of magnitude is what matters for sizing.

For a recommender at 10⁵ QPS × 100 candidates per request = 10⁷ inferences/sec, with each A100 doing 5×10⁴ inferences/sec batched, we need ~200 GPUs per region for ranking alone. This is real money: as of May 2026, on-demand A100 rental averages roughly USD 2/hour (around USD 1.35–1.49/hr at specialized GPU clouds, up to ~USD 3.67/hr on hyperscalers for a single-GPU A100-40GB instance) — so ~200 A100s for one region’s ranking is on the order of USD 400/hour, or several million dollars a year before any redundancy headroom. Cost optimization (quantization, batching, distillation) is therefore not a nice-to-have but a budget line item.

As-of context

A100 cloud pricing figures are point-in-time (May 2026); on-demand rates have drifted down roughly 11% year-over-year and vary widely by provider and commitment term. Treat the dollar figures as order-of-magnitude, not a quote.

3.2 Memory Footprint

Modern model sizes:

small ranker (10M params, fp32)         = 40 MB
medium ranker (100M params, fp32)        = 400 MB
ResNet-50 (25M params, fp32)             = 100 MB
ViT-Large (300M params, fp32)            = 1.2 GB
LLaMA-2 7B (fp16)                        = 14 GB
LLaMA-2 70B (fp16)                       = 140 GB    -- requires multi-GPU
GPT-4-class (fp16, ~1T params)           ≈ 2 TB     -- massive multi-node

A100 has 40 GB or 80 GB; H100 has 80 GB. A 70B-parameter LLM in fp16 doesn’t fit on one A100; you need tensor parallelism splitting the model across 2–4 GPUs.

3.3 Latency Budget Decomposition

For a 50 ms p99 inference budget on a ranker:

network ingress (client → server)          ~ 1-2 ms
authentication, request parsing            ~ 1 ms
feature lookup (online feature store)      ~ 5 ms
batching wait                              ~ 5 ms (controllable)
GPU memcpy host → device                   ~ 1-2 ms
GPU compute                                ~ 20 ms
GPU memcpy device → host                   ~ 1 ms
response serialization                     ~ 1 ms
network egress                             ~ 1-2 ms
buffer for tail                            ~ 10 ms
Total                                       ~ 50 ms p99

Each row is non-trivial. Removing 5 ms from any single component is a meaningful project.

4. API Design

4.1 Predict (REST/gRPC)

TensorFlow Serving exposes both REST and gRPC. The Predict API in REST:

POST /v1/models/recommender:predict
Content-Type: application/json
{
  "instances": [
    {"user_id": 12345, "context_embedding": [0.1, 0.2, ...]},
    {"user_id": 67890, "context_embedding": [0.3, 0.4, ...]}
  ]
}
→ 200 OK
{
  "predictions": [
    [0.873, 0.541, ...],       // raw logits or probability vector per instance
    [0.222, 0.998, ...]
  ]
}

The gRPC variant uses protobuf, halving wire overhead — preferred for production.

4.2 LLM Inference (OpenAI-compatible)

Modern LLM serving converged on the OpenAI-compatible API:

POST /v1/chat/completions
{
  "model": "llama-2-7b",
  "messages": [{"role": "user", "content": "Explain consistent hashing."}],
  "max_tokens": 500,
  "stream": true
}
→ 200 OK (Server-Sent Events stream)
data: {"choices":[{"delta":{"content":"Consistent"}}]}
data: {"choices":[{"delta":{"content":" hashing"}}]}
data: ...
data: [DONE]

The streaming response yields tokens as they generate; the user sees the response progressively rather than waiting for the full answer. This requires a long-lived HTTP/2 or gRPC connection.

4.3 Model Management

Triton Inference Server formalizes model loading/unloading through the model-repository extension of the KServe V2 inference protocol (Triton model-repository extension docs). Note that load/unload are POST operations under a repository path — there is no DELETE verb, and an unload is an explicit POST to a .../unload endpoint:

POST /v2/repository/index                       # list every model in the repository
POST /v2/repository/models/recommender/load     # load (or reload) the recommender model
POST /v2/repository/models/recommender/unload    # unload it (optional unload_dependents flag)
GET  /v2/health/ready                            # is the server ready to serve?
GET  /v2/models/recommender/ready                # is this specific model ready?

The load request optionally carries a JSON config parameter (an inline model configuration) and file:<version>/<file-name> parameters that upload model files directly, so a control plane can push a new version without staging it through the model repository on disk first. KServe wraps these primitives in a Kubernetes Custom Resource Definition (CRD) layer so that “deploy model version N” becomes a declarative InferenceService spec rather than an imperative API call.

5. Data Model

5.1 Model Artifacts

Object Storage (S3 / GCS) layout:
/models/recommender/
    1/                                   <- version 1
        saved_model.pb                   <- TF graph (or model.onnx, model.pt)
        variables/
        assets/
    2/
    3/

The serving server discovers versions in the model directory and loads them per its routing config.

5.2 Model Registry

Table:        models
PK:           model_id
Columns:      name, framework, training_dataset_id, training_run_id, owner_team,
              created_ts, status (production | shadow | staged | retired),
              artifact_url, baseline_metrics (JSON: AUC, log-loss, latency)

Table:        model_deployments
PK:           (model_id, environment, region)
Columns:      version, weight (% traffic), since_ts

The deployment table drives runtime routing. Updates to it are picked up by the serving fleet.

5.3 Inference Logs

For monitoring, every inference (or a sample) is logged:

Topic:        inference.predictions
Schema:       (request_id, model_id, version, ts, input_features, output, latency_ms, error)
Retention:    7 days hot, then aggregated rollups

Used for prediction-drift monitoring, debugging, and offline replay.

6. High-Level Architecture

flowchart TB
    Client[Client / API Service] -->|gRPC or REST| LB[Load Balancer]
    LB --> SvcRouter[Model Router]
    SvcRouter -->|consult deployment table| Reg[(Model Registry)]
    SvcRouter -->|route to right replica| ServerA[Serving Replica A<br/>v1: 90% traffic]
    SvcRouter --> ServerB[Serving Replica B<br/>v2: 10% canary]
    SvcRouter --> Shadow[Shadow Replica<br/>v3: shadow traffic, no return]

    ServerA --> Batcher[Dynamic Batcher]
    Batcher --> Runtime[Inference Runtime<br/>TF / PyTorch / ONNX / TensorRT]
    Runtime --> GPU[(GPU / CPU)]

    ServerA -->|features| FStore[(Feature Store<br/>online)]
    Runtime -->|prediction logs| Kafka[Kafka]
    Kafka --> Drift[Prediction-Drift Monitor]
    Kafka --> Replay[Replay / Eval]

    ArtifactStore[(Model Artifact Store<br/>S3)] -.->|model loading| ServerA & ServerB & Shadow

What this diagram shows. A request flows through a load balancer to a model router that consults the model registry to learn which versions are deployed and at what traffic weights. The router forwards the request to the appropriate serving replica — most replicas run the production version (v1 here, 90 % of traffic), some run a canary (v2, 10 %) for live evaluation, and shadow replicas run the candidate v3 in parallel without returning its prediction (used for offline metric comparison without affecting live behavior). Within a replica, the dynamic batcher coalesces concurrent requests into a single GPU launch — this is the key throughput optimization. The inference runtime abstracts over the framework (TF Serving runs SavedModel; Triton can run TF, PyTorch, ONNX, TensorRT, custom backends). After inference, the result is returned and a sample of (input, output) is logged to Kafka for drift monitoring and replay analysis. Model artifacts live in object storage; replicas pull artifacts on startup or hot-load on a registry update notification.

The key insight is the separation of routing (per-request: which version handles this), batching (per-microsecond: which requests bundle together for one GPU launch), and runtime (per-instruction: how is the model executed). Each operates on a different timescale and is independently optimizable.

7. Request Flow / Sequence

7.1 Online Inference (with Batching)

sequenceDiagram
    participant C as Client
    participant R as Router
    participant B as Batcher
    participant Q as Batch Queue
    participant RT as Runtime/GPU

    C->>R: predict(features_request_1)
    R->>B: enqueue(request_1)
    B->>Q: append request_1 [batch growing]
    C->>R: predict(features_request_2)
    R->>B: enqueue(request_2)
    B->>Q: append request_2
    Note over Q: batch reached size 64 (or 5 ms timer fired)
    Q->>RT: launch(batch of 64)
    RT->>RT: GPU compute
    RT-->>Q: batch results
    Q-->>R: per-request results scattered back
    R-->>C: result_1
    R-->>C: result_2

The batcher is an independent component that holds requests in a queue until either the batch-size threshold (e.g., 64 requests) or batch-timeout (e.g., 5 ms) fires. Larger batches → higher throughput but larger latency tail; the tuning depends on QPS and SLA.

7.2 Canary Routing

sequenceDiagram
    participant C as Client
    participant R as Router
    participant V1 as Replica v1 (90%)
    participant V2 as Replica v2 (10%)

    C->>R: predict(request)
    R->>R: hash(user_id) → bucket; <br/>10% → v2, 90% → v1
    alt routed to v1
        R->>V1: predict(request)
        V1-->>R: result_v1
    else routed to v2
        R->>V2: predict(request)
        V2-->>R: result_v2
    end
    R-->>C: result

The hash on user_id makes the routing sticky per user — the same user consistently sees the same version, which is essential for honest A/B-test attribution. See AB Testing Platform System Design for the full experimentation framework.

7.3 Shadow Traffic

sequenceDiagram
    participant C as Client
    participant R as Router
    participant V1 as Replica v1
    participant Shadow as Shadow v3

    C->>R: predict(request)
    R->>V1: predict(request)
    R->>Shadow: predict(request, shadow=true)
    V1-->>R: result_v1
    Shadow-->>R: result_v3 (logged, not returned)
    R-->>C: result_v1

Shadow traffic compares v3 against v1 offline on real production inputs without influencing user experience. If v3 performs as expected (or better) on shadow, promote it to canary; if not, debug without ever having affected a user.

8. Deep Dive

8.1 Dynamic Batching — the Core Throughput Optimization

GPUs are throughput-optimized; CPU-to-GPU launch overhead is roughly constant per kernel (microseconds), but each kernel can compute thousands of operations in parallel. Running one inference at a time uses 0.1 % of the GPU’s compute capacity and pays the full launch overhead. Batching N concurrent requests into one kernel launch increases per-launch work by N× while paying launch overhead just once.

8.1.1 The Math

Suppose a single inference has:

Launch overhead: 100 µs (constant).
Compute time per inference: 10 µs (with capacity to do many in parallel).

Batch size 1: total = 110 µs; throughput = 9,000 inferences/s. Batch size 64: total ≈ 100 + 64 × 10 = 740 µs (because compute scales sub-linearly until GPU saturates, but for argument sake assume linear); throughput = 64,000 / 0.74 ms ≈ 86,000 inferences/s.

In practice GPUs achieve near-flat throughput across small-to-mid batch sizes because the launch overhead dominates and the SMs (streaming multiprocessors) parallelize the work. So batch sizes of 32–256 give 10–100× throughput improvement vs unbatched.

8.1.2 Latency Cost

Batching introduces a wait: the first request to arrive in a batch must wait for the batch to fill up (or for the timeout to fire). Configured as:

max_batch_size: 64
max_batch_delay_microseconds: 5000 (5 ms)

A batch fires when either condition is met. Tradeoff:

Small max_batch_delay: low latency tail but lower throughput at low QPS.
Large max_batch_delay: better throughput, but slow requests at low QPS.

At sustained high QPS, batches fill quickly and the delay never fires; at low QPS, the delay dominates and per-request latency ≈ delay/2.

8.1.3 Variable-Length Batching (LLMs)

For LLMs, “batching” is more complex because requests have different sequence lengths (and different remaining-token-counts during decoding). Naive padding to the longest sequence wastes compute. Continuous batching (see §8.5) handles this by batching at the per-token granularity rather than the per-request granularity.

8.2 Inference Optimization

A trained model can be 10× to 100× faster in inference with no accuracy loss (or minor controlled loss) via these techniques.

8.2.1 Quantization

Convert model weights from fp32 to fp16, bf16, int8, or even int4. The compute is faster (more operations per cycle on tensor cores) and the memory bandwidth is reduced (fewer bytes to fetch).

fp16 / bf16: ~2× speedup on Volta-and-later GPUs; nearly free in accuracy if the model was trained in fp32.
int8: 2–4× speedup over fp16; requires quantization-aware training or post-training calibration to maintain accuracy. NVIDIA TensorRT and PyTorch’s torch.quantization automate this.
int4 / lower: aggressive; only viable for inference-only scenarios (e.g., LLMs being run on consumer GPUs); accuracy loss is significant unless techniques like GPTQ or AWQ are used.

Han, Mao, & Dally 2016 ICLR “Deep Compression” was the seminal paper; production-quality tooling caught up over the following years.

8.2.2 Kernel Fusion and Graph Optimization

The framework’s eager-mode execution issues separate GPU kernels for each operation (matmul, bias-add, ReLU, etc.). Fusing them into one kernel eliminates intermediate memory allocations and kernel-launch overhead. Tools:

TensorRT (NVIDIA): ahead-of-time compiler from TF/ONNX/PyTorch graphs to optimized GPU kernels. 2–10× speedup typical on CV/NLP models.
OpenVINO (Intel): same idea for Intel CPUs and GPUs.
XLA (TensorFlow / JAX): just-in-time compiler with similar fusion.
ONNX Runtime: has its own optimization passes including operator fusion and graph rewriting.
TorchScript / TorchInductor / TorchCompile (PyTorch 2.x): bringing JIT compilation to PyTorch.

8.2.3 Pruning and Distillation

Pruning: zero out small-magnitude weights; structured pruning (whole channels) yields actual speedup on hardware (unstructured pruning yields zero-fill but no actual speedup on most hardware).
Distillation: train a smaller “student” model to match a larger “teacher” model’s predictions. The student is faster and often nearly as accurate. Used heavily for putting BERT-class models in latency-sensitive paths (DistilBERT, TinyBERT).

8.2.4 Activation Caching

For token-by-token generation in LLMs, the KV cache (key-value tensors from the attention mechanism) is reused across tokens within the same sequence. Without this cache, each new token would require recomputing the entire attention against all prior tokens — quadratic cost. With it, each new token is linear-in-context. KV-cache management is the central memory-pressure problem in LLM serving.

8.3 Multi-Tenancy on GPUs

A single A100 has 40 or 80 GB of memory. A small model (~1 GB) wastes 39 GB if it’s the only tenant. NVIDIA has multiple solutions:

Multi-Instance GPU (MIG) on A100/H100: partition the GPU into up to 7 logically isolated instances, each with its own dedicated streaming multiprocessors (SMs), L2 cache slices, and memory (per the NVIDIA MIG User Guide and the DGX A100 user guide). The partitioning is spatial, not time-sliced, so it is hard isolation — a fault or memory pressure in one instance does not affect another — which is what makes it suitable as a security/multi-tenant boundary. Both Ampere (A100) and Hopper (H100) support the 7-instance maximum; profiles are named by their slice size (e.g. 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb on an 80 GB A100).
Triton instance groups: run multiple replicas of the same model on one GPU to overlap their kernel launches; Triton schedules them via CUDA streams.
Time-slicing: simpler form of sharing; each model gets a time-slice; lower utilization than MIG but no setup.

The trick is balancing isolation (one tenant’s bug doesn’t OOM another) with utilization (idle GPU memory is wasted money).

8.4 Model Versioning and Rollout

Production serving must handle multiple model versions simultaneously. The lifecycle:

Train: produces v_new in the model registry; status “staged”.
Shadow: route a copy of production traffic to v_new with shadow=true; v_new’s predictions are logged but not returned to users. Compare offline against v_old’s predictions on the same inputs.
Canary: route 1 % of real traffic to v_new; compare online metrics (engagement, conversion, latency) against v_old.
Gradual rollout: 1 % → 10 % → 50 % → 100 %, with pause-and-evaluate between steps.
Promote: v_new is the new “production”; v_old is “previous” (kept around for fast rollback).
Retire: after a confidence period (typically days or weeks), unload v_old.

Each transition is gated by metrics (no regression in latency, no regression in business metric); automated rollback on regression is increasingly standard.

8.5 LLM Serving — A New Paradigm

LLM inference differs fundamentally from feed-forward model inference:

Autoregressive: tokens are generated one at a time, each conditioned on all previous tokens.
Two phases: prefill (compute attention over the full input prompt — compute-bound) and decode (generate each subsequent token, mostly memory-bandwidth-bound for the KV cache).
Variable length: prompts can be 100 tokens or 100,000 tokens; outputs can be 1 token or 4,000 tokens.
Long memory footprint: each in-flight request occupies KV-cache memory proportional to its sequence length.

8.5.1 Static Batching is Wrong for LLMs

If we naively batch 32 requests with the longest one having 1000 output tokens, requests that finish their generation early are stalled until the longest one completes. GPU utilization collapses.

8.5.2 Continuous Batching (Yu et al. OSDI 2022, “Orca”)

Instead of batching at the request level, batch at the token level. After each token-generation step, the batch’s composition is re-evaluated: requests that just finished are removed; new arriving requests are added. The batch always runs at near-maximum size; finished requests don’t stall the rest.

Orca introduces this under two named mechanisms — its own paper does not use the phrase “continuous batching,” which was popularized later by vLLM/Anyscale write-ups (per the Orca paper, OSDI 2022). The first is iteration-level scheduling: the scheduler invokes the execution engine to run only a single iteration (one decode step) of the model on the current batch, then returns control so the batch can be re-formed before the next iteration — rather than running a request to completion before touching the batch again. The second is selective batching: because the requests in a batch are at different sequence positions and have different KV-cache lengths, you cannot naively stack their attention computations; Orca applies token-wise batching to all non-attention operations (the dense matmuls, which are shape-uniform) while processing the attention operation for each request individually. Orca reports a 36.9× throughput improvement at the same latency versus NVIDIA FasterTransformer on a GPT-3 175B model (Yu et al. OSDI 2022) — the autoregressive, variable-completion workload is exactly where request-level batching wastes the most GPU.

This is the dominant pattern in modern LLM serving frameworks: vLLM advertises “continuous batching of incoming requests” as a headline feature (per the vLLM docs), and TGI, TensorRT-LLM, and NVIDIA Triton’s TensorRT-LLM backend all implement the same iteration-level idea.

8.5.3 Paged Attention (Kwon et al. SOSP 2023, “vLLM”)

The KV cache for a request can be huge — for a 7B model at 4096-token context, ~2 GB per request. Storing a contiguous reservation for the maximum possible context per request wastes memory (most requests don’t reach max).

vLLM borrows the paging concept from operating systems’ virtual memory: allocate KV cache in fixed-size blocks (e.g., 16 tokens each) and grow the cache block-by-block as the sequence extends. A logical sequence is a sequence of blocks; the blocks live anywhere in physical GPU memory, with a block table mapping logical to physical positions exactly as an OS page table does. The paper’s headline results are near-zero waste in KV-cache memory and flexible sharing of the KV cache within and across requests (e.g., the same system prompt or a shared few-shot prefix is stored once and referenced by many sequences), which together yield a 2–4× throughput improvement at the same latency versus the prior state of the art (FasterTransformer and Orca), with the gain “more pronounced with longer sequences, larger models, and more complex decoding algorithms” (per Kwon et al. SOSP 2023). The practical effect at serving time is roughly 2–4× more concurrent requests packed into the same GPU memory budget.

8.5.4 Tensor and Pipeline Parallelism

Models too large for one GPU are sharded:

Tensor parallelism: split each weight matrix across N GPUs; communication required at every layer (typically NVLink-connected GPUs in the same node).
Pipeline parallelism: split the model’s layers across GPUs (GPU 0 has layers 1-10, GPU 1 has layers 11-20, etc.); inputs flow through in a pipeline.
Production systems usually combine both: tensor-parallel within a node, pipeline-parallel across nodes.

The latency tax is significant — every layer transition incurs network communication — but it’s the only way to serve 70B+ parameter models.

8.6 Warm-Up

A freshly loaded model is slow on the first request:

The framework’s lazy compilation kicks in (TensorRT engine build, XLA HLO compilation).
The GPU’s instruction cache is cold.
Memory allocators haven’t pre-allocated the workspace buffers.

The first request can be 10× slower than subsequent ones. Solution: synthetic warm-up at model-load time. The serving server runs a few canned requests through the loaded model before marking it ready; user requests don’t see the cold-start latency.

8.7 Drift Monitoring at the Serving Layer

A model’s quality degrades silently when:

The input feature distribution shifts (covariate shift).
The label distribution shifts (concept drift).
The model’s prediction distribution shifts (often a downstream symptom).

The serving layer is well-positioned to monitor:

Input-feature distributions — log every (input, prediction) sample; compare distributions to training-time. PSI, KS-test, etc. (See Feature Store System Design for similar concerns at the feature-pipeline layer.)
Prediction distributions — alert if today’s prediction distribution differs significantly from yesterday’s.
Latency distributions — alert if p99 inference time creeps up (often indicates model bloat or hardware degradation).

Sculley et al. 2015 specifically called out monitoring as an under-invested area in production ML; modern serving platforms (Triton, KServe) bake it in.

9. Scaling Strategy

9.1 What Breaks First

GPU memory. Loading a 70B-parameter LLM on insufficient GPUs fails. Tensor parallelism is the only path; planning starts with knowing model size at design time.
Tail latency under load. Average latency stays low until ~70 % capacity, then p99 explodes due to queuing. Standard mitigation: autoscale aggressively (target ~50 % capacity); add request shedding (drop the lowest-priority requests when overloaded).
Cold-start latency on autoscale. Spinning up a new replica with a 10 GB model takes minutes. Pre-warmed replica pool, or graceful capacity ramp ahead of expected demand spikes.
Model-loading thrash. Hot-swapping models takes time and memory. If multiple model deployments happen concurrently, replicas can OOM. Coordinate via a deployment scheduler.
Network egress for LLM responses. Large generated outputs (1000+ tokens) are large bytes; aggressive HTTP/2 multiplexing or gRPC streaming is essential.

9.2 Autoscaling

KServe and Seldon both integrate with Kubernetes HPA (Horizontal Pod Autoscaler) and KEDA. Common metrics for scaling:

Average GPU utilization > 60 % → scale up.
Average GPU utilization < 30 % → scale down (with a cool-down to avoid flapping).
Per-request queue depth.
Latency p95 above target.

LLM serving complicates this because batch sizes are dynamic; the right metric is often “in-flight tokens being processed” rather than QPS.

9.3 Sharding

For very high QPS on a single model, shard by request hash (e.g., user_id mod N) across replicas — this is not for capacity so much as for affinity (preserving the same user’s request → same replica → warm caches).

For very large models, shard the model itself via tensor parallelism (within a node) and pipeline parallelism (across nodes).

9.4 Edge Serving

For some workloads (mobile recommendation, browser-side classification), the model runs on the user’s device:

TensorFlow Lite for mobile.
ONNX Runtime Mobile.
Core ML on iOS.
WebGPU / WebGL in the browser.

This skips network latency entirely but constrains model size (mobile devices have ~2-8 GB RAM and limited GPU). Quantized small models work; LLMs generally don’t. Edge serving doesn’t replace centralized serving; it complements it for latency-sensitive predictions on small models.

10. Real-World Example

TensorFlow Serving (Google, open source). Olston et al. 2017 NeurIPS workshop paper. Originally just for TF models; later extended. Powers many of Google’s internal serving fleets and is widely used externally. Strengths: deep TF integration, mature, gRPC-first.

NVIDIA Triton Inference Server. Multi-framework (TF, PyTorch, ONNX, TensorRT, custom Python backends), GPU-optimized, hardware-aware (works with MIG, multiple instance groups, multi-GPU). The dominant choice for serious GPU-serving fleets.

KServe (Kubernetes-native, formerly KFServing). The Kubernetes CRD layer that wraps any serving runtime (TF Serving, Triton, custom) into a declarative deployment with autoscaling, canary rollout, and integration with knative. Used heavily in Kubeflow stacks.

Seldon Core. Commercial Kubernetes-native serving with strong support for explanation APIs and shadow deployments.

BentoML. Python-friendly serving framework with an emphasis on packaging — bundle the model + Python preprocessing + dependencies into a deployable artifact. Targets data scientists who want to ship without DevOps overhead.

vLLM (Kwon et al. SOSP 2023). The dominant open-source LLM serving framework as of 2024-2026. Paged attention + continuous batching = 2-10× throughput vs naive serving. Supports most popular open LLMs.

HuggingFace Text Generation Inference (TGI). HF’s production-grade LLM serving, with support for popular models. Used to power their Inference Endpoints product.

OpenAI’s serving fleet (proprietary). Powers ChatGPT, Codex, GPT-4 API. No public details, but inferred from public talks: heavy use of continuous batching, multi-node tensor parallelism, sophisticated KV-cache sharing across users with shared system prompts.

Meta’s serving infrastructure (proprietary). Powers feed ranking, ad ranking, integrity classifiers across 3+ billion users. Heavy use of custom accelerators (originally Big Basin GPU servers; now their own AI accelerators).

What is not in flux is the underlying technique set: all of the open frameworks above implement iteration-level/continuous batching and paged KV-cache management, and most now ship speculative decoding and multi-LoRA serving as well (see §8.5 and §13). The differentiation between them in 2026 is largely engineering polish, hardware coverage, and ops integration rather than algorithmic novelty.

Uncertain

Verify: the relative production throughput/latency ranking of TGI vs vLLM vs TensorRT-LLM vs proprietary stacks (OpenAI, Anthropic, Google internal) on a given model and hardware. Reason: head-to-head benchmarks go stale within months as each project lands new kernels and scheduling tricks, and proprietary stacks publish no numbers. To resolve: benchmark the specific model/hardware/workload yourself at decision time; do not trust any published cross-framework comparison older than a few months. uncertain

11. Tradeoffs

Design Choice	Option A	Option B	When A wins	When B wins
Framework abstraction	Single (TF Serving)	Multi (Triton)	Pure-TF shop	Mixed framework portfolio
Hardware	CPU	GPU	Small models, low QPS	Large models, high QPS
Quantization	fp16	int8	Quality-critical	Throughput-critical
Quantization	int8	int4	Need <1 % accuracy loss	Aggressive cost optimization
Batching	Static	Dynamic	Batch jobs	Online with mixed QPS
Model placement	One model per GPU	Multi-tenant	Strict isolation	Cost optimization
Rollout	Direct (replace)	Canary + gradual	Low-stakes	High-stakes (revenue / safety)
Communication	REST	gRPC	Simplest, low QPS	High QPS, low overhead
LLM batching	Static	Continuous	Fixed-length output	Variable-length output (typical)
LLM memory	Reservation per request	Paged attention	Simple	Memory-pressure-sensitive
Autoscaling	Predictive (schedule)	Reactive (metrics)	Predictable load	Unpredictable load
Edge vs cloud	Cloud	Edge (on-device)	Large model, central feature lookup	Privacy-critical, offline-tolerant
Streaming output	All-at-once	Token-by-token (SSE)	Short outputs	LLM, user-facing

12. Pitfalls

No dynamic batching. A naively built serving server runs each request on its own GPU launch. Throughput is 10-100× lower than necessary. Batching is mandatory for any non-trivial GPU workload.
Cold-start hits real users. A newly-spawned replica’s first request is 10× slower. Run synthetic warm-up at startup; mark the replica unhealthy until warm.
Replacing a model in place without canary. A bad model deploys; production breaks; minutes of damage before rollback. Always canary.
No drift monitoring. Model degrades silently for weeks. Bake feature-distribution and prediction-distribution monitoring into the serving layer.
GPU OOM from request size variance. A request with a much larger feature blob than typical OOMs the batch. Set hard limits on per-request input size; reject with 4xx rather than crashing.
Quantization without accuracy validation. Converting fp32 → int8 saves cost but can degrade accuracy. Always validate on a held-out test set; A/B test in shadow before production.
Multi-tenant interference. Tenant A’s spiky traffic causes tenant B’s latency to climb. Either isolate (separate replicas; MIG instances) or carefully size with headroom.
Streaming latency reported wrong. For LLMs, “p99 latency” is meaningless; you need time-to-first-token and inter-token latency separately. Without these metrics, regressions go undetected.
KV-cache fragmentation. Without paged attention, long-running LLM serving accumulates memory fragmentation; concurrent requests drop until restart. vLLM addresses this; naive implementations don’t.
Stale model in canary. v2 was the canary 6 weeks ago and never promoted; it’s now serving stale predictions. Lifecycle discipline: promote or retire within a defined window.
Inconsistent feature lookup paths. Serving fetches features from feature-store v1; training used feature-store v2. Skew. Ensure parity.
Logging-induced latency. Synchronously logging every (input, output, latency) to Kafka adds milliseconds. Use sampling and async batching; never block the inference path on logging.
Auto-scaling with stale model artifacts. New replica starts up but pulls v_old from S3 because the artifact registry hasn’t replicated yet. Race conditions in deployment pipelines.
Unbounded request queue. Under overload, the queue grows; latency explodes; failures cascade. Bounded queue with explicit drop-on-full prevents this.
Failing to partition by user for sticky routing. Without sticky routing, a user might hit v1 and v2 alternately; their experience is confusing and A/B-test attribution is broken.

13. Common Interview Variants and Follow-Ups

“Now design serving for an LLM (e.g., a 7B-parameter chatbot).” Continuous batching, paged attention (vLLM-style), tensor parallelism if larger; streaming responses; OpenAI-compatible API.
“Add real-time A/B testing across model versions.” Sticky routing on user_id; integration with AB Testing Platform System Design; metric collection per variant.
“How do you serve 1000 different small models?” “Many small models, one machine” — each model takes ~100 MB, all loaded in memory; route by model_id. Triton’s instance_groups handles this with concurrent multi-model execution.
“Implement A/B routing without service-level redeploys.” Configuration-driven routing read from a config server; the router watches for changes and updates rules in seconds.
“What if a model needs to call other ML services as part of its inference?” Serving graphs (KServe Inference Graph, Seldon’s prediction graphs) chain multiple models. Each step is a serving call; latency is the sum.
“Add explainability to predictions.” Methods like SHAP, LIME, attention-rollout. Some serving frameworks (Seldon) bake in explanation endpoints.
“How do you debug a slow inference in production?” Per-stage timing instrumentation: feature-lookup, batching wait, GPU time, post-processing. Profiling captures (NVIDIA Nsight Systems for GPU traces). Replay the slow request in a debug environment.
“Build edge serving for mobile.” Quantize the model (often int8); export to TFLite or ONNX Runtime Mobile; bundle in the app; sync model versions periodically.
“How do you achieve sub-10ms inference for a deep model?” Quantize to int8; compile via TensorRT/ONNX Runtime; pre-load on GPU; pin batch size; warm caches; accept lower QPS per replica in exchange for tighter latency.
“Design batch inference for scoring 1B records overnight.” Different system: throughput-only, no latency budget. Spark on GPUs; static batches of optimal size; fault tolerance via checkpointing.
“Serve a base LLM with a per-customer fine-tune for each of 1000 customers.” Don’t load 1000 full models — use multi-LoRA: one shared base model in GPU memory plus 1000 small low-rank adapters, swapped per request. The primary design reference is S-LoRA’s Unified Paging (one memory pool for adapter weights + KV cache); vLLM exposes this via --lora-modules and runtime adapter loading (see §14).
“Make decoding faster without losing quality.” Speculative decoding: a small draft model proposes k tokens, the base model verifies them in one forward pass, mismatches are discarded — lossless speedup, now built into vLLM via the Speculators library (see §14).
“What’s different for serving on TPUs vs GPUs?” TPUs (Google’s accelerator) prefer larger static batches; their compilation model (XLA) is more opinionated; less framework flexibility (TF/JAX-friendly, PyTorch via PyTorch/XLA but historically less smooth).

14. Open Questions / Uncertain

Speculative decoding and attention variants have been commodified. As of late 2025 / early 2026 the question of whether speculative decoding would stay bespoke is settled: it is now a first-class, production-ready feature of vLLM via the Speculators library — a unified library for building, training, and storing speculative-decoding algorithms whose trained draft models run directly in vLLM’s production server (per the vLLM speculative-decoding docs and the Speculators announcement). Speculative decoding is lossless — a small fast draft model proposes several tokens, the large base model verifies them in a single forward pass, and any mismatched tokens are discarded — so it speeds up decoding without changing output quality. Multi-query and grouped-query attention have likewise become standard in modern model architectures rather than serving-stack add-ons.

Multi-LoRA serving is well-studied and supported. The flagship primary reference is S-LoRA (Sheng et al., MLSys 2024), which serves thousands of concurrent LoRA (Low-Rank Adaptation) adapters on one shared base model. Its core mechanism is Unified Paging — a single memory pool that holds both dynamic adapter weights (of differing ranks) and KV-cache tensors (of differing sequence lengths) in the same paged structure — plus custom CUDA kernels for heterogeneous batched LoRA computation; it reports up to 4× throughput over naive vLLM LoRA serving and serving “several orders of magnitude” more adapters than HuggingFace PEFT (per Sheng et al., S-LoRA, arXiv 2311.03285). vLLM itself ships multi-LoRA: multiple adapters at startup via --lora-modules, a max_loras concurrency cap, and dynamic per-request loading via /v1/load_lora_adapter (gated behind VLLM_ALLOW_RUNTIME_LORA_UPDATING, which the docs flag as a security risk for production) (per the vLLM LoRA docs).

Uncertain

Verify: the per-adapter latency/throughput characteristics of multi-LoRA serving at very high QPS with hundreds of distinct adapters in flight. Reason: vLLM’s docs cover configuration but publish no scaling benchmarks, and S-LoRA’s numbers are research-bench, not production telemetry. To resolve: load-test the target adapter count on the target hardware. uncertain

Uncertain

Verify: cost-per-token for serving >70B-parameter LLMs. Reason: inference hardware (NVIDIA H100/H200/B100/B200, Google TPU v5/v6, AWS Inferentia2/Trainium, AMD MI300) and software keep moving, so any specific figure decays in months. To resolve: re-derive from current hardware rental rates and measured tokens/sec at decision time. uncertain

Uncertain

Verify: the “right” split of preprocessing across client / serving layer / feature store. Reason: this is genuinely workload-specific (latency budget, feature freshness, where the data already lives), not a single correct answer. To resolve: decide per workload against the latency budget in §3.3 and the train/serve-skew constraint in §12. uncertain

15. See Also

Major System Designs MOC
SWE Interview Preparation MOC
Feature Store System Design — the input substrate
Recommendation Engine System Design — heavy consumer of model serving
Ad Serving System Design — heavy consumer of model serving with strict latency
Fraud Detection Pipeline System Design — heavy consumer of model serving
AB Testing Platform System Design — for canary and shadow rollout integration
Distributed Log System Design — Kafka, for prediction and feature-drift logging
Distributed Key Value Store System Design — for low-latency model artifact metadata
Consistent Hashing — for sticky-routing and sharding
LRU Cache — for hot-input caching at the inference layer
Bloom Filter — for fast membership tests in routing decisions
Wide & Deep — example deep model commonly served
DLRM — example deep recommender model
BST — sequence-aware ranking model
Recommender Systems MOC

vault

Explorer

ML Model Serving System Design

ML Model Serving System Design

1. Why This System Appears in Interviews

2. Requirements

2.1 Functional Requirements

2.2 Non-Functional Requirements

3. Capacity Estimation

3.1 GPU Throughput as the Constraint

3.2 Memory Footprint

3.3 Latency Budget Decomposition

4. API Design

4.1 Predict (REST/gRPC)

4.2 LLM Inference (OpenAI-compatible)

4.3 Model Management

5. Data Model

5.1 Model Artifacts

5.2 Model Registry

5.3 Inference Logs

6. High-Level Architecture

7. Request Flow / Sequence

7.1 Online Inference (with Batching)

7.2 Canary Routing

7.3 Shadow Traffic

8. Deep Dive

8.1 Dynamic Batching — the Core Throughput Optimization

8.1.1 The Math

8.1.2 Latency Cost

8.1.3 Variable-Length Batching (LLMs)

8.2 Inference Optimization

8.2.1 Quantization

8.2.2 Kernel Fusion and Graph Optimization

8.2.3 Pruning and Distillation

8.2.4 Activation Caching

8.3 Multi-Tenancy on GPUs

8.4 Model Versioning and Rollout

8.5 LLM Serving — A New Paradigm

8.5.1 Static Batching is Wrong for LLMs

8.5.2 Continuous Batching (Yu et al. OSDI 2022, “Orca”)

8.5.3 Paged Attention (Kwon et al. SOSP 2023, “vLLM”)

8.5.4 Tensor and Pipeline Parallelism

8.6 Warm-Up

8.7 Drift Monitoring at the Serving Layer

9. Scaling Strategy

9.1 What Breaks First

9.2 Autoscaling

9.3 Sharding

9.4 Edge Serving

10. Real-World Example

11. Tradeoffs

12. Pitfalls

13. Common Interview Variants and Follow-Ups

14. Open Questions / Uncertain

15. See Also

Graph View

Table of Contents

Backlinks