Two-Tower Retrieval Model

A two-tower retrieval model (also called a dual-encoder model, a bi-encoder, or by its original Microsoft name DSSM — Deep Structured Semantic Model) is a deep recommendation architecture in which two parallel neural networks encode users and items independently into a shared k-dimensional embedding space. The score for a given (user, item) pair is the dot product (or cosine similarity, equivalent for unit-normalized vectors) of the two encoded vectors. This architecture is the dominant retrieval / candidate-generation stage in modern industrial recommender pipelines — the stage whose job is to narrow a catalog of millions or billions of items down to a few hundred plausibly-relevant candidates per request, fast and approximately. Two-tower models are used at YouTube (since the 2016 Covington paper crystallized the multi-stage architecture), Google Search, TikTok, Spotify, Pinterest, and essentially every consumer-scale platform that has built recommendation infrastructure since the late 2010s. They are also the dominant retrieval architecture in semantic search and information retrieval more broadly, where the same dual-encoder pattern serves as the first stage of dense retrieval pipelines.

1. Origin and Motivation

The two-tower architectural pattern has two distinct origins that converged.

Microsoft DSSM (2013). Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero and Larry Heck published “Learning Deep Structured Semantic Models for Web Search using Clickthrough Data” at CIKM 2013. DSSM was designed for web search ranking: given a query and a set of candidate documents, produce a relevance score for each document. The DSSM architecture has a query tower (a deep neural network that encodes the query into a low-dimensional vector) and a document tower (encoding documents the same way), with the score being the cosine similarity of the two vectors. Training was done on click-through data with a cross-entropy loss treating clicked documents as positive and randomly-sampled non-clicked documents as negatives. DSSM is the direct ancestor of every two-tower model in production today, and the architectural pattern (separate query and document towers, dot-product scoring) has been remarkably stable for over a decade.

YouTube candidate generation (2016). Paul Covington, Jay Adams, and Emre Sargin published “Deep Neural Networks for YouTube Recommendations” at RecSys 2016. The paper describes YouTube’s deep-learning-based recommender as having two stages: candidate generation (a deep neural network that scores videos against a user representation, retaining the top-N) and ranking (a heavier model that re-scores those candidates with richer features). The candidate generation stage is structurally a two-tower model — the user network produces a user embedding, the video embeddings are precomputed for the catalog, and the top-N is found by nearest-neighbour search. The Covington paper made the multi-stage retrieval-and-ranking pattern the industry default.

The two origins converged because they were solving structurally the same problem: select a small relevant subset from a large catalog, fast, using deep representations. Web search and video recommendation are the same task at a high enough level of abstraction; the algorithmic answer is the same.

The motivation for two-tower over alternatives — a flat deep model that scores (user, item) pairs directly, or a memory-based neighborhood method — comes from the structural constraint of large-catalog retrieval. Scoring every (user, item) pair with a deep model is O(n) per request in catalog size and is infeasible at billion-item scale. The two-tower architecture’s killer property is that once trained, scoring decomposes into a dot product of independent embeddings, which means the item embeddings can be precomputed offline, indexed in an Approximate Nearest Neighbor structure, and queried in O(log n) time at request. The trade-off — discussed in §6 — is that the model loses the ability to learn arbitrary cross-feature interactions, but for the retrieval stage this is acceptable because precision (correct ordering) is the ranker’s job, not the retriever’s.

The motivation for two-tower over Matrix Factorization is twofold:

1. Cold start. Pure MF embeddings are keyed only on item IDs. A new item with no interactions has no embedding. The two-tower architecture takes item features (text embeddings, category, image embeddings) as inputs to the item tower, so a new item with rich features but no interactions still gets a meaningful tower output from day one.

2. Side features. MF cannot easily incorporate user demographics, request context, item metadata. Two-tower models concatenate these as inputs to the appropriate tower. The model learns end-to-end how to use them.

So a two-tower model is structurally a deep, feature-driven matrix factorization — it preserves MF’s killer property (dot-product scoring → indexable) while adding deep-learning’s flexibility (arbitrary feature inputs, multimodal inputs, rich learned representations).

2. The Architecture

The defining structural feature of the two-tower model is the decoupled design: the user-side network and the item-side network are completely independent until the final dot product. There are no cross-features, no interaction layers, no early fusion. The two networks share nothing except the loss they are optimized against.

flowchart LR
  subgraph User Tower
    UF[User features:<br/>ID, history, geo, device, time, demographics] --> UNN[User encoder<br/>(deep NN)]
    UNN --> UE[User embedding<br/>x_u ∈ ℝ^k]
  end
  subgraph Item Tower
    IF[Item features:<br/>ID, text, image, category, price, tags] --> INN[Item encoder<br/>(deep NN)]
    INN --> IE[Item embedding<br/>y_i ∈ ℝ^k]
  end
  UE --> Dot["Score s(u, i) = x_u · y_i"]
  IE --> Dot

What this diagram shows. Two parallel branches, each a deep neural network. The User Tower (top) takes user-side features — the user’s ID, the recent interaction history (sequence of recently consumed items), geographic and device features, request-time context like time-of-day, and demographic attributes — and produces a k-dimensional embedding x_u. The Item Tower (bottom) takes item-side features — the item’s ID, text content (description, title), image embeddings, category, price, tags — and produces a k-dimensional embedding y_i. The two embeddings are combined only at the very end, by a dot product x_u · y_i ∈ ℝ, which is the predicted relevance score. The key insight from this diagram is that the two towers never see each other’s inputs. This decoupling is what makes the model indexable — once trained, item embeddings can be precomputed and stored without reference to any specific user, and at request time the user embedding is computed once and matched against the precomputed item embeddings via fast nearest-neighbour search. Without the decoupling, every (user, item) score would require a full forward pass, which is infeasible at scale.

2.1 The towers

Each tower is a deep neural network whose architecture is a design choice. Common patterns:

Embedding-and-MLP. Each categorical feature (user ID, item ID, category) goes through an embedding lookup. Continuous features (age, price, time-of-day) are normalized. All are concatenated into a long input vector, then passed through a feed-forward network (typically 2 to 5 hidden layers, each with several hundred units, ReLU activations, optionally with batch normalization and dropout). The final layer projects to a k-dimensional output.

Sequence encoders for the user tower. The user’s recent interaction history is itself a sequence. A common pattern is to encode the last N interactions with a recurrent network (LSTM, GRU) or transformer (a small one), and use the final hidden state as part of the user representation. This brings sequence-aware behaviour into the retrieval stage.

Multi-modal item towers. When items have heterogeneous content (text + image + audio), each modality is encoded by its own sub-network (a sentence-transformer for text, a CNN or vision-transformer for images, a CLAP-style audio encoder for audio), and the modality outputs are fused in a late-fusion layer.

Tower-symmetry decision. Some implementations use structurally identical towers (same depth, same width, same activations); others use asymmetric towers tailored to the input modalities. There is no consensus; both work.

2.2 The score function

The score is the dot product:

s(u, i) = x_u · y_i = Σ_{f=1}^{k}  x_{u,f} · y_{i,f}

where:

• x_u, y_i ∈ ℝ^k are the user and item embeddings produced by their respective towers.
• The summation is over the k dimensions of the embedding space.
• The result is a real number; higher means more relevant.

Some implementations use cosine similarity instead of raw dot product. Cosine is s(u, i) = (x_u · y_i) / (‖x_u‖ · ‖y_i‖), equivalent to the dot product when both embeddings are L2-normalized to unit length. Normalizing both is a common choice because it bounds the score range and helps stability of the softmax loss (see §3.2). The two are mathematically equivalent up to the normalization step.

3. Training

Two-tower models are trained on (user, positive-item) pairs from interaction logs. The training problem is fundamentally an implicit-feedback problem (see Explicit vs Implicit Feedback): we know which items the user interacted with (positives) but we have no labelled negatives. The dominant training paradigm uses in-batch negative sampling with a softmax cross-entropy loss.

3.1 In-batch negative sampling

For a training batch of B user-positive-item pairs (u_1, i_1), (u_2, i_2), ..., (u_B, i_B), the procedure is:

1. Compute all B user embeddings: X = [x_{u_1}, x_{u_2}, ..., x_{u_B}] ∈ ℝ^{B × k}.
2. Compute all B item embeddings: Y = [y_{i_1}, y_{i_2}, ..., y_{i_B}] ∈ ℝ^{B × k}.
3. Compute the full B × B matrix of pairwise scores: S = X · Yᵀ ∈ ℝ^{B × B}. Entry S[a, b] = x_{u_a} · y_{i_b}.

The diagonal entries S[a, a] are the positive scores — u_a’s score for their own positive item i_a. The off-diagonal entries S[a, b] for a ≠ b are implicit negative scores — u_a’s score for some other user’s positive item i_b, which we treat as a negative for u_a because we have no evidence u_a interacted with i_b.

This gives B − 1 negatives per positive at essentially no extra computational cost beyond the single matrix multiplication. The technique is dramatically efficient on GPUs because the per-batch overhead is dominated by the tower forward passes, not the negative sampling.

3.2 The softmax cross-entropy loss

The per-row loss treats row a of S as a B-class classification problem where the correct class is the diagonal entry (the true positive):

L = − (1/B) · Σ_{a=1}^{B}  log( exp(S[a, a]) / Σ_{b=1}^{B} exp(S[a, b]) )

where:

• The numerator exp(S[a, a]) is the unnormalized probability of the positive class.
• The denominator Σ_b exp(S[a, b]) is the normalizing partition function over all classes (the positive plus the in-batch negatives).
• The log of their ratio is the log-likelihood of the correct class under a softmax.
• The negative average over the batch is the cross-entropy loss.

A minimal PyTorch-style implementation:

user_emb = user_tower(user_features)        # shape: [B, k]
item_emb = item_tower(item_features)        # shape: [B, k]
logits = user_emb @ item_emb.T              # shape: [B, B]
labels = torch.arange(B, device=logits.device)  # diagonal indices
loss = F.cross_entropy(logits, labels)

What this code does, line by line:

• The user and item towers are forward-passed independently on the batch’s user features and item features, producing two B × k embedding matrices.
• The matrix product user_emb @ item_emb.T produces the B × B score matrix in a single optimized matmul, leveraging the GPU’s BLAS routines.
• labels is the integer [0, 1, 2, ..., B-1], indicating that for row a, the correct class is column a (the diagonal).
• F.cross_entropy applies softmax to each row and computes the negative log-likelihood of the correct class, averaged over the batch.

The whole training step is a single matrix multiplication and a softmax — extremely efficient for any batch size that fits in GPU memory.

3.3 Sampling-bias correction

A critical detail that the basic in-batch loss gets wrong: popular items appear as positives for many users in any given batch (because they are popular, many users in the random batch sample have them as their positive). They therefore appear as in-batch negatives for all the other users in the batch. Without correction, the model learns to systematically push popular items down, because every gradient update penalizes them as negatives more often than as positives. The result is an under-recommendation of popular items in production.

The fix, due to Yi, Yang, Hong, Cheng, Heldt, Kumthekar, Zhao, Wei, Chi (RecSys 2019), “Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations”, is to subtract log P(item) from each logit before the softmax:

S_corrected[a, b] = S[a, b] − log P(item_b)

where P(item_b) is the empirical sampling probability of item b in batches (estimated online from a streaming-frequency tracker). This is mathematically equivalent to the sampled softmax correction used in language modelling (Bengio & Sénécal 2008) — the bias correction restores the unbiased gradient estimate.

In production at Google, this correction is essential. Forgetting it is a real and frequent production bug.

4. The Inference Architecture — Why Two-Tower Scales

The training architecture and the inference architecture are different. Training computes both towers on every batch. Inference uses them very differently:

flowchart LR
  subgraph Offline (run nightly or as catalog changes)
    Cat[Full catalog<br/>10⁶–10⁹ items] --> IT[Item Tower]
    IT --> EMB[Item embeddings<br/>n × k matrix]
    EMB --> IDX[ANN index build<br/>FAISS / ScaNN]
    IDX --> Persist[(Persistent ANN index)]
  end
  subgraph Online (per request)
    Req[Request: user features + context] --> UT[User Tower]
    UT --> UV[User vector x_u ∈ ℝ^k]
    UV --> Q[Approximate Nearest Neighbour query]
    Persist --> Q
    Q --> TopK[Top-K item IDs]
  end

What this diagram shows. The inference architecture splits into an offline phase (top) and an online phase (bottom). The offline phase runs the item tower over the entire catalog, producing one embedding per item, then builds an Approximate Nearest Neighbor (ANN) index over those embeddings. The index is persisted to a service (often a vector database like FAISS, ScaNN, or a managed product like Pinecone). The online phase runs only the user tower per request, producing a user embedding, then queries the ANN index for the top-K items whose embeddings are closest to the user vector. The key insight from this diagram is the asymmetry: the item tower runs once per item in the catalog (offline, batch); the user tower runs once per request (online, low-latency); the dot-product matching is delegated to the ANN index, which is sub-linear in catalog size. Online latency is dominated by the user-tower forward pass (typically a few milliseconds) plus the ANN query (typically single-digit milliseconds). The total online latency is O(k · L) where L is the user-tower depth — independent of catalog size.

4.1 Approximate Nearest Neighbor search

The ANN index is the engineering component that makes two-tower retrieval feasible. Exact nearest-neighbor search over a billion-vector index is O(n · k) per query — too slow. ANN data structures trade a small amount of recall for orders-of-magnitude speedup.

Two dominant ANN libraries:

FAISS (Facebook AI Similarity Search), github.com/facebookresearch/faiss, is the most-used open-source ANN library. It supports several index structures (IVF — Inverted File; HNSW — Hierarchical Navigable Small World; PQ — Product Quantization) with different speed-recall trade-offs. It is widely deployed in production at Meta and elsewhere.

ScaNN (Scalable Nearest Neighbors, Google), github.com/google-research/google-research/tree/master/scann, uses anisotropic product quantization and is highly optimized for high-dimensional embedding retrieval. It powers many of Google’s deployed retrieval systems.

Both libraries return approximate top-K with sub-millisecond latency on indices of hundreds of millions of vectors, and they scale to billion-vector indices with reasonable hardware.

The ANN index needs to be rebuilt or updated as the model is retrained — item embeddings change as the model learns, so the index becomes stale. Production systems rebuild the index periodically (nightly is common) and use online incremental updates for newly-added items between rebuilds.

5. The Multi-Stage Pipeline Position

Two-tower retrieval is the first stage of a multi-stage recommendation pipeline. It is not the entire system.

flowchart LR
  Cat[Catalog 10⁹ items] --> Two[Two-Tower<br/>Retrieval]
  Two --> Cand[~hundreds of candidates]
  Cand --> Rank[Heavy Ranker<br/>cross-features, GBDT/Deep]
  Rank --> Top[Ordered top-K]
  Top --> Re[Re-ranking<br/>diversity, freshness, business rules]
  Re --> Final[Final list to user]

What this diagram shows. Same multi-stage architecture as in Recommender Systems §5. The two-tower model is Stage 1 (retrieval), responsible for narrowing the catalog to a candidate set of a few hundred items. The ranker (Stage 2) then scores each candidate with a heavier model that uses cross-features (interactions between user features and item features that the two-tower model architecturally cannot use). The re-ranker (Stage 3) adjusts for diversity, freshness, and business rules.

Two-tower’s responsibility is recall — the candidate set must contain most of the genuinely good items. Precision — the precise ordering of those candidates — is the ranker’s responsibility. This division of labour is what allows each stage to use a model architecture suited to its job: a recall-optimized indexable model (two-tower) for retrieval, a precision-optimized cross-feature model (GBDT or deep ranker) for ranking.

The decoupling matters because the ranker can use cross-features the retriever cannot. For instance, a ranker can compute “does this user’s stated interest in genre=horror match this item’s genre?” as an explicit cross-feature, and learn that this interaction is more predictive than either feature alone. The two-tower retriever cannot compute this because the towers do not see each other’s features.

6. The Decoupling Trade-Off

The biggest architectural critique of two-tower models is the lack of cross-features. The user and item towers do not interact until the final dot product. The model cannot learn arbitrary user-item feature interactions like:

• “This user, who is a vegetarian, scores higher on items with category=vegetarian than the additive average of the two features would predict.”
• “This user, who has watched many sci-fi films, scores higher on items that are both sci-fi and less than two hours.”

A flat deep model that takes [x_u ; y_i] as input and learns arbitrary interactions through hidden layers can in principle capture these conditional preferences. The two-tower model cannot — by construction.

This is the explicit price for indexability. The dot product factorizes as x_uᵀ · y_i = Σ_f x_{u,f} · y_{i,f}, and the only way to score items quickly via ANN is for the score to factorize this way. Cross-feature models cannot be indexed because their score is not a dot product of independent embeddings; it is a function of the joint inputs.

The standard production resolution is the multi-stage pipeline: the two-tower retriever sacrifices cross-features for indexability; the downstream ranker recovers cross-features at the cost of operating only on a small candidate set. The two stages together handle both scaling and expressiveness.

Some attempts to add limited cross-features back into a two-tower model exist — for instance, by adding low-rank interaction layers between the towers (Tan et al. 2023, A Learnable Fully Interacted Two-Tower Model) — but they fundamentally compromise indexability and are not yet standard.

7. Strengths

Scales to billions of items via ANN. This is the central engineering property that motivates the architecture. Few alternatives can score against a billion-item catalog at sub-second latency.

Cold-start friendly when towers consume features. A new item with no interaction history but rich features (text, image, category) still gets a meaningful item-tower output from day one. This is one of the architecture’s most-cited advantages over pure matrix factorization. See Cold Start Problem.

Flexible features. Any feature that can be embedded — text, image, audio, structured metadata — fits into a tower. Multi-modal items are handled naturally with multi-modal towers.

Cheap online cost. One user-tower forward pass plus one ANN query per request. Total latency in single-digit milliseconds at production scale.

Dual use as embedding model. The learned item embeddings are useful for downstream tasks (similarity search, clustering, item-to-item recommendation) beyond just user-to-item retrieval.

8. Weaknesses

Limited expressiveness — no cross-features. The structural cost of indexability. The retriever cannot model arbitrary user-item feature interactions; the ranker has to.

Sampling bias from in-batch negatives. Popular items appear as in-batch negatives for many users, biasing the model against them. The Yi 2019 sampling-bias correction is essential and is often forgotten in implementations.

Embedding drift requires re-indexing. Each model update changes the item embeddings, requiring the ANN index to be rebuilt. Production pipelines need infrastructure for this — periodic rebuilds plus online updates for new items.

Hyperparameter sensitivity. Embedding dimension k, tower depth and width, learning rate, batch size, sampling correction strength — all matter. Production teams often spend substantial tuning effort on these.

ANN recall trade-off. The ANN index is approximate; some genuinely good items will be missed. Recall at the chosen approximation level should be measured and tuned against query latency.

Cold-start partial. Two-tower addresses new-item cold start when the item tower consumes features. It does not by itself address new-user cold start, because the user tower still needs user features (which a brand-new user lacks). Onboarding questionnaires or fallback strategies are still required for new users.

9. Pitfalls and Misconceptions

“Two-tower replaces the whole pipeline.” No. It is the retrieval stage. Production quality requires a downstream ranker on top, plus re-ranking for diversity and business rules. Anyone shipping a two-tower retriever as their entire recommender will have worse ranking quality than necessary.

Forgetting the sampling-bias correction. As emphasized in §3.3, the correction is essential. Without it, popular items get under-recommended in production — a real and verified issue that motivated the Yi 2019 paper. The fix is small (subtract log P(item) from each logit) but easy to miss.

Asymmetric towers without justification. It is tempting to make the user tower and item tower structurally different. Usually the right choice is to share most of the architecture and only differ in the input features. Unjustified asymmetry adds complexity without clear benefit.

Indexing stale embeddings. Item-tower output drifts as training progresses. Querying an old ANN index against a new user vector gives wrong results. Production systems must keep the index in sync — either by full rebuilds or by online updates.

Confusing two-tower with NCF. NCF concatenates user and item embeddings into a single MLP and produces a score directly. This is not indexable because there is no dot-product decomposition. Two-tower is indexable; NCF is not. The two are very different architectures with very different operational profiles.

Treating the dot product as a limitation per the NCF claim. Rendle et al. 2020 showed that the dot-product combiner is not a structural limitation — embedding capacity is what matters. Do not over-engineer the combiner; spend effort on the encoders and the features instead.

Not normalizing embeddings. Whether to L2-normalize the tower outputs (yielding cosine similarity rather than raw dot product) is a design choice. Normalization is often beneficial for stability of the softmax loss but adds an extra step. Whichever choice is made, do it consistently between training and inference; mismatch is a common bug.

Negative sampling without considering exposure bias. The model sees what the previous recommender showed users. If the previous recommender systematically excluded certain items, the new model will inherit that bias. Counterfactual evaluation methods or explicit exposure correction are required for robustness.

10. Open Questions

• How to inject limited cross-features into a two-tower model without breaking indexability? Active research area. Partial answers: low-rank interaction layers, multi-task heads, periodic refresh of personalized item embeddings. No clear winner yet.
• Optimal trade-off between recall and latency in ANN indexing. Application-specific; no universal rule.
• Continuous online training of two-tower models. Most current production systems retrain in batch (nightly or weekly). Online incremental updates exist but are operationally fragile.
• Whether transformer-based towers add over MLP-based towers. Mixed empirical results; depends on whether the input features are sequential.

11. See Also

• Recommender Systems — the umbrella where the multi-stage architecture is introduced
• Matrix Factorization — the conceptual ancestor; two-tower is structurally a deep feature-driven MF
• Neural Collaborative Filtering — sibling deep model; not indexable, used for ranking rather than retrieval
• Hybrid Recommender Systems — two-tower implements both feature combination and cascade hybrid patterns
• Cold Start Problem — feature-driven towers are the canonical fix for new-item cold start
• Explicit vs Implicit Feedback — two-tower training assumes implicit feedback with in-batch negatives
• Sequence-Aware Recommenders — transformer-based models that can replace the user tower for session-based scenarios
• Building a Simple Recommender — the staged code progression reaches two-tower as the final architectural step
• Recommender Systems MOC