vault

DSSM

6 min read

DSSM

DSSM (Deep Structured Semantic Model, sometimes also called Deep Semantic Similarity Model) is the dual-encoder neural network architecture introduced by Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck (Microsoft Research) in their 2013 CIKM paper “Learning Deep Structured Semantic Models for Web Search using Clickthrough Data”. The original application was web search ranking: given a query and a set of candidate documents, produce a relevance score for each document. The DSSM architecture has a query tower (a deep neural network encoding queries into a low-dimensional vector) and a document tower (encoding documents the same way), with the score being the cosine similarity of the two vectors. DSSM is the direct ancestor of every modern Two-Tower Retrieval Model and of much of the dense-retrieval literature in modern information retrieval (DPR, ColBERT, etc.). The architectural pattern — two separate encoders feeding into a similarity score — has been remarkably stable: the structure described in the 2013 paper is essentially the structure of YouTube’s candidate generator (2016), Pinterest’s PinSage (2018), and most modern recommendation retrievers.

1. The Original Web Search Application

In 2013, web search ranking was dominated by feature-engineered “learning to rank” models that used dozens or hundreds of hand-crafted query-document features (BM25 scores, link-graph features, click features). DSSM proposed a fundamentally different approach: learn end-to-end neural representations of queries and documents in a shared semantic space, with relevance computed as a vector similarity.

The training data was clickthrough logs: for each search session, the clicked document was treated as a positive (query, document) pair, and unclicked documents shown in the same session were treated as negatives. This is the same implicit-feedback paradigm used in modern recommendation systems.

The empirical contribution was substantial: DSSM substantially outperformed the prior keyword-based and topic-model-based methods on standard web search benchmarks. The architectural contribution — separate deep encoders feeding a similarity score — proved even more important. Within five years, the DSSM pattern had been ported to recommendation (YouTube 2016, Pinterest 2018) and to question answering (DPR 2020) and had become the standard architecture for any task involving “match items in a large collection to a query.”

2. The Architecture

flowchart LR
  subgraph Query Tower
    Q[Query text] --> QH[Word hashing<br/>(letter-trigrams)]
    QH --> QNN[Multi-layer<br/>neural network]
    QNN --> QE[Query embedding<br/>y_Q ∈ ℝ^k]
  end
  subgraph Document Tower
    D[Document text] --> DH[Word hashing]
    DH --> DNN[Multi-layer<br/>neural network]
    DNN --> DE[Document embedding<br/>y_D ∈ ℝ^k]
  end
  QE --> Sim["Score = cos(y_Q, y_D)"]
  DE --> Sim

What this diagram shows. The query tower (top) and document tower (bottom) are independent deep networks. Each consumes a text input, applies a word-hashing preprocessing step to handle the open vocabulary problem (see §3), passes through several fully-connected layers with non-linear activations, and outputs a k-dimensional embedding vector. The score for a (query, document) pair is the cosine similarity of the two embeddings. The key insight from this diagram is the structural decoupling — the two towers do not share parameters and never interact except via the final cosine similarity. This is precisely the property that makes the model indexable at scale: document embeddings can be precomputed offline for the entire corpus and looked up at query time via approximate nearest-neighbour search. The architectural pattern was novel in 2013 and has propagated into nearly every retrieval system since.

3. Word Hashing — Handling the Open Vocabulary

A practical challenge for the original DSSM: web search queries and documents have an effectively unbounded vocabulary (rare words, misspellings, named entities). A naive word-embedding lookup would fail on out-of-vocabulary terms.

DSSM’s solution was word hashing via letter-trigrams: each word is decomposed into the set of overlapping three-character substrings (e.g., “cat” → {#ca, cat, at#} where # marks word boundaries). The trigram vocabulary is much smaller than the word vocabulary (approximately 30,000 letter-trigrams cover most English text), and any new word can be represented as the multi-hot vector of its trigrams. This handles out-of-vocabulary words gracefully — a misspelling shares most trigrams with the correctly-spelled word.

Modern variants of two-tower models use byte-pair encoding (BPE), WordPiece, or sentence embeddings from pretrained language models instead of letter-trigrams. The word-hashing trick was a 2013 innovation that became less central as subword tokenization matured, but the general principle (handle the open vocabulary at the input layer) remains.

4. Training

DSSM is trained on (query, clicked-document, sampled-negative-documents) tuples. The loss is a softmax cross-entropy over the candidates:

L = − Σ_{(Q, D⁺) clicked}  log( exp(γ · cos(y_Q, y_{D⁺})) / Σ_{D ∈ {D⁺} ∪ Negatives} exp(γ · cos(y_Q, y_D)) )

where:

  • γ is a learned scaling factor (since cosine values are bounded in [−1, 1] but the softmax requires logits with reasonable magnitudes).
  • The summation in the denominator is over the clicked document plus a set of sampled negative documents from the session.

This is mathematically the same structure as modern in-batch softmax for two-tower training, just with explicit per-session negatives instead of in-batch negatives. The 2013 paper used about four random negatives per click; modern implementations use larger batches with in-batch negatives.

5. The Lineage to Modern Two-Tower Models

The DSSM-to-modern-two-tower lineage:

  • DSSM (2013) — original dual-encoder for web search, with letter-trigram hashing.
  • C-DSSM / DSM-CNN (2014) — DSSM with convolutional layers.
  • DSSM-LSTM (2015–2016) — DSSM with recurrent (LSTM) encoders.
  • YouTube candidate generation (Covington 2016) — first major adaptation of dual-encoder architecture to large-scale recommendation.
  • DPR — Dense Passage Retrieval (Karpukhin 2020) — dual-encoder for open-domain question answering, with BERT encoders.
  • Modern two-tower recommenders (2018–present) — sentence-transformer-based item towers, multi-modal (CLIP) item towers, in-batch sampled-softmax training.

The structural pattern has remained constant for over a decade; what has changed is the encoder architecture (MLPs → CNNs → LSTMs → transformers) and the training pipeline (per-session negatives → in-batch negatives → sampling-bias-corrected in-batch negatives). The 2013 DSSM paper is the genealogical root.

6. Strengths

  • Established the dual-encoder pattern that dominates retrieval today.
  • Indexable — once trained, document embeddings precomputable; nearest-neighbour search at query time.
  • Scales to large vocabularies via the letter-trigram hashing.
  • Simple architecture that has aged well; modern variants are largely scale-ups of the same idea.

7. Weaknesses

  • Letter-trigram hashing is now superseded by subword tokenization (BPE, WordPiece) and pretrained language model embeddings.
  • Two-tower decoupling means no cross-features. Same trade-off as modern two-tower models — see Two-Tower Retrieval Model §6.
  • MLP encoders are weaker than transformer encoders. A 2013 DSSM with MLP encoders is no longer competitive with a modern BERT-style dual encoder.

8. See Also

Graph View

Backlinks