DLRM

DLRM (Deep Learning Recommendation Model) is the recommendation architecture that Meta (formerly Facebook) developed and open-sourced in 2019, described in the paper “Deep Learning Recommendation Model for Personalization and Recommendation Systems” by Maxim Naumov et al. The model is the foundation of much of Meta’s production recommendation infrastructure (ads ranking, news feed ranking, marketplace) and is included as a benchmark in the MLPerf machine-learning performance suite. DLRM is structurally similar to the Wide & Deep and YouTube Deep Recommender architectures — a deep model with embedding tables for categorical features and dense MLPs for continuous features — but with two important engineering choices: (1) it uses factorization-machine-style pairwise interactions between embeddings (via element-wise dot products) as the primary cross-feature mechanism, removing the need for manually-specified cross-features; (2) it uses model parallelism on the embedding tables (which are huge — Meta’s tables are tens of terabytes) and data parallelism on the dense layers, allowing the model to train at the full scale of Meta’s data. The open-source release (github.com/facebookresearch/dlrm) made the architecture accessible and triggered a wave of subsequent industrial-scale ranker research.

1. Why Meta Built DLRM

Meta’s recommendation problem at scale is dominated by two structural facts:

Massive categorical feature spaces. A user’s history, the items in the catalog, the contextual features, and the metadata each have millions of categorical values. Each categorical value needs an embedding. The total embedding table size for Meta’s ad-ranking system is in the tens of terabytes — far too large for a single machine.

Heterogeneous feature types. Each prediction request involves dozens of features ranging from simple continuous values (predicted CTR from upstream models, recency) to high-cardinality categoricals (user ID, ad ID, country). The architecture has to handle both natively.

DLRM was designed around these constraints. The architecture treats sparse categoricals and dense continuous features through different sub-networks, then combines them through a structured interaction layer.

2. The Architecture

flowchart TD
  subgraph Sparse Features
    S1[Categorical 1] --> E1[Embedding 1]
    S2[Categorical 2] --> E2[Embedding 2]
    S3[Categorical N] --> E3[Embedding N]
  end
  subgraph Dense Features
    D[Dense continuous] --> BMLP[Bottom MLP]
  end
  E1 --> Inter
  E2 --> Inter
  E3 --> Inter
  BMLP --> Inter
  Inter[Pairwise Interaction<br/>(dot products)] --> TMLP[Top MLP]
  BMLP --> TMLP
  TMLP --> Out[Predicted score]

What this diagram shows. The architecture has two input branches: sparse categorical features go through embedding lookups (one embedding table per categorical feature), and dense continuous features go through a bottom MLP that projects them into the same embedding dimension. The embeddings (one per categorical, plus the bottom MLP output) are then passed through a pairwise interaction layer that computes the element-wise dot product between every pair, producing a vector of pairwise interaction values. This interaction vector is concatenated with the bottom MLP output and passed through a top MLP that produces the final score. The key insight from this diagram is that the model captures both first-order effects (through the dense bottom MLP) and second-order pairwise feature interactions (through the explicit dot products in the interaction layer), without requiring manual feature engineering. The structure is conceptually a generalization of Factorization Machines — every feature gets a latent embedding, pairwise interactions are dot products of embeddings — extended with deep MLP feature transformation on the dense input side.

3. The Pairwise Interaction Layer

The interaction layer is where DLRM’s distinctive design choice lives. Given embeddings e_1, e_2, ..., e_N (one per categorical feature) and the bottom-MLP output e_0 = BottomMLP(dense_features), the interaction layer computes:

interaction_output = [ e_iᵀ · e_j  for all pairs (i, j) with i ≤ j ]

This is a vector of N(N+1)/2 scalar values, one per (unordered) feature pair. This vector is then concatenated with e_0 and passed to the top MLP.

Some variants exclude self-interactions (i = j); some include the original embeddings concatenated alongside the interaction vector. The design space is small but matters at scale.

The pairwise-interaction-via-dot-products mechanism is structurally identical to the FM second-order term (see Factorization Machines §1), and DLRM is sometimes described as “FM with deep dense feature processing.” The terminology differs but the math is the same.

4. Model Parallelism on Embedding Tables

The embedding tables are the dominant memory consumer in DLRM. For Meta’s ad-ranking workload, total embedding table size can exceed 10 TB — far beyond what fits on a single GPU (or even a single host).

DLRM addresses this with model parallelism on the embedding tables: each embedding table is sharded across multiple devices, with each device storing a subset of the rows. At forward time, a device looks up its locally-held embeddings and exchanges them with other devices via all-to-all communication. The dense MLP layers, by contrast, are data-parallel (each device runs the MLP on its share of the batch).

The hybrid model-and-data parallelism is the key engineering innovation that lets DLRM scale to Meta’s data sizes. The implementation requires custom communication primitives and careful overlap of compute and communication; the DLRM repository provides a reference implementation.

5. Training

DLRM trains on standard Meta production data — typically billions of (impression, click) pairs per day. The loss is binary cross-entropy:

L = − Σ_{(x, y)}  [ y · log( σ(z(x)) )  +  (1 − y) · log( 1 − σ(z(x)) ) ]

where z(x) is the model’s output logit and σ is the sigmoid.

Training is typically online or near-online — the model is continuously updated on the most recent interaction logs. The optimizer is typically an adaptive method (Adam or AdaGrad). Since the embedding tables are sharded, optimizer state is also sharded.

6. The MLPerf Benchmark

DLRM was added to the MLPerf machine-learning training benchmark suite, where it serves as the canonical recommendation workload. The benchmark uses the Criteo Kaggle Display Advertising Challenge dataset (about 4 billion training examples) and measures time-to-train to a specific accuracy target. Inclusion in MLPerf made DLRM a de facto standard for benchmarking recommender training infrastructure.

7. When to Use DLRM

DLRM is appropriate when:

• You have a rich mix of dense continuous features and sparse categorical features.
• You need a production-grade ranker for CTR prediction or similar binary classification.
• You can leverage distributed training infrastructure (the embedding-table parallelism is the value proposition).

Do not use DLRM when:

• You only have user-item interaction data without rich features — simpler architectures suffice.
• You are doing retrieval rather than ranking — DLRM is too expensive per-pair to score the full catalog. Use Two-Tower Retrieval Model for retrieval and DLRM for ranking.
• You have small data — the architecture’s parallelism is unnecessary.

8. Strengths

• Production-proven at the largest scale in industry (Meta’s ad ranking).
• Hybrid model-and-data parallelism scales to terabyte embedding tables.
• No manual cross-feature engineering — the pairwise interaction layer captures all second-order interactions.
• Open-source — the reference implementation is available.
• Standard benchmark in MLPerf.

9. Weaknesses

• Engineering complexity. The model parallelism requires custom infrastructure; not appropriate for small-scale deployments.
• No native sequence handling. History-based features must be aggregated or encoded externally before being passed in.
• Second-order pairwise only. Higher-order interactions are captured only through the top MLP’s non-linearities. xDeepFM addresses this more directly.
• Tightly coupled to Meta’s infrastructure choices. The reference implementation assumes specific hardware and communication patterns.

10. See Also

• YouTube Deep Recommender — Google’s contemporaneous architecture for the same task class
• Wide & Deep — predecessor; DLRM removes the manual cross-feature engineering
• DeepFM — similar idea (FM + deep), different architectural decisions
• Deep & Cross Network — explicit feature crossing alternative
• Factorization Machines — DLRM’s pairwise interaction layer is structurally an FM
• Two-Tower Retrieval Model — complementary retrieval model upstream of DLRM
• BST — sequence-aware Alibaba ranker
• Recommender Algorithms Catalog
• Recommender Systems