vault

BERT4Rec

10 min read

BERT4Rec

BERT4Rec (Bidirectional Encoder Representations from Transformer for Sequential Recommendation) is a sequence-aware recommender introduced by Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang (Alibaba Group) in their 2019 CIKM paper “BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer” (ACM DOI). The model adapts the BERT architecture from natural language processing to sequential recommendation, with two key choices that distinguish it from SASRec: (1) bidirectional self-attention (each position attends to all positions, not just past ones); (2) masked-item-prediction training under the Cloze objective — randomly mask a fraction ρ of items in the sequence and predict the masked items from full surrounding context (the original paper tunes ρ per dataset to values 0.2, 0.4, and 0.6, not the 15% commonly inherited from BERT — see §3). For roughly four years (2019–2023) the literature treated BERT4Rec as state-of-the-art for sequential recommendation. Two subsequent replication studies refined that picture: Petrov & Macdonald (RecSys 2022) showed BERT4Rec does reach the originally reported numbers — but only when trained roughly an order of magnitude longer than the default configuration, and they introduced a Hugging Face-based implementation that converges in up to 95% less time; then Klenitskiy & Vasilev (RecSys 2023) — “Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?” — showed that the BERT4Rec-vs-SASRec gap is mostly a loss-function artifact: when SASRec is trained with BERT4Rec’s full-softmax / sampled cross-entropy loss instead of the original binary cross-entropy with one negative, SASRec matches or beats BERT4Rec at substantially lower training cost. As of mid-2026 BERT4Rec is treated as a strong baseline rather than the clear default; the consensus has shifted toward SASRec-with-cross-entropy for new deployments.

1. Why Bidirectional — The Argument and Its Limits

SASRec uses causal (left-to-right) self-attention because the next-item prediction task is structurally causal: at recommendation time, the model has only the user’s past history, not the future. Causal masking matches the deployment setting.

BERT4Rec’s argument was that training could benefit from bidirectional context even though inference is unidirectional. Specifically, by training with the BERT-style masked language modeling objective — randomly mask a fraction ρ of items in the sequence (the BERT4Rec paper tunes ρ to 0.2–0.6 per dataset; see §3) and predict them from full surrounding context — the model receives a richer learning signal per training example. Each masked item can attend to both past and future neighbors during training, which provides more information than the causal “predict the next item from past only” setup.

The Cloze task (so-called from psychology, where a participant fills in a missing word) had been shown to produce better representations than next-token prediction in language modeling (BERT versus GPT). BERT4Rec applied the same insight to recommendation.

The 2019 paper reported BERT4Rec outperforming SASRec on standard benchmarks (Amazon Beauty, Steam, MovieLens-1M and MovieLens-20M). The two replication studies revised this picture from different angles. Petrov & Macdonald (RecSys 2022) found that BERT4Rec only matches the originally reported numbers when trained for up to 30× the default training budget — under default settings the model is undertrained and the comparison is unfair to it (Petrov & Macdonald 2022). Klenitskiy & Vasilev (RecSys 2023) then showed that the cross-architecture gap is dominated not by bidirectionality but by loss function: SASRec’s original binary cross-entropy with one sampled negative is an information-poor objective, and once SASRec is trained with the same full-vocabulary (or sampled) cross-entropy that BERT4Rec uses implicitly, SASRec matches or beats BERT4Rec across all four benchmark datasets while converging substantially faster (Klenitskiy & Vasilev 2023; open-source code). The bidirectional-context advantage that the original paper marketed exists but is smaller than the loss-and-sampling advantage the original comparison conflated with it.

2. The Architecture

BERT4Rec is structurally an encoder-only transformer, equivalent to BERT applied to item sequences. The architecture:

flowchart TD
  Items[Item sequence with some MASK tokens] --> Emb[Item embedding lookup]
  Emb --> Pos[Add positional embedding]
  Pos --> Block1[Transformer block 1<br/>(bidirectional self-attention + FFN + residual)]
  Block1 --> BlockN[... transformer blocks ...]
  BlockN --> Hidden[Hidden states at all positions]
  Hidden --> Out[Project mask positions onto<br/>item embedding matrix]
  Out --> Probs[Probability over item vocabulary<br/>at each masked position]

What this diagram shows. The input is a sequence with some positions replaced by a special [MASK] token (during training; at inference, see §4). Items are embedded, positional embeddings are added, and the sequence is passed through N transformer encoder blocks. Unlike SASRec’s causal attention, BERT4Rec’s attention has no mask — every position attends to every other position. The hidden states at the masked positions are projected onto the item vocabulary (with weight tying to the input embedding) to produce a probability distribution over what each masked item should be. The key insight from this diagram is the bidirectional information flow: a masked item’s prediction uses context from both before and after the mask in the sequence. This gives the model more information per training example than the causal setup, at the cost of not being directly applicable to the unidirectional inference task.

3. Masked-Item Training — The Cloze Objective

For each training sequence, randomly mask a fraction ρ of items by replacing them with the [MASK] token. The model is trained to predict the masked items given the rest of the sequence. Writing S for the input sequence and S_mask for the same sequence with the masked positions replaced by [MASK], the per-sequence loss the paper minimizes is

L(S_mask) = (1 / |M|) · Σ_{m ∈ M}  − log P( i_m | S_mask )

where M is the set of masked positions in S_mask, i_m is the ground-truth item that was masked at position m, and P(i_m | S_mask) is computed via softmax over the full item vocabulary at the hidden state of position m. Symbol-by-symbol: |M| normalizes by the number of masks so longer/short masked spans have comparable scale; the negative log of the softmax is standard cross-entropy; the full-vocabulary normalization is what makes the objective information-rich (the model is implicitly contrasted against the entire catalog rather than against one sampled negative).

The masking-ratio convention is the place this note’s earlier draft (and many secondary write-ups) is most often wrong. The 15% figure comes from BERT’s NLP setup; BERT4Rec does not use 15%. The original paper sweeps ρ on a held-out validation set and reports best values of ρ = 0.6 for Amazon Beauty, ρ = 0.4 for Steam, and ρ = 0.2 for MovieLens-1M and MovieLens-20M (per the original paper, §4.4.5 hyperparameter study, also confirmed by the RecBole reference implementation). The systematic pattern — shorter, denser interaction sequences (MovieLens) prefer low ρ; longer, sparser sequences (Beauty) prefer high ρ — is the practical takeaway: ρ is a real hyperparameter that must be tuned per dataset, not a constant inherited from NLP.

4. The Inference Trick

The training task (predict masked items from bidirectional context) does not directly match the inference task (predict the next item given only past history, no future). BERT4Rec resolves this with a clever workaround: at inference, append a [MASK] token to the end of the user’s history and predict the masked item.

Training:  [i_1, i_2, MASK, i_4, MASK, i_6]  →  predict i_3, i_5
Inference: [i_1, i_2, i_3, i_4, i_5, MASK]   →  predict next item

This works because the bidirectional attention treats the position before the trailing [MASK] as past context (same as causal attention would), and there is no future context (the [MASK] is the last position). So the inference setup is effectively unidirectional, even though the training was bidirectional.

The trick is elegant but adds the asymmetry between training and inference distributions — the model has been exposed to “fill in the middle” tasks during training but is asked only to “fill in the end” at inference. Whether this distribution mismatch matters in practice is part of why the SASRec-vs-BERT4Rec comparison is subtle.

5. The Two Replication Studies

The 2019 BERT4Rec-vs-SASRec comparison did not survive contact with two careful replications. They are distinct papers, often confused in secondary write-ups, and worth keeping straight.

5.1 Petrov & Macdonald (RecSys 2022) — “BERT4Rec is fine, but only if you actually train it”

Petrov & Macdonald, “A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation”, University of Glasgow, RecSys 2022 (ACM full HTML). They systematically surveyed papers comparing BERT4Rec to SASRec and found the reported numbers were inconsistent — several follow-up papers used BERT4Rec’s default training configuration and reported it underperforming its own original numbers. Their finding is that BERT4Rec’s defaults (200 epochs in the original Tensorflow code) substantially under-train the model on most datasets; reproducing the original’s published numbers requires training for up to 30× longer. They released a Hugging Face Transformers-based implementation that hits the original published numbers on 3 of 4 datasets in 95% less wall-clock time than the original code, by using better optimizers and a more efficient implementation. Bottom line of the 2022 paper: BERT4Rec is genuinely state-of-the-art when trained long enough; many secondary comparisons against it were unfair to BERT4Rec.

5.2 Klenitskiy & Vasilev (RecSys 2023) — “It was the loss function the whole time”

Klenitskiy & Vasilev, “Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?”, Sber AI Lab, RecSys 2023 (open-source code). They focused on the loss asymmetry: BERT4Rec’s softmax-over-full-vocabulary at masked positions is informationally richer than SASRec’s original binary cross-entropy over one positive and one sampled negative. They re-train SASRec with the same full-softmax (and a sampled-softmax variant for scalability) and find SASRec matches or beats BERT4Rec across all four benchmark datasets, while training and serving substantially faster. The key finding: the cross-entropy-loss SASRec variant “significantly outperform[s] BERT4Rec both in terms of quality and training speed” (per the paper). They also note that sampled cross-entropy with a large number of negatives (≫1) retains most of the gains at lower compute. Bottom line of the 2023 paper: the architecture-vs-architecture comparison is dominated by the loss function; once you control for that, bidirectionality buys little.

5.3 Synthesis

Read together, the two papers say: BERT4Rec the model is fine and reproducible (Petrov & Macdonald 2022), but it does not actually beat SASRec once SASRec is trained with a comparable loss (Klenitskiy & Vasilev 2023). The bidirectional-attention advantage that the original paper marketed exists but is small relative to the loss-function advantage the original comparison conflated with it. As of mid-2026 the field has largely consolidated on this view, and new deployments tend to default to SASRec with cross-entropy loss. This is also the recommendation in Sequence-Aware Recommenders §6 and SASRec §4.

6. When to Use BERT4Rec

BERT4Rec is appropriate when:

  • You need a transformer-based sequence model and have specific reasons to prefer bidirectional context (e.g., empirical evidence on your data).
  • You are reproducing prior research that used BERT4Rec.
  • You want a strong baseline in a sequential-recommendation paper.

For new production deployments without specific reason to choose BERT4Rec, SASRec is generally the simpler and equally-strong choice.

7. Strengths

  • Bidirectional context during training — richer learning signal per example.
  • Strong empirical results — consistently competitive with state-of-the-art on standard benchmarks.
  • Mature implementations — RecBole, BERT4Rec original Tensorflow code, PyTorch ports.

8. Weaknesses

  • Inference trick is a kludge — the [MASK] at the end is awkward and asymmetric with training.
  • Outperformed by tuned SASRec when negative sampling is comparable.
  • More compute per training step — full-vocabulary softmax is expensive.
  • Cold-item start unchanged.

9. See Also

Graph View

Backlinks