vault

Sequence-Aware Recommenders

18 min read

Sequence-Aware Recommenders

Sequence-aware recommenders (also called sequential recommenders or session-based recommenders, depending on whether the input is the user’s full history or just a single session) are recommendation models that treat a user’s interactions as an ordered sequence rather than an unordered set, and that predict the next item conditional on the prefix of previous interactions. The defining contrast with Matrix Factorization and Item-Item Collaborative Filtering is that those treat history as a bag — the order of past interactions does not affect the prediction. Sequence-aware models exploit the temporal ordering: a user who watched Episode 1, then Episode 2, then Episode 3 is in a structurally different state from one who watched the same episodes in a different order, and the next-best recommendation differs accordingly. The dominant modern architectures are transformer-based: SASRec (Self-Attentive Sequential Recommendation, Kang & McAuley ICDM 2018) and BERT4Rec (Bidirectional Encoder Representations from Transformers for Sequential Recommendation, Sun et al. CIKM 2019), both of which adapt the transformer architectures from natural-language processing to sequential recommendation. The recent literature has revisited these methods with more careful tuning and now favours SASRec for many production scenarios.

1. Why Sequence-Aware Recommenders Exist

Three concrete situations make order matter and bag-of-items models inadequate.

Episodic content. A user who watched Breaking Bad Season 1 Episode 1, then Episode 2, then Episode 3 obviously wants Episode 4 next. A bag-of-items model cannot tell the difference between this user and one who watched Episodes 1, 2, and 3 in scrambled order, but the right next-item recommendation differs in subtle ways — completionist viewers want the next episode; pilot-watchers might be open to a different show. Sequential models capture this by conditioning the prediction on the ordered history.

Post-purchase recommendations. After a user buys a phone, the system should recommend cases, chargers, screen protectors — not more phones. A bag model sees the phone in the user’s history and recommends similar items (which are other phones) because that is what user-co-occurrence patterns suggest. A sequential model that knows the phone is the most recent purchase can shift to complementary items.

Session-based behaviour. Music sessions have moods that drift over the next 5 tracks but not the next 50. A user listening to ambient music right now wants the next ambient track even if their long-term profile says they mostly listen to metal. A model that consumes only the long-term profile cannot adapt; a model that conditions on the current session can.

For fully session-based scenarios — anonymous users with no long-term profile, single-visit interactions — sequence is the only signal. Pure CF and MF approaches collapse here because there is no user-ID-keyed embedding to look up. Sequence-aware models are not just better; they are the only viable approach.

2. The Algorithmic Lineage

Sequence-aware recommendation has a clear historical progression:

N-gram models (early 2000s). The simplest sequential model: precompute the most likely next item given each item or each pair of items. Akin to bigram/trigram language models. Cheap, fast, and surprisingly competitive on short-history tasks.

Markov-Chain Recommenders (Rendle, Freudenthaler, Schmidt-Thieme 2010, “Factorizing Personalized Markov Chains for Next-Basket Recommendation”). Combines matrix factorization with first-order Markov chains: the prediction depends on the user’s latent factors and the most recent item. This was the first influential attempt at personalized sequential recommendation.

Recurrent Neural Networks — GRU4Rec (Hidasi, Karatzoglou, Baltrunas, Tikk 2016, “Session-based Recommendations with Recurrent Neural Networks”). The first deep sequential recommender to gain wide attention. Uses a Gated Recurrent Unit (GRU) — a recurrent neural network architecture — to encode the session history into a hidden state, and predicts the next item from that hidden state. Trained with a rank-based loss (TOP1 or BPR-max).

Convolutional Sequential Models — Caser (Jiaxi Tang & Ke Wang, WSDM 2018, “Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding”). Treats the embedding matrix of the most recent items as an “image” and slides horizontal and vertical convolutional filters over it to capture point-level, union-level, and skip patterns, fused with a user embedding for personalization.

Self-Attentive Models — SASRec (Kang & McAuley 2018). First successful application of self-attention (the transformer’s core mechanism) to sequential recommendation. Causal (left-to-right) attention, decoder-only architecture analogous to GPT.

Bidirectional Transformer Models — BERT4Rec (Sun et al. 2019). Encoder-only architecture with masked-item prediction, analogous to BERT. Bidirectional context.

Generative and LLM-based recommenders (2023–present). Models that generate item-ID sequences (TIGER, Rajput et al., NeurIPS 2023) or prompt large language models with user history.

The progression mirrors the natural-language processing field’s progression from n-grams to RNNs to transformers — and indeed, almost every advance in NLP architecture has been ported to sequential recommendation within a year or two of its publication.

3. The Core Mechanism

The shared structural pattern across all sequence-aware methods: treat each interaction as a token in a sequence, train the model to predict the next token given the prefix, deploy by feeding the user’s recent history and asking for the most-likely next item.

flowchart LR
  H[Item history sequence:<br/>i₁, i₂, i₃, ..., iₜ] --> Emb[Item embedding<br/>lookup per position]
  Emb --> Pos[Add positional encoding]
  Pos --> Layers[Stacked self-attention<br/>or recurrent layers]
  Layers --> Hidden[Final hidden state<br/>at position t]
  Hidden --> Proj[Project to vocabulary<br/>(item space)]
  Proj --> Probs[Probability distribution<br/>over next item]
  Probs --> Top[Top-K items as<br/>next-item candidates]

What this diagram shows. The pipeline runs left to right. Input is the user’s recent interaction history as an ordered sequence of item IDs [i₁, i₂, ..., i_t]. Each ID is looked up in an item embedding table to produce a k-dimensional vector per position; positional encodings are added to give the model the position information that pure attention layers lack. The encoded sequence is passed through several layers of self-attention (in transformer-based models) or recurrent computation (in RNN-based models). The final hidden state at the last position is projected back to the item vocabulary (a linear layer with output dimension equal to the catalog size), producing a probability distribution over what item should come next. The top-K items by probability are returned as recommendations. The key insight from this diagram is the structural identity to language modelling: items take the place of word tokens, and the entire architecture is the same as a language model. This identity is why every NLP architectural advance ports directly — items and words are both discrete-vocabulary sequence-token problems, and the model does not care about the semantic content of the tokens.

3.1 Positional encoding

Pure self-attention is permutation-invariant — it produces the same output regardless of the order of its inputs. To make the model order-sensitive, the input embeddings are augmented with positional encodings that encode each token’s position in the sequence. Two common approaches:

Sinusoidal positional encoding (Vaswani et al. 2017, the original transformer paper):

PE(pos, 2k)   = sin(pos / 10000^{2k/d})
PE(pos, 2k+1) = cos(pos / 10000^{2k/d})

where pos is the position index, 2k and 2k+1 are dimension indices in the encoding, and d is the model dimension. The encoding is added to the input embedding before the first attention layer.

Learned positional embedding. A separate embedding table indexed by position (typically up to a maximum sequence length, e.g., 512). At training, each position pos looks up its position embedding and adds it to the item embedding.

For sequence-aware recommendation, learned positional embeddings are more common because the maximum useful sequence length is bounded (usually 50 to 200 items) and the learned embeddings can specialize to the recommendation distribution.

4. SASRec — Self-Attentive Sequential Recommendation

The 2018 paper by Wang-Cheng Kang and Julian McAuley, “Self-Attentive Sequential Recommendation”, introduced the first widely-adopted self-attention-based sequential recommender. Architecture details:

Causal (left-to-right) attention. Each position attends only to past positions, not future ones. The attention mask is the standard lower-triangular mask used in language-model decoders. This ensures the model cannot cheat by looking at future items when predicting the next.

Decoder-only architecture. Like GPT, SASRec is a stack of transformer decoder blocks (without cross-attention). Each block contains: a self-attention sublayer with causal masking, a position-wise feed-forward network, layer normalization, and residual connections.

Training objective: next-item prediction with shifted-sequence labels. For an input sequence [i₁, i₂, ..., i_{t-1}], the targets are the shifted sequence [i₂, i₃, ..., i_t]. At each position, the model predicts the item that came next. This is identical to language model training.

Loss function. The original paper used binary cross-entropy with one negative sample per positive. The model output at position j is paired with positive i_{j+1} and one randomly-sampled negative; the loss penalizes score(positive) − score(negative) being negative. This single-negative training is one of the points that the recent literature has criticized — see §6.

Inference. At recommendation time, feed the user’s recent history into the model and take the hidden state at the last position. Project this state onto the item embedding matrix (same matrix used as input embeddings — weight sharing) and take the top-K items.

The 2018 paper reported that SASRec was approximately ten times faster than RNN-based baselines (because attention parallelizes over the sequence length while recurrence does not) and achieved state-of-the-art results on multiple benchmark datasets.

5. BERT4Rec — Bidirectional Self-Attention with Masked-Item Prediction

The 2019 paper by Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang, “BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer”, adapted the BERT architecture to sequential recommendation.

Bidirectional self-attention. Each position attends to all positions, both past and future. There is no causal mask. This is the BERT-style encoder pattern, structurally different from SASRec’s GPT-style decoder pattern.

Training objective: Cloze task / Masked Item Prediction. For each training example, randomly mask approximately 15% of items in the sequence with a special [MASK] token. The model is trained to predict the masked items from the surrounding (bidirectional) context. This is exactly BERT’s masked language modeling objective.

Why bidirectional helps during training. A real next-item prediction is structurally unidirectional (you cannot use future items to predict the next item). But during training, exposing the model to both past and future context creates a richer learning signal — each masked item can be predicted from both prefix and suffix context. The argument is that this richer signal yields better learned representations, even though the inference task remains unidirectional.

Inference trick. At recommendation time, the user’s history is appended with a [MASK] token at the end. The model’s prediction at the mask position is the next-item recommendation. This is a clever workaround — the bidirectional model is used in a unidirectional way at inference by always having the mask at the right end of the input.

flowchart LR
  subgraph SASRec — Causal (Decoder-Only)
    A1[i₁] --> A2[i₂]
    A2 --> A3[i₃]
    A3 --> A4["i₄ predicted from<br/>attending to i₁, i₂, i₃"]
  end
  subgraph BERT4Rec — Bidirectional + Masked
    B1[i₁] -.-> Bm
    B3[i₃] -.-> Bm
    B4[i₄] -.-> Bm
    Bm["[MASK] at position 2"]
    Bm --> Pred["predict i₂ using<br/>bidirectional context"]
  end

What this diagram shows. Side-by-side comparison of the two architectures’ attention patterns. In SASRec (left), attention flows strictly left-to-right: i₄’s prediction can attend to i₁, i₂, and i₃ but not to itself or any future positions. The dependency is causal. In BERT4Rec (right), a position is masked (e.g., position 2, with i₂ replaced by [MASK]), and the model is asked to predict the masked item using bidirectional context — it attends to all unmasked positions both before and after the mask. The dotted lines indicate bidirectional attention. The key insight from this diagram is that the two architectures are solving structurally different training problems despite using the same model components (transformer blocks). SASRec models P(next | past) directly; BERT4Rec models P(masked | context). The choice of objective drives the choice of attention pattern.

6. The 2023 Reckoning — Has BERT4Rec Actually Beaten SASRec?

For approximately four years (2019–2023), the published consensus was that BERT4Rec outperformed SASRec on standard benchmarks. The conclusion was widely cited and influenced architectural choices in production.

Two distinct replication efforts pulled this consensus apart, and it is worth keeping them separate because they reach the same destination by different routes.

Petrov & Macdonald, RecSys 2022 — the replicability study. Aleksandr Petrov and Craig Macdonald published “A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation” (RecSys 2022, arXiv:2207.07483). They found that BERT4Rec results are not consistent across publications and that the original implementation’s reported numbers were hard to reproduce: a properly converged BERT4Rec needs far longer training than the reference code used (their replication required on the order of ~30× the original training time to reach the paper’s claimed effectiveness), and only then does it actually match its published quality. The headline is a caution about reproducibility, not a verdict that one architecture is intrinsically superior.

Klenitskiy & Vasilev, RecSys 2023 — the loss-function critique. The paper “Turning Dross Into Gold Loss: Is BERT4Rec Really Better than SASRec?” (arXiv:2309.07602) was written by Anton Klenitskiy and Alexey Vasilev — not, as a prior version of this note incorrectly stated, by Petrov & Macdonald. Their argument isolates the loss function as the real confound:

The original SASRec was trained with one negative sample per positive. It computes a binary cross-entropy over one positive and one sampled negative item. BERT4Rec, in contrast, effectively has many implicit negatives because its masked-item-prediction objective applies a full softmax (cross-entropy) over the entire item vocabulary at each masked position. The two were trained under structurally different negative regimes, which biased every head-to-head comparison.

When SASRec is trained with comparable negatives. Klenitskiy & Vasilev retrained SASRec with cross-entropy over the full item set (and with sampled-softmax using many negatives per positive). Under this fairer comparison, SASRec significantly outperformed BERT4Rec in both quality and training speed — the paper’s blunt conclusion is that when both models use the same loss, “SASRec will significantly outperform BERT4Rec.” The architectural difference (causal vs. bidirectional) mattered far less than the loss; “the number of negative examples should be much larger than one” is the operative fix.

The bidirectional context advantage is smaller than reported. The richer training signal does help, but not by enough to overcome the simpler causal architecture’s lower compute requirements and faster inference once SASRec’s loss is fixed.

Taken together: the current practitioner consensus (as of mid-2020s) leans toward causal/SASRec-style models for sequential recommendation — simpler inference, lower training cost, and competitive-or-better accuracy when trained with a proper full/sampled cross-entropy loss. BERT4Rec remains widely cited and is still deployed, and the field has not unanimously consolidated, so the right move when choosing between the two today is to benchmark both with matched negative-sampling regimes rather than trust the pre-2022 comparisons.

7. Beyond SASRec/BERT4Rec — Current Frontier

The sequential-recommendation space has continued to evolve. A short tour of what is current as of mid-2026:

Transformers4Rec (NVIDIA Merlin team, 2021), documentation. A library applying Hugging Face transformer architectures (XLNet, T5, GPT, RoBERTa) to sequential recommendation. Provides industrial-scale infrastructure and pretrained components. Used in some production systems and as a research framework.

Generative recommenders. Predict full token sequences for item IDs rather than scoring a fixed vocabulary. TIGER (Transformer Index for GEnerative Recommenders; Rajput, Mehta, Singh, Keshavan et al., “Recommender Systems with Generative Retrieval”, NeurIPS 2023, arXiv:2305.05065) is the canonical example — items are encoded as sequences of Semantic IDs (hierarchical subtoken sequences derived from item content via a learned quantizer), and the model generates the next item’s Semantic ID with a sequence-to-sequence transformer. The advantage: the model can in principle generate new items not seen in training (by composing novel token sequences), addressing a long-standing limitation of vocabulary-fixed models.

Large Language Model (LLM)-based recommenders. Prompt a large language model with the user’s interaction history (often as a natural-language description) and ask for the next-item recommendation. Promising for cold-start (the LLM has world knowledge to fall back on) and for explanation (the model can articulate its reasoning), but expensive at scale and prone to hallucination (recommending items not in the catalog). An active research frontier; production deployment is rare as of mid-2026 outside specific niches (conversational recommenders, B2B systems with low query volume).

Mixed sequential-and-content models. Recognizing that pure ID-based sequential models cannot handle cold items, many production systems combine sequence-aware modelling with content features. The user-tower of a Two-Tower Retrieval Model is sometimes a transformer over the user’s interaction sequence; the item-tower combines the item ID embedding with content features. This hybrid handles both temporal context and cold items.

8. Strengths

Captures temporal dynamics. Order, recency, sessional context — none of which bag-of-items models can use. For domains where order matters, sequence-aware models can be substantially more accurate.

Handles anonymous users. Session-based scenarios with no user ID at all are tractable because the model conditions on the session sequence rather than on user-keyed embeddings. This is essential for platforms with high anonymous traffic (news sites, e-commerce browsing without login).

Reuses NLP machinery. Every architectural advance in NLP — better attention variants, sparse attention for long contexts, fine-tuning techniques, distillation — transfers directly. Sequential recommendation rides the NLP research wave.

Strong on next-item prediction. This is the production objective for “Up Next” surfaces and continuous play. Sequential models match the task.

Naturally captures session intent shifts. A user whose session shifts from work-related searches to entertainment-related searches has clearly changed intent; a sequential model picks up the shift in real-time without needing to retrain.

9. Weaknesses

Compute heavy. Transformer self-attention is O(L²) in sequence length L. For long histories (hundreds of items), this becomes expensive. Production systems either truncate to a maximum window (last 50 items, last 200 items) or use efficient attention variants (sparse attention, sliding-window attention).

Cold-item problem unchanged at the architectural level. Items are encoded as vocabulary entries; new items have no embedding unless features are folded in. A pure sequential model with ID-only embeddings cannot recommend new items. The fix is to use feature-based item embeddings (combining ID and content features), the same fix as in two-tower models.

Less interpretable. Attention weights are sometimes pointed to as explanations (“the model attended heavily to item X when predicting item Y”), but the literature on attention-as-explanation is contested — attention weights do not reliably correspond to causal importance. Sequential recommenders are deep models and inherit the general opacity.

Distribution shift sensitivity. A sequential model trained on past behavior can lock in stale preferences. A user whose taste has shifted in the last week may continue to receive recommendations based on the old taste until the model is retrained on more recent data. Production systems retrain frequently to mitigate.

Training data is itself biased by the previous recommender. The interactions the model trains on are a function of what previous recommenders showed users. A sequential model trained only on what was previously surfaced will inherit that bias. Counterfactual debiasing is an active research area but not yet routine.

Long-history versus short-history trade-off. The model has to choose how much history to consume. Short windows lose long-term taste signal; long windows are computationally expensive and may be dominated by stale interactions. The optimal window length is domain-specific and requires tuning.

10. Pitfalls and Misconceptions

“BERT4Rec is always better than SASRec.” As discussed in §6, this conclusion does not survive careful negative-sampling-controlled comparison. Tune both before picking.

Treating session-based as long-term personalization. A 5-item session and a year of history are very different distributions. Models trained on one cannot be assumed to perform well on the other without retraining.

Forgetting positional encoding. Without it, attention is permutation-invariant — you have just rebuilt a fancier matrix factorization without the order information. Position is what makes the model sequential.

Window length too short. For some domains (TV series binge-watching, podcast subscriptions, long-running purchases), the relevant context spans hundreds of interactions over months. Truncating to 10 items destroys the signal.

Training only on positives. Sequential models trained without negative sampling collapse to predicting whatever item is most popular. Ensure negatives are sampled (sampled softmax, in-batch negatives, or BPR-style pairwise sampling).

Confusing self-attention with sequence understanding. Self-attention is a learned aggregation; it does not “understand” sequence semantics. The model still has to learn the patterns that matter from data.

Using sequential models for tasks where order does not matter. If the recommendation surface is “what should I buy this week” with no temporal coherence, a sequential model is overkill. Bag-of-items methods are simpler and may be just as good.

11. Open Questions

  • Are LLM-based recommenders production-viable, or just demoware? Current bottlenecks: latency (LLM inference is expensive at scale), cost (LLM API/serving cost is orders of magnitude higher than dedicated recommenders), hallucination (recommending items not in the catalog), and the lack of any clear evaluation methodology that captures the LLM’s strengths over specialized recommenders.
  • How to combine sequential signal with long-term taste in one model without one dominating. The “session-shift versus long-term-profile” trade-off is real and not cleanly resolved. Hybrid models exist but the design choices are still empirical.
  • What is the right sequence length for a given domain? No theory; tune empirically.
  • Causal sequence-aware recommendation. Most current methods are correlational. The causal effect of showing an item conditioned on history is harder to estimate; off-policy methods exist but are noisy.

12. See Also

Graph View

Backlinks