vault

SASRec

7 min read

SASRec

SASRec (Self-Attentive Sequential Recommendation) is a sequence-aware recommender introduced by Wang-Cheng Kang and Julian McAuley (UC San Diego) in their 2018 ICDM paper “Self-Attentive Sequential Recommendation”. It was the first widely-adopted application of self-attention — the core mechanism of the transformer architecture — to recommendation. The model uses causal (left-to-right) self-attention, structurally analogous to the GPT-style decoder used in language modeling: each position in the sequence attends only to past positions, predicting the next item conditional on the prefix. SASRec demonstrated that transformer-style attention substantially outperformed recurrent and convolutional sequential recommenders (GRU4Rec, Caser) and could be trained ten times faster on GPUs because of attention’s parallelizability over the sequence dimension. For several years (2018–2023), the field treated BERT4Rec as superior to SASRec; the 2023 Petrov & Macdonald paper “Turning Dross Into Gold Loss: Is BERT4Rec Really Better than SASRec?” showed the comparison was unfair and that SASRec, properly tuned with comparable negative sampling, matches or beats BERT4Rec at substantially lower compute. SASRec is now widely regarded as the default transformer-based sequential recommender.

1. Why Self-Attention for Sequential Recommendation

The transformer architecture had been introduced one year earlier (Vaswani et al. 2017, “Attention Is All You Need”) and was rapidly displacing recurrent networks in language modeling and machine translation. Three properties made attention attractive for sequential recommendation:

Parallelizable over time. Recurrent networks like GRU4Rec process one time step at a time, which prevents parallelization. Attention computes all positions’ representations simultaneously via matrix operations, so a sequence of length L is processed in O(L²) work but in a small constant number of GPU-parallel steps. The empirical result: SASRec trains roughly 10× faster than GRU4Rec on GPUs.

Direct attention to relevant past items. A recurrent network must compress all past items into a fixed-size hidden state, with information from older items decaying through the recurrent updates. Attention directly attends to every past item with learned attention weights, allowing the model to give arbitrary importance to specific past positions regardless of distance.

Decoupled position and content. Through positional encodings, attention can learn position-dependent patterns separately from content-dependent patterns. This was the same architectural choice that proved successful in language modeling.

The conceptual leap was small: language modeling and next-item recommendation are structurally identical (both predict the next token in a sequence over a fixed vocabulary). Once attention worked for language modeling, applying it to sequential recommendation was a natural step.

2. The Architecture

SASRec is structurally a decoder-only transformer — equivalent to GPT applied to item sequences. The architecture for predicting the next item given a sequence i_1, i_2, ..., i_t:

flowchart TD
  Items[Sequence i_1, ..., i_t] --> Emb[Item embedding lookup]
  Emb --> Pos[Add positional embedding]
  Pos --> Block1[Transformer block 1<br/>(causal self-attention + FFN + residual)]
  Block1 --> BlockN[... transformer blocks ...]
  BlockN --> Hidden[Final hidden state at position t]
  Hidden --> Out[Inner product with item embeddings<br/>(weight tying)]
  Out --> Probs[Probability distribution over next item]

What this diagram shows. Each item in the input sequence is embedded; positional embeddings are added (so the model knows the order); the resulting sequence of vectors is passed through N transformer blocks (each containing causal self-attention, a feed-forward layer, residual connections, and layer normalization). The hidden state at the last sequence position is then projected onto the item embedding matrix (using weight tying — the same embedding matrix is used for input lookup and output projection) to produce logits over the entire item vocabulary. Softmax converts to probabilities; the top-K items are returned as recommendations. The key insight from this diagram is that the architecture is identical to GPT-style language models — items take the role of word tokens, and everything else (attention, position, FFN) is unchanged. This direct port allows the entire NLP toolkit (training tricks, hyperparameter heuristics, efficient implementations) to transfer directly.

3. Causal Self-Attention

The defining property of SASRec is causal masking: each position attends only to past positions, never to future. The attention computation for position t:

Q = X · W^Q    K = X · W^K    V = X · W^V
attention_logits[t, j] = Q[t] · K[j]ᵀ / √d   for j ≤ t,  −∞ for j > t
attention_weights[t] = softmax(attention_logits[t])
output[t] = Σ_{j ≤ t} attention_weights[t, j] · V[j]

where X ∈ ℝ^{L × d} is the input sequence of embeddings, W^Q, W^K, W^V ∈ ℝ^{d × d} are learned weight matrices for queries, keys, and values, and d is the model dimension. The −∞ masking on future positions ensures the softmax assigns them zero weight, enforcing causality.

This is the standard transformer decoder self-attention. In multi-head form, the input is split across h heads (each d/h-dimensional) and the heads’ outputs are concatenated.

4. Training Objective

SASRec uses the shifted-sequence training paradigm: for input sequence [i_1, i_2, ..., i_{t-1}], the targets are the corresponding shifted sequence [i_2, i_3, ..., i_t]. At each position j, the model predicts the item that came next, given the items up to position j.

The original 2018 paper used a binary cross-entropy loss with one negative sample per positive (uniformly sampled from items the user has not interacted with):

L = − Σ_{(u, t)}  [ log σ(ŷ_{u, t, i_{t+1}})  +  log( 1 − σ(ŷ_{u, t, j}) ) ]

where j is the sampled negative.

The 2023 Petrov & Macdonald paper showed that this single-negative training was the source of the apparent SASRec inferiority to BERT4Rec. Switching SASRec to a sampled softmax loss with many negatives per positive (matching what BERT4Rec effectively gets through its masked-LM objective) made SASRec match or beat BERT4Rec.

5. Inference

At recommendation time, the user’s interaction history is fed into the model; the hidden state at the last position is taken; this hidden state is projected onto the item embedding matrix to produce a score per item; the top-K items are returned. The whole inference pipeline is a single forward pass through the model — fast on modern GPUs.

The history can be truncated to a maximum length (e.g., last 50 items) to bound the per-request compute cost. SASRec’s O(L²) attention complexity makes long sequences expensive; truncation or efficient attention variants (sparse, sliding-window) are options.

6. When to Use SASRec

SASRec is the modern default for sequential recommendation. Use it when:

  • You have sequential interaction data and want to predict the next item.
  • The catalog is medium-to-large (up to about 10⁷ items; for larger catalogs use a two-stage architecture with two-tower retrieval feeding SASRec).
  • You can afford GPU training.

Do not use SASRec when:

7. Strengths

  • State-of-the-art accuracy for sequential recommendation when properly tuned.
  • Parallelizable training — much faster than recurrent alternatives on GPUs.
  • Direct attention to relevant past items — captures long-range dependencies.
  • Same architecture as GPT — all NLP tooling and tricks transfer.
  • Simpler inference than BERT4Rec — no masking trick needed; final position’s hidden state is the prediction.

8. Weaknesses

  • O(L²) attention complexity — long sequences are expensive; use efficient attention variants for very long histories.
  • Single-negative training underperforms — use sampled softmax for production.
  • Cold-item start unchanged — vocabulary-based; new items have no embedding.
  • Less interpretable than memory-based methods.

9. See Also

Graph View

Backlinks