vault

AutoRec

5 min read

AutoRec

AutoRec is an autoencoder-based collaborative filtering model introduced by Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie (Australian National University, NICTA) in their 2015 WWW paper “AutoRec: Autoencoders Meet Collaborative Filtering”. The model takes a partially observed rating vector — a row or column of the user-item matrix R — projects it into a low-dimensional latent space, and reconstructs the output to produce predicted ratings for the missing entries. AutoRec was historically important as the first widely-cited application of autoencoders to recommendation, demonstrating that compact non-linear models could outperform Matrix Factorization baselines like RBM-CF and biased MF on standard datasets. The architectural pattern — encode the observed interactions of a single user (or item), reconstruct the full vector — has since been extended into deeper variants (Mult-VAE, EASE) and remains a useful baseline in recommender benchmarks.

1. The Two AutoRec Variants

The paper distinguishes two flavours of AutoRec depending on which axis of the rating matrix is encoded.

User-based AutoRec (U-AutoRec). Each user is represented by their rating vector r^(u) ∈ ℝ^n (length n = number of items, with zeros for unrated items). The autoencoder takes r^(u) as input, encodes it, and reconstructs it. The model has parameters shared across all users — the same encoder and decoder are applied to every user’s vector.

Item-based AutoRec (I-AutoRec). Each item is represented by its rating vector r^(i) ∈ ℝ^m (length m = number of users, with zeros for users who have not rated this item). The autoencoder encodes and reconstructs r^(i). The parameters are shared across all items.

The paper found that I-AutoRec generally outperforms U-AutoRec on standard datasets like MovieLens and Netflix, because items typically have more ratings than users (catalog effect: a few popular items get many ratings; many users rate few items). More observations per input vector means the encoder gets more signal per training example.

2. The Architecture

For I-AutoRec, the model is:

h(r^(i); θ) = f( W · g(V · r^(i) + μ) + b )

where:

  • r^(i) ∈ ℝ^m is the item rating vector (column of R).
  • V ∈ ℝ^{k × m} is the encoder weight matrix; μ ∈ ℝ^k is the encoder bias.
  • g(·) is the encoder activation (the original paper found sigmoid works best for the encoder).
  • W ∈ ℝ^{m × k} is the decoder weight matrix; b ∈ ℝ^m is the decoder bias.
  • f(·) is the decoder activation (the paper found identity works best for the decoder, since ratings are real numbers).
  • h(r^(i); θ) is the reconstructed rating vector — predictions for every user, including users whose original ratings were observed.

The key parameter k is the latent (hidden) dimension. Typical values: 100–500 for MovieLens-scale data.

3. Training

The loss is the squared reconstruction error over observed entries only:

L(θ) = Σ_{i=1}^{n}  ‖ ( r^(i) − h(r^(i); θ) ) ⊙ I[r^(i)] ‖²  +  λ · ( ‖V‖_F² + ‖W‖_F² )

where:

  • I[r^(i)] is an indicator vector that is 1 at observed positions and 0 elsewhere.
  • ⊙ is the element-wise product (so the loss is computed only on positions where the original rating was observed).
  • λ is the L2 regularization coefficient on the weight matrices.

The summation is over all items i. Trained by stochastic gradient descent (or Adam, RMSprop, etc.).

The masked-loss trick — restricting the reconstruction loss to observed entries — is essential. Without it, the autoencoder would learn to predict zero (the missing-data fill-in) for unobserved entries, which is wrong for the same reason Funk SVD excludes missing entries from its loss.

4. Empirical Results

On MovieLens 1M and Netflix datasets, AutoRec (specifically I-AutoRec) outperformed:

  • Biased Matrix Factorization (BiasedMF).
  • Restricted Boltzmann Machines for Collaborative Filtering (RBM-CF).
  • LLORMA (Local Low-Rank Matrix Approximation).

at the time of publication (2015). The improvements were modest but consistent.

The architecture is conceptually appealing because it is so simple: a single hidden layer feed-forward autoencoder with sigmoid encoder and identity decoder. Despite the simplicity, the non-linearity in the encoder allowed AutoRec to capture user/item structure that the linear MF could not.

5. Strengths

  • Simple, single-layer architecture with strong empirical results in 2015.
  • Captures non-linear structure that pure MF cannot.
  • Per-row (or per-column) processing scales naturally — each item’s rating vector is processed independently at training and inference.
  • Easy to extend to deeper architectures (Mult-VAE, denoising autoencoders).

6. Weaknesses

  • No side features. Pure rating-matrix model.
  • Cold start unchanged — a new item’s vector is empty; encoder produces an all-zeros embedding; reconstruction is uninformative.
  • Outdated relative to Mult-VAE. The variational autoencoder formulation in Mult-VAE generally outperforms AutoRec on the same benchmarks.
  • Sigmoid encoder is somewhat arbitrary. The choice was empirical; modern variants use ReLU or other activations.

7. See Also

Graph View

Backlinks