Full search page →

Probabilistic PCA — a density model behind a subspace

Optional · paper guide 1 of 1 for ML 06A · Linear Factor Models and Autoencoders. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Can PCA be expressed as a probabilistic generative model rather than only a geometric projection? Tipping and Bishop show how a Gaussian latent-variable model recovers the principal subspace while providing a likelihood and a route for inference with missing observations [4]. Start with eigenvectors, covariance and Gaussian distributions. This paper supplies a bridge from ML 01A’s geometry to the latent-variable models in this chapter.

Method and figure

The example compares a complete-data projection with a projection inferred from incomplete observations. The numbered groups and outlier let readers judge retained structure.
Source: Tipping and Bishop, JRSS B 61(3), 1999; author manuscript dated September 27, 1999, hosted by Oxford. Figure 1, PDF p. 6. The example compares a complete-data projection with a projection inferred from incomplete observations. The numbered groups and outlier let readers judge retained structure. [4]

The model assumes

x=Wz+μ+ϵ,z∼N(0,I),ϵ∼N(0,σ2I),x∼N(μ,WW⊤+σ2I).x=Wz+\mu+\epsilon,\qquad z\sim\mathcal N(0,I),\quad\epsilon\sim\mathcal N(0,\sigma^2I),\quad x\sim\mathcal N(\mu,WW^\top+\sigma^2I).

Here xx is the observed vector, zz is a lower-dimensional latent vector, WW is a loading matrix, μ\mu is the data mean, ϵ\epsilon is independent isotropic noise, and σ2\sigma^2 is its variance. Each identity matrix has the dimension of the corresponding space. Integrating out zz gives the displayed observation distribution.

Maximum likelihood aligns the learned loading subspace with leading covariance eigenvectors. Latent coordinates are not uniquely identified: rotations can preserve the observation distribution. At finite noise, a posterior latent mean is also not simply the ordinary orthogonal PCA projection. Keep the subspace, loading parameters and inferred coordinates conceptually separate.

Evidence

The central evidence includes a likelihood derivation and an expectation-maximization procedure, not a new neural architecture benchmark. Figure 1 illustrates missing-data inference using 38 observations with 18 dimensions. Values are removed independently with probability 20%; the shown case has 136 missing values. The inferred projection retains recognizable groups and the highlighted outlier. This supports the illustrated use case, not universal robustness to arbitrary missingness.

Design analysis

A probability model makes conditional inference possible using the coordinates that were actually observed. It also exposes assumptions that ordinary projection notation can hide: linear structure, Gaussian latent variables and equal noise variance in every observation direction. The missing-data example tests one practical consequence of the model. It does not demonstrate that the observation mechanism itself is correctly modeled when values are missing for informative reasons.

Reading checkpoint

Explain why changing the orientation of latent axes need not change the modeled observation density. Then consider an observation with one missing coordinate: which observed entries should condition the prediction, and why is replacing the missing value with zero generally a different procedure?

Limits and next step

A single linear Gaussian component cannot represent every curved or multimodal distribution. Its likelihood is useful only to the extent that its assumptions fit the task. Next read the VAE guide in ML 07: compare the linear decoder and tractable posterior here with a nonlinear decoder and learned approximate posterior there.

Continue reading: VAE — learning through a tractable lower bound

References

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 13–14. Book
  2. K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Book
  3. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper
  4. M. E. Tipping and C. M. Bishop, “Probabilistic Principal Component Analysis,” Journal of the Royal Statistical Society: Series B, vol. 61, no. 3, 1999. Paper
  5. A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks, vol. 13, no. 4–5, 2000. Paper
  6. B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, vol. 381, 1996. Paper
  7. D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, 1999. Paper
  8. P. Baldi and K. Hornik, “Neural networks and principal component analysis: Learning from examples without local minima,” Neural Networks, vol. 2, no. 1, 1989. Paper
  9. G. E. Hinton and R. R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, 2006. Paper
  10. P. Vincent, H. Larochelle, Y. Bengio and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” ICML, 2008. Paper
  11. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P.-A. Manzagol, “Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion,” JMLR, vol. 11, 2010. Paper
  12. S. Rifai, P. Vincent, X. Muller, X. Glorot and Y. Bengio, “Contractive Auto-Encoders: Explicit Invariance During Feature Extraction,” ICML, 2011. Paper
  13. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
  14. D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, vol. 12, no. 4, 2019. Paper
  15. I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
  16. D. Bank, N. Koenigstein and R. Giryes, “Autoencoders,” survey chapter, arXiv:2003.05991, 2020. Survey
  17. F. Locatello et al., “Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations,” ICML, 2019. Paper
  18. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
  19. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
  20. H. Cunningham, A. Ewart, L. Riggs, R. Huben and L. Sharkey, “Sparse Autoencoders Find Highly Interpretable Features in Language Models,” arXiv:2309.08600, 2023. Paper
  21. L. Gao et al., “Scaling and evaluating sparse autoencoders,” arXiv:2406.04093, 2024. Paper