Full search page →

PCA — Jolliffe and Cadima’s Review

Optional · paper guide 1 of 1 for ML 01A · Linear Algebra for Learning. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

How can many correlated measurements be summarized without silently changing the meaning of the data? Jolliffe and Cadima review PCA as an adaptive linear representation: the axes come from the observed covariance structure, rather than being chosen in advance [11]. Their central distinction is between an optimal variance-preserving projection and a scientifically useful interpretation. Those can coincide, but one does not prove the other. Begin with this chapter’s centered matrices, eigenvectors and SVD.

Method and figure

Each numbered point is a fossil tooth projected onto two principal components. Axis signs are arbitrary; preserved variance does not establish species labels.
Source: Jolliffe and Cadima, Philosophical Transactions A 374, 2016; Europe PMC published copy. Figure 1, PDF p. 5. Each numbered point is a fossil tooth projected onto two principal components. Axis signs are arbitrary; preserved variance does not establish species labels. [11]

For a centered data matrix XX with nn rows and dd measured variables, define the sample covariance S=X⊤X/(n−1)S=X^\top X/(n-1). The first loading vector solves

a1=arg⁡max⁡∥a∥2=1a⊤Sa.a_1=\arg\max_{\lVert a\rVert_2=1}a^\top Sa.

Here aa is a direction in the dd-dimensional variable space. The maximizing vector a1a_1 is an eigenvector of SS for its largest eigenvalue, and Xa1Xa_1 contains the observations’ scores. Later loading vectors maximize the remaining variance subject to orthogonality. The loadings are directions; the scores are the coordinates of observations along those directions.

The SVD connects this variance problem to minimum squared reconstruction error for a fixed rank. That equivalence applies to the specified linear, centered geometry. It does not say that a low-variance direction is irrelevant to every downstream task. Figure 1 shows scores, not the loading coefficients; its numbered observations should not be mistaken for learned categories.

Evidence

The review illustrates PCA with nine measurements of 88 fossil teeth. The first two components account for 78.8% and 16.7% of variation. This makes the two-dimensional display a strong approximation in the paper’s variance criterion. The first component is interpreted mainly as overall size, while the second contrasts lengths with heights and widths. The proposed biological grouping is an interpretation of this example, not a supervised classification result or a guarantee for other collections.

Design analysis

The covariance-versus-correlation discussion is a substantive design choice. Standardizing variables changes the geometry by giving each variable unit variance; it is not a harmless display adjustment. The review also considers variants that trade variance optimality for interpretability or suitability to different data structures. Read these as different objectives and assumptions, rather than as a single universally superior replacement for ordinary PCA.

Reading checkpoint

Given one loading vector and the centered measurements, compute an observation’s score. Reverse the sign of that vector and explain why the displayed axis flips while reconstruction is unchanged. Then ask whether standardizing a variable with tiny but noisy variation would improve the analysis or amplify an unhelpful direction.

Limits and next step

PCA is linear and sensitive to scale and outliers. Uncorrelated scores need not be statistically independent, and large explained variance need not preserve class information or causal factors. This review supplies geometric reasoning and illustrative applications rather than a controlled benchmark contest. Continue to probabilistic PCA in ML 06A to see what changes when the projection becomes a likelihood model with explicit observation noise.

Continue reading: Probabilistic PCA — a density model behind a subspace

References

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 2 “Linear Algebra”. Book
  2. M. P. Deisenroth, A. A. Faisal and C. S. Ong, Mathematics for Machine Learning, Cambridge University Press, 2020. Book
  3. G. Strang, Introduction to Linear Algebra, 6th ed., Wellesley-Cambridge Press, 2023. Book site
  4. G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013. DOI
  5. L. N. Trefethen and D. Bau III, Numerical Linear Algebra, SIAM, 1997. DOI
  6. C. Eckart and G. Young, “The Approximation of One Matrix by Another of Lower Rank,” Psychometrika, vol. 1, pp. 211–218, 1936. DOI
  7. R. Penrose, “A Generalized Inverse for Matrices,” Mathematical Proceedings of the Cambridge Philosophical Society, vol. 51, pp. 406–413, 1955. DOI
  8. A. E. Hoerl and R. W. Kennard, “Ridge Regression: Biased Estimation for Nonorthogonal Problems,” Technometrics, vol. 12, pp. 55–67, 1970. DOI
  9. K. Pearson, “On Lines and Planes of Closest Fit to Systems of Points in Space,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 2, pp. 559–572, 1901. DOI
  10. H. Hotelling, “Analysis of a Complex of Statistical Variables into Principal Components,” Journal of Educational Psychology, vol. 24, pp. 417–441, 1933. DOI
  11. I. T. Jolliffe and J. Cadima, “Principal Component Analysis: A Review and Recent Developments,” Philosophical Transactions of the Royal Society A, vol. 374, 20150202, 2016. DOI
  12. N. Halko, P.-G. Martinsson and J. A. Tropp, “Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions,” SIAM Review, vol. 53, no. 2, pp. 217–288, 2011. DOI · arXiv
  13. P.-G. Martinsson and J. A. Tropp, “Randomized Numerical Linear Algebra: Foundations and Algorithms,” Acta Numerica, vol. 29, pp. 403–572, 2020. DOI · arXiv
  14. R. Murray, J. Demmel, M. W. Mahoney, N. B. Erichson, M. Melnichenko, O. A. Malik, L. Grigori, P. Luszczek, M. Dereziński, M. E. Lopes, T. Liang, H. Luo and J. Dongarra, “Randomized Numerical Linear Algebra: A Perspective on the Field With an Eye to Software,” LAPACK Working Note 299, 2023. arXiv
  15. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. International Conference on Learning Representations (ICLR), 2022. arXiv
  16. J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar and Y. Tian, “GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection,” ICML, 2024. arXiv
  17. T. Miyato, T. Kataoka, M. Koyama and Y. Yoshida, “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018. arXiv
  18. T. G. Kolda and B. W. Bader, “Tensor Decompositions and Applications,” SIAM Review, vol. 51, no. 3, pp. 455–500, 2009. DOI
  19. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book