PCA — Jolliffe and Cadima’s Review
Optional · paper guide 1 of 1 for ML 01A · Linear Algebra for Learning. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
How can many correlated measurements be summarized without silently changing the meaning of the data? Jolliffe and Cadima review PCA as an adaptive linear representation: the axes come from the observed covariance structure, rather than being chosen in advance [11]. Their central distinction is between an optimal variance-preserving projection and a scientifically useful interpretation. Those can coincide, but one does not prove the other. Begin with this chapter’s centered matrices, eigenvectors and SVD.
Method and figure

For a centered data matrix with rows and measured variables, define the sample covariance . The first loading vector solves
Here is a direction in the -dimensional variable space. The maximizing vector is an eigenvector of for its largest eigenvalue, and contains the observations’ scores. Later loading vectors maximize the remaining variance subject to orthogonality. The loadings are directions; the scores are the coordinates of observations along those directions.
The SVD connects this variance problem to minimum squared reconstruction error for a fixed rank. That equivalence applies to the specified linear, centered geometry. It does not say that a low-variance direction is irrelevant to every downstream task. Figure 1 shows scores, not the loading coefficients; its numbered observations should not be mistaken for learned categories.
Evidence
The review illustrates PCA with nine measurements of 88 fossil teeth. The first two components account for 78.8% and 16.7% of variation. This makes the two-dimensional display a strong approximation in the paper’s variance criterion. The first component is interpreted mainly as overall size, while the second contrasts lengths with heights and widths. The proposed biological grouping is an interpretation of this example, not a supervised classification result or a guarantee for other collections.
Design analysis
The covariance-versus-correlation discussion is a substantive design choice. Standardizing variables changes the geometry by giving each variable unit variance; it is not a harmless display adjustment. The review also considers variants that trade variance optimality for interpretability or suitability to different data structures. Read these as different objectives and assumptions, rather than as a single universally superior replacement for ordinary PCA.
Reading checkpoint
Given one loading vector and the centered measurements, compute an observation’s score. Reverse the sign of that vector and explain why the displayed axis flips while reconstruction is unchanged. Then ask whether standardizing a variable with tiny but noisy variation would improve the analysis or amplify an unhelpful direction.
Limits and next step
PCA is linear and sensitive to scale and outliers. Uncorrelated scores need not be statistically independent, and large explained variance need not preserve class information or causal factors. This review supplies geometric reasoning and illustrative applications rather than a controlled benchmark contest. Continue to probabilistic PCA in ML 06A to see what changes when the projection becomes a likelihood model with explicit observation noise.
Continue reading: Probabilistic PCA — a density model behind a subspace
References
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 2 “Linear Algebra”. Book
- M. P. Deisenroth, A. A. Faisal and C. S. Ong, Mathematics for Machine Learning, Cambridge University Press, 2020. Book
- G. Strang, Introduction to Linear Algebra, 6th ed., Wellesley-Cambridge Press, 2023. Book site
- G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013. DOI
- L. N. Trefethen and D. Bau III, Numerical Linear Algebra, SIAM, 1997. DOI
- C. Eckart and G. Young, “The Approximation of One Matrix by Another of Lower Rank,” Psychometrika, vol. 1, pp. 211–218, 1936. DOI
- R. Penrose, “A Generalized Inverse for Matrices,” Mathematical Proceedings of the Cambridge Philosophical Society, vol. 51, pp. 406–413, 1955. DOI
- A. E. Hoerl and R. W. Kennard, “Ridge Regression: Biased Estimation for Nonorthogonal Problems,” Technometrics, vol. 12, pp. 55–67, 1970. DOI
- K. Pearson, “On Lines and Planes of Closest Fit to Systems of Points in Space,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 2, pp. 559–572, 1901. DOI
- H. Hotelling, “Analysis of a Complex of Statistical Variables into Principal Components,” Journal of Educational Psychology, vol. 24, pp. 417–441, 1933. DOI
- I. T. Jolliffe and J. Cadima, “Principal Component Analysis: A Review and Recent Developments,” Philosophical Transactions of the Royal Society A, vol. 374, 20150202, 2016. DOI
- N. Halko, P.-G. Martinsson and J. A. Tropp, “Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions,” SIAM Review, vol. 53, no. 2, pp. 217–288, 2011. DOI · arXiv
- P.-G. Martinsson and J. A. Tropp, “Randomized Numerical Linear Algebra: Foundations and Algorithms,” Acta Numerica, vol. 29, pp. 403–572, 2020. DOI · arXiv
- R. Murray, J. Demmel, M. W. Mahoney, N. B. Erichson, M. Melnichenko, O. A. Malik, L. Grigori, P. Luszczek, M. Dereziński, M. E. Lopes, T. Liang, H. Luo and J. Dongarra, “Randomized Numerical Linear Algebra: A Perspective on the Field With an Eye to Software,” LAPACK Working Note 299, 2023. arXiv
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. International Conference on Learning Representations (ICLR), 2022. arXiv
- J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar and Y. Tian, “GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection,” ICML, 2024. arXiv
- T. Miyato, T. Kataoka, M. Koyama and Y. Yoshida, “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018. arXiv
- T. G. Kolda and B. W. Bader, “Tensor Decompositions and Applications,” SIAM Review, vol. 51, no. 3, pp. 455–500, 2009. DOI
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book