Probabilistic PCA:子空間背後的機率模型
選讀・ML 06A・線性因子模型與自編碼器 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
PCA 能否不只是一種幾何投影,也是一個機率生成模型?Tipping 與 Bishop 顯示,高斯潛在變數模型可找回主子空間,同時提供概似及缺失觀測的推論方法 [4]。先讀特徵向量、共變異數與高斯分布。本論文把 ML 01A 的幾何觀點連到本章的潛在變數模型。
方法與圖解

模型假設
其中 為觀測向量, 為低維潛在向量, 為負荷矩陣, 為資料平均, 為獨立等向性雜訊, 為其變異數。各單位矩陣的維度對應所在空間。積分消去 ,就得到式中的觀測分布。
最大概似讓學得的負荷子空間對齊主要共變異數特徵向量。但潛在座標不唯一:旋轉可能保留同一觀測分布。雜訊不為零時,潛在後驗平均也不是直接做一般 PCA 正交投影。應區分子空間、負荷參數與推得的座標。
證據
核心證據包含概似推導及期望最大化程序,不是新神經架構的排行榜。圖 1 用 38 筆、18 維觀測示範缺失資料推論。每個值以 20% 機率獨立移除;圖示情況共有 136 個缺失值。推得的投影仍保留可辨識群組及標示的離群點。這支持圖示用途,並非對任意缺失方式都穩健的保證。
設計分析
機率模型能利用真正觀測到的座標進行條件推論,也揭露一般投影記號可能隱藏的假設:線性結構、高斯潛在變數,以及各觀測方向相同的雜訊變異數。缺失資料範例測試其中一項實務效果;若缺失原因本身帶有資訊,它並未證明觀測機制已被正確建模。
閱讀檢查點
說明為何改變潛在軸方向,不一定改變觀測密度。再考慮一筆缺少一個座標的觀測:預測應以哪些已觀測值為條件?為何直接把缺失值補零,通常是不同的程序?
限制與下一步
單一線性高斯成分無法表示所有彎曲或多峰分布;概似是否有用,取決於假設是否適合任務。接著讀 ML 07 的 VAE 導讀,比較此處線性解碼器與可處理的後驗,和後者的非線性解碼器及可學習近似後驗。
接著閱讀: VAE:透過可計算的下界學習
參考文獻
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 13–14. Book
- K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Book
- Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper
- M. E. Tipping and C. M. Bishop, “Probabilistic Principal Component Analysis,” Journal of the Royal Statistical Society: Series B, vol. 61, no. 3, 1999. Paper
- A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks, vol. 13, no. 4–5, 2000. Paper
- B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, vol. 381, 1996. Paper
- D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, 1999. Paper
- P. Baldi and K. Hornik, “Neural networks and principal component analysis: Learning from examples without local minima,” Neural Networks, vol. 2, no. 1, 1989. Paper
- G. E. Hinton and R. R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, 2006. Paper
- P. Vincent, H. Larochelle, Y. Bengio and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” ICML, 2008. Paper
- P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P.-A. Manzagol, “Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion,” JMLR, vol. 11, 2010. Paper
- S. Rifai, P. Vincent, X. Muller, X. Glorot and Y. Bengio, “Contractive Auto-Encoders: Explicit Invariance During Feature Extraction,” ICML, 2011. Paper
- D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
- D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, vol. 12, no. 4, 2019. Paper
- I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
- D. Bank, N. Koenigstein and R. Giryes, “Autoencoders,” survey chapter, arXiv:2003.05991, 2020. Survey
- F. Locatello et al., “Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations,” ICML, 2019. Paper
- A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
- H. Cunningham, A. Ewart, L. Riggs, R. Huben and L. Sharkey, “Sparse Autoencoders Find Highly Interpretable Features in Language Models,” arXiv:2309.08600, 2023. Paper
- L. Gao et al., “Scaling and evaluating sparse autoencoders,” arXiv:2406.04093, 2024. Paper