完整搜尋頁面 →

Probabilistic PCA:子空間背後的機率模型

選讀・ML 06A・線性因子模型與自編碼器 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

PCA 能否不只是一種幾何投影,也是一個機率生成模型?Tipping 與 Bishop 顯示,高斯潛在變數模型可找回主子空間,同時提供概似及缺失觀測的推論方法 [4]。先讀特徵向量、共變異數與高斯分布。本論文把 ML 01A 的幾何觀點連到本章的潛在變數模型。

方法與圖解

範例比較完整資料投影與由不完整觀測推得的投影;編號群組及離群點讓讀者判斷結構是否保留。
來源:Tipping and Bishop, JRSS B 61(3), 1999; author manuscript dated September 27, 1999, hosted by Oxford. 圖 1,PDF 第 6 頁。範例比較完整資料投影與由不完整觀測推得的投影;編號群組及離群點讓讀者判斷結構是否保留。 [4]

模型假設

x=Wz+μ+ϵ,z∼N(0,I),ϵ∼N(0,σ2I),x∼N(μ,WW⊤+σ2I).x=Wz+\mu+\epsilon,\qquad z\sim\mathcal N(0,I),\quad\epsilon\sim\mathcal N(0,\sigma^2I),\quad x\sim\mathcal N(\mu,WW^\top+\sigma^2I).

其中 xx 為觀測向量,zz 為低維潛在向量,WW 為負荷矩陣,μ\mu 為資料平均,ϵ\epsilon 為獨立等向性雜訊,σ2\sigma^2 為其變異數。各單位矩陣的維度對應所在空間。積分消去 zz,就得到式中的觀測分布。

最大概似讓學得的負荷子空間對齊主要共變異數特徵向量。但潛在座標不唯一:旋轉可能保留同一觀測分布。雜訊不為零時,潛在後驗平均也不是直接做一般 PCA 正交投影。應區分子空間、負荷參數與推得的座標。

證據

核心證據包含概似推導及期望最大化程序,不是新神經架構的排行榜。圖 1 用 38 筆、18 維觀測示範缺失資料推論。每個值以 20% 機率獨立移除;圖示情況共有 136 個缺失值。推得的投影仍保留可辨識群組及標示的離群點。這支持圖示用途,並非對任意缺失方式都穩健的保證。

設計分析

機率模型能利用真正觀測到的座標進行條件推論,也揭露一般投影記號可能隱藏的假設:線性結構、高斯潛在變數,以及各觀測方向相同的雜訊變異數。缺失資料範例測試其中一項實務效果;若缺失原因本身帶有資訊,它並未證明觀測機制已被正確建模。

閱讀檢查點

說明為何改變潛在軸方向,不一定改變觀測密度。再考慮一筆缺少一個座標的觀測:預測應以哪些已觀測值為條件?為何直接把缺失值補零,通常是不同的程序?

限制與下一步

單一線性高斯成分無法表示所有彎曲或多峰分布;概似是否有用,取決於假設是否適合任務。接著讀 ML 07 的 VAE 導讀,比較此處線性解碼器與可處理的後驗,和後者的非線性解碼器及可學習近似後驗。

接著閱讀: VAE:透過可計算的下界學習

參考文獻

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 13–14. Book
  2. K. P. Murphy, Probabilistic Machine Learning: An Introduction, MIT Press, 2022. Book
  3. Y. Bengio, A. Courville and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013. Paper
  4. M. E. Tipping and C. M. Bishop, “Probabilistic Principal Component Analysis,” Journal of the Royal Statistical Society: Series B, vol. 61, no. 3, 1999. Paper
  5. A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks, vol. 13, no. 4–5, 2000. Paper
  6. B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, vol. 381, 1996. Paper
  7. D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, 1999. Paper
  8. P. Baldi and K. Hornik, “Neural networks and principal component analysis: Learning from examples without local minima,” Neural Networks, vol. 2, no. 1, 1989. Paper
  9. G. E. Hinton and R. R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, 2006. Paper
  10. P. Vincent, H. Larochelle, Y. Bengio and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” ICML, 2008. Paper
  11. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P.-A. Manzagol, “Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion,” JMLR, vol. 11, 2010. Paper
  12. S. Rifai, P. Vincent, X. Muller, X. Glorot and Y. Bengio, “Contractive Auto-Encoders: Explicit Invariance During Feature Extraction,” ICML, 2011. Paper
  13. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
  14. D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, vol. 12, no. 4, 2019. Paper
  15. I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
  16. D. Bank, N. Koenigstein and R. Giryes, “Autoencoders,” survey chapter, arXiv:2003.05991, 2020. Survey
  17. F. Locatello et al., “Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations,” ICML, 2019. Paper
  18. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
  19. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
  20. H. Cunningham, A. Ewart, L. Riggs, R. Huben and L. Sharkey, “Sparse Autoencoders Find Highly Interpretable Features in Language Models,” arXiv:2309.08600, 2023. Paper
  21. L. Gao et al., “Scaling and evaluating sparse autoencoders,” arXiv:2406.04093, 2024. Paper