完整搜尋頁面 →

VAE:透過可計算的下界學習

選讀・ML 07・生成模型:自編碼器、VAE、GAN 與正規化流 的論文導讀(第 1/2 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

非線性潛在變數模型的真實後驗難以計算時,如何訓練?Kingma 與 Welling 利用變分下界及重參數化梯度估計,同時學習生成模型與近似推論模型 [6]。先讀條件機率、KL 散度與反向傳播。編碼器是推論工具,不是保證找回唯一隱藏原因的解碼器反函數。

方法與圖解

實線表示生成模型,虛線表示辨識模型;由觀測推論潛在變數,與由先驗生成資料,沿著不同方向運作。
來源:Kingma and Welling, ICLR 2014 manuscript, arXiv:1312.6114v11. 圖 1,PDF 第 2 頁。實線表示生成模型,虛線表示辨識模型;由觀測推論潛在變數,與由先驗生成資料,沿著不同方向運作。 [6]

圖 1 區分生成方向與辨識方向。對單筆觀測,

L(x)=Eqϕ(z∣x)[log⁡pθ(x∣z)]−DKL(qϕ(z∣x)∥p(z))≤log⁡pθ(x).\mathcal L(x)=\mathbb E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]-D_{\mathrm{KL}}(q_\phi(z\mid x)\Vert p(z))\leq\log p_\theta(x).

其中 xx 為觀測,zz 為潛在變數,p(z)p(z) 為先驗,pθ(x∣z)p_\theta(x\mid z) 為解碼器概似,qϕ(z∣x)q_\phi(z\mid x) 為近似後驗。θ\theta、ϕ\phi 分別屬於解碼器與編碼器;L\mathcal L 是證據下界。它與對數證據的落差,等於近似後驗到模型真實後驗的 KL 散度。

對角高斯編碼器使用

z=μϕ(x)+σϕ(x)⊙ϵ,ϵ∼N(0,I).z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon,\qquad\epsilon\sim\mathcal N(0,I).

其中 μϕ\mu_\phi、σϕ\sigma_\phi 為預測平均與標準差,⊙\odot 表示逐元素乘法,ϵ\epsilon 是與參數無關的雜訊。隨機性進入可微轉換後,梯度便能通過樣本建構流程。生成則從先驗取樣再執行解碼器,不需要輸入影像或編碼器。

證據

圖 2 在 MNIST 與 Frey faces、不同潛在維度下,比較變分下界最佳化與 wake-sleep。橫軸是處理過的訓練樣本數,不是經過時間。圖 3 另外檢查訓練資料量改變時的估計邊際概似。下界曲線、邊際概似估計與好看的生成影像,是不同證據,不能互換名稱。

設計分析

第一項鼓勵解釋觀測,KL 項把推得的編碼連到生成時使用的先驗。解碼器有彈性,不代表近似後驗就精確。重參數化技巧降低這類連續分布的梯度估計難度,但不能直接用同一公式解決任意離散取樣。

閱讀檢查點

指出訓練與無條件生成時,各由哪個分布提供潛在樣本。再解釋:編碼訓練影像後能良好重建,為何不能證明獨立先驗樣本也會產生同樣好的觀測?

限制與下一步

結果取決於概似家族與近似落差。平滑的二維潛在圖,不代表已證明解耦或不確定性校準。接著讀 GAN,對照明確的變分密度學習與對抗式樣本匹配;再讀 VQ-VAE,理解離散表示及另行學習的先驗。

接著閱讀: GAN:分布賽局與理想條件下的保證

參考文獻

  1. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  2. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks
  3. S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. Paper
  4. G. Papamakarios et al., “Normalizing Flows for Probabilistic Modeling and Inference,” JMLR, 2021. Review
  5. L. Theis, A. van den Oord and M. Bethge, “A note on the evaluation of generative models,” ICLR, 2016. Paper
  6. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
  7. S. R. Bowman et al., “Generating Sentences from a Continuous Space,” CoNLL, 2016. Paper
  8. I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
  9. D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, 2019. Review
  10. A. Vahdat and J. Kautz, “NVAE: A Deep Hierarchical Variational Autoencoder,” NeurIPS, 2020. Paper
  11. I. Goodfellow et al., “Generative Adversarial Nets,” NeurIPS, 2014. Paper
  12. T. Salimans et al., “Improved Techniques for Training GANs,” NeurIPS, 2016. Paper
  13. M. Arjovsky, S. Chintala and L. Bottou, “Wasserstein GAN,” ICML, 2017. Paper
  14. I. Gulrajani et al., “Improved Training of Wasserstein GANs,” NeurIPS, 2017. Paper
  15. T. Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018. Paper
  16. L. Mescheder, A. Geiger and S. Nowozin, “Which Training Methods for GANs do actually Converge?” ICML, 2018. Paper
  17. T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. Paper
  18. L. Dinh, J. Sohl-Dickstein and S. Bengio, “Density estimation using Real NVP,” ICLR, 2017. Paper
  19. D. P. Kingma and P. Dhariwal, “Glow: Generative Flow with Invertible 1x1 Convolutions,” NeurIPS, 2018. Paper
  20. D. J. Rezende and S. Mohamed, “Variational Inference with Normalizing Flows,” ICML, 2015. Paper
  21. R. T. Q. Chen, Y. Rubanova, J. Bettencourt and D. Duvenaud, “Neural Ordinary Differential Equations,” NeurIPS, 2018. Paper
  22. Y. Lipman et al., “Flow Matching for Generative Modeling,” ICLR, 2023. Paper
  23. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 18–20. Book
  24. Y. Song and D. P. Kingma, “How to Train Your Energy-Based Models,” arXiv, 2021. Paper
  25. A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” ICML, 2016. Paper
  26. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. Paper
  27. L. Yang et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. Survey
  28. E. Nalisnick et al., “Do Deep Generative Models Know What They Don’t Know?” ICLR, 2019. Paper
  29. M. Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. Paper
  30. M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet and S. Gelly, “Assessing Generative Models via Precision and Recall,” NeurIPS, 2018. Paper
  31. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. Paper
  32. G. Stein et al., “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models,” NeurIPS, 2023. Paper
  33. S. Jayasumana et al., “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. Paper
  34. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
  35. R. Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
  36. M. Kang et al., “Scaling up GANs for Text-to-Image Synthesis,” CVPR, 2023. Paper
  37. Y. Huang, A. Gokaslan, V. Kuleshov and J. Tompkin, “The GAN is dead; long live the GAN! A Modern GAN Baseline,” NeurIPS, 2024. Paper