Full search page →

VAE — learning through a tractable lower bound

Optional · paper guide 1 of 2 for ML 07 · Generative Models: Autoencoders, VAEs, GANs and Flows. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

How can a nonlinear latent-variable model be trained when its true posterior is hard to compute? Kingma and Welling jointly learn a generative model and an approximate inference model using a variational bound and a reparameterized gradient estimator [6]. Read conditional probability, KL divergence and backpropagation first. The encoder is an inference device, not an inverse decoder guaranteed to recover a unique hidden cause.

Method and figure

Solid arrows define the generative model and dashed arrows the recognition model. Inference from an observation and generation from a prior traverse different directions.
Source: Kingma and Welling, ICLR 2014 manuscript, arXiv:1312.6114v11. Figure 1, PDF p. 2. Solid arrows define the generative model and dashed arrows the recognition model. Inference from an observation and generation from a prior traverse different directions. [6]

Figure 1 separates the generative arrows from the recognition arrows. For one observation,

L(x)=Eqϕ(z∣x)[log⁡pθ(x∣z)]−DKL(qϕ(z∣x)∥p(z))≤log⁡pθ(x).\mathcal L(x)=\mathbb E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]-D_{\mathrm{KL}}(q_\phi(z\mid x)\Vert p(z))\leq\log p_\theta(x).

Here xx is observed data, zz is latent, p(z)p(z) is the prior, pθ(x∣z)p_\theta(x\mid z) is the decoder likelihood and qϕ(z∣x)q_\phi(z\mid x) is the approximate posterior. Parameters θ\theta and ϕ\phi belong to decoder and encoder; L\mathcal L is the evidence lower bound. Its gap to log evidence is the KL divergence from the approximate posterior to the true model posterior.

For a diagonal Gaussian encoder, use

z=μϕ(x)+σϕ(x)⊙ϵ,ϵ∼N(0,I).z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon,\qquad\epsilon\sim\mathcal N(0,I).

Here μϕ\mu_\phi and σϕ\sigma_\phi are predicted means and standard deviations, ⊙\odot is elementwise multiplication, and ϵ\epsilon is parameter-independent noise. Randomness now enters a differentiable transformation, allowing gradients through the sample construction. Generation instead samples the prior and runs the decoder; it does not require an input image or the encoder.

Evidence

Figure 2 compares optimization of the variational lower bound with wake-sleep on MNIST and Frey faces for several latent dimensions. Its horizontal axis counts training examples processed, not elapsed time. Figure 3 separately examines estimated marginal likelihood as training-set size changes. A lower-bound curve, a marginal-likelihood estimate and an attractive generated image are different evidence; none should be relabeled as the others.

Design analysis

The first term rewards explaining observations; the KL term relates inferred codes to the prior used for generation. A flexible decoder alone does not make the approximate posterior exact. The reparameterization trick reduces the difficulty of gradient estimation for this continuous family, but does not solve arbitrary discrete sampling by simply applying the same formula.

Reading checkpoint

Identify which distribution supplies a latent sample during training and which supplies it during unconditional generation. Then explain why a good reconstruction from an encoded training example does not establish that independent prior samples produce equally good observations.

Limits and next step

Results depend on the likelihood family and approximation gap. A smooth two-dimensional latent plot does not prove disentanglement or calibrated uncertainty. Continue to the GAN guide to compare explicit variational density learning with adversarial sample matching, then to VQ-VAE for a discrete latent representation with a separately learned prior.

Continue reading: GAN — a distribution game and its idealized guarantee

References

  1. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  2. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks
  3. S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. Paper
  4. G. Papamakarios et al., “Normalizing Flows for Probabilistic Modeling and Inference,” JMLR, 2021. Review
  5. L. Theis, A. van den Oord and M. Bethge, “A note on the evaluation of generative models,” ICLR, 2016. Paper
  6. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
  7. S. R. Bowman et al., “Generating Sentences from a Continuous Space,” CoNLL, 2016. Paper
  8. I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
  9. D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, 2019. Review
  10. A. Vahdat and J. Kautz, “NVAE: A Deep Hierarchical Variational Autoencoder,” NeurIPS, 2020. Paper
  11. I. Goodfellow et al., “Generative Adversarial Nets,” NeurIPS, 2014. Paper
  12. T. Salimans et al., “Improved Techniques for Training GANs,” NeurIPS, 2016. Paper
  13. M. Arjovsky, S. Chintala and L. Bottou, “Wasserstein GAN,” ICML, 2017. Paper
  14. I. Gulrajani et al., “Improved Training of Wasserstein GANs,” NeurIPS, 2017. Paper
  15. T. Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018. Paper
  16. L. Mescheder, A. Geiger and S. Nowozin, “Which Training Methods for GANs do actually Converge?” ICML, 2018. Paper
  17. T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. Paper
  18. L. Dinh, J. Sohl-Dickstein and S. Bengio, “Density estimation using Real NVP,” ICLR, 2017. Paper
  19. D. P. Kingma and P. Dhariwal, “Glow: Generative Flow with Invertible 1x1 Convolutions,” NeurIPS, 2018. Paper
  20. D. J. Rezende and S. Mohamed, “Variational Inference with Normalizing Flows,” ICML, 2015. Paper
  21. R. T. Q. Chen, Y. Rubanova, J. Bettencourt and D. Duvenaud, “Neural Ordinary Differential Equations,” NeurIPS, 2018. Paper
  22. Y. Lipman et al., “Flow Matching for Generative Modeling,” ICLR, 2023. Paper
  23. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 18–20. Book
  24. Y. Song and D. P. Kingma, “How to Train Your Energy-Based Models,” arXiv, 2021. Paper
  25. A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” ICML, 2016. Paper
  26. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. Paper
  27. L. Yang et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. Survey
  28. E. Nalisnick et al., “Do Deep Generative Models Know What They Don’t Know?” ICLR, 2019. Paper
  29. M. Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. Paper
  30. M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet and S. Gelly, “Assessing Generative Models via Precision and Recall,” NeurIPS, 2018. Paper
  31. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. Paper
  32. G. Stein et al., “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models,” NeurIPS, 2023. Paper
  33. S. Jayasumana et al., “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. Paper
  34. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
  35. R. Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
  36. M. Kang et al., “Scaling up GANs for Text-to-Image Synthesis,” CVPR, 2023. Paper
  37. Y. Huang, A. Gokaslan, V. Kuleshov and J. Tompkin, “The GAN is dead; long live the GAN! A Modern GAN Baseline,” NeurIPS, 2024. Paper