VAE — learning through a tractable lower bound
Optional · paper guide 1 of 2 for ML 07 · Generative Models: Autoencoders, VAEs, GANs and Flows. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
How can a nonlinear latent-variable model be trained when its true posterior is hard to compute? Kingma and Welling jointly learn a generative model and an approximate inference model using a variational bound and a reparameterized gradient estimator [6]. Read conditional probability, KL divergence and backpropagation first. The encoder is an inference device, not an inverse decoder guaranteed to recover a unique hidden cause.
Method and figure

Figure 1 separates the generative arrows from the recognition arrows. For one observation,
Here is observed data, is latent, is the prior, is the decoder likelihood and is the approximate posterior. Parameters and belong to decoder and encoder; is the evidence lower bound. Its gap to log evidence is the KL divergence from the approximate posterior to the true model posterior.
For a diagonal Gaussian encoder, use
Here and are predicted means and standard deviations, is elementwise multiplication, and is parameter-independent noise. Randomness now enters a differentiable transformation, allowing gradients through the sample construction. Generation instead samples the prior and runs the decoder; it does not require an input image or the encoder.
Evidence
Figure 2 compares optimization of the variational lower bound with wake-sleep on MNIST and Frey faces for several latent dimensions. Its horizontal axis counts training examples processed, not elapsed time. Figure 3 separately examines estimated marginal likelihood as training-set size changes. A lower-bound curve, a marginal-likelihood estimate and an attractive generated image are different evidence; none should be relabeled as the others.
Design analysis
The first term rewards explaining observations; the KL term relates inferred codes to the prior used for generation. A flexible decoder alone does not make the approximate posterior exact. The reparameterization trick reduces the difficulty of gradient estimation for this continuous family, but does not solve arbitrary discrete sampling by simply applying the same formula.
Reading checkpoint
Identify which distribution supplies a latent sample during training and which supplies it during unconditional generation. Then explain why a good reconstruction from an encoded training example does not establish that independent prior samples produce equally good observations.
Limits and next step
Results depend on the likelihood family and approximation gap. A smooth two-dimensional latent plot does not prove disentanglement or calibrated uncertainty. Continue to the GAN guide to compare explicit variational density learning with adversarial sample matching, then to VQ-VAE for a discrete latent representation with a separately learned prior.
Continue reading: GAN — a distribution game and its idealized guarantee
References
- C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks
- S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. Paper
- G. Papamakarios et al., “Normalizing Flows for Probabilistic Modeling and Inference,” JMLR, 2021. Review
- L. Theis, A. van den Oord and M. Bethge, “A note on the evaluation of generative models,” ICLR, 2016. Paper
- D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
- S. R. Bowman et al., “Generating Sentences from a Continuous Space,” CoNLL, 2016. Paper
- I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
- D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, 2019. Review
- A. Vahdat and J. Kautz, “NVAE: A Deep Hierarchical Variational Autoencoder,” NeurIPS, 2020. Paper
- I. Goodfellow et al., “Generative Adversarial Nets,” NeurIPS, 2014. Paper
- T. Salimans et al., “Improved Techniques for Training GANs,” NeurIPS, 2016. Paper
- M. Arjovsky, S. Chintala and L. Bottou, “Wasserstein GAN,” ICML, 2017. Paper
- I. Gulrajani et al., “Improved Training of Wasserstein GANs,” NeurIPS, 2017. Paper
- T. Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018. Paper
- L. Mescheder, A. Geiger and S. Nowozin, “Which Training Methods for GANs do actually Converge?” ICML, 2018. Paper
- T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. Paper
- L. Dinh, J. Sohl-Dickstein and S. Bengio, “Density estimation using Real NVP,” ICLR, 2017. Paper
- D. P. Kingma and P. Dhariwal, “Glow: Generative Flow with Invertible 1x1 Convolutions,” NeurIPS, 2018. Paper
- D. J. Rezende and S. Mohamed, “Variational Inference with Normalizing Flows,” ICML, 2015. Paper
- R. T. Q. Chen, Y. Rubanova, J. Bettencourt and D. Duvenaud, “Neural Ordinary Differential Equations,” NeurIPS, 2018. Paper
- Y. Lipman et al., “Flow Matching for Generative Modeling,” ICLR, 2023. Paper
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 18–20. Book
- Y. Song and D. P. Kingma, “How to Train Your Energy-Based Models,” arXiv, 2021. Paper
- A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” ICML, 2016. Paper
- J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. Paper
- L. Yang et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. Survey
- E. Nalisnick et al., “Do Deep Generative Models Know What They Don’t Know?” ICLR, 2019. Paper
- M. Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. Paper
- M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet and S. Gelly, “Assessing Generative Models via Precision and Recall,” NeurIPS, 2018. Paper
- T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. Paper
- G. Stein et al., “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models,” NeurIPS, 2023. Paper
- S. Jayasumana et al., “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. Paper
- A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
- R. Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
- M. Kang et al., “Scaling up GANs for Text-to-Image Synthesis,” CVPR, 2023. Paper
- Y. Huang, A. Gokaslan, V. Kuleshov and J. Tompkin, “The GAN is dead; long live the GAN! A Modern GAN Baseline,” NeurIPS, 2024. Paper