A VAE trades reconstruction against prior agreement
IntermediateML 06A · Linear Factor Models and AutoencodersInteractive plot
Description
Move and widen a Gaussian posterior, then compute expected decoder loss and the exact KL penalty.
Try this: Set q(z|x) equal to the prior. Why is KL zero but reconstruction still poor for x=1? Can shrinking σ improve one term while worsening the other?
Hints
The chapter section that uses this lab:
Lab · A VAE trades reconstruction against prior agreement
- Expected decoder NLL
- 0.806
- KL(q || prior), nats
- 0.638
- NLL + β KL
- 1.444
Reconstruction favors latent values near x=1 and a narrow posterior. KL penalizes both moving away from zero and changing variance away from one. Prior agreement alone does not guarantee useful reconstruction.
Try this: Set q(z|x) equal to the prior. Why is KL zero but reconstruction still poor for x=1? Can shrinking σ improve one term while worsening the other?
One observation x=1, q=N(μ,σ²), prior=N(0,1), fixed decoder p(x|z)=N(z,0.5²). Expected negative log likelihood is analytic; no samples or network training. β=1 gives the negative evidence lower bound for this model; other β values are weighted objectives.
Method sources
- Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning, MIT Press, 2016
- Ashish Vaswani et al., Attention Is All You Need, NeurIPS, 2017
- Jonathan Ho, Ajay Jain & Pieter Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS, 2020
- Tim Salimans & Jonathan Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR, 2022