Full search page →

The ELBO and the reparameterization trick: training a latent state

Optional · supplementary note 1 of 3 for DLCV 13 · World Models: Video, Latent Dynamics and Planning. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 3

  1. The ELBO and the reparameterization trick: training a latent state
  2. More of the JEPA line: I-JEPA, V-JEPA and V-JEPA 2.1
  3. The cross-entropy method: planning by sampling and refitting

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Plain version. The camera never shows the hidden state zz, so we cannot train it directly. The ELBO is a score for a guess about zz: the guess earns points when it explains the image well and loses points when it strays from what the model could have predicted before looking. The reparameterization trick lets gradients flow through a random guess.

Precise version. A latent-variable model has a prior p(z)p(z) and a likelihood p(o∣z)p(o\mid z). The log-evidence log⁡p(o)=log⁡∫p(z) p(o∣z) dz\log p(o)=\log\int p(z)\,p(o\mid z)\,dz is usually intractable. For any distribution q(z)q(z) over the latent,

log⁡p(o)=Eq[log⁡p(o∣z)]−DKL(q(z) ∥ p(z))⏟ELBO(q)+DKL(q(z) ∥ p(z∣o)),\log p(o)=\underbrace{\mathbb E_{q}\bigl[\log p(o\mid z)\bigr]-D_{\mathrm{KL}}\bigl(q(z)\,\|\,p(z)\bigr)}_{\mathrm{ELBO}(q)} +D_{\mathrm{KL}}\bigl(q(z)\,\|\,p(z\mid o)\bigr),

where p(z∣o)p(z\mid o) is the true posterior and DKL≥0D_{\mathrm{KL}}\ge0. So the ELBO (evidence lower bound) never exceeds log⁡p(o)\log p(o), and the gap is exactly how far qq is from the true posterior. Maximizing the ELBO over qq closes the gap; maximizing it over the model parameters raises the evidence. The per-step loss Lt\mathcal L_t in the RSSM section has this shape: reconstruction terms, plus a KL between the posterior qθ(zt∣ht,ot)q_\theta(z_t\mid h_t,o_t) and a prior pθ(zt∣ht)p_\theta(z_t\mid h_t) that is itself predicted by the model; β=1\beta=1 gives the plain bound, other values reweight it.

To learn qq by gradient descent we must differentiate Eq[⋅]\mathbb E_q[\cdot] with respect to the parameters of qq. If q=N(μ,s2)q=\mathcal N(\mu,s^2), write

z=μ+s ε,ε∼N(0,1),z=\mu+s\,\varepsilon,\qquad \varepsilon\sim\mathcal N(0,1),

where μ\mu and ss are the mean and standard deviation of qq and ε\varepsilon is noise that does not depend on them. Then Eq[f(z)]=Eε[f(μ+sε)]\mathbb E_q[f(z)]=\mathbb E_\varepsilon[f(\mu+s\varepsilon)], and the gradient moves inside the expectation. This is the reparameterization trick of Kingma and Welling [33]; Rezende et al. [34] developed the same idea under the name stochastic backpropagation. It needs a continuous zz.

Worked example. Take p(z)=N(0,1)p(z)=\mathcal N(0,1), p(o∣z)=N(o;z,σ2)p(o\mid z)=\mathcal N(o;z,\sigma^2) with σ2=0.5\sigma^2=0.5, and one observation o=2o=2. Everything is Gaussian, so the true posterior is N(4/3, 1/3)\mathcal N(4/3,\,1/3) (precisions add; this is a hand calculation) and the evidence is N(o;0,1+σ2)\mathcal N(o;0,1+\sigma^2) in closed form.

Code · Python · 29 lines
import numpy as np

o, var = 2.0, 0.5                      # observation and likelihood variance sigma^2

def elbo(mu, s):                       # closed form for q = N(mu, s^2), prior N(0, 1)
    recon = -0.5 * np.log(2 * np.pi * var) - ((o - mu) ** 2 + s ** 2) / (2 * var)
    kl = 0.5 * (mu ** 2 + s ** 2 - 1 - np.log(s ** 2))
    return recon - kl

evidence = -0.5 * np.log(2 * np.pi * (1 + var)) - o ** 2 / (2 * (1 + var))  # log N(o; 0, 1+var)
mu_star, s_star = (o / var) / (1 + 1 / var), np.sqrt(1 / (1 + 1 / var))     # exact posterior
print(f"log evidence log p(o):        {evidence:.4f}")
print(f"ELBO at the exact posterior:  {elbo(mu_star, s_star):.4f}")
print(f"ELBO at q = N(0, 1):          {elbo(0.0, 1.0):.4f}")
gap = np.log(s_star / 1.0) + (1.0 + (0.0 - mu_star) ** 2) / (2 * s_star ** 2) - 0.5  # KL(q || posterior)
print(f"evidence minus that ELBO:     {evidence - elbo(0.0, 1.0):.4f}  (= KL(q || posterior) = {gap:.4f})")

# Gradient of the ELBO with respect to mu, estimated from samples of z ~ q.
rng = np.random.default_rng(0)
mu, s, n = 0.0, 1.0, 100_000
true_grad = (o - mu) / var - mu                      # analytic value
eps = rng.standard_normal(n)
z = mu + s * eps                                     # reparameterization: z = mu + s * eps
g_rep = (o - z) / var - mu                           # pathwise gradient (KL term analytic)
logp = -0.5 * np.log(2 * np.pi * var) - (o - z) ** 2 / (2 * var)
g_score = logp * (z - mu) / s ** 2 - mu              # score-function (REINFORCE) estimator
print(f"true gradient {true_grad:.3f}")
print(f"reparameterized: mean {g_rep.mean():.3f}, per-sample std {g_rep.std():.3f}")
print(f"score function:  mean {g_score.mean():.3f}, per-sample std {g_score.std():.3f}")

Output:

Code · 7 lines
log evidence log p(o):        -2.4550
ELBO at the exact posterior:  -2.4550
ELBO at q = N(0, 1):          -5.5724
evidence minus that ELBO:     3.1174  (= KL(q || posterior) = 3.1174)
true gradient 4.000
reparameterized: mean 4.002, per-sample std 2.000
score function:  mean 4.013, per-sample std 9.840

At the exact posterior the ELBO equals the log-evidence (−2.4550-2.4550). A poor guess, q=N(0,1)q=\mathcal N(0,1), scores −5.5724-5.5724, and the shortfall 3.11743.1174 is exactly its KL divergence from the posterior. Both gradient estimators are centered on the true value 4.0004.000 with 10510^5 samples, but a single reparameterized sample is far less noisy (standard deviation 2.0002.000 against 9.8409.840 for the score-function estimator), which is why it is the default for continuous latents.

Common mistakes. Reading the ELBO as the likelihood (it is a lower bound, and a loose one if qq is poor); forgetting that the trick needs the noise ε\varepsilon to be independent of the parameters; and applying it to a discrete latent, where μ+sε\mu+s\varepsilon has no meaning and a different gradient estimator is required.

References

  1. J. Ding et al., “Understanding world or predicting future? A comprehensive survey of world models,” ACM Computing Surveys, vol. 58, no. 3, 2025. doi arXiv
  2. C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, ch. 13 “Sequential Data”. book
  3. R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., Springer, 2022. book
  4. R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM SIGART Bulletin, vol. 2, no. 4, 1991. doi
  5. T. M. Moerland, J. Broekens, A. Plaat and C. M. Jonker, “Model-based reinforcement learning: a survey,” Foundations and Trends in Machine Learning, vol. 16, no. 1, 2023. doi arXiv
  6. D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” NeurIPS, 2018; interactive version “World Models,” arXiv:1803.10122. paper interactive
  7. D. Hafner et al., “Learning latent dynamics for planning from pixels,” ICML, 2019. paper
  8. D. Hafner, T. Lillicrap, J. Ba and M. Norouzi, “Dream to control: learning behaviors by latent imagination,” ICLR, 2020. arXiv
  9. D. Hafner, T. Lillicrap, M. Norouzi and J. Ba, “Mastering Atari with discrete world models,” ICLR, 2021. arXiv
  10. D. Hafner, J. Pasukonis, J. Ba and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature, vol. 640, 2025. doi arXiv
  11. N. Hansen, H. Su and X. Wang, “TD-MPC2: scalable, robust world models for continuous control,” ICLR, 2024. arXiv
  12. V. Micheli, E. Alonso and F. Fleuret, “Transformers are sample-efficient world models,” ICLR, 2023. arXiv
  13. E. Alonso et al., “Diffusion for world modeling: visual details matter in Atari,” NeurIPS, 2024. arXiv
  14. J. Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model,” Nature, vol. 588, 2020. doi arXiv
  15. C. Finn, I. Goodfellow and S. Levine, “Unsupervised learning for physical interaction through video prediction,” NIPS, 2016. arXiv
  16. M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell and S. Levine, “Stochastic variational video prediction,” arXiv:1710.11252, 2017. arXiv
  17. A. Hu et al., “GAIA-1: a generative world model for autonomous driving,” technical report, arXiv:2309.17080, 2023. arXiv
  18. S. Yang et al., “Learning interactive real-world simulators,” arXiv:2310.06114, 2023. arXiv
  19. J. Bruce et al., “Genie: generative interactive environments,” arXiv:2402.15391, 2024. arXiv
  20. Google DeepMind, “Genie 3: a new frontier for world models,” blog post, August 5, 2025. blog
  21. NVIDIA (N. Agarwal et al.), “Cosmos world foundation model platform for physical AI,” arXiv:2501.03575, 2025. arXiv
  22. S. Motamed, L. Culp, K. Swersky, P. Jaini and R. Geirhos, “Do generative video models understand physical principles?,” arXiv:2501.09038, 2025. arXiv
  23. B. Kang et al., “How far is video generation from world model: a physical law perspective,” ICML, 2025. arXiv
  24. M. Assran et al., “Self-supervised learning from images with a joint-embedding predictive architecture,” CVPR, 2023. doi arXiv
  25. A. Bardes et al., “Revisiting feature prediction for learning visual representations from video,” arXiv:2404.08471, 2024. arXiv
  26. M. Assran et al., “V-JEPA 2: self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
  27. L. Mur-Labadia et al., “V-JEPA 2.1: unlocking dense features in video self-supervised learning,” arXiv:2603.14482, 2026. arXiv
  28. G. Zhou, H. Pan, Y. LeCun and L. Pinto, “DINO-WM: world models on pre-trained visual features enable zero-shot planning,” arXiv:2411.04983, 2024. arXiv
  29. K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” NeurIPS, 2018. arXiv
  30. M. Janner, J. Fu, M. Zhang and S. Levine, “When to trust your model: model-based policy optimization,” NeurIPS, 2019. arXiv
  31. Z. Zhu et al., “Is Sora a world simulator? A comprehensive survey on general world models and beyond,” arXiv:2405.03520, 2024. arXiv
  32. Y. Guan et al., “World models for autonomous driving: an initial survey,” arXiv:2403.02622, 2024. arXiv
  33. D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” International Conference on Learning Representations (ICLR), 2014; arXiv:1312.6114. arXiv
  34. D. J. Rezende, S. Mohamed and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” ICML, 2014. paper
  35. R. Y. Rubinstein, “The cross-entropy method for combinatorial and continuous optimization,” Methodology and Computing in Applied Probability, vol. 1, no. 2, 1999. doi
  36. P.-T. de Boer, D. P. Kroese, S. Mannor and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” Annals of Operations Research, vol. 134, no. 1, 2005. doi