The ELBO and the reparameterization trick: training a latent state
Optional · supplementary note 1 of 3 for DLCV 13 · World Models: Video, Latent Dynamics and Planning. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Plain version. The camera never shows the hidden state , so we cannot train it directly. The ELBO is a score for a guess about : the guess earns points when it explains the image well and loses points when it strays from what the model could have predicted before looking. The reparameterization trick lets gradients flow through a random guess.
Precise version. A latent-variable model has a prior and a likelihood . The log-evidence is usually intractable. For any distribution over the latent,
where is the true posterior and . So the ELBO (evidence lower bound) never exceeds , and the gap is exactly how far is from the true posterior. Maximizing the ELBO over closes the gap; maximizing it over the model parameters raises the evidence. The per-step loss in the RSSM section has this shape: reconstruction terms, plus a KL between the posterior and a prior that is itself predicted by the model; gives the plain bound, other values reweight it.
To learn by gradient descent we must differentiate with respect to the parameters of . If , write
where and are the mean and standard deviation of and is noise that does not depend on them. Then , and the gradient moves inside the expectation. This is the reparameterization trick of Kingma and Welling [33]; Rezende et al. [34] developed the same idea under the name stochastic backpropagation. It needs a continuous .
Worked example. Take , with , and one observation . Everything is Gaussian, so the true posterior is (precisions add; this is a hand calculation) and the evidence is in closed form.
Code · Python · 29 lines
import numpy as np
o, var = 2.0, 0.5 # observation and likelihood variance sigma^2
def elbo(mu, s): # closed form for q = N(mu, s^2), prior N(0, 1)
recon = -0.5 * np.log(2 * np.pi * var) - ((o - mu) ** 2 + s ** 2) / (2 * var)
kl = 0.5 * (mu ** 2 + s ** 2 - 1 - np.log(s ** 2))
return recon - kl
evidence = -0.5 * np.log(2 * np.pi * (1 + var)) - o ** 2 / (2 * (1 + var)) # log N(o; 0, 1+var)
mu_star, s_star = (o / var) / (1 + 1 / var), np.sqrt(1 / (1 + 1 / var)) # exact posterior
print(f"log evidence log p(o): {evidence:.4f}")
print(f"ELBO at the exact posterior: {elbo(mu_star, s_star):.4f}")
print(f"ELBO at q = N(0, 1): {elbo(0.0, 1.0):.4f}")
gap = np.log(s_star / 1.0) + (1.0 + (0.0 - mu_star) ** 2) / (2 * s_star ** 2) - 0.5 # KL(q || posterior)
print(f"evidence minus that ELBO: {evidence - elbo(0.0, 1.0):.4f} (= KL(q || posterior) = {gap:.4f})")
# Gradient of the ELBO with respect to mu, estimated from samples of z ~ q.
rng = np.random.default_rng(0)
mu, s, n = 0.0, 1.0, 100_000
true_grad = (o - mu) / var - mu # analytic value
eps = rng.standard_normal(n)
z = mu + s * eps # reparameterization: z = mu + s * eps
g_rep = (o - z) / var - mu # pathwise gradient (KL term analytic)
logp = -0.5 * np.log(2 * np.pi * var) - (o - z) ** 2 / (2 * var)
g_score = logp * (z - mu) / s ** 2 - mu # score-function (REINFORCE) estimator
print(f"true gradient {true_grad:.3f}")
print(f"reparameterized: mean {g_rep.mean():.3f}, per-sample std {g_rep.std():.3f}")
print(f"score function: mean {g_score.mean():.3f}, per-sample std {g_score.std():.3f}")Output:
Code · 7 lines
log evidence log p(o): -2.4550
ELBO at the exact posterior: -2.4550
ELBO at q = N(0, 1): -5.5724
evidence minus that ELBO: 3.1174 (= KL(q || posterior) = 3.1174)
true gradient 4.000
reparameterized: mean 4.002, per-sample std 2.000
score function: mean 4.013, per-sample std 9.840At the exact posterior the ELBO equals the log-evidence (). A poor guess, , scores , and the shortfall is exactly its KL divergence from the posterior. Both gradient estimators are centered on the true value with samples, but a single reparameterized sample is far less noisy (standard deviation against for the score-function estimator), which is why it is the default for continuous latents.
Common mistakes. Reading the ELBO as the likelihood (it is a lower bound, and a loose one if is poor); forgetting that the trick needs the noise to be independent of the parameters; and applying it to a discrete latent, where has no meaning and a different gradient estimator is required.
References
- J. Ding et al., “Understanding world or predicting future? A comprehensive survey of world models,” ACM Computing Surveys, vol. 58, no. 3, 2025. doi arXiv
- C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, ch. 13 “Sequential Data”. book
- R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., Springer, 2022. book
- R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM SIGART Bulletin, vol. 2, no. 4, 1991. doi
- T. M. Moerland, J. Broekens, A. Plaat and C. M. Jonker, “Model-based reinforcement learning: a survey,” Foundations and Trends in Machine Learning, vol. 16, no. 1, 2023. doi arXiv
- D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” NeurIPS, 2018; interactive version “World Models,” arXiv:1803.10122. paper interactive
- D. Hafner et al., “Learning latent dynamics for planning from pixels,” ICML, 2019. paper
- D. Hafner, T. Lillicrap, J. Ba and M. Norouzi, “Dream to control: learning behaviors by latent imagination,” ICLR, 2020. arXiv
- D. Hafner, T. Lillicrap, M. Norouzi and J. Ba, “Mastering Atari with discrete world models,” ICLR, 2021. arXiv
- D. Hafner, J. Pasukonis, J. Ba and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature, vol. 640, 2025. doi arXiv
- N. Hansen, H. Su and X. Wang, “TD-MPC2: scalable, robust world models for continuous control,” ICLR, 2024. arXiv
- V. Micheli, E. Alonso and F. Fleuret, “Transformers are sample-efficient world models,” ICLR, 2023. arXiv
- E. Alonso et al., “Diffusion for world modeling: visual details matter in Atari,” NeurIPS, 2024. arXiv
- J. Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model,” Nature, vol. 588, 2020. doi arXiv
- C. Finn, I. Goodfellow and S. Levine, “Unsupervised learning for physical interaction through video prediction,” NIPS, 2016. arXiv
- M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell and S. Levine, “Stochastic variational video prediction,” arXiv:1710.11252, 2017. arXiv
- A. Hu et al., “GAIA-1: a generative world model for autonomous driving,” technical report, arXiv:2309.17080, 2023. arXiv
- S. Yang et al., “Learning interactive real-world simulators,” arXiv:2310.06114, 2023. arXiv
- J. Bruce et al., “Genie: generative interactive environments,” arXiv:2402.15391, 2024. arXiv
- Google DeepMind, “Genie 3: a new frontier for world models,” blog post, August 5, 2025. blog
- NVIDIA (N. Agarwal et al.), “Cosmos world foundation model platform for physical AI,” arXiv:2501.03575, 2025. arXiv
- S. Motamed, L. Culp, K. Swersky, P. Jaini and R. Geirhos, “Do generative video models understand physical principles?,” arXiv:2501.09038, 2025. arXiv
- B. Kang et al., “How far is video generation from world model: a physical law perspective,” ICML, 2025. arXiv
- M. Assran et al., “Self-supervised learning from images with a joint-embedding predictive architecture,” CVPR, 2023. doi arXiv
- A. Bardes et al., “Revisiting feature prediction for learning visual representations from video,” arXiv:2404.08471, 2024. arXiv
- M. Assran et al., “V-JEPA 2: self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
- L. Mur-Labadia et al., “V-JEPA 2.1: unlocking dense features in video self-supervised learning,” arXiv:2603.14482, 2026. arXiv
- G. Zhou, H. Pan, Y. LeCun and L. Pinto, “DINO-WM: world models on pre-trained visual features enable zero-shot planning,” arXiv:2411.04983, 2024. arXiv
- K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” NeurIPS, 2018. arXiv
- M. Janner, J. Fu, M. Zhang and S. Levine, “When to trust your model: model-based policy optimization,” NeurIPS, 2019. arXiv
- Z. Zhu et al., “Is Sora a world simulator? A comprehensive survey on general world models and beyond,” arXiv:2405.03520, 2024. arXiv
- Y. Guan et al., “World models for autonomous driving: an initial survey,” arXiv:2403.02622, 2024. arXiv
- D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” International Conference on Learning Representations (ICLR), 2014; arXiv:1312.6114. arXiv
- D. J. Rezende, S. Mohamed and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” ICML, 2014. paper
- R. Y. Rubinstein, “The cross-entropy method for combinatorial and continuous optimization,” Methodology and Computing in Applied Probability, vol. 1, no. 2, 1999. doi
- P.-T. de Boer, D. P. Kroese, S. Mannor and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” Annals of Operations Research, vol. 134, no. 1, 2005. doi