ELBO 與重新參數化技巧:訓練潛在狀態
選讀・DLCV 13・世界模型:影片、潛在動態與規劃 的補充資料(第 1/3 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
白話版。 相機從來不會顯示隱藏狀態 ,所以我們無法直接訓練它。ELBO 是替一個對 的猜測打的分數:猜測若能把影像解釋得好就加分,若偏離模型在「還沒看到影像前」本來能預測的結果就扣分。重新參數化技巧(reparameterization trick)則讓梯度能夠穿過這個隨機的猜測。
精確版。 潛在變數模型有先驗 與概似 。對數證據(log-evidence) 通常無法計算。對潛在變數上的任何分布 ,
其中 是真正的後驗,而 。所以 ELBO(evidence lower bound,證據下界)永遠不會超過 ,兩者的差距恰好就是 與真正後驗之間的距離。對 最大化 ELBO 會縮小差距;對模型參數最大化它則會提高證據。RSSM 一節中的單步損失 就是這個形狀:幾個重建項,加上後驗 與先驗 之間的 KL,而這個先驗本身由模型預測; 得到最單純的下界,其他數值則是重新加權。
若要用梯度下降學習 ,就必須對 的參數微分 。當 時,改寫成
其中 與 是 的平均與標準差, 是不依賴它們的雜訊。於是 ,梯度就可以移進期望值裡面。這就是 Kingma 與 Welling 的重新參數化技巧 [33];Rezende 等人 [34] 以「隨機反向傳播」(stochastic backpropagation)為名發展了同樣的想法。它要求 是連續的。
實作範例。 設 、,其中 ,並只有一筆觀測 。全部都是高斯分布,所以真正的後驗是 (精確度相加;這是手算結果),證據則有閉式解 。
程式碼 · Python · 29 行
import numpy as np
o, var = 2.0, 0.5 # observation and likelihood variance sigma^2
def elbo(mu, s): # closed form for q = N(mu, s^2), prior N(0, 1)
recon = -0.5 * np.log(2 * np.pi * var) - ((o - mu) ** 2 + s ** 2) / (2 * var)
kl = 0.5 * (mu ** 2 + s ** 2 - 1 - np.log(s ** 2))
return recon - kl
evidence = -0.5 * np.log(2 * np.pi * (1 + var)) - o ** 2 / (2 * (1 + var)) # log N(o; 0, 1+var)
mu_star, s_star = (o / var) / (1 + 1 / var), np.sqrt(1 / (1 + 1 / var)) # exact posterior
print(f"log evidence log p(o): {evidence:.4f}")
print(f"ELBO at the exact posterior: {elbo(mu_star, s_star):.4f}")
print(f"ELBO at q = N(0, 1): {elbo(0.0, 1.0):.4f}")
gap = np.log(s_star / 1.0) + (1.0 + (0.0 - mu_star) ** 2) / (2 * s_star ** 2) - 0.5 # KL(q || posterior)
print(f"evidence minus that ELBO: {evidence - elbo(0.0, 1.0):.4f} (= KL(q || posterior) = {gap:.4f})")
# Gradient of the ELBO with respect to mu, estimated from samples of z ~ q.
rng = np.random.default_rng(0)
mu, s, n = 0.0, 1.0, 100_000
true_grad = (o - mu) / var - mu # analytic value
eps = rng.standard_normal(n)
z = mu + s * eps # reparameterization: z = mu + s * eps
g_rep = (o - z) / var - mu # pathwise gradient (KL term analytic)
logp = -0.5 * np.log(2 * np.pi * var) - (o - z) ** 2 / (2 * var)
g_score = logp * (z - mu) / s ** 2 - mu # score-function (REINFORCE) estimator
print(f"true gradient {true_grad:.3f}")
print(f"reparameterized: mean {g_rep.mean():.3f}, per-sample std {g_rep.std():.3f}")
print(f"score function: mean {g_score.mean():.3f}, per-sample std {g_score.std():.3f}")輸出:
程式碼 · 7 行
log evidence log p(o): -2.4550
ELBO at the exact posterior: -2.4550
ELBO at q = N(0, 1): -5.5724
evidence minus that ELBO: 3.1174 (= KL(q || posterior) = 3.1174)
true gradient 4.000
reparameterized: mean 4.002, per-sample std 2.000
score function: mean 4.013, per-sample std 9.840在真正的後驗上,ELBO 等於對數證據()。一個很差的猜測 只得到 ,差的 恰好就是它與後驗之間的 KL 散度。使用 個樣本時,兩種梯度估計量都以真值 為中心,但單一個重新參數化的樣本雜訊小得多(標準差 ,對比分數函數(score-function)估計量的 ),這就是連續潛在變數預設採用它的原因。
常見錯誤。 把 ELBO 當成概似(它是下界,若 很差,這個下界會很鬆);忘記這個技巧要求雜訊 與參數無關;以及把它用在離散的潛在變數上—— 對離散變數沒有意義,必須改用別的梯度估計量。
參考文獻
- J. Ding et al., “Understanding world or predicting future? A comprehensive survey of world models,” ACM Computing Surveys, vol. 58, no. 3, 2025. doi arXiv
- C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, ch. 13 “Sequential Data”. book
- R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., Springer, 2022. book
- R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM SIGART Bulletin, vol. 2, no. 4, 1991. doi
- T. M. Moerland, J. Broekens, A. Plaat and C. M. Jonker, “Model-based reinforcement learning: a survey,” Foundations and Trends in Machine Learning, vol. 16, no. 1, 2023. doi arXiv
- D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” NeurIPS, 2018; interactive version “World Models,” arXiv:1803.10122. paper interactive
- D. Hafner et al., “Learning latent dynamics for planning from pixels,” ICML, 2019. paper
- D. Hafner, T. Lillicrap, J. Ba and M. Norouzi, “Dream to control: learning behaviors by latent imagination,” ICLR, 2020. arXiv
- D. Hafner, T. Lillicrap, M. Norouzi and J. Ba, “Mastering Atari with discrete world models,” ICLR, 2021. arXiv
- D. Hafner, J. Pasukonis, J. Ba and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature, vol. 640, 2025. doi arXiv
- N. Hansen, H. Su and X. Wang, “TD-MPC2: scalable, robust world models for continuous control,” ICLR, 2024. arXiv
- V. Micheli, E. Alonso and F. Fleuret, “Transformers are sample-efficient world models,” ICLR, 2023. arXiv
- E. Alonso et al., “Diffusion for world modeling: visual details matter in Atari,” NeurIPS, 2024. arXiv
- J. Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model,” Nature, vol. 588, 2020. doi arXiv
- C. Finn, I. Goodfellow and S. Levine, “Unsupervised learning for physical interaction through video prediction,” NIPS, 2016. arXiv
- M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell and S. Levine, “Stochastic variational video prediction,” arXiv:1710.11252, 2017. arXiv
- A. Hu et al., “GAIA-1: a generative world model for autonomous driving,” technical report, arXiv:2309.17080, 2023. arXiv
- S. Yang et al., “Learning interactive real-world simulators,” arXiv:2310.06114, 2023. arXiv
- J. Bruce et al., “Genie: generative interactive environments,” arXiv:2402.15391, 2024. arXiv
- Google DeepMind, “Genie 3: a new frontier for world models,” blog post, August 5, 2025. blog
- NVIDIA (N. Agarwal et al.), “Cosmos world foundation model platform for physical AI,” arXiv:2501.03575, 2025. arXiv
- S. Motamed, L. Culp, K. Swersky, P. Jaini and R. Geirhos, “Do generative video models understand physical principles?,” arXiv:2501.09038, 2025. arXiv
- B. Kang et al., “How far is video generation from world model: a physical law perspective,” ICML, 2025. arXiv
- M. Assran et al., “Self-supervised learning from images with a joint-embedding predictive architecture,” CVPR, 2023. doi arXiv
- A. Bardes et al., “Revisiting feature prediction for learning visual representations from video,” arXiv:2404.08471, 2024. arXiv
- M. Assran et al., “V-JEPA 2: self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
- L. Mur-Labadia et al., “V-JEPA 2.1: unlocking dense features in video self-supervised learning,” arXiv:2603.14482, 2026. arXiv
- G. Zhou, H. Pan, Y. LeCun and L. Pinto, “DINO-WM: world models on pre-trained visual features enable zero-shot planning,” arXiv:2411.04983, 2024. arXiv
- K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” NeurIPS, 2018. arXiv
- M. Janner, J. Fu, M. Zhang and S. Levine, “When to trust your model: model-based policy optimization,” NeurIPS, 2019. arXiv
- Z. Zhu et al., “Is Sora a world simulator? A comprehensive survey on general world models and beyond,” arXiv:2405.03520, 2024. arXiv
- Y. Guan et al., “World models for autonomous driving: an initial survey,” arXiv:2403.02622, 2024. arXiv
- D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” International Conference on Learning Representations (ICLR), 2014; arXiv:1312.6114. arXiv
- D. J. Rezende, S. Mohamed and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” ICML, 2014. paper
- R. Y. Rubinstein, “The cross-entropy method for combinatorial and continuous optimization,” Methodology and Computing in Applied Probability, vol. 1, no. 2, 1999. doi
- P.-T. de Boer, D. P. Kroese, S. Mannor and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” Annals of Operations Research, vol. 134, no. 1, 2005. doi