完整搜尋頁面 →

ELBO 與重新參數化技巧:訓練潛在狀態

選讀・DLCV 13・世界模型:影片、潛在動態與規劃 的補充資料(第 1/3 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 3

  1. ELBO 與重新參數化技巧:訓練潛在狀態
  2. JEPA 系列的其他模型:I-JEPA、V-JEPA 與 V-JEPA 2.1
  3. 交叉熵方法:用取樣與重新擬合來規劃

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

白話版。 相機從來不會顯示隱藏狀態 zz,所以我們無法直接訓練它。ELBO 是替一個對 zz 的猜測打的分數:猜測若能把影像解釋得好就加分,若偏離模型在「還沒看到影像前」本來能預測的結果就扣分。重新參數化技巧(reparameterization trick)則讓梯度能夠穿過這個隨機的猜測。

精確版。 潛在變數模型有先驗 p(z)p(z) 與概似 p(o∣z)p(o\mid z)。對數證據(log-evidence)log⁡p(o)=log⁡∫p(z) p(o∣z) dz\log p(o)=\log\int p(z)\,p(o\mid z)\,dz 通常無法計算。對潛在變數上的任何分布 q(z)q(z),

log⁡p(o)=Eq[log⁡p(o∣z)]−DKL(q(z) ∥ p(z))⏟ELBO(q)+DKL(q(z) ∥ p(z∣o)),\log p(o)=\underbrace{\mathbb E_{q}\bigl[\log p(o\mid z)\bigr]-D_{\mathrm{KL}}\bigl(q(z)\,\|\,p(z)\bigr)}_{\mathrm{ELBO}(q)} +D_{\mathrm{KL}}\bigl(q(z)\,\|\,p(z\mid o)\bigr),

其中 p(z∣o)p(z\mid o) 是真正的後驗,而 DKL≥0D_{\mathrm{KL}}\ge0。所以 ELBO(evidence lower bound,證據下界)永遠不會超過 log⁡p(o)\log p(o),兩者的差距恰好就是 qq 與真正後驗之間的距離。對 qq 最大化 ELBO 會縮小差距;對模型參數最大化它則會提高證據。RSSM 一節中的單步損失 Lt\mathcal L_t 就是這個形狀:幾個重建項,加上後驗 qθ(zt∣ht,ot)q_\theta(z_t\mid h_t,o_t) 與先驗 pθ(zt∣ht)p_\theta(z_t\mid h_t) 之間的 KL,而這個先驗本身由模型預測;β=1\beta=1 得到最單純的下界,其他數值則是重新加權。

若要用梯度下降學習 qq,就必須對 qq 的參數微分 Eq[⋅]\mathbb E_q[\cdot]。當 q=N(μ,s2)q=\mathcal N(\mu,s^2) 時,改寫成

z=μ+s ε,ε∼N(0,1),z=\mu+s\,\varepsilon,\qquad \varepsilon\sim\mathcal N(0,1),

其中 μ\mu 與 ss 是 qq 的平均與標準差,ε\varepsilon 是不依賴它們的雜訊。於是 Eq[f(z)]=Eε[f(μ+sε)]\mathbb E_q[f(z)]=\mathbb E_\varepsilon[f(\mu+s\varepsilon)],梯度就可以移進期望值裡面。這就是 Kingma 與 Welling 的重新參數化技巧 [33];Rezende 等人 [34] 以「隨機反向傳播」(stochastic backpropagation)為名發展了同樣的想法。它要求 zz 是連續的。

實作範例。 設 p(z)=N(0,1)p(z)=\mathcal N(0,1)、p(o∣z)=N(o;z,σ2)p(o\mid z)=\mathcal N(o;z,\sigma^2),其中 σ2=0.5\sigma^2=0.5,並只有一筆觀測 o=2o=2。全部都是高斯分布,所以真正的後驗是 N(4/3, 1/3)\mathcal N(4/3,\,1/3)(精確度相加;這是手算結果),證據則有閉式解 N(o;0,1+σ2)\mathcal N(o;0,1+\sigma^2)。

程式碼 · Python · 29 行
import numpy as np

o, var = 2.0, 0.5                      # observation and likelihood variance sigma^2

def elbo(mu, s):                       # closed form for q = N(mu, s^2), prior N(0, 1)
    recon = -0.5 * np.log(2 * np.pi * var) - ((o - mu) ** 2 + s ** 2) / (2 * var)
    kl = 0.5 * (mu ** 2 + s ** 2 - 1 - np.log(s ** 2))
    return recon - kl

evidence = -0.5 * np.log(2 * np.pi * (1 + var)) - o ** 2 / (2 * (1 + var))  # log N(o; 0, 1+var)
mu_star, s_star = (o / var) / (1 + 1 / var), np.sqrt(1 / (1 + 1 / var))     # exact posterior
print(f"log evidence log p(o):        {evidence:.4f}")
print(f"ELBO at the exact posterior:  {elbo(mu_star, s_star):.4f}")
print(f"ELBO at q = N(0, 1):          {elbo(0.0, 1.0):.4f}")
gap = np.log(s_star / 1.0) + (1.0 + (0.0 - mu_star) ** 2) / (2 * s_star ** 2) - 0.5  # KL(q || posterior)
print(f"evidence minus that ELBO:     {evidence - elbo(0.0, 1.0):.4f}  (= KL(q || posterior) = {gap:.4f})")

# Gradient of the ELBO with respect to mu, estimated from samples of z ~ q.
rng = np.random.default_rng(0)
mu, s, n = 0.0, 1.0, 100_000
true_grad = (o - mu) / var - mu                      # analytic value
eps = rng.standard_normal(n)
z = mu + s * eps                                     # reparameterization: z = mu + s * eps
g_rep = (o - z) / var - mu                           # pathwise gradient (KL term analytic)
logp = -0.5 * np.log(2 * np.pi * var) - (o - z) ** 2 / (2 * var)
g_score = logp * (z - mu) / s ** 2 - mu              # score-function (REINFORCE) estimator
print(f"true gradient {true_grad:.3f}")
print(f"reparameterized: mean {g_rep.mean():.3f}, per-sample std {g_rep.std():.3f}")
print(f"score function:  mean {g_score.mean():.3f}, per-sample std {g_score.std():.3f}")

輸出:

程式碼 · 7 行
log evidence log p(o):        -2.4550
ELBO at the exact posterior:  -2.4550
ELBO at q = N(0, 1):          -5.5724
evidence minus that ELBO:     3.1174  (= KL(q || posterior) = 3.1174)
true gradient 4.000
reparameterized: mean 4.002, per-sample std 2.000
score function:  mean 4.013, per-sample std 9.840

在真正的後驗上,ELBO 等於對數證據(−2.4550-2.4550)。一個很差的猜測 q=N(0,1)q=\mathcal N(0,1) 只得到 −5.5724-5.5724,差的 3.11743.1174 恰好就是它與後驗之間的 KL 散度。使用 10510^5 個樣本時,兩種梯度估計量都以真值 4.0004.000 為中心,但單一個重新參數化的樣本雜訊小得多(標準差 2.0002.000,對比分數函數(score-function)估計量的 9.8409.840),這就是連續潛在變數預設採用它的原因。

常見錯誤。 把 ELBO 當成概似(它是下界,若 qq 很差,這個下界會很鬆);忘記這個技巧要求雜訊 ε\varepsilon 與參數無關;以及把它用在離散的潛在變數上——μ+sε\mu+s\varepsilon 對離散變數沒有意義,必須改用別的梯度估計量。

參考文獻

  1. J. Ding et al., “Understanding world or predicting future? A comprehensive survey of world models,” ACM Computing Surveys, vol. 58, no. 3, 2025. doi arXiv
  2. C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, ch. 13 “Sequential Data”. book
  3. R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., Springer, 2022. book
  4. R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM SIGART Bulletin, vol. 2, no. 4, 1991. doi
  5. T. M. Moerland, J. Broekens, A. Plaat and C. M. Jonker, “Model-based reinforcement learning: a survey,” Foundations and Trends in Machine Learning, vol. 16, no. 1, 2023. doi arXiv
  6. D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” NeurIPS, 2018; interactive version “World Models,” arXiv:1803.10122. paper interactive
  7. D. Hafner et al., “Learning latent dynamics for planning from pixels,” ICML, 2019. paper
  8. D. Hafner, T. Lillicrap, J. Ba and M. Norouzi, “Dream to control: learning behaviors by latent imagination,” ICLR, 2020. arXiv
  9. D. Hafner, T. Lillicrap, M. Norouzi and J. Ba, “Mastering Atari with discrete world models,” ICLR, 2021. arXiv
  10. D. Hafner, J. Pasukonis, J. Ba and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature, vol. 640, 2025. doi arXiv
  11. N. Hansen, H. Su and X. Wang, “TD-MPC2: scalable, robust world models for continuous control,” ICLR, 2024. arXiv
  12. V. Micheli, E. Alonso and F. Fleuret, “Transformers are sample-efficient world models,” ICLR, 2023. arXiv
  13. E. Alonso et al., “Diffusion for world modeling: visual details matter in Atari,” NeurIPS, 2024. arXiv
  14. J. Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model,” Nature, vol. 588, 2020. doi arXiv
  15. C. Finn, I. Goodfellow and S. Levine, “Unsupervised learning for physical interaction through video prediction,” NIPS, 2016. arXiv
  16. M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell and S. Levine, “Stochastic variational video prediction,” arXiv:1710.11252, 2017. arXiv
  17. A. Hu et al., “GAIA-1: a generative world model for autonomous driving,” technical report, arXiv:2309.17080, 2023. arXiv
  18. S. Yang et al., “Learning interactive real-world simulators,” arXiv:2310.06114, 2023. arXiv
  19. J. Bruce et al., “Genie: generative interactive environments,” arXiv:2402.15391, 2024. arXiv
  20. Google DeepMind, “Genie 3: a new frontier for world models,” blog post, August 5, 2025. blog
  21. NVIDIA (N. Agarwal et al.), “Cosmos world foundation model platform for physical AI,” arXiv:2501.03575, 2025. arXiv
  22. S. Motamed, L. Culp, K. Swersky, P. Jaini and R. Geirhos, “Do generative video models understand physical principles?,” arXiv:2501.09038, 2025. arXiv
  23. B. Kang et al., “How far is video generation from world model: a physical law perspective,” ICML, 2025. arXiv
  24. M. Assran et al., “Self-supervised learning from images with a joint-embedding predictive architecture,” CVPR, 2023. doi arXiv
  25. A. Bardes et al., “Revisiting feature prediction for learning visual representations from video,” arXiv:2404.08471, 2024. arXiv
  26. M. Assran et al., “V-JEPA 2: self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
  27. L. Mur-Labadia et al., “V-JEPA 2.1: unlocking dense features in video self-supervised learning,” arXiv:2603.14482, 2026. arXiv
  28. G. Zhou, H. Pan, Y. LeCun and L. Pinto, “DINO-WM: world models on pre-trained visual features enable zero-shot planning,” arXiv:2411.04983, 2024. arXiv
  29. K. Chua, R. Calandra, R. McAllister and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” NeurIPS, 2018. arXiv
  30. M. Janner, J. Fu, M. Zhang and S. Levine, “When to trust your model: model-based policy optimization,” NeurIPS, 2019. arXiv
  31. Z. Zhu et al., “Is Sora a world simulator? A comprehensive survey on general world models and beyond,” arXiv:2405.03520, 2024. arXiv
  32. Y. Guan et al., “World models for autonomous driving: an initial survey,” arXiv:2403.02622, 2024. arXiv
  33. D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” International Conference on Learning Representations (ICLR), 2014; arXiv:1312.6114. arXiv
  34. D. J. Rezende, S. Mohamed and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” ICML, 2014. paper
  35. R. Y. Rubinstein, “The cross-entropy method for combinatorial and continuous optimization,” Methodology and Computing in Applied Probability, vol. 1, no. 2, 1999. doi
  36. P.-T. de Boer, D. P. Kroese, S. Mannor and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” Annals of Operations Research, vol. 134, no. 1, 2005. doi