完整搜尋頁面 →

因果 3D 卷積與 1 + T 影格排列

選讀・DLCV 12A・影片生成 的補充資料(第 1/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 因果 3D 卷積與 1 + T 影格排列
  2. 從影像模型起步的四種方式
  3. 重建引導:延伸影片
  4. 相機位姿與 Plücker 嵌入

論文導讀 · 2

  1. Blattmann 等人:Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets(2023)
  2. Ge 等人:On the Content Bias in Fréchet Video Distance(CVPR 2024)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則補充說明因果時間卷積如何補零、為什麼第一個影格最後是單獨編碼的,以及步幅層如何產生「1+T/ft1 + T/f_t」的潛在長度。

一般補零與因果補零。 考慮沿影格軸、核大小為 ktk_t 的一個時間卷積(空間部分不變)。一般的「same」卷積在片段前補 ⌊(kt−1)/2⌋\lfloor (k_t-1)/2 \rfloor 個影格、片段後補 ⌊kt/2⌋\lfloor k_t/2 \rfloor 個影格,所以輸出影格 tt 依賴輸入影格 t−⌊(kt−1)/2⌋t - \lfloor (k_t-1)/2 \rfloor 到 t+⌊kt/2⌋t + \lfloor k_t/2 \rfloor,包含未來的影格。因果卷積在片段前補 kt−1k_t - 1 個影格、後面不補,所以輸出 tt 只依賴影格 t−kt+1t - k_t + 1 到 tt [15]、[2]。

為什麼第一個影格是獨立的。 採用因果補零時,輸出影格 0 只看得到補的零和輸入影格 0。疊上因果層仍保有這個性質,所以第一個影格的潛在表徵只是第一個影格本身的函數。這就是同一個編碼器能對單張影像做 token 化的原因 [15]。

步幅降取樣。 步幅為 ss 的因果卷積把 1+s t1 + s\,t 個輸入影格對應到 1+t1 + t 個輸出;解碼器則上取樣 ss 倍,並丟掉得到的前 s−1s - 1 個影格,把 1+t1 + t 對應回 1+s t1 + s\,t [15]。兩個 s=2s = 2 的階段得到 ft=4f_t = 4:17 個影格變成 5 個潛在影格,81 個變成 21 個。

長影片的分塊編碼。 因果層只需要最後 kt−1k_t - 1 個影格的上下文,所以長影片可以一塊一塊地編碼或解碼,同時快取這些影格。Wan 的 VAE 每個潛在影格處理一塊(最多 4 個影格),並在核大小為 3 時,保留前一塊最後兩個影格的特徵,這讓它能在有限記憶體下對任意長度的影片編碼與解碼 [3]。CogVideoX 利用同樣的性質,把長影片分給多個裝置處理,每個裝置只需傳 kt−1k_t - 1 個影格給下一個 [2]。

經過測試的檢查。 這段程式實作一個核大小 4、步幅 4 的因果時間卷積,作用在 17 個「影格」上(為了清楚起見,用純量),並確認輸出長度,以及沒有任何輸出依賴較後面的影格。

程式碼 · Python · 16 行
import numpy as np

def causal_conv1d(x, w, stride):
    """Causal temporal convolution: pad len(w) - 1 zeros in front, none behind."""
    k = len(w)
    xp = np.concatenate([np.zeros(k - 1), x])
    return np.array([xp[i * stride:i * stride + k] @ w for i in range((len(x) - 1) // stride + 1)])

rng = np.random.default_rng(0)
x, w = rng.normal(size=17), rng.normal(size=4)
y = causal_conv1d(x, w, stride=4)
print("17 frames ->", len(y), "latent frames")
for j in [0, 1, 4, 5, 16]:                     # perturb one frame, see which outputs change
    x2 = x.copy(); x2[j] += 1.0
    changed = np.flatnonzero(~np.isclose(causal_conv1d(x2, w, 4), y))
    print(f"frame {j:2d} changes latent frames {changed.tolist()}")

輸出:

程式碼 · 6 行
17 frames -> 5 latent frames
frame  0 changes latent frames [0]
frame  1 changes latent frames [1]
frame  4 changes latent frames [1]
frame  5 changes latent frames [2]
frame 16 changes latent frames [4]

影格 0 只影響潛在影格 0,之後每四個影格一組,只影響它自己的潛在影格,如圖 1 所示。真實的編碼器會疊上好幾層、核互相重疊,所以一個潛在影格也會看到一些更早的組,但絕不會看到更後面的。

陷阱。 影格數若不符合 1+s t1 + s\,t,就必須補齊或裁切;Open-Sora 報告,只用 17 個影格的片段訓練它的自編碼器,會造成其他長度時的模糊,混合長度訓練則解決了這個問題 [22]。沿時間彙整統計量的正規化層會破壞因果性;Wan 因此把 group normalization 換成 RMS normalization [3]。

參考文獻

  1. U. Singer, A. Polyak, T. Hayes et al., “Make-A-Video: Text-to-Video Generation without Text-Video Data,” arXiv:2209.14792, 2022. arXiv
  2. Z. Yang, J. Teng, W. Zheng et al., “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer,” ICLR, 2025. arXiv
  3. Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” arXiv:2503.20314, 2025. arXiv
  4. A. Blattmann, T. Dockhorn, S. Kulal et al., “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets,” arXiv:2311.15127, 2023. arXiv
  5. J. Xing, M. Xia, Y. Zhang et al., “DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors,” ECCV, LNCS, pp. 399–417, 2024. doi · arXiv
  6. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi and D. J. Fleet, “Video Diffusion Models,” NeurIPS, 2022. proceedings · arXiv
  7. A. Blattmann, R. Rombach, H. Ling et al., “Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models,” CVPR, 2023. CVF
  8. R. Henschel, L. Khachatryan, H. Poghosyan et al., “StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text,” CVPR, 2025. CVF
  9. J. Z. Wu, Y. Ge, X. Wang et al., “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation,” ICCV, 2023. CVF
  10. L. Khachatryan, A. Movsisyan, V. Tadevosyan et al., “Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators,” ICCV, 2023. CVF
  11. Z. Wang, Z. Yuan, X. Wang et al., “MotionCtrl: A Unified and Flexible Motion Controller for Video Generation,” SIGGRAPH 2024 Conference Papers, 2024. doi · arXiv
  12. H. He, Y. Xu, Y. Guo et al., “CameraCtrl: Enabling Camera Control for Text-to-Video Generation,” ICLR, 2025. arXiv
  13. S. Yin, C. Wu, J. Liang et al., “DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory,” arXiv:2308.08089, 2023. arXiv
  14. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. CVF
  15. L. Yu, J. Lezama, N. B. Gundavarapu et al., “Language Model Beats Diffusion — Tokenizer is Key to Visual Generation,” ICLR, 2024. arXiv
  16. W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” arXiv:2412.03603, 2024. arXiv
  17. A. Polyak, A. Zohar, A. Brown et al., “Movie Gen: A Cast of Media Foundation Models,” arXiv:2410.13720, 2024. arXiv
  18. Y. HaCohen, N. Chiprut, B. Brazowski et al., “LTX-Video: Realtime Video Latent Diffusion,” arXiv:2501.00103, 2024. arXiv
  19. D. Kondratyuk, L. Yu, X. Gu et al., “VideoPoet: A Large Language Model for Zero-Shot Video Generation,” ICML, PMLR 235, 2024. PMLR · arXiv
  20. L. Yu, Y. Cheng, K. Sohn et al., “MAGVIT: Masked Generative Video Transformer,” CVPR, 2023. CVF
  21. Y. Guo, C. Yang, A. Rao et al., “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning,” ICLR, 2024. arXiv
  22. Z. Zheng, X. Peng, T. Yang et al., “Open-Sora: Democratizing Efficient Video Production for All,” arXiv:2412.20404, 2024. arXiv
  23. G. Bertasius, H. Wang and L. Torresani, “Is Space-Time Attention All You Need for Video Understanding?,” ICML, PMLR 139, 2021. PMLR
  24. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A Video Vision Transformer,” ICCV, 2021. CVF
  25. X. Ma, Y. Wang, X. Chen et al., “Latte: Latent Diffusion Transformer for Video Generation,” Transactions on Machine Learning Research, 2025. arXiv
  26. B. Lin, Y. Ge, X. Cheng et al., “Open-Sora Plan: Open-Source Large Video Generation Model,” arXiv:2412.00131, 2024. arXiv
  27. J. Ho, W. Chan, C. Saharia et al., “Imagen Video: High Definition Video Generation with Diffusion Models,” arXiv:2210.02303, 2022. arXiv
  28. O. Bar-Tal, H. Chefer, O. Tov et al., “Lumiere: A Space-Time Diffusion Model for Video Generation,” SIGGRAPH Asia 2024 Conference Papers, 2024. doi · arXiv
  29. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” arXiv:2210.02747, 2022. arXiv
  30. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. CVF
  31. S. Yin, C. Wu, H. Yang et al., “NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation,” ACL, pp. 1309–1320, 2023. doi · arXiv
  32. H. Qiu, M. Xia, Y. Zhang et al., “FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling,” ICLR, 2024. arXiv
  33. R. Villegas, M. Babaeizadeh, P.-J. Kindermans et al., “Phenaki: Variable Length Video Generation From Open Domain Textual Description,” arXiv:2210.02399, 2022. arXiv
  34. Z. Huang, Y. He, J. Yu et al., “VBench: Comprehensive Benchmark Suite for Video Generative Models,” CVPR, 2024. CVF
  35. T. Yin, Q. Zhang, R. Zhang et al., “From Slow Bidirectional to Fast Autoregressive Video Diffusion Models,” CVPR, 2025. CVF
  36. B. Chen, D. M. Monso, Y. Du, M. Simchowitz, R. Tedrake and V. Sitzmann, “Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion,” NeurIPS, 2024. proceedings · arXiv
  37. X. Huang, Z. Li, G. He, M. Zhou and E. Shechtman, “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion,” NeurIPS, 2025. arXiv
  38. T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards Accurate Generative Models of Video: A New Metric & Challenges,” arXiv:1812.01717, 2018. arXiv
  39. J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset,” CVPR, 2017. CVF
  40. W. Kay, J. Carreira, K. Simonyan et al., “The Kinetics Human Action Video Dataset,” arXiv:1705.06950, 2017. arXiv
  41. K. Soomro, A. R. Zamir and M. Shah, “UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
  42. S. Ge, A. Mahapatra, G. Parmar, J.-Y. Zhu and J.-B. Huang, “On the Content Bias in Fréchet Video Distance,” CVPR, 2024. CVF
  43. L. Wang, B. Huang, Z. Zhao et al., “VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking,” CVPR, 2023. CVF
  44. Z. Huang, F. Zhang, X. Xu et al., “VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 3, pp. 3268–3285, 2026. doi · arXiv
  45. Y. Liu, X. Cun, X. Liu et al., “EvalCrafter: Benchmarking and Evaluating Large Video Generation Models,” CVPR, 2024. CVF
  46. Z. Zheng, X. Peng, Y. Lou et al., “Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k,” arXiv:2503.09642, 2025. arXiv
  47. M. Bain, A. Nagrani, G. Varol and A. Zisserman, “Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval,” ICCV, 2021. CVF · arXiv
  48. T.-S. Chen, A. Siarohin, W. Menapace et al., “Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers,” CVPR, 2024. CVF
  49. A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies and M. Nießner, “FaceForensics++: Learning to Detect Manipulated Facial Images,” ICCV, 2019. doi · arXiv
  50. P. Fernandez, H. Elsahar, I. Z. Yalniz and A. Mourachko, “Video Seal: Open and Efficient Video Watermarking,” arXiv:2412.09492, 2024. arXiv
  51. A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
  52. Z. Xing, Q. Feng, H. Chen et al., “A Survey on Video Diffusion Models,” arXiv:2310.10647, 2023. arXiv
  53. Y. Liu, K. Zhang, Y. Li et al., “Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models,” arXiv:2402.17177, 2024. arXiv
  54. OpenAI, “What to know about the Sora discontinuation,” official service notice, 2026, accessed 5 November 2026. Notice.