因果 3D 卷積與 1 + T 影格排列
選讀・DLCV 12A・影片生成 的補充資料(第 1/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
這則補充說明因果時間卷積如何補零、為什麼第一個影格最後是單獨編碼的,以及步幅層如何產生「」的潛在長度。
一般補零與因果補零。 考慮沿影格軸、核大小為 的一個時間卷積(空間部分不變)。一般的「same」卷積在片段前補 個影格、片段後補 個影格,所以輸出影格 依賴輸入影格 到 ,包含未來的影格。因果卷積在片段前補 個影格、後面不補,所以輸出 只依賴影格 到 [15]、[2]。
為什麼第一個影格是獨立的。 採用因果補零時,輸出影格 0 只看得到補的零和輸入影格 0。疊上因果層仍保有這個性質,所以第一個影格的潛在表徵只是第一個影格本身的函數。這就是同一個編碼器能對單張影像做 token 化的原因 [15]。
步幅降取樣。 步幅為 的因果卷積把 個輸入影格對應到 個輸出;解碼器則上取樣 倍,並丟掉得到的前 個影格,把 對應回 [15]。兩個 的階段得到 :17 個影格變成 5 個潛在影格,81 個變成 21 個。
長影片的分塊編碼。 因果層只需要最後 個影格的上下文,所以長影片可以一塊一塊地編碼或解碼,同時快取這些影格。Wan 的 VAE 每個潛在影格處理一塊(最多 4 個影格),並在核大小為 3 時,保留前一塊最後兩個影格的特徵,這讓它能在有限記憶體下對任意長度的影片編碼與解碼 [3]。CogVideoX 利用同樣的性質,把長影片分給多個裝置處理,每個裝置只需傳 個影格給下一個 [2]。
經過測試的檢查。 這段程式實作一個核大小 4、步幅 4 的因果時間卷積,作用在 17 個「影格」上(為了清楚起見,用純量),並確認輸出長度,以及沒有任何輸出依賴較後面的影格。
程式碼 · Python · 16 行
import numpy as np
def causal_conv1d(x, w, stride):
"""Causal temporal convolution: pad len(w) - 1 zeros in front, none behind."""
k = len(w)
xp = np.concatenate([np.zeros(k - 1), x])
return np.array([xp[i * stride:i * stride + k] @ w for i in range((len(x) - 1) // stride + 1)])
rng = np.random.default_rng(0)
x, w = rng.normal(size=17), rng.normal(size=4)
y = causal_conv1d(x, w, stride=4)
print("17 frames ->", len(y), "latent frames")
for j in [0, 1, 4, 5, 16]: # perturb one frame, see which outputs change
x2 = x.copy(); x2[j] += 1.0
changed = np.flatnonzero(~np.isclose(causal_conv1d(x2, w, 4), y))
print(f"frame {j:2d} changes latent frames {changed.tolist()}")輸出:
程式碼 · 6 行
17 frames -> 5 latent frames
frame 0 changes latent frames [0]
frame 1 changes latent frames [1]
frame 4 changes latent frames [1]
frame 5 changes latent frames [2]
frame 16 changes latent frames [4]影格 0 只影響潛在影格 0,之後每四個影格一組,只影響它自己的潛在影格,如圖 1 所示。真實的編碼器會疊上好幾層、核互相重疊,所以一個潛在影格也會看到一些更早的組,但絕不會看到更後面的。
陷阱。 影格數若不符合 ,就必須補齊或裁切;Open-Sora 報告,只用 17 個影格的片段訓練它的自編碼器,會造成其他長度時的模糊,混合長度訓練則解決了這個問題 [22]。沿時間彙整統計量的正規化層會破壞因果性;Wan 因此把 group normalization 換成 RMS normalization [3]。
參考文獻
- U. Singer, A. Polyak, T. Hayes et al., “Make-A-Video: Text-to-Video Generation without Text-Video Data,” arXiv:2209.14792, 2022. arXiv
- Z. Yang, J. Teng, W. Zheng et al., “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer,” ICLR, 2025. arXiv
- Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” arXiv:2503.20314, 2025. arXiv
- A. Blattmann, T. Dockhorn, S. Kulal et al., “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets,” arXiv:2311.15127, 2023. arXiv
- J. Xing, M. Xia, Y. Zhang et al., “DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors,” ECCV, LNCS, pp. 399–417, 2024. doi · arXiv
- J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi and D. J. Fleet, “Video Diffusion Models,” NeurIPS, 2022. proceedings · arXiv
- A. Blattmann, R. Rombach, H. Ling et al., “Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models,” CVPR, 2023. CVF
- R. Henschel, L. Khachatryan, H. Poghosyan et al., “StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text,” CVPR, 2025. CVF
- J. Z. Wu, Y. Ge, X. Wang et al., “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation,” ICCV, 2023. CVF
- L. Khachatryan, A. Movsisyan, V. Tadevosyan et al., “Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators,” ICCV, 2023. CVF
- Z. Wang, Z. Yuan, X. Wang et al., “MotionCtrl: A Unified and Flexible Motion Controller for Video Generation,” SIGGRAPH 2024 Conference Papers, 2024. doi · arXiv
- H. He, Y. Xu, Y. Guo et al., “CameraCtrl: Enabling Camera Control for Text-to-Video Generation,” ICLR, 2025. arXiv
- S. Yin, C. Wu, J. Liang et al., “DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory,” arXiv:2308.08089, 2023. arXiv
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. CVF
- L. Yu, J. Lezama, N. B. Gundavarapu et al., “Language Model Beats Diffusion — Tokenizer is Key to Visual Generation,” ICLR, 2024. arXiv
- W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” arXiv:2412.03603, 2024. arXiv
- A. Polyak, A. Zohar, A. Brown et al., “Movie Gen: A Cast of Media Foundation Models,” arXiv:2410.13720, 2024. arXiv
- Y. HaCohen, N. Chiprut, B. Brazowski et al., “LTX-Video: Realtime Video Latent Diffusion,” arXiv:2501.00103, 2024. arXiv
- D. Kondratyuk, L. Yu, X. Gu et al., “VideoPoet: A Large Language Model for Zero-Shot Video Generation,” ICML, PMLR 235, 2024. PMLR · arXiv
- L. Yu, Y. Cheng, K. Sohn et al., “MAGVIT: Masked Generative Video Transformer,” CVPR, 2023. CVF
- Y. Guo, C. Yang, A. Rao et al., “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning,” ICLR, 2024. arXiv
- Z. Zheng, X. Peng, T. Yang et al., “Open-Sora: Democratizing Efficient Video Production for All,” arXiv:2412.20404, 2024. arXiv
- G. Bertasius, H. Wang and L. Torresani, “Is Space-Time Attention All You Need for Video Understanding?,” ICML, PMLR 139, 2021. PMLR
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A Video Vision Transformer,” ICCV, 2021. CVF
- X. Ma, Y. Wang, X. Chen et al., “Latte: Latent Diffusion Transformer for Video Generation,” Transactions on Machine Learning Research, 2025. arXiv
- B. Lin, Y. Ge, X. Cheng et al., “Open-Sora Plan: Open-Source Large Video Generation Model,” arXiv:2412.00131, 2024. arXiv
- J. Ho, W. Chan, C. Saharia et al., “Imagen Video: High Definition Video Generation with Diffusion Models,” arXiv:2210.02303, 2022. arXiv
- O. Bar-Tal, H. Chefer, O. Tov et al., “Lumiere: A Space-Time Diffusion Model for Video Generation,” SIGGRAPH Asia 2024 Conference Papers, 2024. doi · arXiv
- Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” arXiv:2210.02747, 2022. arXiv
- W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. CVF
- S. Yin, C. Wu, H. Yang et al., “NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation,” ACL, pp. 1309–1320, 2023. doi · arXiv
- H. Qiu, M. Xia, Y. Zhang et al., “FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling,” ICLR, 2024. arXiv
- R. Villegas, M. Babaeizadeh, P.-J. Kindermans et al., “Phenaki: Variable Length Video Generation From Open Domain Textual Description,” arXiv:2210.02399, 2022. arXiv
- Z. Huang, Y. He, J. Yu et al., “VBench: Comprehensive Benchmark Suite for Video Generative Models,” CVPR, 2024. CVF
- T. Yin, Q. Zhang, R. Zhang et al., “From Slow Bidirectional to Fast Autoregressive Video Diffusion Models,” CVPR, 2025. CVF
- B. Chen, D. M. Monso, Y. Du, M. Simchowitz, R. Tedrake and V. Sitzmann, “Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion,” NeurIPS, 2024. proceedings · arXiv
- X. Huang, Z. Li, G. He, M. Zhou and E. Shechtman, “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion,” NeurIPS, 2025. arXiv
- T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards Accurate Generative Models of Video: A New Metric & Challenges,” arXiv:1812.01717, 2018. arXiv
- J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset,” CVPR, 2017. CVF
- W. Kay, J. Carreira, K. Simonyan et al., “The Kinetics Human Action Video Dataset,” arXiv:1705.06950, 2017. arXiv
- K. Soomro, A. R. Zamir and M. Shah, “UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
- S. Ge, A. Mahapatra, G. Parmar, J.-Y. Zhu and J.-B. Huang, “On the Content Bias in Fréchet Video Distance,” CVPR, 2024. CVF
- L. Wang, B. Huang, Z. Zhao et al., “VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking,” CVPR, 2023. CVF
- Z. Huang, F. Zhang, X. Xu et al., “VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 3, pp. 3268–3285, 2026. doi · arXiv
- Y. Liu, X. Cun, X. Liu et al., “EvalCrafter: Benchmarking and Evaluating Large Video Generation Models,” CVPR, 2024. CVF
- Z. Zheng, X. Peng, Y. Lou et al., “Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k,” arXiv:2503.09642, 2025. arXiv
- M. Bain, A. Nagrani, G. Varol and A. Zisserman, “Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval,” ICCV, 2021. CVF · arXiv
- T.-S. Chen, A. Siarohin, W. Menapace et al., “Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers,” CVPR, 2024. CVF
- A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies and M. Nießner, “FaceForensics++: Learning to Detect Manipulated Facial Images,” ICCV, 2019. doi · arXiv
- P. Fernandez, H. Elsahar, I. Z. Yalniz and A. Mourachko, “Video Seal: Open and Efficient Video Watermarking,” arXiv:2412.09492, 2024. arXiv
- A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
- Z. Xing, Q. Feng, H. Chen et al., “A Survey on Video Diffusion Models,” arXiv:2310.10647, 2023. arXiv
- Y. Liu, K. Zhang, Y. Li et al., “Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models,” arXiv:2402.17177, 2024. arXiv
- OpenAI, “What to know about the Sora discontinuation,” official service notice, 2026, accessed 5 November 2026. Notice.