完整搜尋頁面 →

從影像模型起步的四種方式

選讀・DLCV 12A・影片生成 的補充資料(第 2/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 因果 3D 卷積與 1 + T 影格排列
  2. 從影像模型起步的四種方式
  3. 重建引導:延伸影片
  4. 相機位姿與 Plücker 嵌入

論文導讀 · 2

  1. Blattmann 等人:Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets(2023)
  2. Ge 等人:On the Content Bias in Fréchet Video Distance(CVPR 2024)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則補充逐一系統比較四個影片生成器如何重用預訓練的影像模型:凍結了什麼、加入了什麼,以及新的部分如何初始化。

四者都遵循正文的規則:新的時間部分在訓練一開始,必須不改變影像模型的行為。它們的差別在於凍結了什麼,以及「什麼都不做」的起點如何實作。

  • Make-A-Video [1] 在每個預訓練的 2D 卷積之後堆疊一個 1D 時間卷積(「偽 3D 卷積」),並在每個預訓練的空間注意力層之後加一個時間注意力層。新的 1D 層被初始化為恆等函數,所以初始時網路生成的是各自符合文字、但在時間上不連貫的獨立影像。之後空間層與時間層一起在影片上訓練。
  • Video LDM [7] 凍結預訓練潛在影像模型的空間層,只訓練插入的時間層(時間注意力與 3D 卷積殘差區塊)。每個時間層之後,它的輸出 z′z' 與空間輸出 zz 以 αz+(1−α)z′\alpha z + (1 - \alpha) z' 混合,其中 α∈[0,1]\alpha \in [0, 1] 是學得的;當 α=1\alpha = 1 時,時間層被略過,影像模型被完整還原。由於空間層沒有動,訓練好的時間層也能遷移到同一個影像模型的微調版本。
  • AnimateDiff [21] 分階段訓練。首先,領域轉接器(domain adapter;影像模型注意力中的 LoRA 層,用影片資料的單一影格訓練)吸收影像訓練資料與影片資料之間的視覺品質差距,例如動態模糊、壓縮瑕疵與浮水印,使這個差距不被當成動作的一部分學進去;推論時把它拿掉。接著模型被膨脹,並在影像權重凍結的情況下訓練運動模組(沿時間使用正弦位置編碼的時間 transformer 區塊);它的輸出投影是零初始化的,並透過殘差連接加入。此外還可以選擇用 MotionLoRA 把運動模組調整到新的動作樣式。由於影像層從未改變,運動模組可以接到由同一個基礎模型衍生的個人化模型上。
  • Open-Sora [22] 從預訓練的影像擴散 transformer 出發,在每個區塊加入時間注意力層;新層的投影初始化為零,以便在一開始保存影像生成能力;加入的時間注意力使參數量加倍。
系統影片訓練時的影像層新的時間部分「什麼都不做」的起點
Make-A-Video訓練每層的 1D 時間卷積與時間注意力恆等初始化
Video LDM凍結時間注意力與 3D 卷積區塊學得的混合係數 α\alpha,α=1\alpha = 1 時略過
AnimateDiff凍結運動模組(時間 transformer)零初始化的輸出投影
Open-Sora訓練每個區塊的時間注意力零初始化的投影

陷阱。 凍結影像層能讓模型與影像生態系(個人化檢查點、LoRA)保持相容,但限制了外觀能調整到適應影片的程度,例如動態模糊或影片特有的取景。全部一起訓練適應得較好,卻失去這種相容性。無論哪一種,逐影格的影像自編碼器仍可能在解碼時造成閃爍,Video LDM 的作法是用影片微調解碼器(見正文)。

參考文獻

  1. U. Singer, A. Polyak, T. Hayes et al., “Make-A-Video: Text-to-Video Generation without Text-Video Data,” arXiv:2209.14792, 2022. arXiv
  2. Z. Yang, J. Teng, W. Zheng et al., “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer,” ICLR, 2025. arXiv
  3. Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” arXiv:2503.20314, 2025. arXiv
  4. A. Blattmann, T. Dockhorn, S. Kulal et al., “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets,” arXiv:2311.15127, 2023. arXiv
  5. J. Xing, M. Xia, Y. Zhang et al., “DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors,” ECCV, LNCS, pp. 399–417, 2024. doi · arXiv
  6. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi and D. J. Fleet, “Video Diffusion Models,” NeurIPS, 2022. proceedings · arXiv
  7. A. Blattmann, R. Rombach, H. Ling et al., “Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models,” CVPR, 2023. CVF
  8. R. Henschel, L. Khachatryan, H. Poghosyan et al., “StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text,” CVPR, 2025. CVF
  9. J. Z. Wu, Y. Ge, X. Wang et al., “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation,” ICCV, 2023. CVF
  10. L. Khachatryan, A. Movsisyan, V. Tadevosyan et al., “Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators,” ICCV, 2023. CVF
  11. Z. Wang, Z. Yuan, X. Wang et al., “MotionCtrl: A Unified and Flexible Motion Controller for Video Generation,” SIGGRAPH 2024 Conference Papers, 2024. doi · arXiv
  12. H. He, Y. Xu, Y. Guo et al., “CameraCtrl: Enabling Camera Control for Text-to-Video Generation,” ICLR, 2025. arXiv
  13. S. Yin, C. Wu, J. Liang et al., “DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory,” arXiv:2308.08089, 2023. arXiv
  14. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. CVF
  15. L. Yu, J. Lezama, N. B. Gundavarapu et al., “Language Model Beats Diffusion — Tokenizer is Key to Visual Generation,” ICLR, 2024. arXiv
  16. W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” arXiv:2412.03603, 2024. arXiv
  17. A. Polyak, A. Zohar, A. Brown et al., “Movie Gen: A Cast of Media Foundation Models,” arXiv:2410.13720, 2024. arXiv
  18. Y. HaCohen, N. Chiprut, B. Brazowski et al., “LTX-Video: Realtime Video Latent Diffusion,” arXiv:2501.00103, 2024. arXiv
  19. D. Kondratyuk, L. Yu, X. Gu et al., “VideoPoet: A Large Language Model for Zero-Shot Video Generation,” ICML, PMLR 235, 2024. PMLR · arXiv
  20. L. Yu, Y. Cheng, K. Sohn et al., “MAGVIT: Masked Generative Video Transformer,” CVPR, 2023. CVF
  21. Y. Guo, C. Yang, A. Rao et al., “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning,” ICLR, 2024. arXiv
  22. Z. Zheng, X. Peng, T. Yang et al., “Open-Sora: Democratizing Efficient Video Production for All,” arXiv:2412.20404, 2024. arXiv
  23. G. Bertasius, H. Wang and L. Torresani, “Is Space-Time Attention All You Need for Video Understanding?,” ICML, PMLR 139, 2021. PMLR
  24. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A Video Vision Transformer,” ICCV, 2021. CVF
  25. X. Ma, Y. Wang, X. Chen et al., “Latte: Latent Diffusion Transformer for Video Generation,” Transactions on Machine Learning Research, 2025. arXiv
  26. B. Lin, Y. Ge, X. Cheng et al., “Open-Sora Plan: Open-Source Large Video Generation Model,” arXiv:2412.00131, 2024. arXiv
  27. J. Ho, W. Chan, C. Saharia et al., “Imagen Video: High Definition Video Generation with Diffusion Models,” arXiv:2210.02303, 2022. arXiv
  28. O. Bar-Tal, H. Chefer, O. Tov et al., “Lumiere: A Space-Time Diffusion Model for Video Generation,” SIGGRAPH Asia 2024 Conference Papers, 2024. doi · arXiv
  29. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” arXiv:2210.02747, 2022. arXiv
  30. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. CVF
  31. S. Yin, C. Wu, H. Yang et al., “NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation,” ACL, pp. 1309–1320, 2023. doi · arXiv
  32. H. Qiu, M. Xia, Y. Zhang et al., “FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling,” ICLR, 2024. arXiv
  33. R. Villegas, M. Babaeizadeh, P.-J. Kindermans et al., “Phenaki: Variable Length Video Generation From Open Domain Textual Description,” arXiv:2210.02399, 2022. arXiv
  34. Z. Huang, Y. He, J. Yu et al., “VBench: Comprehensive Benchmark Suite for Video Generative Models,” CVPR, 2024. CVF
  35. T. Yin, Q. Zhang, R. Zhang et al., “From Slow Bidirectional to Fast Autoregressive Video Diffusion Models,” CVPR, 2025. CVF
  36. B. Chen, D. M. Monso, Y. Du, M. Simchowitz, R. Tedrake and V. Sitzmann, “Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion,” NeurIPS, 2024. proceedings · arXiv
  37. X. Huang, Z. Li, G. He, M. Zhou and E. Shechtman, “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion,” NeurIPS, 2025. arXiv
  38. T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards Accurate Generative Models of Video: A New Metric & Challenges,” arXiv:1812.01717, 2018. arXiv
  39. J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset,” CVPR, 2017. CVF
  40. W. Kay, J. Carreira, K. Simonyan et al., “The Kinetics Human Action Video Dataset,” arXiv:1705.06950, 2017. arXiv
  41. K. Soomro, A. R. Zamir and M. Shah, “UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
  42. S. Ge, A. Mahapatra, G. Parmar, J.-Y. Zhu and J.-B. Huang, “On the Content Bias in Fréchet Video Distance,” CVPR, 2024. CVF
  43. L. Wang, B. Huang, Z. Zhao et al., “VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking,” CVPR, 2023. CVF
  44. Z. Huang, F. Zhang, X. Xu et al., “VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 3, pp. 3268–3285, 2026. doi · arXiv
  45. Y. Liu, X. Cun, X. Liu et al., “EvalCrafter: Benchmarking and Evaluating Large Video Generation Models,” CVPR, 2024. CVF
  46. Z. Zheng, X. Peng, Y. Lou et al., “Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k,” arXiv:2503.09642, 2025. arXiv
  47. M. Bain, A. Nagrani, G. Varol and A. Zisserman, “Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval,” ICCV, 2021. CVF · arXiv
  48. T.-S. Chen, A. Siarohin, W. Menapace et al., “Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers,” CVPR, 2024. CVF
  49. A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies and M. Nießner, “FaceForensics++: Learning to Detect Manipulated Facial Images,” ICCV, 2019. doi · arXiv
  50. P. Fernandez, H. Elsahar, I. Z. Yalniz and A. Mourachko, “Video Seal: Open and Efficient Video Watermarking,” arXiv:2412.09492, 2024. arXiv
  51. A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
  52. Z. Xing, Q. Feng, H. Chen et al., “A Survey on Video Diffusion Models,” arXiv:2310.10647, 2023. arXiv
  53. Y. Liu, K. Zhang, Y. Li et al., “Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models,” arXiv:2402.17177, 2024. arXiv
  54. OpenAI, “What to know about the Sora discontinuation,” official service notice, 2026, accessed 5 November 2026. Notice.