Full search page →

Causal 3D convolutions and the 1 + T frame layout

Optional · supplementary note 1 of 4 for DLCV 12A · Video Generation. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 4

  1. Causal 3D convolutions and the 1 + T frame layout
  2. Four ways to start a video model from an image model
  3. Reconstruction guidance for extending a video
  4. Camera poses as Plücker embeddings

Paper guides · 2

  1. Blattmann et al., “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets” (2023)
  2. Ge et al., “On the Content Bias in Fréchet Video Distance” (CVPR 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

This note explains how a causal temporal convolution is padded, why the first frame ends up encoded on its own, and how strided layers produce the "1+T/ft1 + T/f_t" latent length.

Ordinary versus causal padding. Consider one temporal convolution with kernel size ktk_t along the frame axis (the spatial part is unchanged). An ordinary “same” convolution pads ⌊(kt−1)/2⌋\lfloor (k_t-1)/2 \rfloor frames before the clip and ⌊kt/2⌋\lfloor k_t/2 \rfloor after it, so output frame tt depends on input frames t−⌊(kt−1)/2⌋t - \lfloor (k_t-1)/2 \rfloor to t+⌊kt/2⌋t + \lfloor k_t/2 \rfloor, including future frames. A causal convolution pads kt−1k_t - 1 frames before the clip and none after, so output tt depends only on frames t−kt+1t - k_t + 1 to tt [15], [2].

Why the first frame is independent. With causal padding, output frame 0 sees only padding and input frame 0. Stacking causal layers keeps this property, so the latent of the first frame is a function of the first frame alone. That is what lets the same encoder tokenize a single image [15].

Strided downsampling. A causal convolution with stride ss maps 1+s t1 + s\,t input frames to 1+t1 + t outputs; the decoder upsamples by ss and drops the first s−1s - 1 resulting frames to map 1+t1 + t back to 1+s t1 + s\,t [15]. Two such stages with s=2s = 2 give ft=4f_t = 4: 17 frames become 5 latent frames, 81 become 21.

Chunked encoding of long videos. Because a causal layer needs only the last kt−1k_t - 1 frames of context, a long video can be encoded or decoded chunk by chunk while caching those frames. Wan’s VAE processes one chunk per latent frame (at most 4 frames) and keeps, for a kernel of size 3, the features of the last two frames of the previous chunk, which supports encoding and decoding of arbitrarily long videos with bounded memory [3]. CogVideoX uses the same property to split long videos across devices, each device sending only kt−1k_t - 1 frames to the next [2].

A tested check. The snippet implements one causal temporal convolution with kernel 4 and stride 4 on 17 “frames” (scalars, for clarity) and confirms the output length and that no output depends on a later frame.

Code · Python · 16 lines
import numpy as np

def causal_conv1d(x, w, stride):
    """Causal temporal convolution: pad len(w) - 1 zeros in front, none behind."""
    k = len(w)
    xp = np.concatenate([np.zeros(k - 1), x])
    return np.array([xp[i * stride:i * stride + k] @ w for i in range((len(x) - 1) // stride + 1)])

rng = np.random.default_rng(0)
x, w = rng.normal(size=17), rng.normal(size=4)
y = causal_conv1d(x, w, stride=4)
print("17 frames ->", len(y), "latent frames")
for j in [0, 1, 4, 5, 16]:                     # perturb one frame, see which outputs change
    x2 = x.copy(); x2[j] += 1.0
    changed = np.flatnonzero(~np.isclose(causal_conv1d(x2, w, 4), y))
    print(f"frame {j:2d} changes latent frames {changed.tolist()}")

It prints:

Code · 6 lines
17 frames -> 5 latent frames
frame  0 changes latent frames [0]
frame  1 changes latent frames [1]
frame  4 changes latent frames [1]
frame  5 changes latent frames [2]
frame 16 changes latent frames [4]

Frame 0 influences only latent frame 0, and each later group of four frames influences only its own latent frame, as in Figure 1. Real encoders stack several layers with overlapping kernels, so a latent frame also sees some earlier groups, but never later ones.

Pitfalls. Frame counts that do not fit 1+s t1 + s\,t must be padded or trimmed; Open-Sora reports that training its autoencoder only on 17-frame clips caused blurriness at other lengths, which mixed-length training fixed [22]. Normalization layers that pool statistics over time break causality; Wan replaced group normalization with RMS normalization for this reason [3].

References

  1. U. Singer, A. Polyak, T. Hayes et al., “Make-A-Video: Text-to-Video Generation without Text-Video Data,” arXiv:2209.14792, 2022. arXiv
  2. Z. Yang, J. Teng, W. Zheng et al., “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer,” ICLR, 2025. arXiv
  3. Team Wan, A. Wang, B. Ai, B. Wen et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” arXiv:2503.20314, 2025. arXiv
  4. A. Blattmann, T. Dockhorn, S. Kulal et al., “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets,” arXiv:2311.15127, 2023. arXiv
  5. J. Xing, M. Xia, Y. Zhang et al., “DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors,” ECCV, LNCS, pp. 399–417, 2024. doi · arXiv
  6. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi and D. J. Fleet, “Video Diffusion Models,” NeurIPS, 2022. proceedings · arXiv
  7. A. Blattmann, R. Rombach, H. Ling et al., “Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models,” CVPR, 2023. CVF
  8. R. Henschel, L. Khachatryan, H. Poghosyan et al., “StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text,” CVPR, 2025. CVF
  9. J. Z. Wu, Y. Ge, X. Wang et al., “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation,” ICCV, 2023. CVF
  10. L. Khachatryan, A. Movsisyan, V. Tadevosyan et al., “Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators,” ICCV, 2023. CVF
  11. Z. Wang, Z. Yuan, X. Wang et al., “MotionCtrl: A Unified and Flexible Motion Controller for Video Generation,” SIGGRAPH 2024 Conference Papers, 2024. doi · arXiv
  12. H. He, Y. Xu, Y. Guo et al., “CameraCtrl: Enabling Camera Control for Text-to-Video Generation,” ICLR, 2025. arXiv
  13. S. Yin, C. Wu, J. Liang et al., “DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory,” arXiv:2308.08089, 2023. arXiv
  14. R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. CVF
  15. L. Yu, J. Lezama, N. B. Gundavarapu et al., “Language Model Beats Diffusion — Tokenizer is Key to Visual Generation,” ICLR, 2024. arXiv
  16. W. Kong, Q. Tian, Z. Zhang et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models,” arXiv:2412.03603, 2024. arXiv
  17. A. Polyak, A. Zohar, A. Brown et al., “Movie Gen: A Cast of Media Foundation Models,” arXiv:2410.13720, 2024. arXiv
  18. Y. HaCohen, N. Chiprut, B. Brazowski et al., “LTX-Video: Realtime Video Latent Diffusion,” arXiv:2501.00103, 2024. arXiv
  19. D. Kondratyuk, L. Yu, X. Gu et al., “VideoPoet: A Large Language Model for Zero-Shot Video Generation,” ICML, PMLR 235, 2024. PMLR · arXiv
  20. L. Yu, Y. Cheng, K. Sohn et al., “MAGVIT: Masked Generative Video Transformer,” CVPR, 2023. CVF
  21. Y. Guo, C. Yang, A. Rao et al., “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning,” ICLR, 2024. arXiv
  22. Z. Zheng, X. Peng, T. Yang et al., “Open-Sora: Democratizing Efficient Video Production for All,” arXiv:2412.20404, 2024. arXiv
  23. G. Bertasius, H. Wang and L. Torresani, “Is Space-Time Attention All You Need for Video Understanding?,” ICML, PMLR 139, 2021. PMLR
  24. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A Video Vision Transformer,” ICCV, 2021. CVF
  25. X. Ma, Y. Wang, X. Chen et al., “Latte: Latent Diffusion Transformer for Video Generation,” Transactions on Machine Learning Research, 2025. arXiv
  26. B. Lin, Y. Ge, X. Cheng et al., “Open-Sora Plan: Open-Source Large Video Generation Model,” arXiv:2412.00131, 2024. arXiv
  27. J. Ho, W. Chan, C. Saharia et al., “Imagen Video: High Definition Video Generation with Diffusion Models,” arXiv:2210.02303, 2022. arXiv
  28. O. Bar-Tal, H. Chefer, O. Tov et al., “Lumiere: A Space-Time Diffusion Model for Video Generation,” SIGGRAPH Asia 2024 Conference Papers, 2024. doi · arXiv
  29. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le, “Flow Matching for Generative Modeling,” arXiv:2210.02747, 2022. arXiv
  30. W. Peebles and S. Xie, “Scalable Diffusion Models with Transformers,” ICCV, 2023. CVF
  31. S. Yin, C. Wu, H. Yang et al., “NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation,” ACL, pp. 1309–1320, 2023. doi · arXiv
  32. H. Qiu, M. Xia, Y. Zhang et al., “FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling,” ICLR, 2024. arXiv
  33. R. Villegas, M. Babaeizadeh, P.-J. Kindermans et al., “Phenaki: Variable Length Video Generation From Open Domain Textual Description,” arXiv:2210.02399, 2022. arXiv
  34. Z. Huang, Y. He, J. Yu et al., “VBench: Comprehensive Benchmark Suite for Video Generative Models,” CVPR, 2024. CVF
  35. T. Yin, Q. Zhang, R. Zhang et al., “From Slow Bidirectional to Fast Autoregressive Video Diffusion Models,” CVPR, 2025. CVF
  36. B. Chen, D. M. Monso, Y. Du, M. Simchowitz, R. Tedrake and V. Sitzmann, “Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion,” NeurIPS, 2024. proceedings · arXiv
  37. X. Huang, Z. Li, G. He, M. Zhou and E. Shechtman, “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion,” NeurIPS, 2025. arXiv
  38. T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards Accurate Generative Models of Video: A New Metric & Challenges,” arXiv:1812.01717, 2018. arXiv
  39. J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset,” CVPR, 2017. CVF
  40. W. Kay, J. Carreira, K. Simonyan et al., “The Kinetics Human Action Video Dataset,” arXiv:1705.06950, 2017. arXiv
  41. K. Soomro, A. R. Zamir and M. Shah, “UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
  42. S. Ge, A. Mahapatra, G. Parmar, J.-Y. Zhu and J.-B. Huang, “On the Content Bias in Fréchet Video Distance,” CVPR, 2024. CVF
  43. L. Wang, B. Huang, Z. Zhao et al., “VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking,” CVPR, 2023. CVF
  44. Z. Huang, F. Zhang, X. Xu et al., “VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 3, pp. 3268–3285, 2026. doi · arXiv
  45. Y. Liu, X. Cun, X. Liu et al., “EvalCrafter: Benchmarking and Evaluating Large Video Generation Models,” CVPR, 2024. CVF
  46. Z. Zheng, X. Peng, Y. Lou et al., “Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k,” arXiv:2503.09642, 2025. arXiv
  47. M. Bain, A. Nagrani, G. Varol and A. Zisserman, “Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval,” ICCV, 2021. CVF · arXiv
  48. T.-S. Chen, A. Siarohin, W. Menapace et al., “Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers,” CVPR, 2024. CVF
  49. A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies and M. Nießner, “FaceForensics++: Learning to Detect Manipulated Facial Images,” ICCV, 2019. doi · arXiv
  50. P. Fernandez, H. Elsahar, I. Z. Yalniz and A. Mourachko, “Video Seal: Open and Efficient Video Watermarking,” arXiv:2412.09492, 2024. arXiv
  51. A. Melnik, M. Ljubljanac, C. Lu, Q. Yan, W. Ren and H. Ritter, “Video Diffusion Models: A Survey,” Transactions on Machine Learning Research, 2024. arXiv
  52. Z. Xing, Q. Feng, H. Chen et al., “A Survey on Video Diffusion Models,” arXiv:2310.10647, 2023. arXiv
  53. Y. Liu, K. Zhang, Y. Li et al., “Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models,” arXiv:2402.17177, 2024. arXiv
  54. OpenAI, “What to know about the Sora discontinuation,” official service notice, 2026, accessed 5 November 2026. Notice.