The temporal shift as a sparse temporal convolution
Optional · supplementary note 2 of 4 for DLCV 08 · Video Understanding and Action Recognition. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
This note shows why shifting channels in time and then applying a convolution is a temporal convolution in disguise.
The shift. Let be the feature vector at one spatial position in frame , and split the channels into three groups: (the first ), (the next ) and (the rest). The bi-directional shift produces
with zeros where or falls outside the clip. No multiplications are involved, only data movement.
The following convolution. A convolution with weight matrix computes . Writing for the columns of that belong to group , and so on,
This is a temporal convolution of kernel size 3, , whose three kernels are copies of with most columns set to zero: keeps only the columns of group , only those of , and only those of [21]. A full temporal convolution would have free weights; the shifted version has , the cost of the convolution that was already there. The same argument holds for a spatial convolution after the shift, applied position by position.
Code · Python · 27 lines
import numpy as np
rng = np.random.default_rng(0)
T, C = 8, 16
x = rng.standard_normal((T, C)) # one spatial position: T frames, C channels
fold = C // 8 # 1/8 of the channels in each direction
def temporal_shift(x):
out = np.zeros_like(x)
out[:-1, :fold] = x[1:, :fold] # these channels see the next frame
out[1:, fold:2 * fold] = x[:-1, fold:2 * fold] # these channels see the previous frame
out[:, 2 * fold:] = x[:, 2 * fold:] # the rest stay in place
return out
W = rng.standard_normal((C, C)) # the channel mixing of the following 1x1 convolution
y_tsm = temporal_shift(x) @ W.T
# the same result written as a temporal convolution with three sparse kernels
K_next, K_prev, K_now = np.zeros_like(W), np.zeros_like(W), np.zeros_like(W)
K_next[:, :fold] = W[:, :fold]
K_prev[:, fold:2 * fold] = W[:, fold:2 * fold]
K_now[:, 2 * fold:] = W[:, 2 * fold:]
xp = np.vstack([np.zeros((1, C)), x, np.zeros((1, C))]) # zero padding in time
y_conv = xp[2:] @ K_next.T + xp[:-2] @ K_prev.T + xp[1:-1] @ K_now.T
print("max difference:", np.abs(y_tsm - y_conv).max())
print("nonzero kernel weights:", sum((K != 0).sum() for K in (K_next, K_prev, K_now)),
"of", 3 * C * C, "in a full temporal convolution")Output
max difference: 3.552713678800501e-15
nonzero kernel weights: 256 of 768 in a full temporal convolutionDesign choices and pitfalls. (1) Proportion. Too few shifted channels give too little temporal mixing; too many leave each frame with too little of its own information. The paper found 1/4 (1/8 per direction) best for the residual variant [21]. (2) Residual versus in-place. Placing the shift on the residual branch keeps the unshifted features available through the skip connection, which worked better than shifting in place [21]. (3) Online use. For live video, future frames are not available, so only the backward shift (past to present) is used; the shifted channels of the previous frame can be cached, so each new frame costs one 2D forward pass [21]. (4) Receptive field. One shift reaches one frame in each direction; stacking shifted blocks reaches frames, so temporal context grows with depth, as with stacked temporal convolutions.
References
- Y. Zhu, X. Li, C. Liu, M. Zolfaghari, Y. Xiong, C. Wu, Z. Zhang, J. Tighe et al., “A comprehensive study of deep video action recognition,” technical report, arXiv:2012.06567, 2020. arXiv
- Y. Kong and Y. Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision, vol. 130, no. 5, pp. 1366–1401, 2022. doi
- J. Selva, A. S. Johansen, S. Escalera, K. Nasrollahi, T. B. Moeslund and A. Clapés, “Video transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 12922–12943, 2023. doi
- N. Madan, A. Moegelmose, R. Modi, Y. S. Rawat and T. B. Moeslund, “Foundation models for video understanding: A survey,” arXiv:2405.03770, 2024. arXiv
- Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An et al., “Video understanding with large language models: A survey,” arXiv:2312.17432, 2023 (accepted to IEEE Transactions on Circuits and Systems for Video Technology according to the arXiv record). arXiv
- A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proc. IEEE CVPR, pp. 1725–1732, 2014. CVF
- Y. Li, Y. Li and N. Vasconcelos, “RESOUND: Towards action recognition without representation bias,” in Proc. ECCV, 2018. CVF
- L. Sevilla-Lara, S. Zha, Z. Yan, V. Goswami, M. Feiszli and L. Torresani, “Only time can tell: Discovering temporal data for temporal modeling,” in Proc. IEEE WACV, 2021. doi
- R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzyńska, S. Westphal, H. Kim, V. Haenel, I. Fruend et al., “The ‘something something’ video database for learning and evaluating visual common sense,” in Proc. IEEE ICCV, 2017. CVF
- Z. Tong, Y. Song, J. Wang and L. Wang, “VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. NeurIPS · arXiv
- S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. doi
- K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in Advances in Neural Information Processing Systems (NeurIPS), 2014. NeurIPS · arXiv
- L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in Proc. ECCV, pp. 20–36, 2016. doi · arXiv
- J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the Kinetics dataset,” in Proc. IEEE CVPR, 2017. CVF · arXiv
- C. Feichtenhofer, H. Fan, J. Malik and K. He, “SlowFast networks for video recognition,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
- D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in Proc. IEEE ICCV, pp. 4489–4497, 2015. CVF
- W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green et al., “The Kinetics human action video dataset,” arXiv:1705.06950, 2017. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
- T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,” arXiv:1812.01717, 2018. arXiv
- D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- J. Lin, C. Gan and S. Han, “TSM: Temporal shift module for efficient video understanding,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
- X. Wang, R. Girshick, A. Gupta and K. He, “Non-local neural networks,” in Proc. IEEE/CVF CVPR, 2018. CVF
- C. Feichtenhofer, “X3D: Expanding architectures for efficient video recognition,” in Proc. IEEE/CVF CVPR, 2020. CVF
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021. arXiv
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A video vision transformer,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- G. Bertasius, H. Wang and L. Torresani, “Is space-time attention all you need for video understanding?,” in Proc. ICML, PMLR 139, 2021. PMLR · arXiv
- H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik and C. Feichtenhofer, “Multiscale vision transformers,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin and H. Hu, “Video Swin Transformer,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin and B. Guo, “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF ICCV, 2021. doi
- K. Li, X. Li, Y. Wang, Y. He, Y. Wang, L. Wang and Y. Qiao, “VideoMamba: State space model for efficient video understanding,” in Proc. ECCV, pp. 237–255, 2024. doi · arXiv
- A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF CVPR, 2022. doi
- L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang and Y. Qiao, “VideoMAE V2: Scaling video masked autoencoders with dual masking,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv
- A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran and N. Ballas, “Revisiting feature prediction for learning visual representations from video,” Transactions on Machine Learning Research (TMLR), 2024. OpenReview · arXiv
- M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes et al., “V-JEPA 2: Self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, PMLR 139, 2021. PMLR
- H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan and T. Li, “CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning,” Neurocomputing, vol. 508, pp. 293–304, 2022. doi · arXiv
- J. Xu, T. Mei, T. Yao and Y. Rui, “MSR-VTT: A large video description dataset for bridging video and language,” in Proc. IEEE CVPR, 2016. CVF
- Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng et al., “InternVideo2: Scaling foundation models for multimodal video understanding,” in Proc. ECCV, pp. 396–416, 2024. doi · arXiv
- C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou et al., “Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
- T. Lin, X. Liu, X. Li, E. Ding and S. Wen, “BMN: Boundary-matching network for temporal action proposal generation,” in Proc. IEEE/CVF ICCV, 2019. CVF
- C.-L. Zhang, J. Wu and Y. Li, “ActionFormer: Localizing moments of actions with transformers,” in Proc. ECCV, pp. 492–510, 2022. doi · arXiv
- C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici et al., “AVA: A video dataset of spatio-temporally localized atomic visual actions,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- A. Shahroudy, J. Liu, T.-T. Ng and G. Wang, “NTU RGB+D: A large scale dataset for 3D human activity analysis,” in Proc. IEEE CVPR, 2016. doi · arXiv
- S. Yan, Y. Xiong and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Proc. AAAI, vol. 32, no. 1, 2018. doi · arXiv
- P. Weinzaepfel and G. Rogez, “Mimetics: Towards understanding human actions out of context,” International Journal of Computer Vision, vol. 129, no. 5, pp. 1675–1690, 2021. doi · arXiv
- K. Soomro, A. R. Zamir and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
- H. Kuehne, H. Jhuang, E. Garrote, T. Poggio and T. Serre, “HMDB: A large video database for human motion recognition,” in Proc. IEEE ICCV, pp. 2556–2563, 2011. doi
- G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in Proc. ECCV, pp. 510–526, 2016. doi · arXiv
- H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar and M. Shah, “The THUMOS challenge on action recognition for videos ‘in the wild’,” Computer Vision and Image Understanding, vol. 155, pp. 1–23, 2017. doi
- F. Caba Heilbron, V. Escorcia, B. Ghanem and J. C. Niebles, “ActivityNet: A large-scale video benchmark for human activity understanding,” in Proc. IEEE CVPR, pp. 961–970, 2015. CVF
- D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro et al., “Scaling egocentric vision: The EPIC-KITCHENS dataset,” in Proc. ECCV, 2018. CVF
- D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro et al., “Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100,” International Journal of Computer Vision, vol. 130, no. 1, 2022. doi · arXiv
- J. Choi, C. Gao, J. C. E. Messou and J.-B. Huang, “Why can’t I dance in the mall? Learning to mitigate scene bias in action recognition,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. NeurIPS · arXiv
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “Going deeper with convolutions,” in Proc. IEEE CVPR, 2015. doi