Graph convolution on a skeleton sequence (ST-GCN)
Optional · supplementary note 4 of 4 for DLCV 08 · Video Understanding and Action Recognition. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
This note explains the graph convolution used by ST-GCN for skeleton-based recognition.
The graph. A skeleton sequence has joints over frames. ST-GCN connects joints along the bones within each frame (spatial edges) and each joint to itself in the next frame (temporal edges) [45]. Let be the bone adjacency matrix ( if joints and share a bone) and the identity (self-connections).
One spatial layer. The paper adopts the standard degree-normalized graph convolution; the single-frame layer with one neighbor group is [45]
where holds one feature vector per joint (initially its coordinates and the pose estimator’s confidence [45]), is the learned weight matrix shared by all joints, and is the diagonal degree matrix that normalizes by the number of neighbors. Each joint’s output is a degree-normalized average of its own and its neighbors’ features, followed by a linear map.
Partitioning the neighbors. A single treats all neighbors alike, which is like a convolution kernel with one weight. ST-GCN splits each joint’s neighbors into subsets and gives each subset its own weights, with [45]. Its spatial configuration partition uses three subsets: the joint itself; the neighbors closer to the skeleton’s center of gravity (centripetal); and those farther away (centrifugal) [45]. This lets the layer distinguish “the elbow moves toward the body” from “away from it”. A learnable mask multiplies each element-wise to weight the importance of individual edges [45].
Time. The temporal edges are handled by an ordinary 1D convolution along the frame axis for each joint (the paper uses a temporal kernel of 9 frames), so one ST-GCN unit is a graph convolution over joints followed by a temporal convolution over frames, the same space-then-time factorization as (2+1)D [45].
Pitfalls. Coordinates must be normalized (the paper applies batch normalization to the input skeletons) so that the shared weights see similar scales for all joints [45]; missing joints from the pose estimator must be filled or masked; and two-person actions need a convention for which body comes first.
References
- Y. Zhu, X. Li, C. Liu, M. Zolfaghari, Y. Xiong, C. Wu, Z. Zhang, J. Tighe et al., “A comprehensive study of deep video action recognition,” technical report, arXiv:2012.06567, 2020. arXiv
- Y. Kong and Y. Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision, vol. 130, no. 5, pp. 1366–1401, 2022. doi
- J. Selva, A. S. Johansen, S. Escalera, K. Nasrollahi, T. B. Moeslund and A. Clapés, “Video transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 12922–12943, 2023. doi
- N. Madan, A. Moegelmose, R. Modi, Y. S. Rawat and T. B. Moeslund, “Foundation models for video understanding: A survey,” arXiv:2405.03770, 2024. arXiv
- Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An et al., “Video understanding with large language models: A survey,” arXiv:2312.17432, 2023 (accepted to IEEE Transactions on Circuits and Systems for Video Technology according to the arXiv record). arXiv
- A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proc. IEEE CVPR, pp. 1725–1732, 2014. CVF
- Y. Li, Y. Li and N. Vasconcelos, “RESOUND: Towards action recognition without representation bias,” in Proc. ECCV, 2018. CVF
- L. Sevilla-Lara, S. Zha, Z. Yan, V. Goswami, M. Feiszli and L. Torresani, “Only time can tell: Discovering temporal data for temporal modeling,” in Proc. IEEE WACV, 2021. doi
- R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzyńska, S. Westphal, H. Kim, V. Haenel, I. Fruend et al., “The ‘something something’ video database for learning and evaluating visual common sense,” in Proc. IEEE ICCV, 2017. CVF
- Z. Tong, Y. Song, J. Wang and L. Wang, “VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. NeurIPS · arXiv
- S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. doi
- K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in Advances in Neural Information Processing Systems (NeurIPS), 2014. NeurIPS · arXiv
- L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in Proc. ECCV, pp. 20–36, 2016. doi · arXiv
- J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the Kinetics dataset,” in Proc. IEEE CVPR, 2017. CVF · arXiv
- C. Feichtenhofer, H. Fan, J. Malik and K. He, “SlowFast networks for video recognition,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
- D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in Proc. IEEE ICCV, pp. 4489–4497, 2015. CVF
- W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green et al., “The Kinetics human action video dataset,” arXiv:1705.06950, 2017. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
- T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,” arXiv:1812.01717, 2018. arXiv
- D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- J. Lin, C. Gan and S. Han, “TSM: Temporal shift module for efficient video understanding,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
- X. Wang, R. Girshick, A. Gupta and K. He, “Non-local neural networks,” in Proc. IEEE/CVF CVPR, 2018. CVF
- C. Feichtenhofer, “X3D: Expanding architectures for efficient video recognition,” in Proc. IEEE/CVF CVPR, 2020. CVF
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021. arXiv
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A video vision transformer,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- G. Bertasius, H. Wang and L. Torresani, “Is space-time attention all you need for video understanding?,” in Proc. ICML, PMLR 139, 2021. PMLR · arXiv
- H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik and C. Feichtenhofer, “Multiscale vision transformers,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin and H. Hu, “Video Swin Transformer,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin and B. Guo, “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF ICCV, 2021. doi
- K. Li, X. Li, Y. Wang, Y. He, Y. Wang, L. Wang and Y. Qiao, “VideoMamba: State space model for efficient video understanding,” in Proc. ECCV, pp. 237–255, 2024. doi · arXiv
- A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF CVPR, 2022. doi
- L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang and Y. Qiao, “VideoMAE V2: Scaling video masked autoencoders with dual masking,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv
- A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran and N. Ballas, “Revisiting feature prediction for learning visual representations from video,” Transactions on Machine Learning Research (TMLR), 2024. OpenReview · arXiv
- M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes et al., “V-JEPA 2: Self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, PMLR 139, 2021. PMLR
- H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan and T. Li, “CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning,” Neurocomputing, vol. 508, pp. 293–304, 2022. doi · arXiv
- J. Xu, T. Mei, T. Yao and Y. Rui, “MSR-VTT: A large video description dataset for bridging video and language,” in Proc. IEEE CVPR, 2016. CVF
- Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng et al., “InternVideo2: Scaling foundation models for multimodal video understanding,” in Proc. ECCV, pp. 396–416, 2024. doi · arXiv
- C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou et al., “Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
- T. Lin, X. Liu, X. Li, E. Ding and S. Wen, “BMN: Boundary-matching network for temporal action proposal generation,” in Proc. IEEE/CVF ICCV, 2019. CVF
- C.-L. Zhang, J. Wu and Y. Li, “ActionFormer: Localizing moments of actions with transformers,” in Proc. ECCV, pp. 492–510, 2022. doi · arXiv
- C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici et al., “AVA: A video dataset of spatio-temporally localized atomic visual actions,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- A. Shahroudy, J. Liu, T.-T. Ng and G. Wang, “NTU RGB+D: A large scale dataset for 3D human activity analysis,” in Proc. IEEE CVPR, 2016. doi · arXiv
- S. Yan, Y. Xiong and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Proc. AAAI, vol. 32, no. 1, 2018. doi · arXiv
- P. Weinzaepfel and G. Rogez, “Mimetics: Towards understanding human actions out of context,” International Journal of Computer Vision, vol. 129, no. 5, pp. 1675–1690, 2021. doi · arXiv
- K. Soomro, A. R. Zamir and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
- H. Kuehne, H. Jhuang, E. Garrote, T. Poggio and T. Serre, “HMDB: A large video database for human motion recognition,” in Proc. IEEE ICCV, pp. 2556–2563, 2011. doi
- G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in Proc. ECCV, pp. 510–526, 2016. doi · arXiv
- H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar and M. Shah, “The THUMOS challenge on action recognition for videos ‘in the wild’,” Computer Vision and Image Understanding, vol. 155, pp. 1–23, 2017. doi
- F. Caba Heilbron, V. Escorcia, B. Ghanem and J. C. Niebles, “ActivityNet: A large-scale video benchmark for human activity understanding,” in Proc. IEEE CVPR, pp. 961–970, 2015. CVF
- D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro et al., “Scaling egocentric vision: The EPIC-KITCHENS dataset,” in Proc. ECCV, 2018. CVF
- D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro et al., “Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100,” International Journal of Computer Vision, vol. 130, no. 1, 2022. doi · arXiv
- J. Choi, C. Gao, J. C. E. Messou and J.-B. Huang, “Why can’t I dance in the mall? Learning to mitigate scene bias in action recognition,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. NeurIPS · arXiv
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “Going deeper with convolutions,” in Proc. IEEE CVPR, 2015. doi