骨架序列上的圖卷積(ST-GCN)
選讀・DLCV 08・影片理解與動作辨識 的補充資料(第 4/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
這則筆記說明 ST-GCN 為骨架式辨識所用的圖卷積。
圖。 一段骨架序列在 張影格中有 個關節。ST-GCN 沿著每張影格內的骨頭連接關節(空間邊),並把每個關節連到下一張影格中的自己(時間邊)[45]。令 為骨頭鄰接矩陣(若關節 與 共用一根骨頭,則 ), 為單位矩陣(自連接)。
一個空間層。 論文採用標準的、以度數正規化的圖卷積;單張影格、只有一個鄰居群組的層是 [45]
其中 儲存每個關節一個特徵向量(起初是它的座標與姿態估計器的信心 [45]), 是所有關節共用的學習權重矩陣, 是以鄰居數正規化的對角度數矩陣。每個關節的輸出,是它自己與鄰居的特徵經度數正規化的平均,再接一個線性映射。
劃分鄰居。 單一個 把所有鄰居一視同仁,就像只有一個權重的卷積核。ST-GCN 把每個關節的鄰居分成幾個子集,並為每個子集配上自己的權重,,其中 [45]。它的空間配置(spatial configuration)劃分使用三個子集:關節自己;比較靠近骨架重心的鄰居(向心);以及比較遠離重心的鄰居(離心)[45]。這讓該層能區分「手肘朝身體移動」與「遠離身體」。一個可學習的遮罩 以逐元素方式乘上每個 ,來衡量個別邊的重要性 [45]。
時間。 時間邊由沿影格軸、對每個關節各做一次的一般 1D 卷積處理(論文使用 9 張影格的時間核),所以一個 ST-GCN 單元就是先對關節做圖卷積、再對影格做時間卷積,與 (2+1)D 同樣是先空間後時間的分解 [45]。
陷阱。 座標必須正規化(論文對輸入骨架做批次正規化),讓共用權重看到所有關節相近的尺度 [45];姿態估計器漏掉的關節必須補上或遮罩;而雙人動作需要約定哪一個身體排在前面。
參考文獻
- Y. Zhu, X. Li, C. Liu, M. Zolfaghari, Y. Xiong, C. Wu, Z. Zhang, J. Tighe et al., “A comprehensive study of deep video action recognition,” technical report, arXiv:2012.06567, 2020. arXiv
- Y. Kong and Y. Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision, vol. 130, no. 5, pp. 1366–1401, 2022. doi
- J. Selva, A. S. Johansen, S. Escalera, K. Nasrollahi, T. B. Moeslund and A. Clapés, “Video transformers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 12922–12943, 2023. doi
- N. Madan, A. Moegelmose, R. Modi, Y. S. Rawat and T. B. Moeslund, “Foundation models for video understanding: A survey,” arXiv:2405.03770, 2024. arXiv
- Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An et al., “Video understanding with large language models: A survey,” arXiv:2312.17432, 2023 (accepted to IEEE Transactions on Circuits and Systems for Video Technology according to the arXiv record). arXiv
- A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proc. IEEE CVPR, pp. 1725–1732, 2014. CVF
- Y. Li, Y. Li and N. Vasconcelos, “RESOUND: Towards action recognition without representation bias,” in Proc. ECCV, 2018. CVF
- L. Sevilla-Lara, S. Zha, Z. Yan, V. Goswami, M. Feiszli and L. Torresani, “Only time can tell: Discovering temporal data for temporal modeling,” in Proc. IEEE WACV, 2021. doi
- R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzyńska, S. Westphal, H. Kim, V. Haenel, I. Fruend et al., “The ‘something something’ video database for learning and evaluating visual common sense,” in Proc. IEEE ICCV, 2017. CVF
- Z. Tong, Y. Song, J. Wang and L. Wang, “VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. NeurIPS · arXiv
- S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. doi
- K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in Advances in Neural Information Processing Systems (NeurIPS), 2014. NeurIPS · arXiv
- L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in Proc. ECCV, pp. 20–36, 2016. doi · arXiv
- J. Carreira and A. Zisserman, “Quo vadis, action recognition? A new model and the Kinetics dataset,” in Proc. IEEE CVPR, 2017. CVF · arXiv
- C. Feichtenhofer, H. Fan, J. Malik and K. He, “SlowFast networks for video recognition,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
- D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in Proc. IEEE ICCV, pp. 4489–4497, 2015. CVF
- W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green et al., “The Kinetics human action video dataset,” arXiv:1705.06950, 2017. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
- T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,” arXiv:1812.01717, 2018. arXiv
- D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- J. Lin, C. Gan and S. Han, “TSM: Temporal shift module for efficient video understanding,” in Proc. IEEE/CVF ICCV, 2019. CVF · arXiv
- X. Wang, R. Girshick, A. Gupta and K. He, “Non-local neural networks,” in Proc. IEEE/CVF CVPR, 2018. CVF
- C. Feichtenhofer, “X3D: Expanding architectures for efficient video recognition,” in Proc. IEEE/CVF CVPR, 2020. CVF
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021. arXiv
- A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić and C. Schmid, “ViViT: A video vision transformer,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- G. Bertasius, H. Wang and L. Torresani, “Is space-time attention all you need for video understanding?,” in Proc. ICML, PMLR 139, 2021. PMLR · arXiv
- H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik and C. Feichtenhofer, “Multiscale vision transformers,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin and H. Hu, “Video Swin Transformer,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin and B. Guo, “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF ICCV, 2021. doi
- K. Li, X. Li, Y. Wang, Y. He, Y. Wang, L. Wang and Y. Qiao, “VideoMamba: State space model for efficient video understanding,” in Proc. ECCV, pp. 237–255, 2024. doi · arXiv
- A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF CVPR, 2022. doi
- L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang and Y. Qiao, “VideoMAE V2: Scaling video masked autoencoders with dual masking,” in Proc. IEEE/CVF CVPR, 2023. CVF · arXiv
- A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran and N. Ballas, “Revisiting feature prediction for learning visual representations from video,” Transactions on Machine Learning Research (TMLR), 2024. OpenReview · arXiv
- M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes et al., “V-JEPA 2: Self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025. arXiv
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, PMLR 139, 2021. PMLR
- H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan and T. Li, “CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning,” Neurocomputing, vol. 508, pp. 293–304, 2022. doi · arXiv
- J. Xu, T. Mei, T. Yao and Y. Rui, “MSR-VTT: A large video description dataset for bridging video and language,” in Proc. IEEE CVPR, 2016. CVF
- Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng et al., “InternVideo2: Scaling foundation models for multimodal video understanding,” in Proc. ECCV, pp. 396–416, 2024. doi · arXiv
- C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou et al., “Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
- T. Lin, X. Liu, X. Li, E. Ding and S. Wen, “BMN: Boundary-matching network for temporal action proposal generation,” in Proc. IEEE/CVF ICCV, 2019. CVF
- C.-L. Zhang, J. Wu and Y. Li, “ActionFormer: Localizing moments of actions with transformers,” in Proc. ECCV, pp. 492–510, 2022. doi · arXiv
- C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici et al., “AVA: A video dataset of spatio-temporally localized atomic visual actions,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- A. Shahroudy, J. Liu, T.-T. Ng and G. Wang, “NTU RGB+D: A large scale dataset for 3D human activity analysis,” in Proc. IEEE CVPR, 2016. doi · arXiv
- S. Yan, Y. Xiong and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Proc. AAAI, vol. 32, no. 1, 2018. doi · arXiv
- P. Weinzaepfel and G. Rogez, “Mimetics: Towards understanding human actions out of context,” International Journal of Computer Vision, vol. 129, no. 5, pp. 1675–1690, 2021. doi · arXiv
- K. Soomro, A. R. Zamir and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” technical report CRCV-TR-12-01, arXiv:1212.0402, 2012. arXiv
- H. Kuehne, H. Jhuang, E. Garrote, T. Poggio and T. Serre, “HMDB: A large video database for human motion recognition,” in Proc. IEEE ICCV, pp. 2556–2563, 2011. doi
- G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in Proc. ECCV, pp. 510–526, 2016. doi · arXiv
- H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar and M. Shah, “The THUMOS challenge on action recognition for videos ‘in the wild’,” Computer Vision and Image Understanding, vol. 155, pp. 1–23, 2017. doi
- F. Caba Heilbron, V. Escorcia, B. Ghanem and J. C. Niebles, “ActivityNet: A large-scale video benchmark for human activity understanding,” in Proc. IEEE CVPR, pp. 961–970, 2015. CVF
- D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro et al., “Scaling egocentric vision: The EPIC-KITCHENS dataset,” in Proc. ECCV, 2018. CVF
- D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro et al., “Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100,” International Journal of Computer Vision, vol. 130, no. 1, 2022. doi · arXiv
- J. Choi, C. Gao, J. C. E. Messou and J.-B. Huang, “Why can’t I dance in the mall? Learning to mitigate scene bias in action recognition,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. NeurIPS · arXiv
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “Going deeper with convolutions,” in Proc. IEEE CVPR, 2015. doi