完整搜尋頁面 →

SMPL 細說

選讀・DLCV 05・人體姿態與關鍵點估計 的補充資料(第 5/5 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 5

  1. 以 Taylor 步驟做子像素解碼
  2. 座標分類與以似然為基礎的迴歸
  3. 翻轉測試與左右對調的記帳
  4. 深入 OKS:未標註的人、擁擠區域與尺度 s
  5. SMPL 細說

論文導讀 · 3

  1. Simple Baselines(Xiao 等人,2018)
  2. OpenPose:部分親和場(Cao 等人,2017)
  3. 「現代觀點」綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則補充資料說明正文概述的 SMPL 人體模型 [39] 各項的意義。

模板與蒙皮。 SMPL 從藝術家製作的模板網格 Tˉ∈R3N\bar{\mathbf{T}} \in \mathbb{R}^{3N}(休息姿態,有 N=6890N = 6890 個頂點)、有 K=23K = 23 個關節加一個根的骨架,以及混合權重 W∈RN×K\mathcal{W} \in \mathbb{R}^{N \times K} 出發,混合權重說明每個關節的旋轉對每個頂點的牽動有多強。線性混合蒙皮 W(⋅)W(\cdot) 用影響某個頂點的各關節剛體變換的加權平均,來移動每個頂點。單獨使用混合蒙皮會產生眾所周知的瑕疵,例如關節大幅彎曲時,手肘與肩膀會塌陷。

體型混合形狀。 BS(β)=∑n=1∣β∣βnSnB_S(\boldsymbol{\beta}) = \sum_{n=1}^{|\boldsymbol{\beta}|} \beta_n \mathbf{S}_n 加上身體形狀變異的正交主成分 Sn∈R3N\mathbf{S}_n \in \mathbb{R}^{3N} 的線性組合,這些主成分是從許多人的註冊掃描學來的。少數幾個係數就已涵蓋身高、體重與比例的大部分變異。

姿態混合形狀。 BP(θ)B_P(\boldsymbol{\theta}) 加上線性依賴於各關節旋轉矩陣元素(相對於休息姿態計算)的頂點偏移。這些偏移修正混合蒙皮的瑕疵,並建模隨姿態變化的形變。因為休息姿態的旋轉被扣掉了,所以 BPB_P 在休息姿態下恰好為零 [39],使體型與姿態的效果保持分離。

由體型得到關節。 體型不同的人,關節位置也不同,所以 SMPL 用學到的線性迴歸器 J\mathcal{J},從上了體型的網格計算休息姿態的關節位置:J(β)=J(Tˉ+BS(β))J(\boldsymbol{\beta}) = \mathcal{J}\big(\bar{\mathbf{T}} + B_S(\boldsymbol{\beta})\big) [39]。擺好姿態之後,3D 關節(以及任何定義為由頂點迴歸出的關鍵點)都跟著網格走,所以永遠遵守那個人的骨長。

為什麼這對姿態估計有幫助。 預測 (β,θ)(\boldsymbol{\beta}, \boldsymbol{\theta}) 的網路只能產生模型能表達的人體,而且那 72 個姿態數字是旋轉,而不是自由的 3D 位置。用估計的相機把模型的關節投影到影像中,就得到可以和 2D 標籤比較的 2D 關鍵點,這就是 HMR [40] 在沒有 3D 真值的影像上訓練所用的重投影損失。

參考文獻

  1. A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis and M. Bethge, “DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning,” Nature Neuroscience, vol. 21, no. 9, 2018. doi · arXiv
  2. C. Zheng, W. Wu, C. Chen, T. Yang, S. Zhu, J. Shen, N. Kehtarnavaz and M. Shah, “Deep learning-based human pose estimation: A survey,” ACM Computing Surveys, vol. 56, no. 1, 2024. doi · arXiv
  3. Y. Chen, Y. Tian and M. He, “Monocular human pose estimation: A survey of deep learning-based methods,” Computer Vision and Image Understanding, vol. 192, 102897, 2020. doi · arXiv
  4. COCO Consortium, “Keypoint evaluation,” COCO dataset official documentation (accessed 2026-11-01). cocodataset.org
  5. Y. Tian, H. Zhang, Y. Liu and L. Wang, “Recovering 3D human mesh from monocular images: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, 2023. doi · arXiv
  6. A. Toshev and C. Szegedy, “DeepPose: Human pose estimation via deep neural networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. doi · arXiv
  7. J. Tompson, A. Jain, Y. LeCun and C. Bregler, “Joint training of a convolutional network and a graphical model for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), 2014. NeurIPS · arXiv
  8. A. Newell, K. Yang and J. Deng, “Stacked hourglass networks for human pose estimation,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2016. doi · arXiv
  9. B. Xiao, H. Wu and Y. Wei, “Simple baselines for human pose estimation and tracking,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2018. doi · arXiv
  10. F. Zhang, X. Zhu, H. Dai, M. Ye and C. Zhu, “Distribution-aware coordinate representation for human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  11. X. Sun, B. Xiao, F. Wei, S. Liang and Y. Wei, “Integral human pose regression,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2018. doi · arXiv
  12. Y. Li, S. Yang, P. Liu, S. Zhang, Y. Wang, Z. Wang, W. Yang and S.-T. Xia, “SimCC: A simple coordinate classification perspective for human pose estimation,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2022. doi · arXiv
  13. J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu and C. Lu, “Human pose regression with residual log-likelihood estimation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi · arXiv
  14. S.-E. Wei, V. Ramakrishna, T. Kanade and Y. Sheikh, “Convolutional pose machines,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi · arXiv
  15. K. Sun, B. Xiao, D. Liu and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  16. Y. Xu, J. Zhang, Q. Zhang and D. Tao, “ViTPose: Simple vision transformer baselines for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. NeurIPS · arXiv
  17. K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  18. Z. Cao, T. Simon, S.-E. Wei and Y. Sheikh, “Realtime multi-person 2D pose estimation using part affinity fields,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · arXiv
  19. A. Newell, Z. Huang and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. NeurIPS · arXiv
  20. B. Cheng, B. Xiao, J. Wang, H. Shi, T. S. Huang and L. Zhang, “HigherHRNet: Scale-aware representation learning for bottom-up human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  21. Z. Geng, K. Sun, B. Xiao, Z. Zhang and J. Wang, “Bottom-up human pose estimation via disentangled keypoint regression,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi · arXiv
  22. X. Zhou, D. Wang and P. Krähenbühl, “Objects as points,” arXiv:1904.07850, 2019. arXiv
  23. P. Lu, T. Jiang, Y. Li, X. Li, K. Chen and W. Yang, “RTMO: Towards high-performance one-stage real-time multi-person pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi · arXiv
  24. J. Yang, A. Zeng, S. Liu, F. Li, R. Zhang and L. Zhang, “Explicit box detection unifies end-to-end multi-person pose estimation,” in Proc. International Conference on Learning Representations (ICLR), 2023. OpenReview · arXiv
  25. Y. Yang and D. Ramanan, “Articulated human detection with flexible mixtures of parts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 12, 2013. doi
  26. M. Andriluka, L. Pishchulin, P. Gehler and B. Schiele, “2D human pose estimation: New benchmark and state of the art analysis,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. doi · CVF
  27. COCO Consortium, cocoeval.py (keypoint OKS computation and the per-keypoint sigmas), official COCO API source code (accessed 2026-11-01). GitHub
  28. J. Martinez, R. Hossain, J. Romero and J. J. Little, “A simple yet effective baseline for 3d human pose estimation,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  29. M. R. Ronchi and P. Perona, “Benchmarking and error diagnosis in multi-instance pose estimation,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  30. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2014. doi · arXiv
  31. J. Li, C. Wang, H. Zhu, Y. Mao, H.-S. Fang and C. Lu, “CrowdPose: Efficient crowded scenes pose estimation and a new benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  32. S. Jin, L. Xu, J. Xu, C. Wang, W. Liu, C. Qian, W. Ouyang and P. Luo, “Whole-body human pose estimation in the wild,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2020. doi · arXiv
  33. M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall and B. Schiele, “PoseTrack: A benchmark for human pose estimation and tracking,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  34. C. Ionescu, D. Papava, V. Olaru and C. Sminchisescu, “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 7, 2014. doi · project page
  35. H. Yu, Y. Xu, J. Zhang, W. Zhao, Z. Guan and D. Tao, “AP-10K: A benchmark for animal pose estimation in the wild,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. NeurIPS · arXiv
  36. Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei and Y. Sheikh, “OpenPose: Realtime multi-person 2D pose estimation using part affinity fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, 2021. doi · arXiv
  37. D. Pavllo, C. Feichtenhofer, D. Grangier and M. Auli, “3D human pose estimation in video with temporal convolutions and semi-supervised training,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  38. K. Iskakov, E. Burkov, V. Lempitsky and Y. Malkov, “Learnable triangulation of human pose,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019. doi · arXiv
  39. M. Loper, N. Mahmood, J. Romero, G. Pons-Moll and M. J. Black, “SMPL: A skinned multi-person linear model,” ACM Transactions on Graphics (Proc. SIGGRAPH Asia), vol. 34, no. 6, 2015. doi
  40. A. Kanazawa, M. J. Black, D. W. Jacobs and J. Malik, “End-to-end recovery of human shape and pose,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  41. S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa and J. Malik, “Humans in 4D: Reconstructing and tracking humans with transformers,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi · arXiv
  42. R. Khirodkar, T. Bagautdinov, J. Martinez, S. Zhaoen, A. James, P. Selednik, S. Anderson and S. Saito, “Sapiens: Foundation for human vision models,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2024. doi · arXiv
  43. T. Jiang, P. Lu, L. Zhang, N. Ma, R. Han, C. Lyu, Y. Li and K. Chen, “RTMPose: Real-time multi-person pose estimation based on MMPose,” arXiv:2303.07399, 2023. arXiv