完整搜尋頁面 →

以 Taylor 步驟做子像素解碼

選讀・DLCV 05・人體姿態與關鍵點估計 的補充資料(第 1/5 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 5

  1. 以 Taylor 步驟做子像素解碼
  2. 座標分類與以似然為基礎的迴歸
  3. 翻轉測試與左右對調的記帳
  4. 深入 OKS:未標註的人、擁擠區域與尺度 s
  5. SMPL 細說

論文導讀 · 3

  1. Simple Baselines(Xiao 等人,2018)
  2. OpenPose:部分親和場(Cao 等人,2017)
  3. 「現代觀點」綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則補充資料推導 DARK [10] 使用、並在解碼程式碼中實作的子像素修正,也列出它會失效的情況。

設定。 令 H^\hat{H} 為一張預測熱圖,m\mathbf{m} 為值最大的格子。假設在峰值附近,熱圖是一個縮放過的高斯,中心 μ\boldsymbol{\mu} 未知,共變異數為 Σ\Sigma:

H^(p)≈cexp⁡ ⁣(−12(p−μ)⊤Σ−1(p−μ)),\hat{H}(\mathbf{p}) \approx c \exp\!\Big(-\tfrac{1}{2} (\mathbf{p} - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\mathbf{p} - \boldsymbol{\mu})\Big),

其中 c>0c \gt 0 是峰值高度,p\mathbf{p} 是以熱圖單位表示的位置。取對數,D(p)=log⁡H^(p)=log⁡c−12(p−μ)⊤Σ−1(p−μ)D(\mathbf{p}) = \log \hat{H}(\mathbf{p}) = \log c - \tfrac{1}{2} (\mathbf{p} - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\mathbf{p} - \boldsymbol{\mu})。這是 p\mathbf{p} 的精確二次函數,梯度為 D′(p)=−Σ−1(p−μ)\mathbf{D}'(\mathbf{p}) = -\Sigma^{-1}(\mathbf{p} - \boldsymbol{\mu}),Hessian 為常數 D′′=−Σ−1\mathbf{D}'' = -\Sigma^{-1}。

步驟。 在最亮的格子上計算兩者並解出中心:D′(m)=−Σ−1(m−μ)=D′′(m−μ)\mathbf{D}'(\mathbf{m}) = -\Sigma^{-1}(\mathbf{m} - \boldsymbol{\mu}) = \mathbf{D}''(\mathbf{m} - \boldsymbol{\mu}),所以

μ=m−D′′(m)−1 D′(m).\boldsymbol{\mu} = \mathbf{m} - \mathbf{D}''(\mathbf{m})^{-1}\, \mathbf{D}'(\mathbf{m}).

這是朝 DD 的最大值走的一步 Newton 步驟。因為高斯的 DD 恰好是二次函數,一步就是精確的,不需要迭代。DARK 把同樣的結果寫成 DD 在 m\mathbf{m} 附近的二階 Taylor 展開 [10]。SIFT 也用同類的步驟,在高斯差函數上修正關鍵點(見 IP 12 的關鍵點定位補充)。

有限差分。 在格點上,令 Dx,yD_{x,y} 為第 xx 欄、第 yy 列的對數熱圖值,且 (x,y)=m(x, y) = \mathbf{m}:

Dx=12(Dx+1,y−Dx−1,y),Dxx=Dx+1,y−2Dx,y+Dx−1,y,Dy=12(Dx,y+1−Dx,y−1),Dyy=Dx,y+1−2Dx,y+Dx,y−1,Dxy=14(Dx+1,y+1−Dx−1,y+1−Dx+1,y−1+Dx−1,y−1),\begin{aligned} D_x &= \tfrac{1}{2}(D_{x+1,y} - D_{x-1,y}), & D_{xx} &= D_{x+1,y} - 2D_{x,y} + D_{x-1,y}, \\ D_y &= \tfrac{1}{2}(D_{x,y+1} - D_{x,y-1}), & D_{yy} &= D_{x,y+1} - 2D_{x,y} + D_{x,y-1}, \\ D_{xy} &= \tfrac{1}{4}(D_{x+1,y+1} - D_{x-1,y+1} - D_{x+1,y-1} + D_{x-1,y-1}), \end{aligned}

其中 Dx,DyD_x, D_y 組成 D′\mathbf{D}',Dxx,Dxy,DyyD_{xx}, D_{xy}, D_{yy} 組成 2×22 \times 2 矩陣 D′′\mathbf{D}''。對二次函數而言,這些中央差分是精確的,這就是為什麼程式碼中的 Taylor 解碼器在無雜訊的高斯上能還原真實位置。

為什麼對數很重要。 直接對 H^\hat{H}(而不是它的對數)擬合拋物線,也能得到子像素估計,但高斯並不是拋物線,所以除非光斑很寬,否則該估計會偏向格子中心。對數剛好把高斯變成拋物線。

特殊情況與陷阱。

  • 峰值在邊界上。 中央差分需要全部八個鄰格;在圖的邊界上,解碼器必須退回格子中心(或使用單邊規則),如程式碼所做的。
  • 平坦或鞍形的峰值。 如果 D′′\mathbf{D}'' 不是負定的(行列式不為正,或 Dxx≥0D_{xx} \ge 0),這一步可能跳得很遠。程式碼會檢查這一點,並保留格子中心。
  • 多個峰值。 訓練好的網路常產生凹凸不平的圖,在關鍵點附近有好幾個局部最大值。DARK 先用訓練時寬度的高斯核平滑預測熱圖,再重新縮放回原本的最大值 [10],讓局部形狀更接近單一高斯。
  • 接近零的值。 極小或負的預測值取對數時未定義或會爆炸;取對數前要先把熱圖截到一個小的正數下限。
  • 有偏的目標。 如果訓練目標以四捨五入後的關鍵點位置為中心,網路會學著把峰值放在格子中心,任何解碼器都無法找回遺失的小數部分。因此 DARK 把目標高斯以精確、未四捨五入的位置為中心 [10]。

參考文獻

  1. A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis and M. Bethge, “DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning,” Nature Neuroscience, vol. 21, no. 9, 2018. doi · arXiv
  2. C. Zheng, W. Wu, C. Chen, T. Yang, S. Zhu, J. Shen, N. Kehtarnavaz and M. Shah, “Deep learning-based human pose estimation: A survey,” ACM Computing Surveys, vol. 56, no. 1, 2024. doi · arXiv
  3. Y. Chen, Y. Tian and M. He, “Monocular human pose estimation: A survey of deep learning-based methods,” Computer Vision and Image Understanding, vol. 192, 102897, 2020. doi · arXiv
  4. COCO Consortium, “Keypoint evaluation,” COCO dataset official documentation (accessed 2026-11-01). cocodataset.org
  5. Y. Tian, H. Zhang, Y. Liu and L. Wang, “Recovering 3D human mesh from monocular images: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, 2023. doi · arXiv
  6. A. Toshev and C. Szegedy, “DeepPose: Human pose estimation via deep neural networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. doi · arXiv
  7. J. Tompson, A. Jain, Y. LeCun and C. Bregler, “Joint training of a convolutional network and a graphical model for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), 2014. NeurIPS · arXiv
  8. A. Newell, K. Yang and J. Deng, “Stacked hourglass networks for human pose estimation,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2016. doi · arXiv
  9. B. Xiao, H. Wu and Y. Wei, “Simple baselines for human pose estimation and tracking,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2018. doi · arXiv
  10. F. Zhang, X. Zhu, H. Dai, M. Ye and C. Zhu, “Distribution-aware coordinate representation for human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  11. X. Sun, B. Xiao, F. Wei, S. Liang and Y. Wei, “Integral human pose regression,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2018. doi · arXiv
  12. Y. Li, S. Yang, P. Liu, S. Zhang, Y. Wang, Z. Wang, W. Yang and S.-T. Xia, “SimCC: A simple coordinate classification perspective for human pose estimation,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2022. doi · arXiv
  13. J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu and C. Lu, “Human pose regression with residual log-likelihood estimation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi · arXiv
  14. S.-E. Wei, V. Ramakrishna, T. Kanade and Y. Sheikh, “Convolutional pose machines,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi · arXiv
  15. K. Sun, B. Xiao, D. Liu and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  16. Y. Xu, J. Zhang, Q. Zhang and D. Tao, “ViTPose: Simple vision transformer baselines for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. NeurIPS · arXiv
  17. K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  18. Z. Cao, T. Simon, S.-E. Wei and Y. Sheikh, “Realtime multi-person 2D pose estimation using part affinity fields,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · arXiv
  19. A. Newell, Z. Huang and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. NeurIPS · arXiv
  20. B. Cheng, B. Xiao, J. Wang, H. Shi, T. S. Huang and L. Zhang, “HigherHRNet: Scale-aware representation learning for bottom-up human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  21. Z. Geng, K. Sun, B. Xiao, Z. Zhang and J. Wang, “Bottom-up human pose estimation via disentangled keypoint regression,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi · arXiv
  22. X. Zhou, D. Wang and P. Krähenbühl, “Objects as points,” arXiv:1904.07850, 2019. arXiv
  23. P. Lu, T. Jiang, Y. Li, X. Li, K. Chen and W. Yang, “RTMO: Towards high-performance one-stage real-time multi-person pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi · arXiv
  24. J. Yang, A. Zeng, S. Liu, F. Li, R. Zhang and L. Zhang, “Explicit box detection unifies end-to-end multi-person pose estimation,” in Proc. International Conference on Learning Representations (ICLR), 2023. OpenReview · arXiv
  25. Y. Yang and D. Ramanan, “Articulated human detection with flexible mixtures of parts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 12, 2013. doi
  26. M. Andriluka, L. Pishchulin, P. Gehler and B. Schiele, “2D human pose estimation: New benchmark and state of the art analysis,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. doi · CVF
  27. COCO Consortium, cocoeval.py (keypoint OKS computation and the per-keypoint sigmas), official COCO API source code (accessed 2026-11-01). GitHub
  28. J. Martinez, R. Hossain, J. Romero and J. J. Little, “A simple yet effective baseline for 3d human pose estimation,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  29. M. R. Ronchi and P. Perona, “Benchmarking and error diagnosis in multi-instance pose estimation,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  30. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2014. doi · arXiv
  31. J. Li, C. Wang, H. Zhu, Y. Mao, H.-S. Fang and C. Lu, “CrowdPose: Efficient crowded scenes pose estimation and a new benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  32. S. Jin, L. Xu, J. Xu, C. Wang, W. Liu, C. Qian, W. Ouyang and P. Luo, “Whole-body human pose estimation in the wild,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2020. doi · arXiv
  33. M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall and B. Schiele, “PoseTrack: A benchmark for human pose estimation and tracking,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  34. C. Ionescu, D. Papava, V. Olaru and C. Sminchisescu, “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 7, 2014. doi · project page
  35. H. Yu, Y. Xu, J. Zhang, W. Zhao, Z. Guan and D. Tao, “AP-10K: A benchmark for animal pose estimation in the wild,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. NeurIPS · arXiv
  36. Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei and Y. Sheikh, “OpenPose: Realtime multi-person 2D pose estimation using part affinity fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, 2021. doi · arXiv
  37. D. Pavllo, C. Feichtenhofer, D. Grangier and M. Auli, “3D human pose estimation in video with temporal convolutions and semi-supervised training,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  38. K. Iskakov, E. Burkov, V. Lempitsky and Y. Malkov, “Learnable triangulation of human pose,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019. doi · arXiv
  39. M. Loper, N. Mahmood, J. Romero, G. Pons-Moll and M. J. Black, “SMPL: A skinned multi-person linear model,” ACM Transactions on Graphics (Proc. SIGGRAPH Asia), vol. 34, no. 6, 2015. doi
  40. A. Kanazawa, M. J. Black, D. W. Jacobs and J. Malik, “End-to-end recovery of human shape and pose,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  41. S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa and J. Malik, “Humans in 4D: Reconstructing and tracking humans with transformers,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi · arXiv
  42. R. Khirodkar, T. Bagautdinov, J. Martinez, S. Zhaoen, A. James, P. Selednik, S. Anderson and S. Saito, “Sapiens: Foundation for human vision models,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2024. doi · arXiv
  43. T. Jiang, P. Lu, L. Zhang, N. Ma, R. Han, C. Lyu, Y. Li and K. Chen, “RTMPose: Real-time multi-person pose estimation based on MMPose,” arXiv:2303.07399, 2023. arXiv