Full search page →

Sub-pixel decoding by a Taylor step

Optional · supplementary note 1 of 5 for DLCV 05 · Human Pose and Keypoint Estimation. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 5

  1. Sub-pixel decoding by a Taylor step
  2. Coordinate classification and likelihood-based regression
  3. Flip testing and left–right bookkeeping
  4. Inside OKS: unlabeled people, crowds and the scale s
  5. SMPL in more detail

Paper guides · 3

  1. Simple Baselines (Xiao et al., 2018)
  2. OpenPose: part affinity fields (Cao et al., 2017)
  3. Guide to the Modern view surveys

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

This note derives the sub-pixel refinement used by DARK [10] and implemented in the decoding code, and lists the cases where it fails.

Setup. Let H^\hat{H} be one predicted heatmap and m\mathbf{m} the cell with the largest value. Assume that near its peak the heatmap is a scaled Gaussian with an unknown centre μ\boldsymbol{\mu} and covariance Σ\Sigma:

H^(p)≈cexp⁡ ⁣(−12(p−μ)⊤Σ−1(p−μ)),\hat{H}(\mathbf{p}) \approx c \exp\!\Big(-\tfrac{1}{2} (\mathbf{p} - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\mathbf{p} - \boldsymbol{\mu})\Big),

where c>0c \gt 0 is the peak height and p\mathbf{p} is a position in heatmap units. Take the logarithm, D(p)=log⁡H^(p)=log⁡c−12(p−μ)⊤Σ−1(p−μ)D(\mathbf{p}) = \log \hat{H}(\mathbf{p}) = \log c - \tfrac{1}{2} (\mathbf{p} - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\mathbf{p} - \boldsymbol{\mu}). This is an exact quadratic in p\mathbf{p}, with gradient D′(p)=−Σ−1(p−μ)\mathbf{D}'(\mathbf{p}) = -\Sigma^{-1}(\mathbf{p} - \boldsymbol{\mu}) and constant Hessian D′′=−Σ−1\mathbf{D}'' = -\Sigma^{-1}.

The step. Evaluate both at the brightest cell and solve for the centre: D′(m)=−Σ−1(m−μ)=D′′(m−μ)\mathbf{D}'(\mathbf{m}) = -\Sigma^{-1}(\mathbf{m} - \boldsymbol{\mu}) = \mathbf{D}''(\mathbf{m} - \boldsymbol{\mu}), so

μ=m−D′′(m)−1 D′(m).\boldsymbol{\mu} = \mathbf{m} - \mathbf{D}''(\mathbf{m})^{-1}\, \mathbf{D}'(\mathbf{m}).

This is one Newton step toward the maximum of DD. Because DD is exactly quadratic for a Gaussian, one step is exact; there is no need to iterate. DARK writes the same result as a second-order Taylor expansion of DD around m\mathbf{m} [10]. SIFT refines its keypoints with the same kind of step on the difference-of-Gaussians function (see the keypoint localization note of IP 12).

Finite differences. On the grid, with Dx,yD_{x,y} the log-heatmap at column xx and row yy and (x,y)=m(x, y) = \mathbf{m}:

Dx=12(Dx+1,y−Dx−1,y),Dxx=Dx+1,y−2Dx,y+Dx−1,y,Dy=12(Dx,y+1−Dx,y−1),Dyy=Dx,y+1−2Dx,y+Dx,y−1,Dxy=14(Dx+1,y+1−Dx−1,y+1−Dx+1,y−1+Dx−1,y−1),\begin{aligned} D_x &= \tfrac{1}{2}(D_{x+1,y} - D_{x-1,y}), & D_{xx} &= D_{x+1,y} - 2D_{x,y} + D_{x-1,y}, \\ D_y &= \tfrac{1}{2}(D_{x,y+1} - D_{x,y-1}), & D_{yy} &= D_{x,y+1} - 2D_{x,y} + D_{x,y-1}, \\ D_{xy} &= \tfrac{1}{4}(D_{x+1,y+1} - D_{x-1,y+1} - D_{x+1,y-1} + D_{x-1,y-1}), \end{aligned}

where Dx,DyD_x, D_y form D′\mathbf{D}' and Dxx,Dxy,DyyD_{xx}, D_{xy}, D_{yy} form the 2×22 \times 2 matrix D′′\mathbf{D}''. For a quadratic function these central differences are exact, which is why the code’s Taylor decoder recovers the true position on a noise-free Gaussian.

Why the logarithm matters. Fitting a parabola to H^\hat{H} itself (not its log) also gives a sub-pixel estimate, but a Gaussian is not a parabola, so that estimate is biased toward the cell centre unless the blob is wide. The log turns the Gaussian into a parabola exactly.

Edge cases and pitfalls.

  • Peak on the border. The central differences need all eight neighbours; at the map border the decoder must fall back to the cell centre (or a one-sided rule), as the code does.
  • Flat or saddle-shaped peaks. If D′′\mathbf{D}'' is not negative definite (determinant not positive, or Dxx≥0D_{xx} \ge 0), the step can jump far away. The code checks this and keeps the cell centre.
  • Multiple peaks. Trained networks often produce lumpy maps with several local maxima near the keypoint. DARK first smooths the predicted heatmap with a Gaussian kernel of the training width and rescales it to the original maximum [10], which makes the local shape closer to a single Gaussian.
  • Values near zero. The log of tiny or negative predicted values is undefined or huge; clip the heatmap to a small positive floor before taking the log.
  • Biased targets. If the training targets were centred on the rounded keypoint position, the network learns to put the peak on the cell centre, and no decoder can recover the lost fraction. DARK therefore centres the target Gaussian on the exact, unrounded position [10].

References

  1. A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis and M. Bethge, “DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning,” Nature Neuroscience, vol. 21, no. 9, 2018. doi · arXiv
  2. C. Zheng, W. Wu, C. Chen, T. Yang, S. Zhu, J. Shen, N. Kehtarnavaz and M. Shah, “Deep learning-based human pose estimation: A survey,” ACM Computing Surveys, vol. 56, no. 1, 2024. doi · arXiv
  3. Y. Chen, Y. Tian and M. He, “Monocular human pose estimation: A survey of deep learning-based methods,” Computer Vision and Image Understanding, vol. 192, 102897, 2020. doi · arXiv
  4. COCO Consortium, “Keypoint evaluation,” COCO dataset official documentation (accessed 2026-11-01). cocodataset.org
  5. Y. Tian, H. Zhang, Y. Liu and L. Wang, “Recovering 3D human mesh from monocular images: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, 2023. doi · arXiv
  6. A. Toshev and C. Szegedy, “DeepPose: Human pose estimation via deep neural networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. doi · arXiv
  7. J. Tompson, A. Jain, Y. LeCun and C. Bregler, “Joint training of a convolutional network and a graphical model for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), 2014. NeurIPS · arXiv
  8. A. Newell, K. Yang and J. Deng, “Stacked hourglass networks for human pose estimation,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2016. doi · arXiv
  9. B. Xiao, H. Wu and Y. Wei, “Simple baselines for human pose estimation and tracking,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2018. doi · arXiv
  10. F. Zhang, X. Zhu, H. Dai, M. Ye and C. Zhu, “Distribution-aware coordinate representation for human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  11. X. Sun, B. Xiao, F. Wei, S. Liang and Y. Wei, “Integral human pose regression,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2018. doi · arXiv
  12. Y. Li, S. Yang, P. Liu, S. Zhang, Y. Wang, Z. Wang, W. Yang and S.-T. Xia, “SimCC: A simple coordinate classification perspective for human pose estimation,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2022. doi · arXiv
  13. J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu and C. Lu, “Human pose regression with residual log-likelihood estimation,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi · arXiv
  14. S.-E. Wei, V. Ramakrishna, T. Kanade and Y. Sheikh, “Convolutional pose machines,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi · arXiv
  15. K. Sun, B. Xiao, D. Liu and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  16. Y. Xu, J. Zhang, Q. Zhang and D. Tao, “ViTPose: Simple vision transformer baselines for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. NeurIPS · arXiv
  17. K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  18. Z. Cao, T. Simon, S.-E. Wei and Y. Sheikh, “Realtime multi-person 2D pose estimation using part affinity fields,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · arXiv
  19. A. Newell, Z. Huang and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. NeurIPS · arXiv
  20. B. Cheng, B. Xiao, J. Wang, H. Shi, T. S. Huang and L. Zhang, “HigherHRNet: Scale-aware representation learning for bottom-up human pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  21. Z. Geng, K. Sun, B. Xiao, Z. Zhang and J. Wang, “Bottom-up human pose estimation via disentangled keypoint regression,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi · arXiv
  22. X. Zhou, D. Wang and P. Krähenbühl, “Objects as points,” arXiv:1904.07850, 2019. arXiv
  23. P. Lu, T. Jiang, Y. Li, X. Li, K. Chen and W. Yang, “RTMO: Towards high-performance one-stage real-time multi-person pose estimation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi · arXiv
  24. J. Yang, A. Zeng, S. Liu, F. Li, R. Zhang and L. Zhang, “Explicit box detection unifies end-to-end multi-person pose estimation,” in Proc. International Conference on Learning Representations (ICLR), 2023. OpenReview · arXiv
  25. Y. Yang and D. Ramanan, “Articulated human detection with flexible mixtures of parts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 12, 2013. doi
  26. M. Andriluka, L. Pishchulin, P. Gehler and B. Schiele, “2D human pose estimation: New benchmark and state of the art analysis,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. doi · CVF
  27. COCO Consortium, cocoeval.py (keypoint OKS computation and the per-keypoint sigmas), official COCO API source code (accessed 2026-11-01). GitHub
  28. J. Martinez, R. Hossain, J. Romero and J. J. Little, “A simple yet effective baseline for 3d human pose estimation,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  29. M. R. Ronchi and P. Perona, “Benchmarking and error diagnosis in multi-instance pose estimation,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  30. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2014. doi · arXiv
  31. J. Li, C. Wang, H. Zhu, Y. Mao, H.-S. Fang and C. Lu, “CrowdPose: Efficient crowded scenes pose estimation and a new benchmark,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  32. S. Jin, L. Xu, J. Xu, C. Wang, W. Liu, C. Qian, W. Ouyang and P. Luo, “Whole-body human pose estimation in the wild,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2020. doi · arXiv
  33. M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall and B. Schiele, “PoseTrack: A benchmark for human pose estimation and tracking,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  34. C. Ionescu, D. Papava, V. Olaru and C. Sminchisescu, “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 7, 2014. doi · project page
  35. H. Yu, Y. Xu, J. Zhang, W. Zhao, Z. Guan and D. Tao, “AP-10K: A benchmark for animal pose estimation in the wild,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021. NeurIPS · arXiv
  36. Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei and Y. Sheikh, “OpenPose: Realtime multi-person 2D pose estimation using part affinity fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, 2021. doi · arXiv
  37. D. Pavllo, C. Feichtenhofer, D. Grangier and M. Auli, “3D human pose estimation in video with temporal convolutions and semi-supervised training,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  38. K. Iskakov, E. Burkov, V. Lempitsky and Y. Malkov, “Learnable triangulation of human pose,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2019. doi · arXiv
  39. M. Loper, N. Mahmood, J. Romero, G. Pons-Moll and M. J. Black, “SMPL: A skinned multi-person linear model,” ACM Transactions on Graphics (Proc. SIGGRAPH Asia), vol. 34, no. 6, 2015. doi
  40. A. Kanazawa, M. J. Black, D. W. Jacobs and J. Malik, “End-to-end recovery of human shape and pose,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  41. S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa and J. Malik, “Humans in 4D: Reconstructing and tracking humans with transformers,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi · arXiv
  42. R. Khirodkar, T. Bagautdinov, J. Martinez, S. Zhaoen, A. James, P. Selednik, S. Anderson and S. Saito, “Sapiens: Foundation for human vision models,” in Proc. European Conference on Computer Vision (ECCV), LNCS, 2024. doi · arXiv
  43. T. Jiang, P. Lu, L. Zhang, N. Ma, R. Han, C. Lyu, Y. Li and K. Chen, “RTMPose: Real-time multi-person pose estimation based on MMPose,” arXiv:2303.07399, 2023. arXiv