DLCV 05 · Is the argmax of a heatmap the keypoint?
AdvancedDLCV 05 · Human Pose and Keypoint EstimationImage lab
Description
Render a Gaussian heatmap at output stride 1, 2, 4 or 8 for 500 random sub-pixel keypoints and compare three ways to read the position back: the brightest cell, a quarter-cell offset and a Taylor step.
Try this: At the default setting (stride 4, plain argmax) the 500 keypoints are off by 1.58 px on average, and the table of means shows the error doubling with every doubling of the stride: 0.37, 0.78, 1.58 and 3.12 px at strides 1, 2, 4 and 8, close to 0.383 times the stride. Switch the decoding to the quarter offset (about half of that) and then to the Taylor step: at stride 8 it brings 3.12 px down to 0.41 px. Now keep the Taylor step and raise σ from 2 to 4 cells: at stride 8 its error grows from 0.41 to 1.58 px. A wider blob should be easier to read, so why does the same noise hurt more?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
About 1.5 px. The peak can only sit on a grid with 4 px spacing, but the keypoint can fall anywhere between grid points, so the brightest cell is the nearest grid point and is off by up to half a cell on each axis. The mean distance is 0.383 of a cell: 1.58 px in this run (0.37, 0.78, 1.58, 3.12 px at strides 1, 2, 4, 8), before the network makes any mistake. A Taylor step on the log of the heatmap cuts this to 0.21 px.
Lab · DLCV 05 · Is the argmax of a heatmap the keypoint?
Loading image…
Try this: At the default setting (stride 4, plain argmax) the 500 keypoints are off by 1.58 px on average, and the table of means shows the error doubling with every doubling of the stride: 0.37, 0.78, 1.58 and 3.12 px at strides 1, 2, 4 and 8, close to 0.383 times the stride. Switch the decoding to the quarter offset (about half of that) and then to the Taylor step: at stride 8 it brings 3.12 px down to 0.41 px. Now keep the Taylor step and raise σ from 2 to 4 cells: at stride 8 its error grows from 0.41 to 1.58 px. A wider blob should be easier to read, so why does the same noise hurt more?
A toy that mirrors the chapter’s code, with its own random numbers (so the means differ in the second decimal from the chapter’s run). 500 keypoints are drawn uniformly in [16, 240] × [16, 240] pixels of a 256 × 256 input. The heatmap at stride s is exp(−‖p − x/s‖² / (2σ²)) on integer cells p, centred on the exact unrounded position, plus Gaussian noise of standard deviation 0.01 per cell, floored at 10⁻⁶; a cell’s centre is the integer position, as in the chapter, so the brightest cell i maps to s · i. Argmax decodes s · (brightest cell). Quarter offset moves the brightest cell by ¼ cell on each axis toward the brighter neighbour. The Taylor step is one Newton step on the logarithm of the heatmap at the brightest cell, with central differences, kept only if the 2 × 2 Hessian is negative definite. The error is the Euclidean distance to the true position in input pixels. Only the (2R + 1)² cells around the true cell, R = ⌈2σ⌉ + 2, are generated; outside them the Gaussian is below 0.14 (against a peak of at least 0.77 and noise of 0.01), so the rest of the map cannot change the answer. The grid shows one fixed keypoint at (102.6, 78.9), the same noise for every decoder. Real networks add errors that this toy leaves out; the lab isolates the error that comes from the grid alone.
Method and image sources
- J. Tompson, A. Jain, Y. LeCun and C. Bregler, Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation, NeurIPS, 2014 (heatmap targets: a 2D Gaussian at the joint)
- A. Newell, K. Yang and J. Deng, Stacked Hourglass Networks for Human Pose Estimation, ECCV, 2016 (quarter-pixel offset toward the next highest neighbour)
- B. Xiao, H. Wu and Y. Wei, Simple Baselines for Human Pose Estimation and Tracking, ECCV, 2018 (quarter offset from the highest to the second highest response)
- F. Zhang, X. Zhu, H. Dai, M. Ye and C. Zhu, Distribution-Aware Coordinate Representation for Human Pose Estimation, CVPR, 2020 (Taylor-expansion decoding of the heatmap)