RoI pooling 與 RoIAlign:不捨入的取樣方式
選讀・DLCV 03・物件偵測:從 R-CNN 到 DETR 的補充資料(第 3/5 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
白話版。 RoI pooling 像是用只能沿著格線剪的剪刀從特徵圖上剪出一塊區域,所以剪痕會貼齊最近的格線。RoIAlign 則剪在框真正所在的位置,並把格線之間的值當成相鄰格子的混合來讀取。
設特徵圖 的步幅(stride)為 (一個格子涵蓋 個影像像素),RoI 以影像像素給定。像素座標 對應到特徵座標 。這裡約定第 格涵蓋 ,其值位於格子中心 。
RoI pooling [9] 先把 RoI 捨入到整數格,切成 個格子(bin),把每個 bin 的邊緣也捨入到整數格,再在每個 bin 內取最大值。每次捨入都會讓邊界移動最多半格,也就是 個影像像素,而且每個 RoI 的誤差都不同 [16]。
RoIAlign [16] 讓所有座標保持實數。每個 bin 內放一組規則排列的取樣點(論文每個 bin 用四個等距的點;下面的程式在 方向用兩個),每個點以雙線性內插讀取。對特徵座標中的點 ,令 、、、、、,則
其中 是第 列、第 行格子的特徵值,、 是該點相對於格子中心 的小數偏移量,四個項分別以權重混合周圍四個格子。bin 的輸出是其取樣點的最大值或平均值;論文使用平均值。論文指出,只要不做任何捨入,結果對取樣位置與點數並不敏感 [16]。梯度會傳到每個取樣點的四個鄰格,所以這一層和其他層一樣可以訓練。
下面的程式用一張「值等於各格中心 座標」的特徵圖,所以正確的池化應該回傳每個 bin 中心的 座標。為了把位置誤差和取最大值的效果分開,兩種方法這裡都用平均。RoI 寬度固定為 64 像素,每一列向右移動 4 像素。
程式碼 · Python · 36 行
import numpy as np
STRIDE = 16
F = np.tile(np.arange(8) + 0.5, (8, 1)) # F[i, j] = x-coordinate of the centre of cell j (in cells)
def roi_pool_avg(F, x1, x2, bins=2):
"""Quantized pooling along x (average). Returns the mean cell-centre read by each bin."""
s, e = int(round(x1 / STRIDE)), int(round(x2 / STRIDE)) # quantize the RoI
edges = s + (e - s) * np.arange(bins + 1) / bins
out = []
for a, b in zip(edges[:-1], edges[1:]):
lo, hi = int(np.floor(a)), int(np.ceil(b)) # quantize the bin edges
out.append(F[0, lo:hi].mean())
return np.array(out)
def bilinear(F, x, y=0.0):
"""Sample F at a continuous point (x, y); cell j has its centre at x = j + 0.5."""
u, v = x - 0.5, y - 0.5
j0, i0 = int(np.floor(u)), int(np.floor(v))
du, dv = u - j0, v - i0
j0c, j1c = np.clip([j0, j0 + 1], 0, F.shape[1] - 1)
i0c, i1c = np.clip([i0, i0 + 1], 0, F.shape[0] - 1)
return ((1 - dv) * ((1 - du) * F[i0c, j0c] + du * F[i0c, j1c])
+ dv * ((1 - du) * F[i1c, j0c] + du * F[i1c, j1c]))
def roi_align_avg(F, x1, x2, bins=2, n=2):
"""RoIAlign along x: n sample points per bin on a regular grid, no rounding, then average."""
a, b = x1 / STRIDE, x2 / STRIDE
w = (b - a) / bins
return np.array([np.mean([bilinear(F, a + k * w + (t + 0.5) * w / n) for t in range(n)])
for k in range(bins)])
for x1, x2 in [(22, 86), (26, 90), (30, 94)]: # same width, moved right by 4 px each step
true_c = (np.arange(2) + 0.5) / 2 * (x2 - x1) / STRIDE + x1 / STRIDE # true bin centres (cells)
print(f"RoI x = [{x1}, {x2}] px true bin centres {np.round(true_c, 3)} "
f"RoIPool {np.round(roi_pool_avg(F, x1, x2), 3)} RoIAlign {np.round(roi_align_avg(F, x1, x2), 3)}")輸出
RoI x = [22, 86] px true bin centres [2.375 4.375] RoIPool [2. 4.] RoIAlign [2.375 4.375]
RoI x = [26, 90] px true bin centres [2.625 4.625] RoIPool [3. 5.] RoIAlign [2.625 4.625]
RoI x = [30, 94] px true bin centres [2.875 4.875] RoIPool [3. 5.] RoIAlign [2.875 4.875]RoIAlign 完全跟著框走,因為雙線性內插能精確重現線性斜坡。RoI pooling 的誤差分別是 0.375、0.375 與 0.125 格,在步幅 16 下相當於 6、6 與 2 像素。RoI 從第一列移到第二列時只移動了 4 像素,輸出卻整整跳了一格(16 像素);再移 4 像素,輸出完全沒有變化。真實的特徵圖不是斜坡,但 RoI pooling 對特徵造成的就是這種階梯式的反應。
如何閱讀論文的證據。 先看 [16] 的 Table 2 (c) 與 (d)。使用步幅為 32 的 ResNet-50-C5 骨幹時,RoIPool 與 RoIAlign 兩列的遮罩 AP 為 23.6 與 30.9(+7.3),AP75 為 21.6 與 32.1(+10.5);框的 AP75 從 26.9 升到 36.4。論文也測試了 RoIWarp:它同樣用雙線性內插,但仍然捨入 RoI,結果與 RoI pooling 相當。因此增益來自移除捨入,而不是來自內插。閱讀這類消融實驗時,要確認被比較的幾列使用相同的骨幹、資料切分與訓練排程(表格說明寫明在 trainval35k 上訓練、在 minival 上測試),也要注意表中只有單一數值;我在文中沒有找到重複實驗或變異數的報告。
常見錯誤。 忘記半格的偏移。如果把格子的值當成位於角落,而 RoI 的運算卻假設它位於中心(或反過來),每個取樣點都會讀到偏離半格的位置,其大小與 RoIAlign 原本要消除的捨入誤差相當。
參考文獻
- Z. Zou, K. Chen, Z. Shi, Y. Guo and J. Ye, “Object detection in 20 years: A survey,” Proceedings of the IEEE, vol. 111, no. 3, 2023. doi
- L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu and M. Pietikäinen, “Deep learning for generic object detection: A survey,” International Journal of Computer Vision, 2020. doi
- M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, 2010. doi
- T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick et al., “Microsoft COCO: Common objects in context,” in Proc. ECCV, 2014. doi
- P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proc. IEEE CVPR, 2001. doi
- N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. IEEE CVPR, 2005. doi
- P. F. Felzenszwalb, R. B. Girshick, D. McAllester and D. Ramanan, “Object detection with discriminatively trained part-based models,” IEEE TPAMI, 2010. doi
- R. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE CVPR, 2014. arXiv
- R. Girshick, “Fast R-CNN,” in Proc. IEEE ICCV, 2015. arXiv
- S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE TPAMI, 2017 (conference version 2015). doi
- H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in Proc. IEEE/CVF CVPR, 2019. arXiv
- Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye and D. Ren, “Distance-IoU loss: Faster and better learning for bounding box regression,” in Proc. AAAI, 2020. arXiv
- N. Bodla, B. Singh, R. Chellappa and L. S. Davis, “Soft-NMS — Improving object detection with one line of code,” in Proc. IEEE ICCV, 2017. arXiv
- T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, “Feature pyramid networks for object detection,” in Proc. IEEE CVPR, 2017. doi
- T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
- K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE ICCV, 2017. doi
- Z. Cai and N. Vasconcelos, “Cascade R-CNN: Delving into high quality object detection,” in Proc. IEEE/CVF CVPR, 2018. doi
- J. Redmon, S. Divvala, R. Girshick and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. IEEE CVPR, 2016. doi
- J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7263–7271, 2017. CVF · arXiv
- J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” tech. report, arXiv:1804.02767, 2018. arXiv
- A. Bochkovskiy, C.-Y. Wang and H.-Y. M. Liao, “YOLOv4: Optimal speed and accuracy of object detection,” arXiv:2004.10934, 2020. arXiv
- C.-Y. Wang, A. Bochkovskiy and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” in Proc. IEEE/CVF CVPR, 2023. doi
- C.-Y. Wang, I-H. Yeh and H.-Y. M. Liao, “YOLOv9: Learning what you want to learn using programmable gradient information,” in Proc. ECCV, 2024. doi
- A. Wang, H. Chen, L. Liu, K. Chen, Z. Lin et al., “YOLOv10: Real-time end-to-end object detection,” in Proc. NeurIPS, 2024. arXiv
- Y. Tian, Q. Ye and D. Doermann, “YOLOv12: Attention-centric real-time object detectors,” arXiv:2502.12524, 2025. arXiv
- W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu and A. C. Berg, “SSD: Single shot multibox detector,” in Proc. ECCV, 2016. arXiv
- H. Law and J. Deng, “CornerNet: Detecting objects as paired keypoints,” in Proc. ECCV, 2018. doi
- X. Zhou, D. Wang and P. Krähenbühl, “Objects as points,” arXiv:1904.07850, 2019. arXiv
- Z. Tian, C. Shen, H. Chen and T. He, “FCOS: Fully convolutional one-stage object detection,” in Proc. IEEE/CVF ICCV, 2019. arXiv
- N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov and S. Zagoruyko, “End-to-end object detection with transformers,” in Proc. ECCV, 2020. doi
- H. W. Kuhn, “The Hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, 1955. doi
- X. Zhu, W. Su, L. Lu, B. Li, X. Wang and J. Dai, “Deformable DETR: Deformable transformers for end-to-end object detection,” in Proc. ICLR, 2021. arXiv
- H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni and H.-Y. Shum, “DINO: DETR with improved denoising anchor boxes for end-to-end object detection,” arXiv:2203.03605, 2022. arXiv
- Y. Zhao, W. Lv, S. Xu, J. Wei, G. Wang et al., “DETRs beat YOLOs on real-time object detection,” in Proc. IEEE/CVF CVPR, 2024. doi
- M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn et al., “Simple open-vocabulary object detection with vision transformers,” in Proc. ECCV, 2022. arXiv
- S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang et al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,” in Proc. ECCV, 2024. doi
- T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang et al., “YOLO-World: Real-time open-vocabulary object detection,” in Proc. IEEE/CVF CVPR, 2024. doi
- N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu et al., “SAM 3: Segment anything with concepts,” arXiv:2511.16719, 2025. arXiv
- G. Cheng, X. Yuan, X. Yao, K. Yan, Q. Zeng et al., “Towards large-scale small object detection: Survey and benchmarks,” IEEE TPAMI, vol. 45, no. 11, 2023. arXiv
- COCO Consortium, “COCO detection evaluation,” official evaluation description (12 metrics, 101-point interpolation). link · official evaluator
- M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes challenge: A retrospective,” International Journal of Computer Vision, vol. 111, 2015. doi
- S. Shao, Z. Li, T. Zhang, C. Peng et al., “Objects365: A large-scale, high-quality dataset for object detection,” in Proc. IEEE/CVF ICCV, 2019. doi
- A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin et al., “The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale,” International Journal of Computer Vision, 2020. arXiv
- A. Gupta, P. Dollár and R. Girshick, “LVIS: A dataset for large vocabulary instance segmentation,” in Proc. IEEE/CVF CVPR, 2019. doi
- T. Shehzadi, K. A. Hashmi, M. Liwicki, D. Stricker and M. Z. Afzal, “Object detection with transformers: A review,” Sensors, vol. 25, no. 19, 2025. doi
- C. Zhu and L. Chen, “A survey on open-vocabulary detection and segmentation: Past, present, and future,” IEEE TPAMI, 2024. doi