完整搜尋頁面 →

RoI pooling 與 RoIAlign:不捨入的取樣方式

選讀・DLCV 03・物件偵測:從 R-CNN 到 DETR 的補充資料(第 3/5 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 5

  1. 傳統偵測器細說:Viola–Jones、HOG 與 DPM
  2. 以 k-means 決定錨框形狀:用 IoU 當距離
  3. RoI pooling 與 RoIAlign:不捨入的取樣方式
  4. YOLO 家族逐版介紹
  5. DETR 後繼者細說

論文導讀 · 1

  1. 現代觀點綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

白話版。 RoI pooling 像是用只能沿著格線剪的剪刀從特徵圖上剪出一塊區域,所以剪痕會貼齊最近的格線。RoIAlign 則剪在框真正所在的位置,並把格線之間的值當成相鄰格子的混合來讀取。

設特徵圖 FF 的步幅(stride)為 ss(一個格子涵蓋 s×ss \times s 個影像像素),RoI 以影像像素給定。像素座標 xx 對應到特徵座標 x/sx/s。這裡約定第 jj 格涵蓋 [j,j+1)[j, j+1),其值位於格子中心 j+12j + \tfrac12。

RoI pooling [9] 先把 RoI 捨入到整數格,切成 H×WH \times W 個格子(bin),把每個 bin 的邊緣也捨入到整數格,再在每個 bin 內取最大值。每次捨入都會讓邊界移動最多半格,也就是 s/2s/2 個影像像素,而且每個 RoI 的誤差都不同 [16]。

RoIAlign [16] 讓所有座標保持實數。每個 bin 內放一組規則排列的取樣點(論文每個 bin 用四個等距的點;下面的程式在 xx 方向用兩個),每個點以雙線性內插讀取。對特徵座標中的點 (x,y)(x, y),令 u=x−12u = x - \tfrac12、v=y−12v = y - \tfrac12、j=⌊u⌋j = \lfloor u \rfloor、i=⌊v⌋i = \lfloor v \rfloor、a=u−ja = u - j、b=v−ib = v - i,則

F(x,y)=(1−b)[(1−a)Fi,j+aFi,j+1]+b[(1−a)Fi+1,j+aFi+1,j+1],F(x, y) = (1 - b)\big[(1 - a) F_{i,j} + a F_{i,j+1}\big] + b\big[(1 - a) F_{i+1,j} + a F_{i+1,j+1}\big],

其中 Fi,jF_{i,j} 是第 ii 列、第 jj 行格子的特徵值,aa、bb 是該點相對於格子中心 (j,i)(j, i) 的小數偏移量,四個項分別以權重混合周圍四個格子。bin 的輸出是其取樣點的最大值或平均值;論文使用平均值。論文指出,只要不做任何捨入,結果對取樣位置與點數並不敏感 [16]。梯度會傳到每個取樣點的四個鄰格,所以這一層和其他層一樣可以訓練。

下面的程式用一張「值等於各格中心 xx 座標」的特徵圖,所以正確的池化應該回傳每個 bin 中心的 xx 座標。為了把位置誤差和取最大值的效果分開,兩種方法這裡都用平均。RoI 寬度固定為 64 像素,每一列向右移動 4 像素。

程式碼 · Python · 36 行
import numpy as np

STRIDE = 16
F = np.tile(np.arange(8) + 0.5, (8, 1))   # F[i, j] = x-coordinate of the centre of cell j (in cells)

def roi_pool_avg(F, x1, x2, bins=2):
    """Quantized pooling along x (average). Returns the mean cell-centre read by each bin."""
    s, e = int(round(x1 / STRIDE)), int(round(x2 / STRIDE))     # quantize the RoI
    edges = s + (e - s) * np.arange(bins + 1) / bins
    out = []
    for a, b in zip(edges[:-1], edges[1:]):
        lo, hi = int(np.floor(a)), int(np.ceil(b))              # quantize the bin edges
        out.append(F[0, lo:hi].mean())
    return np.array(out)

def bilinear(F, x, y=0.0):
    """Sample F at a continuous point (x, y); cell j has its centre at x = j + 0.5."""
    u, v = x - 0.5, y - 0.5
    j0, i0 = int(np.floor(u)), int(np.floor(v))
    du, dv = u - j0, v - i0
    j0c, j1c = np.clip([j0, j0 + 1], 0, F.shape[1] - 1)
    i0c, i1c = np.clip([i0, i0 + 1], 0, F.shape[0] - 1)
    return ((1 - dv) * ((1 - du) * F[i0c, j0c] + du * F[i0c, j1c])
            + dv * ((1 - du) * F[i1c, j0c] + du * F[i1c, j1c]))

def roi_align_avg(F, x1, x2, bins=2, n=2):
    """RoIAlign along x: n sample points per bin on a regular grid, no rounding, then average."""
    a, b = x1 / STRIDE, x2 / STRIDE
    w = (b - a) / bins
    return np.array([np.mean([bilinear(F, a + k * w + (t + 0.5) * w / n) for t in range(n)])
                     for k in range(bins)])

for x1, x2 in [(22, 86), (26, 90), (30, 94)]:                    # same width, moved right by 4 px each step
    true_c = (np.arange(2) + 0.5) / 2 * (x2 - x1) / STRIDE + x1 / STRIDE   # true bin centres (cells)
    print(f"RoI x = [{x1}, {x2}] px   true bin centres {np.round(true_c, 3)}  "
          f"RoIPool {np.round(roi_pool_avg(F, x1, x2), 3)}  RoIAlign {np.round(roi_align_avg(F, x1, x2), 3)}")

輸出

RoI x = [22, 86] px   true bin centres [2.375 4.375]  RoIPool [2. 4.]  RoIAlign [2.375 4.375]
RoI x = [26, 90] px   true bin centres [2.625 4.625]  RoIPool [3. 5.]  RoIAlign [2.625 4.625]
RoI x = [30, 94] px   true bin centres [2.875 4.875]  RoIPool [3. 5.]  RoIAlign [2.875 4.875]

RoIAlign 完全跟著框走,因為雙線性內插能精確重現線性斜坡。RoI pooling 的誤差分別是 0.375、0.375 與 0.125 格,在步幅 16 下相當於 6、6 與 2 像素。RoI 從第一列移到第二列時只移動了 4 像素,輸出卻整整跳了一格(16 像素);再移 4 像素,輸出完全沒有變化。真實的特徵圖不是斜坡,但 RoI pooling 對特徵造成的就是這種階梯式的反應。

如何閱讀論文的證據。 先看 [16] 的 Table 2 (c) 與 (d)。使用步幅為 32 的 ResNet-50-C5 骨幹時,RoIPool 與 RoIAlign 兩列的遮罩 AP 為 23.6 與 30.9(+7.3),AP75 為 21.6 與 32.1(+10.5);框的 AP75 從 26.9 升到 36.4。論文也測試了 RoIWarp:它同樣用雙線性內插,但仍然捨入 RoI,結果與 RoI pooling 相當。因此增益來自移除捨入,而不是來自內插。閱讀這類消融實驗時,要確認被比較的幾列使用相同的骨幹、資料切分與訓練排程(表格說明寫明在 trainval35k 上訓練、在 minival 上測試),也要注意表中只有單一數值;我在文中沒有找到重複實驗或變異數的報告。

常見錯誤。 忘記半格的偏移。如果把格子的值當成位於角落,而 RoI 的運算卻假設它位於中心(或反過來),每個取樣點都會讀到偏離半格的位置,其大小與 RoIAlign 原本要消除的捨入誤差相當。

參考文獻

  1. Z. Zou, K. Chen, Z. Shi, Y. Guo and J. Ye, “Object detection in 20 years: A survey,” Proceedings of the IEEE, vol. 111, no. 3, 2023. doi
  2. L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu and M. Pietikäinen, “Deep learning for generic object detection: A survey,” International Journal of Computer Vision, 2020. doi
  3. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, 2010. doi
  4. T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick et al., “Microsoft COCO: Common objects in context,” in Proc. ECCV, 2014. doi
  5. P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proc. IEEE CVPR, 2001. doi
  6. N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. IEEE CVPR, 2005. doi
  7. P. F. Felzenszwalb, R. B. Girshick, D. McAllester and D. Ramanan, “Object detection with discriminatively trained part-based models,” IEEE TPAMI, 2010. doi
  8. R. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE CVPR, 2014. arXiv
  9. R. Girshick, “Fast R-CNN,” in Proc. IEEE ICCV, 2015. arXiv
  10. S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE TPAMI, 2017 (conference version 2015). doi
  11. H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in Proc. IEEE/CVF CVPR, 2019. arXiv
  12. Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye and D. Ren, “Distance-IoU loss: Faster and better learning for bounding box regression,” in Proc. AAAI, 2020. arXiv
  13. N. Bodla, B. Singh, R. Chellappa and L. S. Davis, “Soft-NMS — Improving object detection with one line of code,” in Proc. IEEE ICCV, 2017. arXiv
  14. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, “Feature pyramid networks for object detection,” in Proc. IEEE CVPR, 2017. doi
  15. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
  16. K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE ICCV, 2017. doi
  17. Z. Cai and N. Vasconcelos, “Cascade R-CNN: Delving into high quality object detection,” in Proc. IEEE/CVF CVPR, 2018. doi
  18. J. Redmon, S. Divvala, R. Girshick and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. IEEE CVPR, 2016. doi
  19. J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7263–7271, 2017. CVF · arXiv
  20. J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” tech. report, arXiv:1804.02767, 2018. arXiv
  21. A. Bochkovskiy, C.-Y. Wang and H.-Y. M. Liao, “YOLOv4: Optimal speed and accuracy of object detection,” arXiv:2004.10934, 2020. arXiv
  22. C.-Y. Wang, A. Bochkovskiy and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” in Proc. IEEE/CVF CVPR, 2023. doi
  23. C.-Y. Wang, I-H. Yeh and H.-Y. M. Liao, “YOLOv9: Learning what you want to learn using programmable gradient information,” in Proc. ECCV, 2024. doi
  24. A. Wang, H. Chen, L. Liu, K. Chen, Z. Lin et al., “YOLOv10: Real-time end-to-end object detection,” in Proc. NeurIPS, 2024. arXiv
  25. Y. Tian, Q. Ye and D. Doermann, “YOLOv12: Attention-centric real-time object detectors,” arXiv:2502.12524, 2025. arXiv
  26. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu and A. C. Berg, “SSD: Single shot multibox detector,” in Proc. ECCV, 2016. arXiv
  27. H. Law and J. Deng, “CornerNet: Detecting objects as paired keypoints,” in Proc. ECCV, 2018. doi
  28. X. Zhou, D. Wang and P. Krähenbühl, “Objects as points,” arXiv:1904.07850, 2019. arXiv
  29. Z. Tian, C. Shen, H. Chen and T. He, “FCOS: Fully convolutional one-stage object detection,” in Proc. IEEE/CVF ICCV, 2019. arXiv
  30. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov and S. Zagoruyko, “End-to-end object detection with transformers,” in Proc. ECCV, 2020. doi
  31. H. W. Kuhn, “The Hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, 1955. doi
  32. X. Zhu, W. Su, L. Lu, B. Li, X. Wang and J. Dai, “Deformable DETR: Deformable transformers for end-to-end object detection,” in Proc. ICLR, 2021. arXiv
  33. H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni and H.-Y. Shum, “DINO: DETR with improved denoising anchor boxes for end-to-end object detection,” arXiv:2203.03605, 2022. arXiv
  34. Y. Zhao, W. Lv, S. Xu, J. Wei, G. Wang et al., “DETRs beat YOLOs on real-time object detection,” in Proc. IEEE/CVF CVPR, 2024. doi
  35. M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn et al., “Simple open-vocabulary object detection with vision transformers,” in Proc. ECCV, 2022. arXiv
  36. S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang et al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection,” in Proc. ECCV, 2024. doi
  37. T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang et al., “YOLO-World: Real-time open-vocabulary object detection,” in Proc. IEEE/CVF CVPR, 2024. doi
  38. N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu et al., “SAM 3: Segment anything with concepts,” arXiv:2511.16719, 2025. arXiv
  39. G. Cheng, X. Yuan, X. Yao, K. Yan, Q. Zeng et al., “Towards large-scale small object detection: Survey and benchmarks,” IEEE TPAMI, vol. 45, no. 11, 2023. arXiv
  40. COCO Consortium, “COCO detection evaluation,” official evaluation description (12 metrics, 101-point interpolation). link · official evaluator
  41. M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes challenge: A retrospective,” International Journal of Computer Vision, vol. 111, 2015. doi
  42. S. Shao, Z. Li, T. Zhang, C. Peng et al., “Objects365: A large-scale, high-quality dataset for object detection,” in Proc. IEEE/CVF ICCV, 2019. doi
  43. A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin et al., “The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale,” International Journal of Computer Vision, 2020. arXiv
  44. A. Gupta, P. Dollár and R. Girshick, “LVIS: A dataset for large vocabulary instance segmentation,” in Proc. IEEE/CVF CVPR, 2019. doi
  45. T. Shehzadi, K. A. Hashmi, M. Liwicki, D. Stricker and M. Z. Afzal, “Object detection with transformers: A review,” Sensors, vol. 25, no. 19, 2025. doi
  46. C. Zhu and L. Chen, “A survey on open-vocabulary detection and segmentation: Past, present, and future,” IEEE TPAMI, 2024. doi