完整搜尋頁面 →

堆疊空洞卷積中的網格效應(gridding)

選讀・DLCV 04・影像分割:語意、實例與全景分割 的補充資料(第 1/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 堆疊空洞卷積中的網格效應(gridding)
  2. 密集 CRF 後處理
  3. 軟 Dice 損失的梯度
  4. Boundary IoU 詳解

論文導讀 · 2

  1. Panoptic Segmentation:論文導讀
  2. Mask2Former:論文導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則補充說明哪些膨脹率序列會在感受野中留下洞,以及一條選擇不會留洞的比率的簡單規則。

問題。 取一個由 nn 層膨脹卷積堆疊而成的網路的某個輸出,卷積核大小為 kk、比率為 r1,…,rnr_1, \dots, r_n,問哪些輸入位置會影響它。在一維中,可到達的偏移集合是和集 {∑lrldl:dl∈{−c,…,c}}\{\sum_l r_l d_l : d_l \in \{-c, \dots, c\}\},其中 c=(k−1)/2c = (k - 1)/2。如果每個 rlr_l 都是某個整數 q>1q \gt 1 的倍數,那麼每個可到達的偏移也都是 qq 的倍數:輸出沿每個軸向,每 qq 個輸入只看到一個,形成帶洞的規則格點。相鄰的輸出於是讀取互不相交的格點,輸出圖就出現棋盤格狀的圖案。Wang 等人 [11] 把這稱為網格效應(gridding),並指出在高比率時更嚴重,這時稀疏的卷積核已經涵蓋不到任何局部鄰域。

規則。 他們的混合膨脹卷積(hybrid dilated convolution, HDC)在連續各層使用不同的比率,排成鋸齒狀,選擇的目標是讓最終的感受野涵蓋一塊沒有洞的方形區域。把透過第 ii 層到第 nn 層所見的兩個非零取樣點之間的最大距離,遞迴地定義為

Mi=max⁡[Mi+1−2ri,  Mi+1−2(Mi+1−ri),  ri],Mn=rn,M_i = \max\big[M_{i+1} - 2 r_i,\; M_{i+1} - 2(M_{i+1} - r_i),\; r_i\big], \qquad M_n = r_n,

其中 rir_i 是第 ii 層的比率。設計目標是 M2≤kM_2 \le k。當 k=3k = 3 時,比率 (1,2,5)(1, 2, 5) 給出 M2=2M_2 = 2,可行;而 (1,2,9)(1, 2, 9) 給出 M2=5M_2 = 5,不可行 [11]。他們也指出,同一組內的比率不應有 2、4、8 這類共同因數。

用程式碼檢查。 數出從每個輸入偏移到一個輸出的路徑數,洞就看得見了。

程式碼 · Python · 13 行
import numpy as np

def coverage(rates, k=3):
    """How many paths connect each input offset to one output of a stack of 1-D dilated convs."""
    reach = np.array([1])
    for r in rates:
        taps = np.zeros(r * (k - 1) + 1, int); taps[::r] = 1      # a k-tap kernel with rate r
        reach = np.convolve(reach, taps)
    return reach

for rates in [(2, 2, 2), (1, 2, 4), (1, 2, 5), (1, 2, 9)]:
    c = coverage(rates)
    print(f"rates {rates}: span {len(c)}, holes {int((c == 0).sum())}, paths {c}")

輸出

rates (2, 2, 2): span 13, holes 6, paths [1 0 3 0 6 0 7 0 6 0 3 0 1]
rates (1, 2, 4): span 15, holes 0, paths [1 1 2 1 3 2 3 1 3 2 3 1 2 1 1]
rates (1, 2, 5): span 17, holes 0, paths [1 1 2 1 2 2 2 2 1 2 2 2 2 1 2 1 1]
rates (1, 2, 9): span 25, holes 4, paths [1 1 2 1 2 1 1 0 0 1 1 2 1 2 1 1 0 0 1 1 2 1 2 1 1]

比率都是 2 時,每隔一個輸入就看不到;從 2 跳到 9,則在內層圖樣的各個複本之間留下縫隙。路徑數也顯示,即使沒有洞,覆蓋也不均勻:有些偏移有三條路徑到達,有些只有一條。

邊界情況。 ASPP 不會以同樣的方式受影響,因為它的各個分支是在同一張圖上平行執行再結合,而不是堆疊。另外,如果骨幹網路較早、沒有膨脹的層已經混合了相鄰的像素,實務上會減輕網格效應;問題在好幾個高度膨脹的層連續出現時最嚴重。

參考文獻

  1. S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, 2022. doi · arXiv
  2. X. Li, H. Ding, H. Yuan, W. Zhang, J. Pang, G. Cheng, K. Chen, Z. Liu and C. C. Loy, “Transformer-based visual segmentation: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, 2024. doi · arXiv
  3. T. Zhou, W. Xia, F. Zhang, B. Chang, W. Wang, Y. Yuan, E. Konukoglu and D. Cremers, “Image segmentation in foundation model era: A survey,” arXiv:2408.12957, 2024. arXiv
  4. A. Kirillov, K. He, R. Girshick, C. Rother and P. Dollár, “Panoptic segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  5. J. Long, E. Shelhamer and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. doi · arXiv
  6. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS vol. 9351, 2015. doi
  7. L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in Proc. European Conference on Computer Vision (ECCV), 2018. doi · arXiv
  8. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, 2018. doi · arXiv
  9. F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in Proc. International Conference on Learning Representations (ICLR), 2016. arXiv
  10. L.-C. Chen, G. Papandreou, F. Schroff and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv:1706.05587, 2017. arXiv
  11. P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou and G. Cottrell, “Understanding convolution for semantic segmentation,” in Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), 2018. doi · arXiv
  12. H. Zhao, J. Shi, X. Qi, X. Wang and J. Jia, “Pyramid scene parsing network,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · arXiv
  13. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. International Conference on Learning Representations (ICLR), 2021. arXiv
  14. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez and P. Luo, “SegFormer: Simple and efficient design for semantic segmentation with transformers,” in Advances in Neural Information Processing Systems (NeurIPS), 2021. NeurIPS · arXiv
  15. F. Milletari, N. Navab and S.-A. Ahmadi, “V-Net: Fully convolutional neural networks for volumetric medical image segmentation,” in Proc. Fourth International Conference on 3D Vision (3DV), 2016. doi · arXiv
  16. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi
  17. B. Cheng, A. G. Schwing and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” in Advances in Neural Information Processing Systems (NeurIPS), 2021. NeurIPS · arXiv
  18. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth and B. Schiele, “The Cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi · arXiv
  19. B. Cheng, R. Girshick, P. Dollár, A. C. Berg and A. Kirillov, “Boundary IoU: Improving object-centric image segmentation evaluation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi · arXiv
  20. K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  21. S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, 2017. doi
  22. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, “Feature pyramid networks for object detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi
  23. A. Kirillov, Y. Wu, K. He and R. Girshick, “PointRend: Image segmentation as rendering,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  24. A. Kirillov, R. Girshick, K. He and P. Dollár, “Panoptic feature pyramid networks,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  25. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov and S. Zagoruyko, “End-to-end object detection with transformers,” in Proc. European Conference on Computer Vision (ECCV), 2020. doi · arXiv
  26. B. Cheng, I. Misra, A. G. Schwing, A. Kirillov and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. doi · arXiv
  27. J. Jain, J. Li, M. Chiu, A. Hassani, N. Orlov and H. Shi, “OneFormer: One transformer to rule universal image segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. doi · arXiv
  28. T. Kerssies, N. Cavagnero, A. Hermans, N. Norouzi, G. Averta, B. Leibe, G. Dubbelman and D. de Geus, “Your ViT is secretly an image segmentation model,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. CVF · arXiv
  29. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson et al., “Segment anything,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi · arXiv
  30. N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma et al., “SAM 2: Segment anything in images and videos,” in Proc. International Conference on Learning Representations (ICLR), 2025. OpenReview · arXiv
  31. N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris et al., “SAM 3: Segment anything with concepts,” in Proc. International Conference on Learning Representations (ICLR), 2026. OpenReview · arXiv
  32. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al., “Learning transferable visual models from natural language supervision,” in Proc. International Conference on Machine Learning (ICML), PMLR 139, 2021. PMLR
  33. B. Li, K. Q. Weinberger, S. Belongie, V. Koltun and R. Ranftl, “Language-driven semantic segmentation,” in Proc. International Conference on Learning Representations (ICLR), 2022. OpenReview · arXiv
  34. S. Cho, H. Shin, S. Hong, A. Arnab, P. H. Seo and S. Kim, “CAT-Seg: Cost aggregation for open-vocabulary semantic segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi · arXiv
  35. C. Zhu and L. Chen, “A survey on open-vocabulary detection and segmentation: Past, present, and future,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, 2024. doi
  36. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, no. 2, 2010. doi
  37. B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso and A. Torralba, “Scene parsing through ADE20K dataset,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · CVF
  38. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan et al., “Microsoft COCO: Common objects in context,” in Proc. European Conference on Computer Vision (ECCV), 2014. doi · arXiv
  39. H. Caesar, J. Uijlings and V. Ferrari, “COCO-Stuff: Thing and stuff classes in context,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  40. G. Neuhold, T. Ollmann, S. Rota Bulò and P. Kontschieder, “The Mapillary Vistas dataset for semantic understanding of street scenes,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi
  41. P. Krähenbühl and V. Koltun, “Efficient inference in fully connected CRFs with Gaussian edge potentials,” in Advances in Neural Information Processing Systems (NeurIPS), 2011. NeurIPS · arXiv