完整搜尋頁面 →

ResNet:學習殘差而非整個映射

選讀・ML 03・卷積、影像濾波與 CNN 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

為何更深的網路反而可能有較高的訓練誤差?He 等人研究這個退化問題,並提出殘差參數化 [5]。因為連訓練誤差都變差,不能只用過擬合解釋。先讀卷積、張量形狀與反向傳播,再比較相同深度的普通網路與殘差網路。

方法與圖解

學習分支產生殘差,捷徑傳遞輸入;原始基本區塊在相加後套用 ReLU,因此兩條路徑的張量形狀必須相容。
來源:He et al., CVPR 2016 proceedings. 圖 2,PDF 第 2 頁。學習分支產生殘差,捷徑傳遞輸入;原始基本區塊在相加後套用 ReLU,因此兩條路徑的張量形狀必須相容。 [5]

圖 2 的原始基本區塊計算

y=ReLU⁡(F(x;W)+x).y=\operatorname{ReLU}(F(x;W)+x).

其中 xx 為區塊輸入,WW 為學習分支的權重,FF 為殘差轉換,yy 為輸出。恆等捷徑不增加可學習參數;若空間尺寸或通道數改變,兩條路徑就需要明確的對齊方式,例如論文的補零或投影選項。投影捷徑會以形狀相容的可學習轉換取代 xx。

直覺是讓「接近恆等映射」可由小殘差達成,不必要求多層網路從頭重建整個映射。圖中的相加後 ReLU 屬於原始區塊;解讀實驗時,不要默默換成後來的區塊設計。

證據

表 2 報告 ImageNet 驗證集、10-crop 協定下的 top-1 錯誤率。普通 18 層與 34 層網路分別為 27.94% 與 28.54%;對應殘差網路為 27.88% 與 25.03%。加深普通網路反而變差,加深殘差網路則改善。圖 4 包含訓練曲線;要區分最佳化退化與只出現在測試端的泛化落差,必須一起看。

設計分析

相同深度的比較,比最大模型對上無關基準更能隔離捷徑的效果。表 3 再比較不同形狀對齊方案。論文亦觀察到,採正規化的普通網路仍有正常梯度範數,因此把每次失敗都歸因於梯度消失,解釋太窄。殘差路徑改變最佳化問題;實驗並未證明所有效益都來自單一機制。

閱讀檢查點

沿圖 2 的兩條路徑追蹤輸入。若殘差變成零,ReLU 前後各剩下什麼?再改變通道數,說明為何未對齊就無法逐元素相加。最後指出:要主張深度造成最佳化困難,而非只有過擬合,還需要哪一條訓練曲線?

限制與下一步

論文更深的 CIFAR 實驗也顯示,增加深度不一定改善測試誤差。容量、資料、正則化與訓練方式仍有影響。接著讀 ML 05 的 Transformer 殘差路徑,比較正規化與非線性的位置,不要把所有使用相加的架構視為同一種區塊。

接著閱讀: ViT:影像區塊與預訓練規模的角色

參考文獻

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 9. Book
  2. L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of Big Data, 2021. Review
  3. W. Luo, Y. Li, R. Urtasun and R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” NeurIPS, 2016. Paper
  4. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, 2015. Paper
  5. K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. Paper
  6. Y. Wu and K. He, “Group Normalization,” ECCV, 2018. Paper
  7. M. Sandler et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” CVPR, 2018. Paper
  8. M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ICML, 2019. Paper
  9. Z. Liu et al., “A ConvNet for the 2020s,” CVPR, 2022. Paper
  10. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023, ch. 10. Book and notebooks
  11. OpenCV, “Image Filtering” (filter2D, border types), documentation, version 4.x. Documentation
  12. V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv:1603.07285, 2016. arXiv
  13. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, 1998. DOI
  14. A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS, 2012. Paper
  15. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ICLR, 2015. arXiv
  16. C. Szegedy et al., “Going Deeper with Convolutions,” CVPR, 2015. arXiv
  17. M. Lin, Q. Chen and S. Yan, “Network In Network,” ICLR, 2014. arXiv
  18. F. Yu and V. Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” ICLR, 2016. arXiv
  19. S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” ICML, 2015. arXiv
  20. R. Zhang, “Making Convolutional Networks Shift-Invariant Again,” ICML, 2019. arXiv
  21. Z. Li, F. Liu, W. Yang, S. Peng and J. Zhou, “A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects,” IEEE Transactions on Neural Networks and Learning Systems, 2022. DOI
  22. A. Khan, A. Sohail, U. Zahoora and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artificial Intelligence Review, 2020. arXiv
  23. J. Gu et al., “Recent Advances in Convolutional Neural Networks,” Pattern Recognition, 2018. arXiv
  24. S. Woo et al., “ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders,” CVPR, 2023. arXiv
  25. W. Wang et al., “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions,” CVPR, 2023. arXiv
  26. X. Ding et al., “Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs” (RepLKNet), CVPR, 2022. arXiv
  27. X. Ding et al., “UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition,” CVPR, 2024. arXiv
  28. D. Qin et al., “MobileNetV4: Universal Models for the Mobile Ecosystem,” ECCV, 2024. arXiv
  29. S. L. Smith, A. Brock, L. Berrada and S. De, “ConvNets Match Vision Transformers at Scale,” arXiv:2310.16764, 2023. arXiv
  30. M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” ECCV, 2014. arXiv
  31. PyTorch, torch.nn.Conv2d documentation. Documentation