ResNet:學習殘差而非整個映射
選讀・ML 03・卷積、影像濾波與 CNN 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
為何更深的網路反而可能有較高的訓練誤差?He 等人研究這個退化問題,並提出殘差參數化 [5]。因為連訓練誤差都變差,不能只用過擬合解釋。先讀卷積、張量形狀與反向傳播,再比較相同深度的普通網路與殘差網路。
方法與圖解

圖 2 的原始基本區塊計算
其中 為區塊輸入, 為學習分支的權重, 為殘差轉換, 為輸出。恆等捷徑不增加可學習參數;若空間尺寸或通道數改變,兩條路徑就需要明確的對齊方式,例如論文的補零或投影選項。投影捷徑會以形狀相容的可學習轉換取代 。
直覺是讓「接近恆等映射」可由小殘差達成,不必要求多層網路從頭重建整個映射。圖中的相加後 ReLU 屬於原始區塊;解讀實驗時,不要默默換成後來的區塊設計。
證據
表 2 報告 ImageNet 驗證集、10-crop 協定下的 top-1 錯誤率。普通 18 層與 34 層網路分別為 27.94% 與 28.54%;對應殘差網路為 27.88% 與 25.03%。加深普通網路反而變差,加深殘差網路則改善。圖 4 包含訓練曲線;要區分最佳化退化與只出現在測試端的泛化落差,必須一起看。
設計分析
相同深度的比較,比最大模型對上無關基準更能隔離捷徑的效果。表 3 再比較不同形狀對齊方案。論文亦觀察到,採正規化的普通網路仍有正常梯度範數,因此把每次失敗都歸因於梯度消失,解釋太窄。殘差路徑改變最佳化問題;實驗並未證明所有效益都來自單一機制。
閱讀檢查點
沿圖 2 的兩條路徑追蹤輸入。若殘差變成零,ReLU 前後各剩下什麼?再改變通道數,說明為何未對齊就無法逐元素相加。最後指出:要主張深度造成最佳化困難,而非只有過擬合,還需要哪一條訓練曲線?
限制與下一步
論文更深的 CIFAR 實驗也顯示,增加深度不一定改善測試誤差。容量、資料、正則化與訓練方式仍有影響。接著讀 ML 05 的 Transformer 殘差路徑,比較正規化與非線性的位置,不要把所有使用相加的架構視為同一種區塊。
接著閱讀: ViT:影像區塊與預訓練規模的角色
參考文獻
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 9. Book
- L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of Big Data, 2021. Review
- W. Luo, Y. Li, R. Urtasun and R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” NeurIPS, 2016. Paper
- O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, 2015. Paper
- K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. Paper
- Y. Wu and K. He, “Group Normalization,” ECCV, 2018. Paper
- M. Sandler et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” CVPR, 2018. Paper
- M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ICML, 2019. Paper
- Z. Liu et al., “A ConvNet for the 2020s,” CVPR, 2022. Paper
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023, ch. 10. Book and notebooks
- OpenCV, “Image Filtering” (
filter2D, border types), documentation, version 4.x. Documentation - V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv:1603.07285, 2016. arXiv
- Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, 1998. DOI
- A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS, 2012. Paper
- K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ICLR, 2015. arXiv
- C. Szegedy et al., “Going Deeper with Convolutions,” CVPR, 2015. arXiv
- M. Lin, Q. Chen and S. Yan, “Network In Network,” ICLR, 2014. arXiv
- F. Yu and V. Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” ICLR, 2016. arXiv
- S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” ICML, 2015. arXiv
- R. Zhang, “Making Convolutional Networks Shift-Invariant Again,” ICML, 2019. arXiv
- Z. Li, F. Liu, W. Yang, S. Peng and J. Zhou, “A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects,” IEEE Transactions on Neural Networks and Learning Systems, 2022. DOI
- A. Khan, A. Sohail, U. Zahoora and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artificial Intelligence Review, 2020. arXiv
- J. Gu et al., “Recent Advances in Convolutional Neural Networks,” Pattern Recognition, 2018. arXiv
- S. Woo et al., “ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders,” CVPR, 2023. arXiv
- W. Wang et al., “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions,” CVPR, 2023. arXiv
- X. Ding et al., “Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs” (RepLKNet), CVPR, 2022. arXiv
- X. Ding et al., “UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition,” CVPR, 2024. arXiv
- D. Qin et al., “MobileNetV4: Universal Models for the Mobile Ecosystem,” ECCV, 2024. arXiv
- S. L. Smith, A. Brock, L. Berrada and S. De, “ConvNets Match Vision Transformers at Scale,” arXiv:2310.16764, 2023. arXiv
- M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” ECCV, 2014. arXiv
- PyTorch,
torch.nn.Conv2ddocumentation. Documentation