ResNet — learning a residual instead of a whole mapping
Optional · paper guide 1 of 1 for ML 03 · Convolution, Image Filtering and CNNs. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Why can a deeper network have higher training error than a shallower one? He et al. study this degradation problem and propose residual parameterization [5]. The issue is not explained merely by overfitting, because the training error itself worsens. Read convolution, tensor shapes and backpropagation first; then compare plain and residual networks at the same depth.
Method and figure

The original basic block in Figure 2 computes
Here is the block input, contains the learned branch’s weights, is its residual transformation and is the block output. The identity shortcut adds no learned parameters. If spatial size or channel count changes, the branches require an explicit matching rule, such as the paper’s padding or projection options. A projection shortcut replaces by a learned compatible transformation.
The intuition is to make a useful near-identity mapping accessible through a small residual, rather than requiring stacked layers to reconstruct that mapping from scratch. The post-addition ReLU shown here belongs to the original block; do not silently substitute a later block design when interpreting its experiments.
Evidence
Table 2 reports ImageNet validation top-1 error under its 10-crop protocol. The plain 18-layer and 34-layer networks obtain 27.94% and 28.54%; their residual counterparts obtain 27.88% and 25.03%. The deeper plain model gets worse, while the deeper residual model improves. Figure 4 includes training curves, which are needed to distinguish optimization degradation from a test-only generalization gap.
Design analysis
The depth-matched comparison isolates the shortcut more usefully than comparing only the largest model with an unrelated baseline. Table 3 then compares shortcut matching choices. The paper also observes healthy gradient norms with normalization in the plain networks, so “every failure is vanishing gradients” is too narrow an explanation. Residual paths change the optimization problem; the experiment does not prove that one simple mechanism explains all benefits.
Reading checkpoint
Trace an input through both branches of Figure 2. If the residual becomes zero, what remains before and after ReLU? Then change the channel count and explain why elementwise addition fails without matching. Finally, identify which training curve would be necessary to claim that depth causes optimization difficulty rather than only overfitting.
Limits and next step
The paper’s much deeper CIFAR experiment also shows that additional depth need not improve test error. Capacity, data, regularization and training remain relevant. Move next to ML 05’s Transformer residual paths: compare where normalization and nonlinearities occur, rather than assuming every architecture using addition implements the same block.
Continue reading: ViT — image patches and the role of pretraining scale
References
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 9. Book
- L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of Big Data, 2021. Review
- W. Luo, Y. Li, R. Urtasun and R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” NeurIPS, 2016. Paper
- O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, 2015. Paper
- K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. Paper
- Y. Wu and K. He, “Group Normalization,” ECCV, 2018. Paper
- M. Sandler et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” CVPR, 2018. Paper
- M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ICML, 2019. Paper
- Z. Liu et al., “A ConvNet for the 2020s,” CVPR, 2022. Paper
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023, ch. 10. Book and notebooks
- OpenCV, “Image Filtering” (
filter2D, border types), documentation, version 4.x. Documentation - V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv:1603.07285, 2016. arXiv
- Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, 1998. DOI
- A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS, 2012. Paper
- K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ICLR, 2015. arXiv
- C. Szegedy et al., “Going Deeper with Convolutions,” CVPR, 2015. arXiv
- M. Lin, Q. Chen and S. Yan, “Network In Network,” ICLR, 2014. arXiv
- F. Yu and V. Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” ICLR, 2016. arXiv
- S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” ICML, 2015. arXiv
- R. Zhang, “Making Convolutional Networks Shift-Invariant Again,” ICML, 2019. arXiv
- Z. Li, F. Liu, W. Yang, S. Peng and J. Zhou, “A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects,” IEEE Transactions on Neural Networks and Learning Systems, 2022. DOI
- A. Khan, A. Sohail, U. Zahoora and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artificial Intelligence Review, 2020. arXiv
- J. Gu et al., “Recent Advances in Convolutional Neural Networks,” Pattern Recognition, 2018. arXiv
- S. Woo et al., “ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders,” CVPR, 2023. arXiv
- W. Wang et al., “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions,” CVPR, 2023. arXiv
- X. Ding et al., “Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs” (RepLKNet), CVPR, 2022. arXiv
- X. Ding et al., “UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition,” CVPR, 2024. arXiv
- D. Qin et al., “MobileNetV4: Universal Models for the Mobile Ecosystem,” ECCV, 2024. arXiv
- S. L. Smith, A. Brock, L. Berrada and S. De, “ConvNets Match Vision Transformers at Scale,” arXiv:2310.16764, 2023. arXiv
- M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” ECCV, 2014. arXiv
- PyTorch,
torch.nn.Conv2ddocumentation. Documentation