Full search page →

Backpropagation step by step

Optional · supplementary note 3 of 4 for IP 13 · Image Pattern Classification. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Backpropagation [5] computes the gradient of the loss EE with respect to every weight of a multilayer network in one backward sweep. The notation (z(ℓ)\mathbf{z}^{(\ell)}, a(ℓ)\mathbf{a}^{(\ell)}, W(ℓ)\mathbf{W}^{(\ell)}, hh, p\mathbf{p}, t\mathbf{t}) is that of the forward pass in the main text.

Define the error signal of layer ℓ\ell as δ(ℓ)=∂E/∂z(ℓ)\boldsymbol{\delta}^{(\ell)} = \partial E / \partial \mathbf{z}^{(\ell)}.

Step 1: output layer. For softmax with cross-entropy the derivative is remarkably simple:

δ(L)=p−t.\boldsymbol{\delta}^{(L)} = \mathbf{p} - \mathbf{t} .

(For a sigmoid output with squared error it is (a(L)−t)⊙h′(z(L))(\mathbf{a}^{(L)} - \mathbf{t}) \odot h'(\mathbf{z}^{(L)}), where ⊙\odot is element-wise multiplication.)

Step 2: propagate backward. Unit ii of layer ℓ\ell affects EE only through the net inputs of layer ℓ+1\ell+1, and ∂zk(ℓ+1)/∂ai(ℓ)=Wki(ℓ+1)\partial z^{(\ell+1)}_k / \partial a^{(\ell)}_i = W^{(\ell+1)}_{ki}. The chain rule gives

δ(ℓ)=h′(z(ℓ))⊙(W(ℓ+1)⊤δ(ℓ+1)),ℓ=L−1,…,1.\boldsymbol{\delta}^{(\ell)} = h'\big(\mathbf{z}^{(\ell)}\big) \odot \Big(\mathbf{W}^{(\ell+1)\top} \boldsymbol{\delta}^{(\ell+1)}\Big), \qquad \ell = L-1, \dots, 1 .

The errors travel backward through the same weights that carried the signal forward, transposed.

Step 3: gradients. Because zi(ℓ)=∑kWik(ℓ)ak(ℓ−1)+bi(ℓ)z^{(\ell)}_i = \sum_k W^{(\ell)}_{ik} a^{(\ell-1)}_k + b^{(\ell)}_i,

∂E∂W(ℓ)=δ(ℓ) a(ℓ−1)⊤,∂E∂b(ℓ)=δ(ℓ).\frac{\partial E}{\partial \mathbf{W}^{(\ell)}} = \boldsymbol{\delta}^{(\ell)} \, \mathbf{a}^{(\ell-1)\top}, \qquad \frac{\partial E}{\partial \mathbf{b}^{(\ell)}} = \boldsymbol{\delta}^{(\ell)} .

Each weight’s gradient is “error at its output end” times “activation at its input end”.

Step 4: update. Gradient descent with learning rate α\alpha:

W(ℓ)←W(ℓ)−α∂E∂W(ℓ),b(ℓ)←b(ℓ)−α∂E∂b(ℓ).\mathbf{W}^{(\ell)} \leftarrow \mathbf{W}^{(\ell)} - \alpha \frac{\partial E}{\partial \mathbf{W}^{(\ell)}}, \qquad \mathbf{b}^{(\ell)} \leftarrow \mathbf{b}^{(\ell)} - \alpha \frac{\partial E}{\partial \mathbf{b}^{(\ell)}} .

The cost of the backward pass is about the same as the forward pass, which is why networks with millions of weights can be trained at all. The perceptron learning rule [4] is a separate mistake-driven update, not ordinary backpropagation through a hard step: the step derivative is zero away from its threshold and undefined at the threshold. Differentiable activations let the chain rule in [5] supply useful gradients.

References

  1. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018, Ch. 13. publisher page
  2. A. K. Jain, R. P. W. Duin, and J. Mao, “Statistical Pattern Recognition: A Review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4–37, 2000. doi:10.1109/34.824819
  3. D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision, vol. 60, 2004. doi:10.1023/B:VISI.0000029664.99615.94
  4. F. Rosenblatt, “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain,” Psychological Review, vol. 65, 1958. doi:10.1037/h0042519
  5. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323, 1986. doi:10.1038/323533a0
  6. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, 1998. doi:10.1109/5.726791
  7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25 (NIPS), 2012. paper
  8. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv:1409.1556, 2014. arXiv
  9. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” arXiv:1512.03385, 2015. arXiv
  10. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, et al., “ImageNet Large Scale Visual Recognition Challenge,” arXiv:1409.0575, 2014. arXiv
  11. W. Rawat and Z. Wang, “Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review,” Neural Computation, vol. 29, no. 9, 2017. doi:10.1162/neco_a_00990
  12. A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  13. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, 2022. arXiv
  14. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022. arXiv
  15. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning Transferable Visual Models From Natural Language Supervision,” arXiv:2103.00020, 2021. arXiv
  16. M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning Robust Visual Features without Supervision,” arXiv:2304.07193, 2023. arXiv
  17. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. JMLR
  18. C. Shorten and T. M. Khoshgoftaar, “A Survey on Image Data Augmentation for Deep Learning,” Journal of Big Data, vol. 6, 2019. doi:10.1186/s40537-019-0197-0
  19. A. Torralba and A. A. Efros, “Unbiased Look at Dataset Bias,” CVPR, 2011. paper page
  20. Z. Liu and K. He, “A Decade’s Battle on Dataset Bias: Are We There Yet?,” ICLR, 2025. arXiv