完整搜尋頁面 →

逐步說明反向傳播

選讀・IP 13・影像圖樣分類 的補充資料(第 3/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

反向傳播 [5] 用一次由後往前的計算,求出損失 EE 對多層網路中每個權重的梯度。符號(z(ℓ)\mathbf{z}^{(\ell)}、a(ℓ)\mathbf{a}^{(\ell)}、W(ℓ)\mathbf{W}^{(\ell)}、hh、p\mathbf{p}、t\mathbf{t})沿用正文前向傳遞的定義。

定義第 ℓ\ell 層的誤差訊號為 δ(ℓ)=∂E/∂z(ℓ)\boldsymbol{\delta}^{(\ell)} = \partial E / \partial \mathbf{z}^{(\ell)}。

步驟 1:輸出層。 softmax 搭配交叉熵時,導數出奇地簡單:

δ(L)=p−t.\boldsymbol{\delta}^{(L)} = \mathbf{p} - \mathbf{t} .

(sigmoid 輸出搭配平方誤差時則是 (a(L)−t)⊙h′(z(L))(\mathbf{a}^{(L)} - \mathbf{t}) \odot h'(\mathbf{z}^{(L)}),⊙\odot 表示逐元素相乘。)

步驟 2:往回傳。 第 ℓ\ell 層的單元 ii 只透過第 ℓ+1\ell+1 層的淨輸入影響 EE,且 ∂zk(ℓ+1)/∂ai(ℓ)=Wki(ℓ+1)\partial z^{(\ell+1)}_k / \partial a^{(\ell)}_i = W^{(\ell+1)}_{ki}。由連鎖律得到

δ(ℓ)=h′(z(ℓ))⊙(W(ℓ+1)⊤δ(ℓ+1)),ℓ=L−1,…,1.\boldsymbol{\delta}^{(\ell)} = h'\big(\mathbf{z}^{(\ell)}\big) \odot \Big(\mathbf{W}^{(\ell+1)\top} \boldsymbol{\delta}^{(\ell+1)}\Big), \qquad \ell = L-1, \dots, 1 .

誤差沿著前向時傳遞訊號的同一組權重(轉置後)往回走。

步驟 3:梯度。 因為 zi(ℓ)=∑kWik(ℓ)ak(ℓ−1)+bi(ℓ)z^{(\ell)}_i = \sum_k W^{(\ell)}_{ik} a^{(\ell-1)}_k + b^{(\ell)}_i,

∂E∂W(ℓ)=δ(ℓ) a(ℓ−1)⊤,∂E∂b(ℓ)=δ(ℓ).\frac{\partial E}{\partial \mathbf{W}^{(\ell)}} = \boldsymbol{\delta}^{(\ell)} \, \mathbf{a}^{(\ell-1)\top}, \qquad \frac{\partial E}{\partial \mathbf{b}^{(\ell)}} = \boldsymbol{\delta}^{(\ell)} .

每個權重的梯度就是「輸出端的誤差」乘以「輸入端的激活值」。

步驟 4:更新。 以學習率 α\alpha 做梯度下降:

W(ℓ)←W(ℓ)−α∂E∂W(ℓ),b(ℓ)←b(ℓ)−α∂E∂b(ℓ).\mathbf{W}^{(\ell)} \leftarrow \mathbf{W}^{(\ell)} - \alpha \frac{\partial E}{\partial \mathbf{W}^{(\ell)}}, \qquad \mathbf{b}^{(\ell)} \leftarrow \mathbf{b}^{(\ell)} - \alpha \frac{\partial E}{\partial \mathbf{b}^{(\ell)}} .

反向傳遞的計算量和前向傳遞差不多,這正是擁有數百萬個權重的網路能被訓練的原因。感知器學習規則 [4] 是另一種依分類錯誤更新的規則,不是對硬階梯函數直接做一般反向傳播:階梯函數的導數在門檻之外為零,在門檻處沒有定義。可微分的激活函數,才讓 [5] 的連鎖律提供有用的梯度。

參考文獻

  1. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018, Ch. 13. publisher page
  2. A. K. Jain, R. P. W. Duin, and J. Mao, “Statistical Pattern Recognition: A Review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4–37, 2000. doi:10.1109/34.824819
  3. D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision, vol. 60, 2004. doi:10.1023/B:VISI.0000029664.99615.94
  4. F. Rosenblatt, “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain,” Psychological Review, vol. 65, 1958. doi:10.1037/h0042519
  5. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323, 1986. doi:10.1038/323533a0
  6. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, 1998. doi:10.1109/5.726791
  7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25 (NIPS), 2012. paper
  8. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv:1409.1556, 2014. arXiv
  9. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” arXiv:1512.03385, 2015. arXiv
  10. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, et al., “ImageNet Large Scale Visual Recognition Challenge,” arXiv:1409.0575, 2014. arXiv
  11. W. Rawat and Z. Wang, “Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review,” Neural Computation, vol. 29, no. 9, 2017. doi:10.1162/neco_a_00990
  12. A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  13. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, 2022. arXiv
  14. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022. arXiv
  15. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning Transferable Visual Models From Natural Language Supervision,” arXiv:2103.00020, 2021. arXiv
  16. M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning Robust Visual Features without Supervision,” arXiv:2304.07193, 2023. arXiv
  17. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. JMLR
  18. C. Shorten and T. M. Khoshgoftaar, “A Survey on Image Data Augmentation for Deep Learning,” Journal of Big Data, vol. 6, 2019. doi:10.1186/s40537-019-0197-0
  19. A. Torralba and A. A. Efros, “Unbiased Look at Dataset Bias,” CVPR, 2011. paper page
  20. Z. Liu and K. He, “A Decade’s Battle on Dataset Bias: Are We There Yet?,” ICLR, 2025. arXiv