完整搜尋頁面 →

Adam:自適應更新與收斂論證的限制

選讀・ML 02・類神經網路:從感知器到深度網路 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

最佳化器如何同時平滑更新方向,並為各參數調整步長尺度?Adam 維護兩個移動平均,再修正零初始化造成的偏差 [19]。先理解梯度與動量。它的實務貢獻是精簡的更新規則;原始收斂討論則必須搭配 Reddi、Kale 與 Kumar 的後續修正閱讀 [30]。

方法與圖解

原始消融比較不同步長與動量設定下,有無偏差修正的訓練表現;縱軸不是未見資料的泛化能力。
來源:Kingma and Ba, ICLR 2015 manuscript, arXiv:1412.6980v9. 圖 4,PDF 第 8 頁。原始消融比較不同步長與動量設定下,有無偏差修正的訓練表現;縱軸不是未見資料的泛化能力。 [19]

動量緩衝區從零開始時,更新式為

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,m^t=mt/(1−β1t),v^t=vt/(1−β2t),θt=θt−1−αm^t/(v^t+ϵ).\begin{aligned} m_t&=\beta_1m_{t-1}+(1-\beta_1)g_t, &v_t&=\beta_2v_{t-1}+(1-\beta_2)g_t^2,\\ \widehat m_t&=m_t/(1-\beta_1^t), &\widehat v_t&=v_t/(1-\beta_2^t),\\ \theta_t&=\theta_{t-1}-\alpha\widehat m_t/(\sqrt{\widehat v_t}+\epsilon). \end{aligned}

其中 tt 為更新次數,gtg_t 為當前梯度,mtm_t、vtv_t 分別估計一階矩與二階原點矩,上帽表示偏差修正。衰減係數 β1,β2\beta_1,\beta_2 介於零與一之間;α\alpha 為步長,ϵ\epsilon 用來穩定除法。平方、平方根與除法都逐座標進行。尤其要注意:vtv_t 估計的是未置中的二階矩,不是梯度變異數。

圖 4 在不同超參數下比較有無偏差修正。修正補償緩衝區從零開始的影響,並不消除訓練資料的偏誤,也不會讓梯度沒有雜訊。

證據

原始實驗包含分類目標與變分自編碼器。圖 4 報告自編碼器訓練 10 與 100 個 epoch 後的訓練損失;二階矩衰減係數較大時,未修正版本可能不穩定。圖 1 同樣追蹤訓練目標。這些曲線支持特定設定下的最佳化表現,不能保證測試準確度更好。

設計分析

分子平滑方向,分母依近期平方梯度調整各座標尺度。兩者都記住歷史,所以單看當前梯度無法解釋更新。後續論文 [30] 提出凸問題反例,並指出早期收斂推論的缺陷。因此,不能用原始定理或成功訓練曲線宣稱一般 Adam 必然收斂,甚至不能宣稱對所有凸問題都成立。實務效用與普遍保證是兩種不同主張。

閱讀檢查點

假設梯度固定且非零,從零緩衝區算出第一次修正後的兩個矩,解釋修正係數如何抵消初始縮減。再考慮梯度突然改變:哪些儲存量仍保有前幾步資訊?為何偏差修正不會抹去那些歷史?

限制與下一步

學習率、衰減係數、數值穩定性與目標函數仍然重要。某個訓練損失的好結果,不代表泛化最佳;調參預算不同時也未必公平。回到本章最佳化診斷,再讀 ML 11 評估,避免只憑一條曲線選最佳化器。討論收斂時,請把 [30] 與 [19] 放在一起。

接著閱讀: Temperature Scaling:不改類別的信心校準

參考文獻

  1. Shyandram, “機器學習及類神經網路筆記,” Medium post, 2024; updated 2026. link
  2. Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. doi
  3. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. online book
  4. F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, pp. 386–408, 1958. doi
  5. G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, pp. 303–314, 1989. doi
  6. D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986. doi
  7. S. R. Dubey, S. K. Singh and B. B. Chaudhuri, “Activation functions in deep learning: A comprehensive survey and benchmark,” Neurocomputing, vol. 503, pp. 92–108, 2022. arXiv
  8. D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” arXiv:1606.08415, 2016. arXiv
  9. S. Elfwing, E. Uchibe and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural Networks, vol. 107, pp. 3–11, 2018. arXiv
  10. P. Ramachandran, B. Zoph and Q. V. Le, “Searching for activation functions,” arXiv:1710.05941, 2017. arXiv
  11. N. Shazeer, “GLU variants improve Transformer,” arXiv:2002.05202, 2020. arXiv
  12. H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023. arXiv
  13. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. International Conference on Machine Learning (ICML), PMLR 37, pp. 448–456, 2015. PMLR
  14. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer normalization,” arXiv:1607.06450, 2016. arXiv
  15. B. Zhang and R. Sennrich, “Root mean square layer normalization,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv
  16. L. Huang, J. Qin, Y. Zhou, F. Zhu, L. Liu and L. Shao, “Normalization techniques in training DNNs: Methodology, analysis and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10173–10196, 2023. arXiv
  17. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9, pp. 249–256, 2010. PMLR
  18. K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015. arXiv
  19. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
  20. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. International Conference on Learning Representations (ICLR), 2019. arXiv
  21. X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu and Q. V. Le, “Symbolic discovery of optimization algorithms,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv
  22. K. Jordan, “Muon: An optimizer for hidden layers in neural networks,” Medium post, December 2024. link
  23. J. Liu, J. Su, X. Yao et al., “Muon is scalable for LLM training,” arXiv:2502.16982, 2025. arXiv
  24. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
  25. N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
  26. R. Sun, “Optimization for deep learning: Theory and algorithms,” arXiv:1912.08957, 2019. arXiv
  27. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. JMLR
  28. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. arXiv
  29. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv
  30. S. J. Reddi, S. Kale and S. Kumar, “On the Convergence of Adam and Beyond,” ICLR, 2018. arXiv