Adam:自適應更新與收斂論證的限制
選讀・ML 02・類神經網路:從感知器到深度網路 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
最佳化器如何同時平滑更新方向,並為各參數調整步長尺度?Adam 維護兩個移動平均,再修正零初始化造成的偏差 [19]。先理解梯度與動量。它的實務貢獻是精簡的更新規則;原始收斂討論則必須搭配 Reddi、Kale 與 Kumar 的後續修正閱讀 [30]。
方法與圖解

動量緩衝區從零開始時,更新式為
其中 為更新次數, 為當前梯度,、 分別估計一階矩與二階原點矩,上帽表示偏差修正。衰減係數 介於零與一之間; 為步長, 用來穩定除法。平方、平方根與除法都逐座標進行。尤其要注意: 估計的是未置中的二階矩,不是梯度變異數。
圖 4 在不同超參數下比較有無偏差修正。修正補償緩衝區從零開始的影響,並不消除訓練資料的偏誤,也不會讓梯度沒有雜訊。
證據
原始實驗包含分類目標與變分自編碼器。圖 4 報告自編碼器訓練 10 與 100 個 epoch 後的訓練損失;二階矩衰減係數較大時,未修正版本可能不穩定。圖 1 同樣追蹤訓練目標。這些曲線支持特定設定下的最佳化表現,不能保證測試準確度更好。
設計分析
分子平滑方向,分母依近期平方梯度調整各座標尺度。兩者都記住歷史,所以單看當前梯度無法解釋更新。後續論文 [30] 提出凸問題反例,並指出早期收斂推論的缺陷。因此,不能用原始定理或成功訓練曲線宣稱一般 Adam 必然收斂,甚至不能宣稱對所有凸問題都成立。實務效用與普遍保證是兩種不同主張。
閱讀檢查點
假設梯度固定且非零,從零緩衝區算出第一次修正後的兩個矩,解釋修正係數如何抵消初始縮減。再考慮梯度突然改變:哪些儲存量仍保有前幾步資訊?為何偏差修正不會抹去那些歷史?
限制與下一步
學習率、衰減係數、數值穩定性與目標函數仍然重要。某個訓練損失的好結果,不代表泛化最佳;調參預算不同時也未必公平。回到本章最佳化診斷,再讀 ML 11 評估,避免只憑一條曲線選最佳化器。討論收斂時,請把 [30] 與 [19] 放在一起。
接著閱讀: Temperature Scaling:不改類別的信心校準
參考文獻
- Shyandram, “機器學習及類神經網路筆記,” Medium post, 2024; updated 2026. link
- Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. doi
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. online book
- F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, pp. 386–408, 1958. doi
- G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, pp. 303–314, 1989. doi
- D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986. doi
- S. R. Dubey, S. K. Singh and B. B. Chaudhuri, “Activation functions in deep learning: A comprehensive survey and benchmark,” Neurocomputing, vol. 503, pp. 92–108, 2022. arXiv
- D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” arXiv:1606.08415, 2016. arXiv
- S. Elfwing, E. Uchibe and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural Networks, vol. 107, pp. 3–11, 2018. arXiv
- P. Ramachandran, B. Zoph and Q. V. Le, “Searching for activation functions,” arXiv:1710.05941, 2017. arXiv
- N. Shazeer, “GLU variants improve Transformer,” arXiv:2002.05202, 2020. arXiv
- H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023. arXiv
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. International Conference on Machine Learning (ICML), PMLR 37, pp. 448–456, 2015. PMLR
- J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer normalization,” arXiv:1607.06450, 2016. arXiv
- B. Zhang and R. Sennrich, “Root mean square layer normalization,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv
- L. Huang, J. Qin, Y. Zhou, F. Zhu, L. Liu and L. Shao, “Normalization techniques in training DNNs: Methodology, analysis and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10173–10196, 2023. arXiv
- X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9, pp. 249–256, 2010. PMLR
- K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015. arXiv
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
- I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. International Conference on Learning Representations (ICLR), 2019. arXiv
- X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu and Q. V. Le, “Symbolic discovery of optimization algorithms,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv
- K. Jordan, “Muon: An optimizer for hidden layers in neural networks,” Medium post, December 2024. link
- J. Liu, J. Su, X. Yao et al., “Muon is scalable for LLM training,” arXiv:2502.16982, 2025. arXiv
- I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
- N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
- R. Sun, “Optimization for deep learning: Theory and algorithms,” arXiv:1912.08957, 2019. arXiv
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. JMLR
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv
- S. J. Reddi, S. Kale and S. Kumar, “On the Convergence of Adam and Beyond,” ICLR, 2018. arXiv