Full search page →

Adam — adaptive updates and the convergence caveat

Optional · paper guide 1 of 1 for ML 02 · Neural Networks from Perceptron to Deep Networks. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

How can an optimizer combine a smoothed direction with a separate step scale for each parameter? Adam maintains two moving averages and corrects their initialization bias [19]. Start with gradients and momentum. Its practical contribution is a compact update rule; its original convergence discussion must also be read alongside the later correction by Reddi, Kale and Kumar [30].

Method and figure

The original ablation varies step sizes and moment settings with and without bias correction. The vertical axis measures training behavior, not held-out generalization.
Source: Kingma and Ba, ICLR 2015 manuscript, arXiv:1412.6980v9. Figure 4, PDF p. 8. The original ablation varies step sizes and moment settings with and without bias correction. The vertical axis measures training behavior, not held-out generalization. [19]

With zero-initialized moment buffers, the update is

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,m^t=mt/(1−β1t),v^t=vt/(1−β2t),θt=θt−1−αm^t/(v^t+ϵ).\begin{aligned} m_t&=\beta_1m_{t-1}+(1-\beta_1)g_t, &v_t&=\beta_2v_{t-1}+(1-\beta_2)g_t^2,\\ \widehat m_t&=m_t/(1-\beta_1^t), &\widehat v_t&=v_t/(1-\beta_2^t),\\ \theta_t&=\theta_{t-1}-\alpha\widehat m_t/(\sqrt{\widehat v_t}+\epsilon). \end{aligned}

Here tt counts updates, gtg_t is the current gradient, mtm_t and vtv_t are first- and second-raw-moment estimates, and hats denote bias correction. The decay factors β1,β2\beta_1,\beta_2 lie between zero and one; α\alpha is the step size and ϵ\epsilon stabilizes division. Squaring, square roots and division act coordinatewise. Importantly, vtv_t estimates an uncentered second moment, not gradient variance.

Figure 4 compares bias-corrected and uncorrected variants across hyperparameters. The correction compensates for buffers starting at zero; it does not remove bias in the training dataset or make gradients noiseless.

Evidence

The original experiments include classification objectives and a variational autoencoder. Figure 4 reports the autoencoder’s training loss after 10 and 100 epochs; its large-second-moment-decay settings expose instability without correction. Figure 1 likewise tracks training objectives. Faster loss reduction in these experiments is evidence about optimization under those settings, not a guarantee of superior test accuracy.

Design analysis

The numerator smooths direction; the denominator rescales coordinates using recent squared gradients. Both retain history, so inspecting only the current gradient cannot explain an update. The later paper [30] constructs a convex counterexample and identifies a flaw in earlier convergence reasoning. Consequently, neither the original theorem nor the successful training curves justify saying that ordinary Adam always converges, even for every convex problem. Empirical utility and a universal guarantee are different claims.

Reading checkpoint

For a constant nonzero gradient, compute the first corrected moments from zero buffers. Explain why the correction factors cancel the initial attenuation. Then consider a suddenly changing gradient: which stored quantities keep information from previous steps, and why does bias correction not erase that history?

Limits and next step

Learning rate, decay factors, numerical stability and the objective still matter. A good result on one training loss does not establish the best generalization or a fair comparison when tuning budgets differ. Return to the chapter’s optimization diagnostics, then read ML 11 evaluation before selecting an optimizer from a single curve. Keep reference [30] beside [19] whenever discussing convergence.

Continue reading: Temperature Scaling — confidence without changing the class

References

  1. Shyandram, “機器學習及類神經網路筆記 (Notes on machine learning and neural networks),” Medium post (in Traditional Chinese), 2024; updated 2026. link
  2. Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. doi
  3. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. online book
  4. F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, pp. 386–408, 1958. doi
  5. G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, pp. 303–314, 1989. doi
  6. D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986. doi
  7. S. R. Dubey, S. K. Singh and B. B. Chaudhuri, “Activation functions in deep learning: A comprehensive survey and benchmark,” Neurocomputing, vol. 503, pp. 92–108, 2022. arXiv
  8. D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” arXiv:1606.08415, 2016. arXiv
  9. S. Elfwing, E. Uchibe and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural Networks, vol. 107, pp. 3–11, 2018. arXiv
  10. P. Ramachandran, B. Zoph and Q. V. Le, “Searching for activation functions,” arXiv:1710.05941, 2017. arXiv
  11. N. Shazeer, “GLU variants improve Transformer,” arXiv:2002.05202, 2020. arXiv
  12. H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023. arXiv
  13. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. International Conference on Machine Learning (ICML), PMLR 37, pp. 448–456, 2015. PMLR
  14. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer normalization,” arXiv:1607.06450, 2016. arXiv
  15. B. Zhang and R. Sennrich, “Root mean square layer normalization,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv
  16. L. Huang, J. Qin, Y. Zhou, F. Zhu, L. Liu and L. Shao, “Normalization techniques in training DNNs: Methodology, analysis and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10173–10196, 2023. arXiv
  17. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9, pp. 249–256, 2010. PMLR
  18. K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015. arXiv
  19. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
  20. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. International Conference on Learning Representations (ICLR), 2019. arXiv
  21. X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu and Q. V. Le, “Symbolic discovery of optimization algorithms,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv
  22. K. Jordan, “Muon: An optimizer for hidden layers in neural networks,” Medium post, December 2024. link
  23. J. Liu, J. Su, X. Yao et al., “Muon is scalable for LLM training,” arXiv:2502.16982, 2025. arXiv
  24. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
  25. N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
  26. R. Sun, “Optimization for deep learning: Theory and algorithms,” arXiv:1912.08957, 2019. arXiv
  27. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. JMLR
  28. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. arXiv
  29. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv
  30. S. J. Reddi, S. Kale and S. Kumar, “On the Convergence of Adam and Beyond,” ICLR, 2018. arXiv