Adam — adaptive updates and the convergence caveat
Optional · paper guide 1 of 1 for ML 02 · Neural Networks from Perceptron to Deep Networks. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
How can an optimizer combine a smoothed direction with a separate step scale for each parameter? Adam maintains two moving averages and corrects their initialization bias [19]. Start with gradients and momentum. Its practical contribution is a compact update rule; its original convergence discussion must also be read alongside the later correction by Reddi, Kale and Kumar [30].
Method and figure

With zero-initialized moment buffers, the update is
Here counts updates, is the current gradient, and are first- and second-raw-moment estimates, and hats denote bias correction. The decay factors lie between zero and one; is the step size and stabilizes division. Squaring, square roots and division act coordinatewise. Importantly, estimates an uncentered second moment, not gradient variance.
Figure 4 compares bias-corrected and uncorrected variants across hyperparameters. The correction compensates for buffers starting at zero; it does not remove bias in the training dataset or make gradients noiseless.
Evidence
The original experiments include classification objectives and a variational autoencoder. Figure 4 reports the autoencoder’s training loss after 10 and 100 epochs; its large-second-moment-decay settings expose instability without correction. Figure 1 likewise tracks training objectives. Faster loss reduction in these experiments is evidence about optimization under those settings, not a guarantee of superior test accuracy.
Design analysis
The numerator smooths direction; the denominator rescales coordinates using recent squared gradients. Both retain history, so inspecting only the current gradient cannot explain an update. The later paper [30] constructs a convex counterexample and identifies a flaw in earlier convergence reasoning. Consequently, neither the original theorem nor the successful training curves justify saying that ordinary Adam always converges, even for every convex problem. Empirical utility and a universal guarantee are different claims.
Reading checkpoint
For a constant nonzero gradient, compute the first corrected moments from zero buffers. Explain why the correction factors cancel the initial attenuation. Then consider a suddenly changing gradient: which stored quantities keep information from previous steps, and why does bias correction not erase that history?
Limits and next step
Learning rate, decay factors, numerical stability and the objective still matter. A good result on one training loss does not establish the best generalization or a fair comparison when tuning budgets differ. Return to the chapter’s optimization diagnostics, then read ML 11 evaluation before selecting an optimizer from a single curve. Keep reference [30] beside [19] whenever discussing convergence.
Continue reading: Temperature Scaling — confidence without changing the class
References
- Shyandram, “機器學習及類神經網路筆記 (Notes on machine learning and neural networks),” Medium post (in Traditional Chinese), 2024; updated 2026. link
- Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. doi
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016. online book
- F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, vol. 65, pp. 386–408, 1958. doi
- G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, pp. 303–314, 1989. doi
- D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986. doi
- S. R. Dubey, S. K. Singh and B. B. Chaudhuri, “Activation functions in deep learning: A comprehensive survey and benchmark,” Neurocomputing, vol. 503, pp. 92–108, 2022. arXiv
- D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” arXiv:1606.08415, 2016. arXiv
- S. Elfwing, E. Uchibe and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural Networks, vol. 107, pp. 3–11, 2018. arXiv
- P. Ramachandran, B. Zoph and Q. V. Le, “Searching for activation functions,” arXiv:1710.05941, 2017. arXiv
- N. Shazeer, “GLU variants improve Transformer,” arXiv:2002.05202, 2020. arXiv
- H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023. arXiv
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. International Conference on Machine Learning (ICML), PMLR 37, pp. 448–456, 2015. PMLR
- J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer normalization,” arXiv:1607.06450, 2016. arXiv
- B. Zhang and R. Sennrich, “Root mean square layer normalization,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv
- L. Huang, J. Qin, Y. Zhou, F. Zhu, L. Liu and L. Shao, “Normalization techniques in training DNNs: Methodology, analysis and application,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 10173–10196, 2023. arXiv
- X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9, pp. 249–256, 2010. PMLR
- K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015. arXiv
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
- I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. International Conference on Learning Representations (ICLR), 2019. arXiv
- X. Chen, C. Liang, D. Huang, E. Real, K. Wang, H. Pham, X. Dong, T. Luong, C.-J. Hsieh, Y. Lu and Q. V. Le, “Symbolic discovery of optimization algorithms,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv
- K. Jordan, “Muon: An optimizer for hidden layers in neural networks,” Medium post, December 2024. link
- J. Liu, J. Su, X. Yao et al., “Muon is scalable for LLM training,” arXiv:2502.16982, 2025. arXiv
- I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
- N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” in Proc. International Conference on Learning Representations (ICLR), 2017. arXiv
- R. Sun, “Optimization for deep learning: Theory and algorithms,” arXiv:1912.08957, 2019. arXiv
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014. JMLR
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv
- S. J. Reddi, S. Kale and S. Kumar, “On the Convergence of Adam and Beyond,” ICLR, 2018. arXiv