Full search page →

Cosine annealing with warm restarts

Optional · supplementary note 1 of 4 for DLCV 01A · Training Deep Vision Models in Practice. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 4

  1. Cosine annealing with warm restarts
  2. Gradient accumulation when the loss is not a plain mean
  3. EMA of weights: choosing the decay
  4. Discriminative learning rates and gradual unfreezing

Paper guides · 2

  1. Accurate, Large Minibatch SGD (Goyal et al.)
  2. Fine-Tuning can Distort Pretrained Features (Kumar et al.)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

This note gives the full SGDR schedule, of which the main text uses only the single cosine decay.

Loshchilov and Hutter’s SGDR (stochastic gradient descent with warm restarts) runs several cosine decays one after another [15]. Within the ii-th run, the rate is

ηt=ηmin⁡i+12(ηmax⁡i−ηmin⁡i)(1+cos⁡ ⁣(π TcurTi)),\eta_t = \eta_{\min}^{i} + \tfrac12\left(\eta_{\max}^{i} - \eta_{\min}^{i}\right)\left(1 + \cos\!\left(\pi\,\frac{T_{\text{cur}}}{T_i}\right)\right),

where ηmax⁡i\eta_{\max}^{i} and ηmin⁡i\eta_{\min}^{i} are the upper and lower rates of that run, TcurT_{\text{cur}} is the number of epochs since the last restart (updated at every batch, so it takes fractional values), and TiT_i is the length of run ii in epochs. When Tcur=0T_{\text{cur}} = 0 the rate is ηmax⁡i\eta_{\max}^{i}; when Tcur=TiT_{\text{cur}} = T_i it reaches ηmin⁡i\eta_{\min}^{i}, and the next run restarts at the top. The lengths grow geometrically, Ti+1=Tmult⋅TiT_{i+1} = T_{\text{mult}} \cdot T_i, starting from T0T_0; the paper studies settings such as T0=1,Tmult=2T_0 = 1, T_{\text{mult}} = 2 and T0=10,Tmult=2T_0 = 10, T_{\text{mult}} = 2 [15].

Steps. (1) Choose T0T_0 and TmultT_{\text{mult}}. (2) Track TcurT_{\text{cur}} in fractional epochs. (3) At Tcur=TiT_{\text{cur}} = T_i, set Tcur=0T_{\text{cur}} = 0 and Ti+1=TmultTiT_{i+1} = T_{\text{mult}} T_i.

Why restart? The paper’s motivation is anytime performance: at the end of every run the rate is small, so the model is in a good state at many points of training, not only at the end. The paper also builds an ensemble from the snapshots taken just before each restart [15].

Pitfalls. The total length should end exactly at the end of a run; with T0=1T_0 = 1 and Tmult=2T_{\text{mult}} = 2 that means 1+2+4+…1 + 2 + 4 + \dots epochs, for example 31 or 63. A run that is cut in the middle of a cycle ends at a high rate and gives a poor final model. The jump back to ηmax⁡\eta_{\max} at each restart briefly increases the loss, which is expected; if you combine restarts with a warmup, decide explicitly whether each restart repeats it.

References

  1. T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proc. IEEE/CVF CVPR, 2019. doi
  2. A. Karpathy, “A recipe for training neural networks,” blog post, 2019. link
  3. scikit-learn developers, “load_digits,” scikit-learn documentation, version 1.9.1. link
  4. M. Buda, A. Maki and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, 2018. doi
  5. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
  6. A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, 2020. doi
  7. C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, 2019. doi
  8. K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE ICCV, 2015. doi
  9. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia and K. He, “Accurate, large minibatch SGD: Training ImageNet in 1 hour,” arXiv:1706.02677 (technical report), 2017. arXiv
  10. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, PMLR 37, 2015. link
  11. Y. Wu and K. He, “Group normalization,” in Proc. ECCV, 2018. doi
  12. Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
  13. L. N. Smith, “Cyclical learning rates for training neural networks,” in Proc. IEEE WACV, 2017. doi
  14. L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay,” US Naval Research Laboratory Technical Report 5510-026, arXiv:1803.09820, 2018. arXiv
  15. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. ICLR, 2017. arXiv
  16. L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Proc. SPIE 11006, 2019. doi
  17. PyTorch contributors, “torch.optim.lr_scheduler.OneCycleLR,” PyTorch 2.15 documentation, 2026. link
  18. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE CVPR, 2016. doi
  19. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
  20. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015. arXiv
  21. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019. arXiv
  22. PyTorch contributors, “torch.optim.Adam,” PyTorch 2.15 documentation, 2026. link
  23. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, 2014. link
  24. G. Huang, Y. Sun, Z. Liu, D. Sedra and K. Q. Weinberger, “Deep networks with stochastic depth,” in Proc. ECCV, 2016. doi
  25. L. Prechelt, “Early stopping — but when?,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
  26. V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue and Z. Nado, “Deep learning tuning playbook,” Google Research, GitHub repository, 2023. link
  27. B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM Journal on Control and Optimization, vol. 30, no. 4, 1992. doi
  28. P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Proc. UAI, 2018. arXiv
  29. D. Morales-Brotons, T. Vogels and H. Hendrikx, “Exponential moving average of weights in deep learning: Dynamics and benefits,” Transactions on Machine Learning Research, 2024. arXiv
  30. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, “Mixed precision training,” in Proc. ICLR, 2018. arXiv
  31. PyTorch contributors, “Automatic mixed precision package — torch.amp,” PyTorch 2.15 documentation, 2026. link
  32. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems 27, 2014. link
  33. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. arXiv
  34. J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in Proc. ACL, 2018. doi
  35. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” in Proc. ICLR, 2022. arXiv
  36. C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in Proc. ICLR, 2017. arXiv
  37. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, 2020. doi
  38. PyTorch contributors, “Reproducibility,” PyTorch 2.15 documentation, 2026. link
  39. Y. Xin, J. Yang, S. Luo, Y. Du, Q. Qin, K. Cen, Y. He, Z. Zhang, B. Fu, X. Yang, G. Zhai, M.-H. Yang and X. Liu, “Parameter-efficient fine-tuning for pre-trained vision models: A survey and benchmark,” arXiv:2402.02242, 2024. arXiv
  40. G. E. Dahl, F. Schneider, Z. Nado, N. Agarwal, C. S. Sastry, P. Hennig, S. Medapati, R. Eschenhagen, P. Kasimbeg, D. Suo et al., “Benchmarking neural network training algorithms,” arXiv:2306.07179, 2023. arXiv
  41. P. Kasimbeg, F. Schneider, R. Eschenhagen, J. Bae, C. S. Sastry, M. Saroufim, B. Feng, L. Wright, E. Z. Yang, Z. Nado, S. Medapati, P. Hennig, M. Rabbat and G. E. Dahl, “Accelerating neural network training: An analysis of the AlgoPerf competition,” in Proc. ICLR, 2025. arXiv
  42. A. Defazio, X. A. Yang, H. Mehta, K. Mishchenko, A. Khaled and A. Cutkosky, “The road less scheduled,” in Advances in Neural Information Processing Systems 37, 2024. link
  43. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao et al., “MiniCPM: Unveiling the potential of small language models with scalable training strategies,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
  44. A. Hägele, E. Bakouch, A. Kosson, L. Ben Allal, L. Von Werra and M. Jaggi, “Scaling laws and compute-optimal training beyond fixed training durations,” in Advances in Neural Information Processing Systems 37, 2024. link
  45. M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. Gontijo Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong and L. Schmidt, “Robust fine-tuning of zero-shot models,” in Proc. IEEE/CVF CVPR, 2022. doi
  46. M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith and L. Schmidt, “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proc. ICML, PMLR 162, 2022. link
  47. PyTorch contributors, “Automatic mixed precision examples,” PyTorch 2.15 documentation, 2026. link
  48. PyTorch contributors, “DistributedDataParallel,” PyTorch 2.15.0 source documentation, 2026. Official source.