完整搜尋頁面 →

Accurate, Large Minibatch SGD(Goyal 等人)

選讀・DLCV 01A・深度視覺模型的訓練實務 的論文導讀(第 1/2 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 帶重啟的餘弦退火
  2. 損失不是單純平均時的梯度累積
  3. 權重的 EMA:如何選擇衰減率
  4. 逐層學習率與逐步解凍

論文導讀 · 2

  1. Accurate, Large Minibatch SGD(Goyal 等人)
  2. Fine-Tuning can Distort Pretrained Features(Kumar 等人)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

同時用許多 GPU 訓練,需要很大的總批次,因為每張 GPU 都需要足夠的工作量才有效率。但先前已有報告指出,較大的批次會損害準確率。Goyal 等人問:這個損失是不是一個可以修好的最佳化問題 [9]。

比較同一大批次的無暖身、固定暖身與漸進暖身;比較最終準確率前,也要看初期最佳化路徑。
原論文圖 2 — 比較同一大批次的無暖身、固定暖身與漸進暖身;比較最終準確率前,也要看初期最佳化路徑。

圖片來源: [9], arXiv:1706.02677v2, PDF 第 8 頁. 原始 PDF.

方法

兩條規則:讓學習率隨批次大小線性縮放,並在最初的幾個 epoch 中逐漸升高。縮放規則來自一個簡單的比較。以小批次與學習率 η\eta 做 kk 步 SGD,權重移動的量是 η\eta 乘上 kk 個小批次平均梯度的總和,每個梯度都取自稍有不同的點;用大 kk 倍的批次做一步,移動的量是 η^\hat{\eta} 乘上同樣這些梯度在起點取得的平均。如果梯度在這 kk 步之間幾乎不變,那麼當 η^=kη\hat{\eta} = k\eta 時兩個更新相符。論文指出這個假設不成立的兩種情況:訓練早期網路變化很快(熱身處理了這點),以及超過某個批次大小之後,準確率開始退化。

結果

ResNet-50 在 ImageNet 上,用 256 張 GPU、8,192 張影像的批次,一小時內訓練完成,準確率與批次 256 的基準相當 [9]。

判讀證據

關鍵的表格比較批次 8k 下的各種熱身策略與基準,每項都是 5 次試驗的 top-1 驗證誤差的平均值與標準差:基準 23.60% ± 0.12、無熱身 24.84% ± 0.37、固定熱身 25.88% ± 0.56、漸進熱身 23.74% ± 0.09 [9]。分三步讀它。第一,看每一列的分散程度:小於約兩個標準差的差異沒有意義。第二,看意外之處:用固定低學習率熱身,反而比沒有熱身更差,因為切換到完整學習率會造成尖峰,訓練從此無法完全恢復。第三,作者把無熱身的差距歸因於較高的訓練誤差,而不是較差的泛化,這正是它被視為最佳化問題的原因。

實作陷阱

有權重衰減時,縮放交叉熵損失並不等同於縮放學習率;如果實作把學習率折進動量緩衝區,學習率改變時動量需要修正;BN 統計量是在每張 GPU 的 32 張影像上計算的,而這個每個工作行程的批次是損失定義的一部分;每個殘差區塊中最後一個 BN 採用零 γ\gamma 初始化是有幫助的,在大批次下尤其如此。

限制與問題

實驗是以 SGD 加動量做 ImageNet 分類,另外把預訓練模型遷移到偵測與實例分割;這條規則是一個近似,其極限(他們的設定中約 8k 張影像)取決於任務與模型,而且它對自適應最佳化器沒有直接的說法。

結論

它是本篇兩個預設值的來源,也就是搭配熱身的線性縮放,以及零 γ\gamma 的殘差初始化,同時也是跨多次試驗報告變異的範例。

參考文獻

  1. T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proc. IEEE/CVF CVPR, 2019. doi
  2. A. Karpathy, “A recipe for training neural networks,” blog post, 2019. link
  3. scikit-learn developers, “load_digits,” scikit-learn documentation, version 1.9.1. link
  4. M. Buda, A. Maki and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, 2018. doi
  5. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
  6. A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, 2020. doi
  7. C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, 2019. doi
  8. K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE ICCV, 2015. doi
  9. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia and K. He, “Accurate, large minibatch SGD: Training ImageNet in 1 hour,” arXiv:1706.02677 (technical report), 2017. arXiv
  10. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, PMLR 37, 2015. link
  11. Y. Wu and K. He, “Group normalization,” in Proc. ECCV, 2018. doi
  12. Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
  13. L. N. Smith, “Cyclical learning rates for training neural networks,” in Proc. IEEE WACV, 2017. doi
  14. L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay,” US Naval Research Laboratory Technical Report 5510-026, arXiv:1803.09820, 2018. arXiv
  15. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. ICLR, 2017. arXiv
  16. L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Proc. SPIE 11006, 2019. doi
  17. PyTorch contributors, “torch.optim.lr_scheduler.OneCycleLR,” PyTorch 2.15 documentation, 2026. link
  18. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE CVPR, 2016. doi
  19. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
  20. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015. arXiv
  21. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019. arXiv
  22. PyTorch contributors, “torch.optim.Adam,” PyTorch 2.15 documentation, 2026. link
  23. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, 2014. link
  24. G. Huang, Y. Sun, Z. Liu, D. Sedra and K. Q. Weinberger, “Deep networks with stochastic depth,” in Proc. ECCV, 2016. doi
  25. L. Prechelt, “Early stopping — but when?,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
  26. V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue and Z. Nado, “Deep learning tuning playbook,” Google Research, GitHub repository, 2023. link
  27. B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM Journal on Control and Optimization, vol. 30, no. 4, 1992. doi
  28. P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Proc. UAI, 2018. arXiv
  29. D. Morales-Brotons, T. Vogels and H. Hendrikx, “Exponential moving average of weights in deep learning: Dynamics and benefits,” Transactions on Machine Learning Research, 2024. arXiv
  30. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, “Mixed precision training,” in Proc. ICLR, 2018. arXiv
  31. PyTorch contributors, “Automatic mixed precision package — torch.amp,” PyTorch 2.15 documentation, 2026. link
  32. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems 27, 2014. link
  33. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. arXiv
  34. J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in Proc. ACL, 2018. doi
  35. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” in Proc. ICLR, 2022. arXiv
  36. C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in Proc. ICLR, 2017. arXiv
  37. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, 2020. doi
  38. PyTorch contributors, “Reproducibility,” PyTorch 2.15 documentation, 2026. link
  39. Y. Xin, J. Yang, S. Luo, Y. Du, Q. Qin, K. Cen, Y. He, Z. Zhang, B. Fu, X. Yang, G. Zhai, M.-H. Yang and X. Liu, “Parameter-efficient fine-tuning for pre-trained vision models: A survey and benchmark,” arXiv:2402.02242, 2024. arXiv
  40. G. E. Dahl, F. Schneider, Z. Nado, N. Agarwal, C. S. Sastry, P. Hennig, S. Medapati, R. Eschenhagen, P. Kasimbeg, D. Suo et al., “Benchmarking neural network training algorithms,” arXiv:2306.07179, 2023. arXiv
  41. P. Kasimbeg, F. Schneider, R. Eschenhagen, J. Bae, C. S. Sastry, M. Saroufim, B. Feng, L. Wright, E. Z. Yang, Z. Nado, S. Medapati, P. Hennig, M. Rabbat and G. E. Dahl, “Accelerating neural network training: An analysis of the AlgoPerf competition,” in Proc. ICLR, 2025. arXiv
  42. A. Defazio, X. A. Yang, H. Mehta, K. Mishchenko, A. Khaled and A. Cutkosky, “The road less scheduled,” in Advances in Neural Information Processing Systems 37, 2024. link
  43. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao et al., “MiniCPM: Unveiling the potential of small language models with scalable training strategies,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
  44. A. Hägele, E. Bakouch, A. Kosson, L. Ben Allal, L. Von Werra and M. Jaggi, “Scaling laws and compute-optimal training beyond fixed training durations,” in Advances in Neural Information Processing Systems 37, 2024. link
  45. M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. Gontijo Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong and L. Schmidt, “Robust fine-tuning of zero-shot models,” in Proc. IEEE/CVF CVPR, 2022. doi
  46. M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith and L. Schmidt, “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proc. ICML, PMLR 162, 2022. link
  47. PyTorch contributors, “Automatic mixed precision examples,” PyTorch 2.15 documentation, 2026. link
  48. PyTorch contributors, “DistributedDataParallel,” PyTorch 2.15.0 source documentation, 2026. Official source.