完整搜尋頁面 →

損失不是單純平均時的梯度累積

選讀・DLCV 01A・深度視覺模型的訓練實務 的補充資料(第 2/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 4

  1. 帶重啟的餘弦退火
  2. 損失不是單純平均時的梯度累積
  3. 權重的 EMA:如何選擇衰減率
  4. 逐層學習率與逐步解凍

論文導讀 · 2

  1. Accurate, Large Minibatch SGD(Goyal 等人)
  2. Fine-Tuning can Distort Pretrained Features(Kumar 等人)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則補充給出當各微批次對損失貢獻的項目數不同時的正確加權,並列出還有哪些東西必須跟著最佳化器更新而不是微批次。

假設微批次 jj 含有 njn_j 個損失項目(有效像素、配對的框或 token),其總和損失為 Sj=∑i=1njℓijS_j = \sum_{i=1}^{n_j} \ell_{ij},其中 ℓij\ell_{ij} 是微批次 jj 中項目 ii 的損失。完整批次的損失是所有項目的平均,

L=∑j=1kSj∑j=1knj,∇θL=∑j=1k∇θSjN,N=∑j=1knj,L = \frac{\sum_{j=1}^{k} S_j}{\sum_{j=1}^{k} n_j}, \qquad \nabla_\theta L = \frac{\sum_{j=1}^{k} \nabla_\theta S_j}{N}, \quad N = \sum_{j=1}^{k} n_j ,

其中 kk 是微批次數,NN 是項目總數。對微批次平均值再取平均,1k∑jSj/nj\tfrac{1}{k}\sum_j S_j / n_j,不論大小,每個微批次的權重都相同,只要 njn_j 不同,這就是不同的目標(練習 4)。

步驟。(1)對每個微批次,計算總和損失 SjS_j 與數量 njn_j。(2)對 SjS_j 做反向傳播並把梯度相加。(3)在最佳化器更新前,把累積的梯度除以 NN。如果 NN 要等所有微批次都看過才知道(通常如此),梯度就先不縮放。必須滿足 N>0N>0;若整個累積週期沒有有效損失項目,就無法定義這個平均值。

跨裝置時,要確認歸約慣例。 令 RR 為參與的程序數、NN 為全域有效項目數、GrG_r 為程序 rr 上總和損失的梯度。加總歸約得到 ∑rGr\sum_r G_r,應除以 NN。PyTorch DistributedDataParallel 預設會平均梯度 [48],得到 1R∑rGr\frac{1}{R}\sum_r G_r;這時應乘以 R/NR/N,才能還原全域逐項平均。若把已經平均的梯度再除以 NN,會額外縮小 RR 倍。自訂通訊掛鉤或不均勻輸入政策可能改變歸約方式,因此使用此係數前必須確認慣例。

跟著最佳化器更新的計數器。學習率排程、權重衰減、梯度裁剪與 EMA 更新,都應該每次最佳化器更新前進一次,而不是每個微批次一次。裁剪必須套用在累積的梯度上,也就是除以 NN 之後;對每個微批次分別裁剪,裁剪的是不同的量。

累積無法改變的事。批次正規化看到的仍然只是微批次;對比式損失中的批次內負樣本也一樣,每個範例的負樣本數是由微批次大小決定的。每個微批次的 dropout 遮罩不同,這是無害的。使用混合精度與損失縮放器時,PyTorch 文件規定,在一個累積週期內,梯度保持縮放狀態,縮放係數保持不變,而 inf/NaN 檢查、跳過更新與縮放係數更新,每個有效批次只發生一次 [47]。

參考文獻

  1. T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proc. IEEE/CVF CVPR, 2019. doi
  2. A. Karpathy, “A recipe for training neural networks,” blog post, 2019. link
  3. scikit-learn developers, “load_digits,” scikit-learn documentation, version 1.9.1. link
  4. M. Buda, A. Maki and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, 2018. doi
  5. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
  6. A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, 2020. doi
  7. C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, 2019. doi
  8. K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE ICCV, 2015. doi
  9. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia and K. He, “Accurate, large minibatch SGD: Training ImageNet in 1 hour,” arXiv:1706.02677 (technical report), 2017. arXiv
  10. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, PMLR 37, 2015. link
  11. Y. Wu and K. He, “Group normalization,” in Proc. ECCV, 2018. doi
  12. Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
  13. L. N. Smith, “Cyclical learning rates for training neural networks,” in Proc. IEEE WACV, 2017. doi
  14. L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay,” US Naval Research Laboratory Technical Report 5510-026, arXiv:1803.09820, 2018. arXiv
  15. I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. ICLR, 2017. arXiv
  16. L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Proc. SPIE 11006, 2019. doi
  17. PyTorch contributors, “torch.optim.lr_scheduler.OneCycleLR,” PyTorch 2.15 documentation, 2026. link
  18. K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE CVPR, 2016. doi
  19. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
  20. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015. arXiv
  21. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019. arXiv
  22. PyTorch contributors, “torch.optim.Adam,” PyTorch 2.15 documentation, 2026. link
  23. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, 2014. link
  24. G. Huang, Y. Sun, Z. Liu, D. Sedra and K. Q. Weinberger, “Deep networks with stochastic depth,” in Proc. ECCV, 2016. doi
  25. L. Prechelt, “Early stopping — but when?,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
  26. V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue and Z. Nado, “Deep learning tuning playbook,” Google Research, GitHub repository, 2023. link
  27. B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM Journal on Control and Optimization, vol. 30, no. 4, 1992. doi
  28. P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Proc. UAI, 2018. arXiv
  29. D. Morales-Brotons, T. Vogels and H. Hendrikx, “Exponential moving average of weights in deep learning: Dynamics and benefits,” Transactions on Machine Learning Research, 2024. arXiv
  30. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, “Mixed precision training,” in Proc. ICLR, 2018. arXiv
  31. PyTorch contributors, “Automatic mixed precision package — torch.amp,” PyTorch 2.15 documentation, 2026. link
  32. J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems 27, 2014. link
  33. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. arXiv
  34. J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in Proc. ACL, 2018. doi
  35. A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” in Proc. ICLR, 2022. arXiv
  36. C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in Proc. ICLR, 2017. arXiv
  37. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, 2020. doi
  38. PyTorch contributors, “Reproducibility,” PyTorch 2.15 documentation, 2026. link
  39. Y. Xin, J. Yang, S. Luo, Y. Du, Q. Qin, K. Cen, Y. He, Z. Zhang, B. Fu, X. Yang, G. Zhai, M.-H. Yang and X. Liu, “Parameter-efficient fine-tuning for pre-trained vision models: A survey and benchmark,” arXiv:2402.02242, 2024. arXiv
  40. G. E. Dahl, F. Schneider, Z. Nado, N. Agarwal, C. S. Sastry, P. Hennig, S. Medapati, R. Eschenhagen, P. Kasimbeg, D. Suo et al., “Benchmarking neural network training algorithms,” arXiv:2306.07179, 2023. arXiv
  41. P. Kasimbeg, F. Schneider, R. Eschenhagen, J. Bae, C. S. Sastry, M. Saroufim, B. Feng, L. Wright, E. Z. Yang, Z. Nado, S. Medapati, P. Hennig, M. Rabbat and G. E. Dahl, “Accelerating neural network training: An analysis of the AlgoPerf competition,” in Proc. ICLR, 2025. arXiv
  42. A. Defazio, X. A. Yang, H. Mehta, K. Mishchenko, A. Khaled and A. Cutkosky, “The road less scheduled,” in Advances in Neural Information Processing Systems 37, 2024. link
  43. S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao et al., “MiniCPM: Unveiling the potential of small language models with scalable training strategies,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
  44. A. Hägele, E. Bakouch, A. Kosson, L. Ben Allal, L. Von Werra and M. Jaggi, “Scaling laws and compute-optimal training beyond fixed training durations,” in Advances in Neural Information Processing Systems 37, 2024. link
  45. M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. Gontijo Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong and L. Schmidt, “Robust fine-tuning of zero-shot models,” in Proc. IEEE/CVF CVPR, 2022. doi
  46. M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith and L. Schmidt, “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proc. ICML, PMLR 162, 2022. link
  47. PyTorch contributors, “Automatic mixed precision examples,” PyTorch 2.15 documentation, 2026. link
  48. PyTorch contributors, “DistributedDataParallel,” PyTorch 2.15.0 source documentation, 2026. Official source.