損失不是單純平均時的梯度累積
選讀・DLCV 01A・深度視覺模型的訓練實務 的補充資料(第 2/4 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
這則補充給出當各微批次對損失貢獻的項目數不同時的正確加權,並列出還有哪些東西必須跟著最佳化器更新而不是微批次。
假設微批次 含有 個損失項目(有效像素、配對的框或 token),其總和損失為 ,其中 是微批次 中項目 的損失。完整批次的損失是所有項目的平均,
其中 是微批次數, 是項目總數。對微批次平均值再取平均,,不論大小,每個微批次的權重都相同,只要 不同,這就是不同的目標(練習 4)。
步驟。(1)對每個微批次,計算總和損失 與數量 。(2)對 做反向傳播並把梯度相加。(3)在最佳化器更新前,把累積的梯度除以 。如果 要等所有微批次都看過才知道(通常如此),梯度就先不縮放。必須滿足 ;若整個累積週期沒有有效損失項目,就無法定義這個平均值。
跨裝置時,要確認歸約慣例。 令 為參與的程序數、 為全域有效項目數、 為程序 上總和損失的梯度。加總歸約得到 ,應除以 。PyTorch DistributedDataParallel 預設會平均梯度 [48],得到 ;這時應乘以 ,才能還原全域逐項平均。若把已經平均的梯度再除以 ,會額外縮小 倍。自訂通訊掛鉤或不均勻輸入政策可能改變歸約方式,因此使用此係數前必須確認慣例。
跟著最佳化器更新的計數器。學習率排程、權重衰減、梯度裁剪與 EMA 更新,都應該每次最佳化器更新前進一次,而不是每個微批次一次。裁剪必須套用在累積的梯度上,也就是除以 之後;對每個微批次分別裁剪,裁剪的是不同的量。
累積無法改變的事。批次正規化看到的仍然只是微批次;對比式損失中的批次內負樣本也一樣,每個範例的負樣本數是由微批次大小決定的。每個微批次的 dropout 遮罩不同,這是無害的。使用混合精度與損失縮放器時,PyTorch 文件規定,在一個累積週期內,梯度保持縮放狀態,縮放係數保持不變,而 inf/NaN 檢查、跳過更新與縮放係數更新,每個有效批次只發生一次 [47]。
參考文獻
- T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proc. IEEE/CVF CVPR, 2019. doi
- A. Karpathy, “A recipe for training neural networks,” blog post, 2019. link
- scikit-learn developers, “load_digits,” scikit-learn documentation, version 1.9.1. link
- M. Buda, A. Maki and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, 2018. doi
- T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
- A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, 2020. doi
- C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, 2019. doi
- K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE ICCV, 2015. doi
- P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia and K. He, “Accurate, large minibatch SGD: Training ImageNet in 1 hour,” arXiv:1706.02677 (technical report), 2017. arXiv
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, PMLR 37, 2015. link
- Y. Wu and K. He, “Group normalization,” in Proc. ECCV, 2018. doi
- Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
- L. N. Smith, “Cyclical learning rates for training neural networks,” in Proc. IEEE WACV, 2017. doi
- L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay,” US Naval Research Laboratory Technical Report 5510-026, arXiv:1803.09820, 2018. arXiv
- I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. ICLR, 2017. arXiv
- L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Proc. SPIE 11006, 2019. doi
- PyTorch contributors, “torch.optim.lr_scheduler.OneCycleLR,” PyTorch 2.15 documentation, 2026. link
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE CVPR, 2016. doi
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015. arXiv
- I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019. arXiv
- PyTorch contributors, “torch.optim.Adam,” PyTorch 2.15 documentation, 2026. link
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, 2014. link
- G. Huang, Y. Sun, Z. Liu, D. Sedra and K. Q. Weinberger, “Deep networks with stochastic depth,” in Proc. ECCV, 2016. doi
- L. Prechelt, “Early stopping — but when?,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
- V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue and Z. Nado, “Deep learning tuning playbook,” Google Research, GitHub repository, 2023. link
- B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM Journal on Control and Optimization, vol. 30, no. 4, 1992. doi
- P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Proc. UAI, 2018. arXiv
- D. Morales-Brotons, T. Vogels and H. Hendrikx, “Exponential moving average of weights in deep learning: Dynamics and benefits,” Transactions on Machine Learning Research, 2024. arXiv
- P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, “Mixed precision training,” in Proc. ICLR, 2018. arXiv
- PyTorch contributors, “Automatic mixed precision package — torch.amp,” PyTorch 2.15 documentation, 2026. link
- J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems 27, 2014. link
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. arXiv
- J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in Proc. ACL, 2018. doi
- A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” in Proc. ICLR, 2022. arXiv
- C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in Proc. ICLR, 2017. arXiv
- R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, 2020. doi
- PyTorch contributors, “Reproducibility,” PyTorch 2.15 documentation, 2026. link
- Y. Xin, J. Yang, S. Luo, Y. Du, Q. Qin, K. Cen, Y. He, Z. Zhang, B. Fu, X. Yang, G. Zhai, M.-H. Yang and X. Liu, “Parameter-efficient fine-tuning for pre-trained vision models: A survey and benchmark,” arXiv:2402.02242, 2024. arXiv
- G. E. Dahl, F. Schneider, Z. Nado, N. Agarwal, C. S. Sastry, P. Hennig, S. Medapati, R. Eschenhagen, P. Kasimbeg, D. Suo et al., “Benchmarking neural network training algorithms,” arXiv:2306.07179, 2023. arXiv
- P. Kasimbeg, F. Schneider, R. Eschenhagen, J. Bae, C. S. Sastry, M. Saroufim, B. Feng, L. Wright, E. Z. Yang, Z. Nado, S. Medapati, P. Hennig, M. Rabbat and G. E. Dahl, “Accelerating neural network training: An analysis of the AlgoPerf competition,” in Proc. ICLR, 2025. arXiv
- A. Defazio, X. A. Yang, H. Mehta, K. Mishchenko, A. Khaled and A. Cutkosky, “The road less scheduled,” in Advances in Neural Information Processing Systems 37, 2024. link
- S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao et al., “MiniCPM: Unveiling the potential of small language models with scalable training strategies,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
- A. Hägele, E. Bakouch, A. Kosson, L. Ben Allal, L. Von Werra and M. Jaggi, “Scaling laws and compute-optimal training beyond fixed training durations,” in Advances in Neural Information Processing Systems 37, 2024. link
- M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. Gontijo Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong and L. Schmidt, “Robust fine-tuning of zero-shot models,” in Proc. IEEE/CVF CVPR, 2022. doi
- M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith and L. Schmidt, “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proc. ICML, PMLR 162, 2022. link
- PyTorch contributors, “Automatic mixed precision examples,” PyTorch 2.15 documentation, 2026. link
- PyTorch contributors, “DistributedDataParallel,” PyTorch 2.15.0 source documentation, 2026. Official source.