完整搜尋頁面 →

Temperature Scaling:不改類別的信心校準

選讀・ML 11・評估、效率與部署 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

分類器說自己很有信心時,實際有多常答對?Guo 等人顯示現代神經分類器可能校準不佳,並研究簡單的訓練後修正 [11]。先讀 logits、softmax、保留驗證資料與準確度。校準關心宣告信心與觀察頻率是否一致,與選對類別是不同性質。

方法與圖解

可靠度圖比較校準前後的信心與經驗準確度;溫度縮放改善兩者一致性,同時保留正溫度下的最大值類別。
來源:Guo et al., ICML 2017, PMLR 70 proceedings. 圖 4,PDF 第 8 頁。可靠度圖比較校準前後的信心與經驗準確度;溫度縮放改善兩者一致性,同時保留正溫度下的最大值類別。 [11]

溫度縮放使用

pT(y=k∣x)=exp⁡(zk(x)/T)∑jexp⁡(zj(x)/T).p_T(y=k\mid x)=\frac{\exp(z_k(x)/T)}{\sum_j\exp(z_j(x)/T)}.

其中 xx 為輸入,zj(x)z_j(x) 為第 jj 類的 logit,kk 為查詢類別,TT 為正純量溫度。固定分類器,在保留驗證資料上最小化負對數概似來擬合 TT,再用獨立測試資料評估。正溫度保留 logit 排序及預測類別,但改變信心。

論文也報告期望校準誤差:

ECE=∑m=1M∣Bm∣n∣acc⁡(Bm)−conf⁡(Bm)∣.\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|.

其中 nn 是評估樣本數,BmB_m 是第 mm 個信心區間,MM 是區間數,acc⁡\operatorname{acc}、conf⁡\operatorname{conf} 分別為該區間經驗準確度與平均預測信心;空區間貢獻為零。這個有限分箱摘要依賴分箱方式與樣本數,不等同擬合目標。

證據

圖 4 使用 CIFAR-100 上採 stochastic depth 的 ResNet-110。未校準 ECE 為 12.67%,溫度縮放面板為 0.96%。必須保留 stochastic depth 限定,不能誤當成文中其他地方的普通 ResNet-110。表 1 比較多種校準方法與資料集;整體有用,不表示每個個別設定都改善。

設計分析

單一參數限制彈性,容易擬合並保留分類決策。更靈活的校準映射可能改變排序,也可能過擬合有限驗證資料。可靠度圖應搭配準確度與負對數概似閱讀;低整體校準誤差可能隱藏不同信心區域或群體的差異。不可使用測試標籤來挑選被評估的溫度。

閱讀檢查點

解釋為何每個 logit 都除以同一正數,預測類別仍不變。再想像部署族群與驗證族群不同:即使溫度沒有改變,可靠度圖量測的哪種一致性仍可能失效?

限制與下一步

溫度縮放不會修復錯誤特徵,不會單靠自身提高分類準確度,也不保證分布外可靠性。驗證資料擬合的修正,需要在資料偏移時持續監測。回到本章部署檢查清單,在準確度之外,也記錄校準切分、擬合目標、分箱規則與評估族群。

接著閱讀: ML 11・回到章節

參考文獻

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 11 “Practical Methodology”. book
  2. G. James, D. Witten, T. Hastie, R. Tibshirani and J. Taylor, An Introduction to Statistical Learning: with Applications in Python, Springer, 2023. book
  3. scikit-learn developers, “Common pitfalls and recommended practices,” scikit-learn documentation. docs
  4. S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, 2023. doi
  5. S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection,” BMC Bioinformatics, vol. 7, art. 91, 2006. doi
  6. G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” Journal of Machine Learning Research, vol. 11, pp. 2079–2107, 2010. paper
  7. M. Bińkowski, D. J. Sutherland, M. Arbel and A. Gretton, “Demystifying MMD GANs,” ICLR, 2018. arXiv
  8. T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, 1998. doi
  9. B. Efron, “Bootstrap methods: another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, 1979. doi
  10. X. Bouthillier et al., “Accounting for variance in machine learning benchmarks,” MLSys, 2021. arXiv
  11. C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On calibration of modern neural networks,” ICML, 2017. arXiv
  12. M. Minderer et al., “Revisiting the calibration of modern neural networks,” NeurIPS, 2021. arXiv
  13. D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” ICLR, 2019. arXiv
  14. B. Recht, R. Roelofs, L. Schmidt and V. Shankar, “Do ImageNet classifiers generalize to ImageNet?,” ICML, 2019. paper
  15. P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” ICML, 2021. paper
  16. Y. Ovadia et al., “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” NeurIPS, 2019. arXiv
  17. Z. C. Lipton, Y.-X. Wang and A. Smola, “Detecting and correcting for label shift with black box predictors,” ICML, 2018. arXiv
  18. M. Dehghani, A. Arnab, L. Beyer, A. Vaswani and Y. Tay, “The efficiency misnomer,” ICLR, 2022. arXiv
  19. PyTorch, “torch.export,” PyTorch documentation. docs
  20. ONNX, “Open Neural Network Exchange: the open standard for machine learning interoperability.” site
  21. ONNX Runtime, “Welcome to ONNX Runtime,” documentation. docs
  22. NVIDIA, “NVIDIA TensorRT documentation.” docs
  23. B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” CVPR, 2018. doi
  24. M. Nagel et al., “A white paper on neural network quantization,” arXiv:2106.08295, 2021. arXiv
  25. A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” book chapter in Low-Power Computer Vision; arXiv:2103.13630, 2021. arXiv
  26. S. Han, H. Mao and W. J. Dally, “Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding,” ICLR, 2016. arXiv
  27. D. Blalock, J. J. Gonzalez Ortiz, J. Frankle and J. Guttag, “What is the state of neural network pruning?,” MLSys, 2020. arXiv
  28. T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden and A. Peste, “Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks,” Journal of Machine Learning Research, vol. 22, 2021. arXiv
  29. G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” NIPS Deep Learning Workshop, 2014; arXiv:1503.02531. arXiv
  30. J. Gou, B. Yu, S. J. Maybank and D. Tao, “Knowledge distillation: a survey,” International Journal of Computer Vision, vol. 129, 2021. doi
  31. Y. Cheng, D. Wang, P. Zhou and T. Zhang, “Model compression and acceleration for deep neural networks: the principles, progress, and challenges,” IEEE Signal Processing Magazine, vol. 35, no. 1, 2018. doi arXiv
  32. S. Rabanser, S. Günnemann and Z. C. Lipton, “Failing loudly: an empirical study of methods for detecting dataset shift,” NeurIPS, 2019. arXiv
  33. D. Sculley et al., “Hidden technical debt in machine learning systems,” NeurIPS, 2015. paper
  34. PyTorch, “Reproducibility,” PyTorch documentation. docs
  35. M. Mitchell et al., “Model cards for model reporting,” ACM FAT*, 2019. arXiv
  36. T. Gebru et al., “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, 2021. doi
  37. J. Pineau et al., “Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program),” Journal of Machine Learning Research, vol. 22, 2021. paper
  38. J. Gawlikowski et al., “A survey of uncertainty in deep neural networks,” Artificial Intelligence Review, vol. 56, 2023. doi
  39. A. Paleyes, R.-G. Urma and N. D. Lawrence, “Challenges in deploying machine learning: a survey of case studies,” ACM Computing Surveys, vol. 55, no. 6, 2022. doi
  40. P. Liang et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023. arXiv
  41. X. Zhu, J. Li, Y. Liu, C. Ma and W. Wang, “A survey on model compression for large language models,” Transactions of the Association for Computational Linguistics, vol. 12, 2024. doi
  42. E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh, “GPTQ: accurate post-training quantization for generative pre-trained transformers,” ICLR, 2023. arXiv
  43. J. Lin et al., “AWQ: activation-aware weight quantization for LLM compression and acceleration,” MLSys, 2024. arXiv
  44. J. Hoffmann et al., “Training compute-optimal large language models,” arXiv:2203.15556, 2022; NeurIPS 2022 version titled “An empirical analysis of compute-optimal large language model training.” arXiv
  45. X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling vision transformers,” CVPR, 2022. doi
  46. PyTorch, “torchao documentation.” docs