Temperature Scaling:不改類別的信心校準
選讀・ML 11・評估、效率與部署 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
分類器說自己很有信心時,實際有多常答對?Guo 等人顯示現代神經分類器可能校準不佳,並研究簡單的訓練後修正 [11]。先讀 logits、softmax、保留驗證資料與準確度。校準關心宣告信心與觀察頻率是否一致,與選對類別是不同性質。
方法與圖解

溫度縮放使用
其中 為輸入, 為第 類的 logit, 為查詢類別, 為正純量溫度。固定分類器,在保留驗證資料上最小化負對數概似來擬合 ,再用獨立測試資料評估。正溫度保留 logit 排序及預測類別,但改變信心。
論文也報告期望校準誤差:
其中 是評估樣本數, 是第 個信心區間, 是區間數,、 分別為該區間經驗準確度與平均預測信心;空區間貢獻為零。這個有限分箱摘要依賴分箱方式與樣本數,不等同擬合目標。
證據
圖 4 使用 CIFAR-100 上採 stochastic depth 的 ResNet-110。未校準 ECE 為 12.67%,溫度縮放面板為 0.96%。必須保留 stochastic depth 限定,不能誤當成文中其他地方的普通 ResNet-110。表 1 比較多種校準方法與資料集;整體有用,不表示每個個別設定都改善。
設計分析
單一參數限制彈性,容易擬合並保留分類決策。更靈活的校準映射可能改變排序,也可能過擬合有限驗證資料。可靠度圖應搭配準確度與負對數概似閱讀;低整體校準誤差可能隱藏不同信心區域或群體的差異。不可使用測試標籤來挑選被評估的溫度。
閱讀檢查點
解釋為何每個 logit 都除以同一正數,預測類別仍不變。再想像部署族群與驗證族群不同:即使溫度沒有改變,可靠度圖量測的哪種一致性仍可能失效?
限制與下一步
溫度縮放不會修復錯誤特徵,不會單靠自身提高分類準確度,也不保證分布外可靠性。驗證資料擬合的修正,需要在資料偏移時持續監測。回到本章部署檢查清單,在準確度之外,也記錄校準切分、擬合目標、分箱規則與評估族群。
接著閱讀: ML 11・回到章節
參考文獻
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 11 “Practical Methodology”. book
- G. James, D. Witten, T. Hastie, R. Tibshirani and J. Taylor, An Introduction to Statistical Learning: with Applications in Python, Springer, 2023. book
- scikit-learn developers, “Common pitfalls and recommended practices,” scikit-learn documentation. docs
- S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, 2023. doi
- S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection,” BMC Bioinformatics, vol. 7, art. 91, 2006. doi
- G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” Journal of Machine Learning Research, vol. 11, pp. 2079–2107, 2010. paper
- M. Bińkowski, D. J. Sutherland, M. Arbel and A. Gretton, “Demystifying MMD GANs,” ICLR, 2018. arXiv
- T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, 1998. doi
- B. Efron, “Bootstrap methods: another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, 1979. doi
- X. Bouthillier et al., “Accounting for variance in machine learning benchmarks,” MLSys, 2021. arXiv
- C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On calibration of modern neural networks,” ICML, 2017. arXiv
- M. Minderer et al., “Revisiting the calibration of modern neural networks,” NeurIPS, 2021. arXiv
- D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” ICLR, 2019. arXiv
- B. Recht, R. Roelofs, L. Schmidt and V. Shankar, “Do ImageNet classifiers generalize to ImageNet?,” ICML, 2019. paper
- P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” ICML, 2021. paper
- Y. Ovadia et al., “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” NeurIPS, 2019. arXiv
- Z. C. Lipton, Y.-X. Wang and A. Smola, “Detecting and correcting for label shift with black box predictors,” ICML, 2018. arXiv
- M. Dehghani, A. Arnab, L. Beyer, A. Vaswani and Y. Tay, “The efficiency misnomer,” ICLR, 2022. arXiv
- PyTorch, “torch.export,” PyTorch documentation. docs
- ONNX, “Open Neural Network Exchange: the open standard for machine learning interoperability.” site
- ONNX Runtime, “Welcome to ONNX Runtime,” documentation. docs
- NVIDIA, “NVIDIA TensorRT documentation.” docs
- B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” CVPR, 2018. doi
- M. Nagel et al., “A white paper on neural network quantization,” arXiv:2106.08295, 2021. arXiv
- A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” book chapter in Low-Power Computer Vision; arXiv:2103.13630, 2021. arXiv
- S. Han, H. Mao and W. J. Dally, “Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding,” ICLR, 2016. arXiv
- D. Blalock, J. J. Gonzalez Ortiz, J. Frankle and J. Guttag, “What is the state of neural network pruning?,” MLSys, 2020. arXiv
- T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden and A. Peste, “Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks,” Journal of Machine Learning Research, vol. 22, 2021. arXiv
- G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” NIPS Deep Learning Workshop, 2014; arXiv:1503.02531. arXiv
- J. Gou, B. Yu, S. J. Maybank and D. Tao, “Knowledge distillation: a survey,” International Journal of Computer Vision, vol. 129, 2021. doi
- Y. Cheng, D. Wang, P. Zhou and T. Zhang, “Model compression and acceleration for deep neural networks: the principles, progress, and challenges,” IEEE Signal Processing Magazine, vol. 35, no. 1, 2018. doi arXiv
- S. Rabanser, S. Günnemann and Z. C. Lipton, “Failing loudly: an empirical study of methods for detecting dataset shift,” NeurIPS, 2019. arXiv
- D. Sculley et al., “Hidden technical debt in machine learning systems,” NeurIPS, 2015. paper
- PyTorch, “Reproducibility,” PyTorch documentation. docs
- M. Mitchell et al., “Model cards for model reporting,” ACM FAT*, 2019. arXiv
- T. Gebru et al., “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, 2021. doi
- J. Pineau et al., “Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program),” Journal of Machine Learning Research, vol. 22, 2021. paper
- J. Gawlikowski et al., “A survey of uncertainty in deep neural networks,” Artificial Intelligence Review, vol. 56, 2023. doi
- A. Paleyes, R.-G. Urma and N. D. Lawrence, “Challenges in deploying machine learning: a survey of case studies,” ACM Computing Surveys, vol. 55, no. 6, 2022. doi
- P. Liang et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023. arXiv
- X. Zhu, J. Li, Y. Liu, C. Ma and W. Wang, “A survey on model compression for large language models,” Transactions of the Association for Computational Linguistics, vol. 12, 2024. doi
- E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh, “GPTQ: accurate post-training quantization for generative pre-trained transformers,” ICLR, 2023. arXiv
- J. Lin et al., “AWQ: activation-aware weight quantization for LLM compression and acceleration,” MLSys, 2024. arXiv
- J. Hoffmann et al., “Training compute-optimal large language models,” arXiv:2203.15556, 2022; NeurIPS 2022 version titled “An empirical analysis of compute-optimal large language model training.” arXiv
- X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling vision transformers,” CVPR, 2022. doi
- PyTorch, “torchao documentation.” docs