Temperature Scaling — confidence without changing the class
Optional · paper guide 1 of 1 for ML 11 · Evaluation, Efficiency and Deployment. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
When a classifier says it is confident, how often is it actually correct? Guo et al. show that modern neural classifiers can be miscalibrated and study simple post-training corrections [11]. Read logits, softmax, held-out validation and accuracy first. Calibration concerns agreement between stated confidence and observed frequency; it is a different property from choosing the right class.
Method and figure

Temperature scaling uses
Here is an input, is the classifier logit for class , is the queried class and is a positive scalar temperature. Fit by minimizing negative log likelihood on held-out validation data while keeping the classifier fixed. Evaluate on separate test data. A positive temperature preserves the logit ranking and hence the predicted class, while changing confidence.
The paper also reports expected calibration error:
Here is the number of evaluated examples, is confidence bin , is the number of bins, and and are that bin’s empirical accuracy and mean predicted confidence. Empty bins contribute zero. This finite-bin summary depends on binning and sample size; it is not identical to the fitting objective.
Evidence
Figure 4 uses a ResNet-110 with stochastic depth on CIFAR-100. Its uncalibrated ECE is 12.67%, and the temperature-scaled panel reports 0.96%. Preserve the stochastic-depth qualifier: this is not the plain ResNet-110 row elsewhere in the paper. Table 1 compares several calibration methods and datasets; the overall usefulness of temperature scaling does not imply that every individual setting improves.
Design analysis
One parameter limits flexibility and makes the correction easy to fit, while preserving classification decisions. More flexible calibration maps can alter rankings or overfit limited validation data. Read reliability diagrams together with accuracy and negative log likelihood: a low aggregate calibration error can hide uneven behavior across confidence regions or groups. Test labels must not be used to choose the temperature being evaluated.
Reading checkpoint
Explain why dividing every logit by the same positive number leaves the predicted class unchanged. Then imagine a deployment population unlike the validation population. Which agreement measured by the reliability diagram might fail, even though the fitted temperature has not changed?
Limits and next step
Temperature scaling does not repair incorrect features, increase classification accuracy by itself, or establish out-of-distribution reliability. A validation-fit correction needs monitoring when data shift. Return to the chapter’s deployment checklist and record the calibration split, fitting objective, binning rule and evaluation population alongside accuracy.
Continue reading: ML 11 · Return to the chapter
References
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 11 “Practical Methodology”. book
- G. James, D. Witten, T. Hastie, R. Tibshirani and J. Taylor, An Introduction to Statistical Learning: with Applications in Python, Springer, 2023. book
- scikit-learn developers, “Common pitfalls and recommended practices,” scikit-learn documentation. docs
- S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, 2023. doi
- S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection,” BMC Bioinformatics, vol. 7, art. 91, 2006. doi
- G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” Journal of Machine Learning Research, vol. 11, pp. 2079–2107, 2010. paper
- M. Bińkowski, D. J. Sutherland, M. Arbel and A. Gretton, “Demystifying MMD GANs,” ICLR, 2018. arXiv
- T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, 1998. doi
- B. Efron, “Bootstrap methods: another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, 1979. doi
- X. Bouthillier et al., “Accounting for variance in machine learning benchmarks,” MLSys, 2021. arXiv
- C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On calibration of modern neural networks,” ICML, 2017. arXiv
- M. Minderer et al., “Revisiting the calibration of modern neural networks,” NeurIPS, 2021. arXiv
- D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” ICLR, 2019. arXiv
- B. Recht, R. Roelofs, L. Schmidt and V. Shankar, “Do ImageNet classifiers generalize to ImageNet?,” ICML, 2019. paper
- P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” ICML, 2021. paper
- Y. Ovadia et al., “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” NeurIPS, 2019. arXiv
- Z. C. Lipton, Y.-X. Wang and A. Smola, “Detecting and correcting for label shift with black box predictors,” ICML, 2018. arXiv
- M. Dehghani, A. Arnab, L. Beyer, A. Vaswani and Y. Tay, “The efficiency misnomer,” ICLR, 2022. arXiv
- PyTorch, “torch.export,” PyTorch documentation. docs
- ONNX, “Open Neural Network Exchange: the open standard for machine learning interoperability.” site
- ONNX Runtime, “Welcome to ONNX Runtime,” documentation. docs
- NVIDIA, “NVIDIA TensorRT documentation.” docs
- B. Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” CVPR, 2018. doi
- M. Nagel et al., “A white paper on neural network quantization,” arXiv:2106.08295, 2021. arXiv
- A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” book chapter in Low-Power Computer Vision; arXiv:2103.13630, 2021. arXiv
- S. Han, H. Mao and W. J. Dally, “Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding,” ICLR, 2016. arXiv
- D. Blalock, J. J. Gonzalez Ortiz, J. Frankle and J. Guttag, “What is the state of neural network pruning?,” MLSys, 2020. arXiv
- T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden and A. Peste, “Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks,” Journal of Machine Learning Research, vol. 22, 2021. arXiv
- G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” NIPS Deep Learning Workshop, 2014; arXiv:1503.02531. arXiv
- J. Gou, B. Yu, S. J. Maybank and D. Tao, “Knowledge distillation: a survey,” International Journal of Computer Vision, vol. 129, 2021. doi
- Y. Cheng, D. Wang, P. Zhou and T. Zhang, “Model compression and acceleration for deep neural networks: the principles, progress, and challenges,” IEEE Signal Processing Magazine, vol. 35, no. 1, 2018. doi arXiv
- S. Rabanser, S. Günnemann and Z. C. Lipton, “Failing loudly: an empirical study of methods for detecting dataset shift,” NeurIPS, 2019. arXiv
- D. Sculley et al., “Hidden technical debt in machine learning systems,” NeurIPS, 2015. paper
- PyTorch, “Reproducibility,” PyTorch documentation. docs
- M. Mitchell et al., “Model cards for model reporting,” ACM FAT*, 2019. arXiv
- T. Gebru et al., “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, 2021. doi
- J. Pineau et al., “Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program),” Journal of Machine Learning Research, vol. 22, 2021. paper
- J. Gawlikowski et al., “A survey of uncertainty in deep neural networks,” Artificial Intelligence Review, vol. 56, 2023. doi
- A. Paleyes, R.-G. Urma and N. D. Lawrence, “Challenges in deploying machine learning: a survey of case studies,” ACM Computing Surveys, vol. 55, no. 6, 2022. doi
- P. Liang et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023. arXiv
- X. Zhu, J. Li, Y. Liu, C. Ma and W. Wang, “A survey on model compression for large language models,” Transactions of the Association for Computational Linguistics, vol. 12, 2024. doi
- E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh, “GPTQ: accurate post-training quantization for generative pre-trained transformers,” ICLR, 2023. arXiv
- J. Lin et al., “AWQ: activation-aware weight quantization for LLM compression and acceleration,” MLSys, 2024. arXiv
- J. Hoffmann et al., “Training compute-optimal large language models,” arXiv:2203.15556, 2022; NeurIPS 2022 version titled “An empirical analysis of compute-optimal large language model training.” arXiv
- X. Zhai, A. Kolesnikov, N. Houlsby and L. Beyer, “Scaling vision transformers,” CVPR, 2022. doi
- PyTorch, “torchao documentation.” docs