Same predictions, different probability quality
BeginnerNLP 03 · Neural Networks for NLPInteractive plot
Description
Apply one positive temperature to fixed validation logits and compare accuracy, log loss and squared probability error.
Try this: Find a temperature with lower log loss than T=1. Why does accuracy stay fixed even while confidence changes?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Dividing every logit by T never changes its sign, so every predicted class is unchanged (accuracy stays 0.75 here). Only the confidence moves, and with it the log loss.
Lab · Same predictions, different probability quality
- Accuracy
- 75%
- Mean log loss, nats
- 0.498
- Brier score
- 0.177
Positive temperature preserves logit ordering and binary decisions. It can adjust overconfidence or underconfidence, but cannot correct which examples the classifier gets wrong.
Try this: Find a temperature with lower log loss than T=1. Why does accuracy stay fixed even while confidence changes?
Eight synthetic validation observations, not a benchmark. Binary probabilities use σ(logit/T), with a fixed 0.5 decision threshold. Choosing T here illustrates validation tuning; it does not establish performance on an unseen test set.