Temperature, top-k and top-p change the distribution
IntermediateNLP 08 · Sequence-to-Sequence Models with Transformers (Machine Translation, Chatbots)Interactive plot
Description
Follow fixed logits through temperature scaling, top-k truncation, nucleus truncation and renormalization.
Try this: Keep temperature fixed and lower top-p. Why can the output entropy change abruptly even though the slider moves continuously?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Top-p keeps a whole number of tokens, so the kept set changes only when the cumulative mass crosses p. With the default logits the entropy is flat, then steps from 0 to 0.84 near p = 0.65, and again near 0.9.
Lab · Temperature, top-k and top-p change the distribution
- Candidates retained
- 3
- Entropy, bits
- 1.201
- Probability sum
- 1
Temperature rescales relative logits. Truncation sets excluded candidates to zero and renormalizes the survivors; it cannot improve the information available to the model.
Try this: Keep temperature fixed and lower top-p. Why can the output entropy change abruptly even though the slider moves continuously?
Five synthetic candidate tokens with logits [3,2,1,0,−1]. This applies top-k first, then top-p to its renormalized distribution. No model is trained and no quality score is inferred from entropy.