Full search page →

Temperature, top-k and top-p change the distribution

IntermediateNLP 08 · Sequence-to-Sequence Models with Transformers (Machine Translation, Chatbots)Interactive plot

Description

Follow fixed logits through temperature scaling, top-k truncation, nucleus truncation and renormalization.

Try this: Keep temperature fixed and lower top-p. Why can the output entropy change abruptly even though the slider moves continuously?

Predict first

Hold the temperature at 1 and lower top-p slowly from 1.0. How does the output entropy change?

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Top-p keeps a whole number of tokens, so the kept set changes only when the cumulative mass crosses p. With the default logits the entropy is flat, then steps from 0 to 0.84 near p = 0.65, and again near 0.9.

Lab · Temperature, top-k and top-p change the distribution

Predict first

Hold the temperature at 1 and lower top-p slowly from 1.0. How does the output entropy change?

Candidates retained
3
Entropy, bits
1.201
Probability sum
1

Temperature rescales relative logits. Truncation sets excluded candidates to zero and renormalizes the survivors; it cannot improve the information available to the model.

Try this: Keep temperature fixed and lower top-p. Why can the output entropy change abruptly even though the slider moves continuously?

Five synthetic candidate tokens with logits [3,2,1,0,−1]. This applies top-k first, then top-p to its renormalized distribution. No model is trained and no quality score is inferred from entropy.

Method sources

Open in the Playground →