Full search page →

Attention: which tokens are visible?

IntermediateNLP 07 · Attention Mechanism and TransformersInteractive plot

Description

Compare bidirectional and causal attention using the same query, keys and values.

Try this: Choose the first query in causal mode: its weight must be 1 on itself. Switch to bidirectional mode, then lower the temperature.

Lab · Attention: which tokens are visible?

Visible tokens
3
Weight sum
1
Attention output
[0.7517, 0.7517]

The score column includes temperature scaling. A masked score is −∞ and its weight is exactly zero. Changing the mask changes normalization over the remaining keys.

Try this: Choose the first query in causal mode: its weight must be 1 on itself. Switch to bidirectional mode, then lower the temperature.

All Q, K and V vectors are fixed, synthetic two-dimensional numbers, not learned language representations. Scores are Q · K / (√2 τ); τ is an extra temperature control and τ = 1 gives the usual scaling. Causal attention masks positions after the query before softmax. The output is the weighted sum of V. These weights illustrate routing, not an explanation of linguistic meaning.

Method sources

Open in the Playground →