Full search page →

What may a masked predictor see?

IntermediateNLP 07 · Attention Mechanism and TransformersInteractive plot

Description

Change a right-context clue, hide it causally, then attach different output heads to the same small contextual vector.

Try this: Change the right-hand clue under each mask. When must the masked-position prediction remain identical, and why?

Lab · What may a masked predictor see?

Context positions used
3
First output probability
0.931
Output probabilities sum
1

The target content is always absent. A bidirectional objective can use right context; a causal next-token objective cannot. An output head maps representations to the labels required by a task.

Try this: Change the right-hand clue under each mask. When must the masked-position prediction remain identical, and why?

A hand-built uniform context encoder and two linear heads illustrate BERT-style visibility and task interfaces. These are not BERT embeddings or pretrained predictions. The sequence head reads this toy pooled context, not an actual [CLS] representation.

Method sources

Open in the Playground →