What may a masked predictor see?
IntermediateNLP 07 · Attention Mechanism and TransformersInteractive plot
Description
Change a right-context clue, hide it causally, then attach different output heads to the same small contextual vector.
Try this: Change the right-hand clue under each mask. When must the masked-position prediction remain identical, and why?
Hints
The chapter section that uses this lab:
Lab · What may a masked predictor see?
- Context positions used
- 3
- First output probability
- 0.931
- Output probabilities sum
- 1
The target content is always absent. A bidirectional objective can use right context; a causal next-token objective cannot. An output head maps representations to the labels required by a task.
Try this: Change the right-hand clue under each mask. When must the masked-position prediction remain identical, and why?
A hand-built uniform context encoder and two linear heads illustrate BERT-style visibility and task interfaces. These are not BERT embeddings or pretrained predictions. The sequence head reads this toy pooled context, not an actual [CLS] representation.