Alignment, hard negatives and collapsed features
BeginnerNLP 04 · Lexical Semantics and Word EmbeddingsInteractive plot
Description
Move paired and unpaired unit vectors, then inspect the normalized matching loss used by contrastive learning.
Try this: Bring a negative closer to the anchor than the positive. Does a lower temperature help the correct pair? Then collapse all vectors.
Hints
The chapter section that uses this lab:
Lab · Alignment, hard negatives and collapsed features
- Positive probability
- 0.896
- Matching loss, nats
- 0.109
- Positive cosine
- 0.94
Loss = −log(exp(sim⁺/τ) / Σ exp(sim/τ)). Temperature sharpens whichever candidate already has the highest similarity. Collapsed unit vectors give equal probabilities and loss log(4).
Try this: Bring a negative closer to the anchor than the positive. Does a lower temperature help the correct pair? Then collapse all vectors.
One anchor, one positive and three negative candidates; vectors are controlled directly rather than trained. The same matching geometry explains image–text retrieval, but these are not CLIP features or semantic predictions.