Does padding change a mean-pooled sentence?
BeginnerNLP 03 · Neural Networks for NLPInteractive plot
Description
Pad a three-word sentence with zero vectors and compare a correct masked mean with a mean that counts the padding in its denominator.
Try this: Keep the padding bug selected and drag the padding count from 0 to 9: predict h and P(class 1) before each step, and explain why the classifier drifts toward 0.5 although the sentence never changed. Then switch to the masked mean and confirm that padding cannot matter. Last, try position weights on the reversed sentence: that is the only mode where word order changes the answer.
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
The sum of the embeddings is unchanged (zeros add nothing), but it is divided by 3 + 9 = 12 instead of 3, so h shrinks by a factor of 3/12 = 1/4: from 0.7 to 0.175. The classifier output falls from 0.971 to 0.706, toward the undecided value 0.5, even though the sentence is identical.
Lab · Does padding change a mean-pooled sentence?
- Pooled h
- 0.233
- Masked-mean h (reference)
- 0.7
- P(class 1)
- 0.763
- Reference P(class 1)
- 0.971
- Real / counted length
- 3 / 9
Zero padding must not change a masked mean. Counting the padding in the denominator shrinks h by 3 divided by the counted length and pulls P(class 1) toward 0.5, although the sentence is the same. An unweighted mean also ignores word order; only the position-weighted diagnostic tells the two sentences apart.
Try this: Keep the padding bug selected and drag the padding count from 0 to 9: predict h and P(class 1) before each step, and explain why the classifier drifts toward 0.5 although the sentence never changed. Then switch to the masked mean and confirm that padding cannot matter. Last, try position weights on the reversed sentence: that is the only mode where word order changes the answer.
Fixed scalar embeddings: dogs=1, chase=0.4, cats=0.7; classifier p=σ(5h). The masked mean divides by the 3 real tokens, so zero padding adds nothing. The bug divides by 3 + padding. The reference bar always shows the masked mean of the same sentence. A plain mean ignores word order by construction, so swapping the sentence changes nothing there; position weights 1,2,3 are a diagnostic, not a trained syntax model.