Selective state updates: remember the useful inputs
IntermediateNLP 05 · Recurrent Neural Networks for LanguageInteractive plot
Description
Compare a fixed recurrent decay with input-dependent write decisions while distractors pass through a sequence.
Try this: Compare the two policies: with selective updates the state written at position 1 survives the distractors, with the same update at every step the distractors overwrite it. Then raise the distractor amplitude and see which state changes. Finally set retention to one: a state that never changes keeps its old value perfectly, so nothing new can be written either. Why is that not memory?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
With a = 1 the update is hₜ = 1·hₜ₋₁ + 0·xₜ, so no input ever enters the state and it stays at its starting value 0 at every position. The derivative ∂h₁₂/∂h₁ is exactly 1, the best possible, but there is nothing in h₁ to remember. Try a = 0.25: the state is 0.75 after position 1, stays 0.75 through the distractors, and ends at −0.5625 after position 9.
Lab · Selective state updates: remember the useful inputs
- State h₁ → h₈ (before the second write)
- 0.75 → 0.75
- Final state
- -0.563
- ∂h₁₂ / ∂h₁
- 0.25
- Largest |state|
- 0.75
- Write opportunities
- 2
Fixed decay trades persistence against interference at every step. Selection can preserve state through irrelevant inputs, but the gate must identify relevance correctly; the oracle here removes that learning problem. Retention 1 preserves perfectly and writes nothing.
Try this: Compare the two policies: with selective updates the state written at position 1 survives the distractors, with the same update at every step the distractors overwrite it. Then raise the distractor amplitude and see which state changes. Finally set retention to one: a state that never changes keeps its old value perfectly, so nothing new can be written either. Why is that not memory?
Scalar recurrence hₜ=aₜhₜ₋₁+(1−aₜ)xₜ with h₀=0. Retention aₜ=1 keeps the old state, aₜ=0 replaces it with the input. Relevant positions 1 and 9 are provided by an oracle; in the selective policy every other position has aₜ=1. With retention 1 at the relevant positions the state stays at its starting value 0: perfect preservation, nothing written. This illustrates gating and Mamba’s input-selectivity motivation; it is not an implementation of Mamba, LSTM or GRU.