Full search page →

Selective state updates: remember the useful inputs

IntermediateNLP 05 · Recurrent Neural Networks for LanguageInteractive plot

Description

Compare a fixed recurrent decay with input-dependent write decisions while distractors pass through a sequence.

Try this: Compare the two policies: with selective updates the state written at position 1 survives the distractors, with the same update at every step the distractors overwrite it. Then raise the distractor amplitude and see which state changes. Finally set retention to one: a state that never changes keeps its old value perfectly, so nothing new can be written either. Why is that not memory?

Predict first

Policy = skip marked distractors, and you set retention a to 1 at the write positions (1 and 9). What is the state after the last position?

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

With a = 1 the update is hₜ = 1·hₜ₋₁ + 0·xₜ, so no input ever enters the state and it stays at its starting value 0 at every position. The derivative ∂h₁₂/∂h₁ is exactly 1, the best possible, but there is nothing in h₁ to remember. Try a = 0.25: the state is 0.75 after position 1, stays 0.75 through the distractors, and ends at −0.5625 after position 9.

Lab · Selective state updates: remember the useful inputs

Predict first

Policy = skip marked distractors, and you set retention a to 1 at the write positions (1 and 9). What is the state after the last position?

State h₁ → h₈ (before the second write)
0.75 → 0.75
Final state
-0.563
∂h₁₂ / ∂h₁
0.25
Largest |state|
0.75
Write opportunities
2

Fixed decay trades persistence against interference at every step. Selection can preserve state through irrelevant inputs, but the gate must identify relevance correctly; the oracle here removes that learning problem. Retention 1 preserves perfectly and writes nothing.

Try this: Compare the two policies: with selective updates the state written at position 1 survives the distractors, with the same update at every step the distractors overwrite it. Then raise the distractor amplitude and see which state changes. Finally set retention to one: a state that never changes keeps its old value perfectly, so nothing new can be written either. Why is that not memory?

Scalar recurrence hₜ=aₜhₜ₋₁+(1−aₜ)xₜ with h₀=0. Retention aₜ=1 keeps the old state, aₜ=0 replaces it with the input. Relevant positions 1 and 9 are provided by an oracle; in the selective policy every other position has aₜ=1. With retention 1 at the relevant positions the state stays at its starting value 0: perfect preservation, nothing written. This illustrates gating and Mamba’s input-selectivity motivation; it is not an implementation of Mamba, LSTM or GRU.

Method sources

Open in the Playground →