LSTM — protecting a state across long delays
Optional · paper guide 1 of 1 for ML 04 · Recurrent and Recursive Networks. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
How can a recurrent network retain information when the useful supervision arrives much later? Hochreiter and Schmidhuber propose a memory cell with a protected linear state path and learned gates [2]. Read recurrent states and backpropagation through time first. The historical contribution is easier to understand if the original 1997 cell is kept separate from the modern variant described in the main chapter.
Method and figure

Figure 1 shows the constant error carousel: a state self-connection with fixed weight one. In simplified scalar notation, its central computation is
Here is the internal state at step , and are input and output gate activations, is the candidate input preactivation, transforms that candidate, transforms the state for output, and is the exposed cell output. Gates determine when to write and when to reveal stored information.
There is no learned forget multiplier on in this original equation. The direct linear state path has derivative one; this is not a claim that every path through the entire recurrent computation has unit derivative. Gate learning and interference still matter.
Evidence
Section 5.4 supplies an especially readable test: add two marked real-valued inputs and report the scaled result only at sequence end. Distractor values must be ignored while the marked values are retained. Table 7 varies minimum sequence length and reports a minimum relevant delay of half that length. For minimum length 1000, the minimum delay is therefore 500, not 1000. Results average ten trials, and test errors count sequences exceeding an absolute-error threshold of 0.04. Read these task definitions before repeating a headline about memory length.
Design analysis
The input gate protects the state from irrelevant writes; the output gate controls when memory affects other units. Section 5.4 also examines state drift and gate-bias initialization. This connects the architecture to a concrete failure mechanism: tiny unwanted additions can accumulate during a long delay. Synthetic marked-input tasks isolate retention well, but do not measure all the semantic or computational demands of a natural sequence.
Reading checkpoint
Trace a marked value into the cell, then close the input gate during distractor steps. Explain what happens to the state and what can still change in the visible output. Identify exactly where a modern forget gate would change the displayed recurrence, without attributing that later design to the 1997 figure.
Limits and next step
Long-delay success depends on the task, training criterion and initialization; it is not an unlimited memory guarantee. Sequential computation also remains sequential. Continue with the chapter’s modern gated recurrence, then the NLP sequence-to-sequence material: distinguish storing a useful state from retrieving arbitrary details of a long input on demand.
Continue reading: ML 04 · Return to the chapter
References
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 10. Book
- S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, 1997. DOI
- K. Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
- K. S. Tai, R. Socher and C. D. Manning, “Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks,” ACL, 2015. arXiv
- F. A. Gers, J. Schmidhuber and F. Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation, 2000. DOI
- Y. Bengio, P. Simard and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, 1994. DOI
- R. Pascanu, T. Mikolov and Y. Bengio, “On the difficulty of training Recurrent Neural Networks,” ICML, 2013. arXiv
- P. J. Werbos, “Backpropagation through time: what it does and how to do it,” Proceedings of the IEEE, 1990. DOI
- J. L. Elman, “Finding Structure in Time,” Cognitive Science, 1990. DOI
- M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing, 1997. DOI
- I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. arXiv
- D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. arXiv
- O. Vinyals, A. Toshev, S. Bengio and D. Erhan, “Show and Tell: A Neural Image Caption Generator,” CVPR, 2015. arXiv
- K. Xu et al., “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” ICML, 2015. arXiv
- J. Donahue et al., “Long-term Recurrent Convolutional Networks for Visual Recognition and Description,” CVPR, 2015. arXiv
- X. Shi et al., “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting,” NeurIPS, 2015. arXiv
- R. Socher et al., “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,” EMNLP, 2013. Paper
- R. Socher, C. C.-Y. Lin, A. Y. Ng and C. D. Manning, “Parsing Natural Scenes and Natural Language with Recursive Neural Networks,” ICML, 2011. Paper
- K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink and J. Schmidhuber, “LSTM: A Search Space Odyssey,” IEEE Transactions on Neural Networks and Learning Systems, 2017. arXiv
- Z. C. Lipton, J. Berkowitz and C. Elkan, “A Critical Review of Recurrent Neural Networks for Sequence Learning,” arXiv:1506.00019, 2015. arXiv
- A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
- M. Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
- L. Feng et al., “Were RNNs All We Needed?,” arXiv:2410.01201, 2024. arXiv
- A. Orvieto et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” ICML, 2023. arXiv
- S. Bengio, O. Vinyals, N. Jaitly and N. Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” NeurIPS, 2015. arXiv
- M. Tiezzi, M. Casoni, A. Betti, T. Guidi, M. Gori and S. Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
- M. Tiezzi, M. Casoni, A. Betti, T. Guidi, M. Gori and S. Melacci, “Back to recurrent processing at the crossroad of transformers and state-space models,” Nature Machine Intelligence, 2025. DOI
- A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. arXiv
- T. Dao and A. Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,” ICML, PMLR 235, 2024. Paper
- J. Y.-C. Hu, X. Zhang, A. Elsheikh, W. Wu and H. Liu, “On Structured State-Space Duality,” ICML, PMLR 306, 2026. Paper