LSTM:跨越長時間間隔保存狀態
選讀・ML 04・循環與遞迴神經網路 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
有用的監督訊號很晚才到達時,循環網路如何保留資訊?Hochreiter 與 Schmidhuber 提出具有受保護線性狀態路徑及可學習閘門的記憶單元 [2]。先讀循環狀態與時間反向傳播。把 1997 年原始單元和正文中的現代變體分開,較容易理解歷史貢獻。
方法與圖解

圖 1 的 constant error carousel 是權重固定為一的狀態自連接。以簡化純量記號表示核心運算:
其中 為第 步內部狀態,、 為輸入閘及輸出閘活化值, 為候選輸入的活化前數值, 轉換候選輸入, 將狀態轉成可輸出形式, 為外顯輸出。閘門決定何時寫入、何時顯露已儲存資訊。
原始式子沒有乘在 上的可學習遺忘係數。直接線性狀態路徑的導數為一,並不代表整個循環計算中每條路徑的導數都是一。閘門學習與訊號干擾仍然重要。
證據
第 5.4 節的測試很適合入門:把兩個標記過的實數輸入相加,到序列結尾才輸出縮放後答案。模型必須忽略干擾值,同時保存標記值。表 7 改變最短序列長度,而最短相關延遲是該長度的一半。因此最短長度 1000 對應的最短延遲為 500,不是 1000。結果平均十次試驗;測試錯誤計算絕對誤差超過 0.04 的序列。引用記憶長度之前,應先看清楚任務定義。
設計分析
輸入閘阻擋無關寫入;輸出閘決定記憶何時影響其他單元。第 5.4 節也檢查狀態漂移及閘門偏差初始化,讓架構對應到具體失敗機制:長延遲期間,微小的非預期加項可能累積。人工標記輸入任務能有效隔離保存能力,卻無法涵蓋自然序列所有語意與計算需求。
閱讀檢查點
追蹤標記值進入單元,接著在干擾步驟關閉輸入閘。解釋狀態如何變化,以及外顯輸出仍可能如何改變。指出現代遺忘閘會修改遞迴式的哪裡,但不要把後來設計歸到 1997 年原圖。
限制與下一步
長延遲成功取決於任務、訓練停止條件與初始化,不是無限記憶保證。循序運算本身也仍是循序的。接著讀本章現代閘控遞迴,再看 NLP 序列到序列內容:保存有用狀態,與隨時取回長輸入中的任意細節,是不同能力。
接著閱讀: ML 04・回到章節
參考文獻
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 10. Book
- S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, 1997. DOI
- K. Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
- K. S. Tai, R. Socher and C. D. Manning, “Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks,” ACL, 2015. arXiv
- F. A. Gers, J. Schmidhuber and F. Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation, 2000. DOI
- Y. Bengio, P. Simard and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, 1994. DOI
- R. Pascanu, T. Mikolov and Y. Bengio, “On the difficulty of training Recurrent Neural Networks,” ICML, 2013. arXiv
- P. J. Werbos, “Backpropagation through time: what it does and how to do it,” Proceedings of the IEEE, 1990. DOI
- J. L. Elman, “Finding Structure in Time,” Cognitive Science, 1990. DOI
- M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing, 1997. DOI
- I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. arXiv
- D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. arXiv
- O. Vinyals, A. Toshev, S. Bengio and D. Erhan, “Show and Tell: A Neural Image Caption Generator,” CVPR, 2015. arXiv
- K. Xu et al., “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” ICML, 2015. arXiv
- J. Donahue et al., “Long-term Recurrent Convolutional Networks for Visual Recognition and Description,” CVPR, 2015. arXiv
- X. Shi et al., “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting,” NeurIPS, 2015. arXiv
- R. Socher et al., “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,” EMNLP, 2013. Paper
- R. Socher, C. C.-Y. Lin, A. Y. Ng and C. D. Manning, “Parsing Natural Scenes and Natural Language with Recursive Neural Networks,” ICML, 2011. Paper
- K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink and J. Schmidhuber, “LSTM: A Search Space Odyssey,” IEEE Transactions on Neural Networks and Learning Systems, 2017. arXiv
- Z. C. Lipton, J. Berkowitz and C. Elkan, “A Critical Review of Recurrent Neural Networks for Sequence Learning,” arXiv:1506.00019, 2015. arXiv
- A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
- M. Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
- L. Feng et al., “Were RNNs All We Needed?,” arXiv:2410.01201, 2024. arXiv
- A. Orvieto et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” ICML, 2023. arXiv
- S. Bengio, O. Vinyals, N. Jaitly and N. Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” NeurIPS, 2015. arXiv
- M. Tiezzi, M. Casoni, A. Betti, T. Guidi, M. Gori and S. Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
- M. Tiezzi, M. Casoni, A. Betti, T. Guidi, M. Gori and S. Melacci, “Back to recurrent processing at the crossroad of transformers and state-space models,” Nature Machine Intelligence, 2025. DOI
- A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. arXiv
- T. Dao and A. Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,” ICML, PMLR 235, 2024. Paper
- J. Y.-C. Hu, X. Zhang, A. Elsheikh, W. Wu and H. Liu, “On Structured State-Space Duality,” ICML, PMLR 306, 2026. Paper