完整搜尋頁面 →

LSTM:跨越長時間間隔保存狀態

選讀・ML 04・循環與遞迴神經網路 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

有用的監督訊號很晚才到達時,循環網路如何保留資訊?Hochreiter 與 Schmidhuber 提出具有受保護線性狀態路徑及可學習閘門的記憶單元 [2]。先讀循環狀態與時間反向傳播。把 1997 年原始單元和正文中的現代變體分開,較容易理解歷史貢獻。

方法與圖解

1997 年記憶單元包含固定自連接、輸入閘與輸出閘,尚未包含許多現代實作使用的遺忘閘。
來源:Hochreiter and Schmidhuber, Neural Computation 9(8), 1997; published PDF hosted by Stanford. 圖 1,PDF 第 10 頁。1997 年記憶單元包含固定自連接、輸入閘與輸出閘,尚未包含許多現代實作使用的遺忘閘。 [2]

圖 1 的 constant error carousel 是權重固定為一的狀態自連接。以簡化純量記號表示核心運算:

st=st−1+itg(at),yt=oth(st).s_t=s_{t-1}+i_t g(a_t),\qquad y_t=o_t h(s_t).

其中 sts_t 為第 tt 步內部狀態,iti_t、oto_t 為輸入閘及輸出閘活化值,ata_t 為候選輸入的活化前數值,gg 轉換候選輸入,hh 將狀態轉成可輸出形式,yty_t 為外顯輸出。閘門決定何時寫入、何時顯露已儲存資訊。

原始式子沒有乘在 st−1s_{t-1} 上的可學習遺忘係數。直接線性狀態路徑的導數為一,並不代表整個循環計算中每條路徑的導數都是一。閘門學習與訊號干擾仍然重要。

證據

第 5.4 節的測試很適合入門:把兩個標記過的實數輸入相加,到序列結尾才輸出縮放後答案。模型必須忽略干擾值,同時保存標記值。表 7 改變最短序列長度,而最短相關延遲是該長度的一半。因此最短長度 1000 對應的最短延遲為 500,不是 1000。結果平均十次試驗;測試錯誤計算絕對誤差超過 0.04 的序列。引用記憶長度之前,應先看清楚任務定義。

設計分析

輸入閘阻擋無關寫入;輸出閘決定記憶何時影響其他單元。第 5.4 節也檢查狀態漂移及閘門偏差初始化,讓架構對應到具體失敗機制:長延遲期間,微小的非預期加項可能累積。人工標記輸入任務能有效隔離保存能力,卻無法涵蓋自然序列所有語意與計算需求。

閱讀檢查點

追蹤標記值進入單元,接著在干擾步驟關閉輸入閘。解釋狀態如何變化,以及外顯輸出仍可能如何改變。指出現代遺忘閘會修改遞迴式的哪裡,但不要把後來設計歸到 1997 年原圖。

限制與下一步

長延遲成功取決於任務、訓練停止條件與初始化,不是無限記憶保證。循序運算本身也仍是循序的。接著讀本章現代閘控遞迴,再看 NLP 序列到序列內容:保存有用狀態,與隨時取回長輸入中的任意細節,是不同能力。

接著閱讀: ML 04・回到章節

參考文獻

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 10. Book
  2. S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, 1997. DOI
  3. K. Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
  4. K. S. Tai, R. Socher and C. D. Manning, “Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks,” ACL, 2015. arXiv
  5. F. A. Gers, J. Schmidhuber and F. Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation, 2000. DOI
  6. Y. Bengio, P. Simard and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, 1994. DOI
  7. R. Pascanu, T. Mikolov and Y. Bengio, “On the difficulty of training Recurrent Neural Networks,” ICML, 2013. arXiv
  8. P. J. Werbos, “Backpropagation through time: what it does and how to do it,” Proceedings of the IEEE, 1990. DOI
  9. J. L. Elman, “Finding Structure in Time,” Cognitive Science, 1990. DOI
  10. M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Transactions on Signal Processing, 1997. DOI
  11. I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. arXiv
  12. D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. arXiv
  13. O. Vinyals, A. Toshev, S. Bengio and D. Erhan, “Show and Tell: A Neural Image Caption Generator,” CVPR, 2015. arXiv
  14. K. Xu et al., “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” ICML, 2015. arXiv
  15. J. Donahue et al., “Long-term Recurrent Convolutional Networks for Visual Recognition and Description,” CVPR, 2015. arXiv
  16. X. Shi et al., “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting,” NeurIPS, 2015. arXiv
  17. R. Socher et al., “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,” EMNLP, 2013. Paper
  18. R. Socher, C. C.-Y. Lin, A. Y. Ng and C. D. Manning, “Parsing Natural Scenes and Natural Language with Recursive Neural Networks,” ICML, 2011. Paper
  19. K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink and J. Schmidhuber, “LSTM: A Search Space Odyssey,” IEEE Transactions on Neural Networks and Learning Systems, 2017. arXiv
  20. Z. C. Lipton, J. Berkowitz and C. Elkan, “A Critical Review of Recurrent Neural Networks for Sequence Learning,” arXiv:1506.00019, 2015. arXiv
  21. A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
  22. M. Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
  23. L. Feng et al., “Were RNNs All We Needed?,” arXiv:2410.01201, 2024. arXiv
  24. A. Orvieto et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” ICML, 2023. arXiv
  25. S. Bengio, O. Vinyals, N. Jaitly and N. Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” NeurIPS, 2015. arXiv
  26. M. Tiezzi, M. Casoni, A. Betti, T. Guidi, M. Gori and S. Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
  27. M. Tiezzi, M. Casoni, A. Betti, T. Guidi, M. Gori and S. Melacci, “Back to recurrent processing at the crossroad of transformers and state-space models,” Nature Machine Intelligence, 2025. DOI
  28. A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. arXiv
  29. T. Dao and A. Gu, “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,” ICML, PMLR 235, 2024. Paper
  30. J. Y.-C. Hu, X. Zhang, A. Elsheikh, W. Wu and H. Liu, “On Structured State-Space Duality,” ICML, PMLR 306, 2026. Paper