Paper guides
Optional reading guides to the papers behind NLP 05 · Recurrent Neural Networks for Language, one page each: the problem, the idea, what it showed and what to check when you read it.
Navigate: this page and this track: Navigate: this page and this track
These optional guides connect the chapter to original papers. Read the challenge and figure first, then unfold the method, evidence and limitations.
- Seq2Seq — Sequence to Sequence Learning with Neural Networks
A recurrent tagger can emit one label per input token, but translation may reorder words and produce a different number of tokens.
References
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 5, Manning, 2021. Publisher
- Susan McRoy, Principles of Natural Language Processing, 2021. Chapter
- Daniel Jurafsky and James H. Martin, Speech and Language Processing, third-edition draft, Chapter 3, August 19, 2026. Chapter
- Stanley F. Chen and Joshua Goodman, “An Empirical Study of Smoothing Techniques for Language Modeling,” ACL, 1996. Paper
- Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Paper
- Jeffrey L. Elman, “Finding Structure in Time,” Cognitive Science 14(2), 1990. DOI
- Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký and Sanjeev Khudanpur, “Recurrent Neural Network Based Language Model,” Interspeech, 2010. DOI
- Paul J. Werbos, “Backpropagation Through Time: What It Does and How to Do It,” Proceedings of the IEEE 78(10), 1990. DOI
- Yoshua Bengio, Patrice Simard and Paolo Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” IEEE Transactions on Neural Networks 5(2), 1994. DOI
- Razvan Pascanu, Tomas Mikolov and Yoshua Bengio, “On the Difficulty of Training Recurrent Neural Networks,” ICML, 2013. arXiv
- Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation 9(8), 1997. DOI
- Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation 12(10), 2000. DOI
- Kyunghyun Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
- Mike Schuster and Kuldip K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing 45(11), 1997. DOI
- Alex Graves, “Generating Sequences With Recurrent Neural Networks,” arXiv:1308.0850, 2013. arXiv
- Wojciech Zaremba, Ilya Sutskever and Oriol Vinyals, “Recurrent Neural Network Regularization,” arXiv:1409.2329, 2014. arXiv
- Gábor Melis, Chris Dyer and Phil Blunsom, “On the State of the Art of Evaluation in Neural Language Models,” arXiv:1707.05589, 2017. arXiv
- Stephen Merity, Nitish Shirish Keskar and Richard Socher, “Regularizing and Optimizing LSTM Language Models,” arXiv:1708.02182, 2017. arXiv
- Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer and Yonghui Wu, “Exploring the Limits of Language Modeling,” arXiv:1602.02410, 2016. arXiv
- Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori and Stefano Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
- Bo Peng et al., “RWKV: Reinventing RNNs for the Transformer Era,” Findings of EMNLP, 2023. Paper
- Albert Gu and Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
- Maximilian Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
- Lewis Carroll, Alice’s Adventures in Wonderland, 1865; Project Gutenberg eBook #11. Project Gutenberg
- Ilya Sutskever, Oriol Vinyals and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper