Full search page →

Paper guides

Optional reading guides to the papers behind NLP 05 · Recurrent Neural Networks for Language, one page each: the problem, the idea, what it showed and what to check when you read it.

Navigate: this page and this track: Navigate: this page and this track

These optional guides connect the chapter to original papers. Read the challenge and figure first, then unfold the method, evidence and limitations.

  1. Seq2Seq — Sequence to Sequence Learning with Neural Networks

    A recurrent tagger can emit one label per input token, but translation may reorder words and produce a different number of tokens.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 5, Manning, 2021. Publisher
  2. Susan McRoy, Principles of Natural Language Processing, 2021. Chapter
  3. Daniel Jurafsky and James H. Martin, Speech and Language Processing, third-edition draft, Chapter 3, August 19, 2026. Chapter
  4. Stanley F. Chen and Joshua Goodman, “An Empirical Study of Smoothing Techniques for Language Modeling,” ACL, 1996. Paper
  5. Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Paper
  6. Jeffrey L. Elman, “Finding Structure in Time,” Cognitive Science 14(2), 1990. DOI
  7. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký and Sanjeev Khudanpur, “Recurrent Neural Network Based Language Model,” Interspeech, 2010. DOI
  8. Paul J. Werbos, “Backpropagation Through Time: What It Does and How to Do It,” Proceedings of the IEEE 78(10), 1990. DOI
  9. Yoshua Bengio, Patrice Simard and Paolo Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” IEEE Transactions on Neural Networks 5(2), 1994. DOI
  10. Razvan Pascanu, Tomas Mikolov and Yoshua Bengio, “On the Difficulty of Training Recurrent Neural Networks,” ICML, 2013. arXiv
  11. Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation 9(8), 1997. DOI
  12. Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation 12(10), 2000. DOI
  13. Kyunghyun Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
  14. Mike Schuster and Kuldip K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing 45(11), 1997. DOI
  15. Alex Graves, “Generating Sequences With Recurrent Neural Networks,” arXiv:1308.0850, 2013. arXiv
  16. Wojciech Zaremba, Ilya Sutskever and Oriol Vinyals, “Recurrent Neural Network Regularization,” arXiv:1409.2329, 2014. arXiv
  17. Gábor Melis, Chris Dyer and Phil Blunsom, “On the State of the Art of Evaluation in Neural Language Models,” arXiv:1707.05589, 2017. arXiv
  18. Stephen Merity, Nitish Shirish Keskar and Richard Socher, “Regularizing and Optimizing LSTM Language Models,” arXiv:1708.02182, 2017. arXiv
  19. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer and Yonghui Wu, “Exploring the Limits of Language Modeling,” arXiv:1602.02410, 2016. arXiv
  20. Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori and Stefano Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
  21. Bo Peng et al., “RWKV: Reinventing RNNs for the Transformer Era,” Findings of EMNLP, 2023. Paper
  22. Albert Gu and Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
  23. Maximilian Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
  24. Lewis Carroll, Alice’s Adventures in Wonderland, 1865; Project Gutenberg eBook #11. Project Gutenberg
  25. Ilya Sutskever, Oriol Vinyals and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper