Seq2Seq — Sequence to Sequence Learning with Neural Networks
Optional · paper guide 1 of 1 for NLP 05 · Recurrent Neural Networks for Language. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
A recurrent tagger can emit one label per input token, but translation may reorder words and produce a different number of tokens. Sutskever, Vinyals and Le show that an encoder and a separate decoder can learn such variable-length mappings [25]. Their thesis is that deep LSTMs, trained on suitable sequence arrangements, can translate through a fixed-dimensional sentence representation. Read this chapter’s recurrent state, LSTM gates and teacher forcing first.
Method

The encoder consumes the source and passes its final representation to a decoder. The decoder predicts a distribution over the next target token, then continues until an end-of-sequence token. The probability of a target sequence factors as
Here is the source sequence, its fixed-dimensional encoder representation, the target sequence including its end token, the target length, and the earlier target tokens. Each factor is the decoder’s next-token probability. Training maximizes this probability using the known target prefix; inference must use previously generated tokens instead.
Read Figure 1 from left to right: the input updates a recurrent state, then output generation reuses the accumulated information. The actual system uses distinct encoder and decoder LSTMs and four recurrent layers. Reversing only the source brings many corresponding source–target words closer along the computation path. It does not remove recurrence or give the decoder direct access to every source state. Beam search retains several partial hypotheses but remains an approximate search.
Results
Table 1 evaluates WMT’14 English→French on ntst14. A five-model reversed-input ensemble with beam size 12 obtains 34.81 BLEU, compared with the reported phrase-based baseline’s 33.30. This is an ensemble result, not a single LSTM score. Table 2’s 36.5 instead comes from rescoring the baseline’s 1000-best candidates with an ensemble. Candidate rescoring and generating translations from scratch are different tasks and must not share one unlabeled “best score.”
Ablations
The single forward-input LSTM scores 26.17 BLEU and the single reversed-input LSTM 30.59, both with beam size 12 in Table 1. This directly supports the importance of sequence arrangement in that training setup. Increasing ensemble size and changing beam width have separate effects: the five-model ensemble scores 33.00 with beam 1 and 34.50 with beam 2. Do not attribute all of the final gain to reversal or assume a wider beam always yields a proportionate benefit.
Conclusion
The central abstraction is conditional sequence generation: an encoder summarizes one sequence and a decoder unfolds another. Trace where information about the first input word must travel before the first output is predicted. Then read the Bahdanau guide in NLP 08 to see how attention gives the decoder repeated access to source representations.
Problems
A fixed representation can lose details, especially as inputs grow or differ from training examples. Source reversal eases some dependencies; it does not eliminate the bottleneck. Training with correct prefixes differs from inference with imperfect generated prefixes. Vocabulary limits, approximate decoding and data-specific word order also constrain the reported results. This paper establishes a useful architecture, not a guarantee that arbitrary long sequences can be faithfully compressed.
References
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 5, Manning, 2021. Publisher
- Susan McRoy, Principles of Natural Language Processing, 2021. Chapter
- Daniel Jurafsky and James H. Martin, Speech and Language Processing, third-edition draft, Chapter 3, August 19, 2026. Chapter
- Stanley F. Chen and Joshua Goodman, “An Empirical Study of Smoothing Techniques for Language Modeling,” ACL, 1996. Paper
- Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Paper
- Jeffrey L. Elman, “Finding Structure in Time,” Cognitive Science 14(2), 1990. DOI
- Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký and Sanjeev Khudanpur, “Recurrent Neural Network Based Language Model,” Interspeech, 2010. DOI
- Paul J. Werbos, “Backpropagation Through Time: What It Does and How to Do It,” Proceedings of the IEEE 78(10), 1990. DOI
- Yoshua Bengio, Patrice Simard and Paolo Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” IEEE Transactions on Neural Networks 5(2), 1994. DOI
- Razvan Pascanu, Tomas Mikolov and Yoshua Bengio, “On the Difficulty of Training Recurrent Neural Networks,” ICML, 2013. arXiv
- Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation 9(8), 1997. DOI
- Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation 12(10), 2000. DOI
- Kyunghyun Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
- Mike Schuster and Kuldip K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing 45(11), 1997. DOI
- Alex Graves, “Generating Sequences With Recurrent Neural Networks,” arXiv:1308.0850, 2013. arXiv
- Wojciech Zaremba, Ilya Sutskever and Oriol Vinyals, “Recurrent Neural Network Regularization,” arXiv:1409.2329, 2014. arXiv
- Gábor Melis, Chris Dyer and Phil Blunsom, “On the State of the Art of Evaluation in Neural Language Models,” arXiv:1707.05589, 2017. arXiv
- Stephen Merity, Nitish Shirish Keskar and Richard Socher, “Regularizing and Optimizing LSTM Language Models,” arXiv:1708.02182, 2017. arXiv
- Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer and Yonghui Wu, “Exploring the Limits of Language Modeling,” arXiv:1602.02410, 2016. arXiv
- Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori and Stefano Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
- Bo Peng et al., “RWKV: Reinventing RNNs for the Transformer Era,” Findings of EMNLP, 2023. Paper
- Albert Gu and Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
- Maximilian Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
- Lewis Carroll, Alice’s Adventures in Wonderland, 1865; Project Gutenberg eBook #11. Project Gutenberg
- Ilya Sutskever, Oriol Vinyals and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper