Full search page →

Seq2Seq — Sequence to Sequence Learning with Neural Networks

Optional · paper guide 1 of 1 for NLP 05 · Recurrent Neural Networks for Language. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

A recurrent tagger can emit one label per input token, but translation may reorder words and produce a different number of tokens. Sutskever, Vinyals and Le show that an encoder and a separate decoder can learn such variable-length mappings [25]. Their thesis is that deep LSTMs, trained on suitable sequence arrangements, can translate through a fixed-dimensional sentence representation. Read this chapter’s recurrent state, LSTM gates and teacher forcing first.

Method

Original recurrent encoder reading ABC and decoder generating WXYZ and an end token
Source: Sutskever et al., NeurIPS 2014 author manuscript, Figure 1, PDF p. 2. The input and output can have different lengths. The end token separates encoding from generation and later ends the output; the experimental models reverse source order, not target order. [25]

The encoder consumes the source and passes its final representation to a decoder. The decoder predicts a distribution over the next target token, then continues until an end-of-sequence token. The probability of a target sequence factors as

p(y∣x)=∏t=1T′p(yt∣v,y<t),v=Encoder⁡(x).p(y\mid x)=\prod_{t=1}^{T'}p(y_t\mid v,y_{<t}),\qquad v=\operatorname{Encoder}(x).

Here xx is the source sequence, vv its fixed-dimensional encoder representation, yy the target sequence including its end token, T′T' the target length, and y<ty_{<t} the earlier target tokens. Each factor is the decoder’s next-token probability. Training maximizes this probability using the known target prefix; inference must use previously generated tokens instead.

Read Figure 1 from left to right: the input updates a recurrent state, then output generation reuses the accumulated information. The actual system uses distinct encoder and decoder LSTMs and four recurrent layers. Reversing only the source brings many corresponding source–target words closer along the computation path. It does not remove recurrence or give the decoder direct access to every source state. Beam search retains several partial hypotheses but remains an approximate search.

Results

Table 1 evaluates WMT’14 English→French on ntst14. A five-model reversed-input ensemble with beam size 12 obtains 34.81 BLEU, compared with the reported phrase-based baseline’s 33.30. This is an ensemble result, not a single LSTM score. Table 2’s 36.5 instead comes from rescoring the baseline’s 1000-best candidates with an ensemble. Candidate rescoring and generating translations from scratch are different tasks and must not share one unlabeled “best score.”

Ablations

The single forward-input LSTM scores 26.17 BLEU and the single reversed-input LSTM 30.59, both with beam size 12 in Table 1. This directly supports the importance of sequence arrangement in that training setup. Increasing ensemble size and changing beam width have separate effects: the five-model ensemble scores 33.00 with beam 1 and 34.50 with beam 2. Do not attribute all of the final gain to reversal or assume a wider beam always yields a proportionate benefit.

Conclusion

The central abstraction is conditional sequence generation: an encoder summarizes one sequence and a decoder unfolds another. Trace where information about the first input word must travel before the first output is predicted. Then read the Bahdanau guide in NLP 08 to see how attention gives the decoder repeated access to source representations.

Problems

A fixed representation can lose details, especially as inputs grow or differ from training examples. Source reversal eases some dependencies; it does not eliminate the bottleneck. Training with correct prefixes differs from inference with imperfect generated prefixes. Vocabulary limits, approximate decoding and data-specific word order also constrain the reported results. This paper establishes a useful architecture, not a guarantee that arbitrary long sequences can be faithfully compressed.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 5, Manning, 2021. Publisher
  2. Susan McRoy, Principles of Natural Language Processing, 2021. Chapter
  3. Daniel Jurafsky and James H. Martin, Speech and Language Processing, third-edition draft, Chapter 3, August 19, 2026. Chapter
  4. Stanley F. Chen and Joshua Goodman, “An Empirical Study of Smoothing Techniques for Language Modeling,” ACL, 1996. Paper
  5. Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Paper
  6. Jeffrey L. Elman, “Finding Structure in Time,” Cognitive Science 14(2), 1990. DOI
  7. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký and Sanjeev Khudanpur, “Recurrent Neural Network Based Language Model,” Interspeech, 2010. DOI
  8. Paul J. Werbos, “Backpropagation Through Time: What It Does and How to Do It,” Proceedings of the IEEE 78(10), 1990. DOI
  9. Yoshua Bengio, Patrice Simard and Paolo Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” IEEE Transactions on Neural Networks 5(2), 1994. DOI
  10. Razvan Pascanu, Tomas Mikolov and Yoshua Bengio, “On the Difficulty of Training Recurrent Neural Networks,” ICML, 2013. arXiv
  11. Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation 9(8), 1997. DOI
  12. Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, “Learning to Forget: Continual Prediction with LSTM,” Neural Computation 12(10), 2000. DOI
  13. Kyunghyun Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. arXiv
  14. Mike Schuster and Kuldip K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing 45(11), 1997. DOI
  15. Alex Graves, “Generating Sequences With Recurrent Neural Networks,” arXiv:1308.0850, 2013. arXiv
  16. Wojciech Zaremba, Ilya Sutskever and Oriol Vinyals, “Recurrent Neural Network Regularization,” arXiv:1409.2329, 2014. arXiv
  17. Gábor Melis, Chris Dyer and Phil Blunsom, “On the State of the Art of Evaluation in Neural Language Models,” arXiv:1707.05589, 2017. arXiv
  18. Stephen Merity, Nitish Shirish Keskar and Richard Socher, “Regularizing and Optimizing LSTM Language Models,” arXiv:1708.02182, 2017. arXiv
  19. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer and Yonghui Wu, “Exploring the Limits of Language Modeling,” arXiv:1602.02410, 2016. arXiv
  20. Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori and Stefano Melacci, “On the Resurgence of Recurrent Models for Long Sequences: Survey and Research Opportunities in the Transformer Era,” arXiv:2402.08132, 2024. arXiv
  21. Bo Peng et al., “RWKV: Reinventing RNNs for the Transformer Era,” Findings of EMNLP, 2023. Paper
  22. Albert Gu and Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” COLM, 2024. arXiv
  23. Maximilian Beck et al., “xLSTM: Extended Long Short-Term Memory,” NeurIPS, 2024. Paper
  24. Lewis Carroll, Alice’s Adventures in Wonderland, 1865; Project Gutenberg eBook #11. Project Gutenberg
  25. Ilya Sutskever, Oriol Vinyals and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper