Bahdanau attention — Jointly Learning to Align and Translate
Optional · paper guide 1 of 1 for NLP 08 · Sequence-to-Sequence Models with Transformers (Machine Translation, Chatbots). Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
A fixed-vector encoder–decoder must compress the source before generating any translation. Bahdanau, Cho and Bengio ask whether the decoder can instead consult different source locations at each output step [7]. Their thesis is to learn translation and a soft alignment jointly, using only sentence-pair supervision. Read the chapter’s encoder–decoder setup first. This recurrent attention model explains the motivation for attention even if you already studied the later Transformer in NLP 07.
Method

A bidirectional recurrent encoder creates an annotation for every source position. Each annotation combines left and right context. Before the decoder predicts a target word, a small learned alignment network scores how useful each annotation is for its previous state:
Here is a target step, a source position, the source length, the previous decoder state, and a bidirectional source annotation. The learned feed-forward function gives score ; is its normalized attention weight and the resulting context vector. The index ranges over source positions in the normalization. The decoder uses this context along with its previous state and previous target token.
Figure 1’s arrows make the difference from a fixed-vector model concrete: source annotations remain available throughout generation. The weighted sum is differentiable, so the translation loss can train the alignment mechanism without separately supplied word alignments. It is a soft distribution over positions, not a discrete commitment that each target word translates exactly one source word.
Results
Table 1 reports 17.82 BLEU for RNNencdec-50 and 26.75 for RNNsearch-50 on the full English→French test set. Longer-trained RNNsearch-50 reaches 28.45, while the reported Moses baseline reaches 33.30. On the filtered subset with no unknown words in source or references, the longer-trained attention model reaches 36.15 versus Moses’s 35.63. The filtered evaluation also disallows generated unknown tokens. That result does not establish superiority on the full test distribution.
Ablations
The comparisons cross two model types with maximum training sentence lengths of 30 or 50. Figure 2 plots performance against sentence length and supports the claim that attention handles longer inputs better in this setup. However, the starred Table 1 model was trained longer, so its gain is not an attention-only ablation. Keep architecture, training-length filtering, training duration and unknown-word filtering distinct when reading the evidence.
Conclusion
The key step is replacing one fixed source summary with a decoder-dependent summary at every output position. Explain why two target steps can assign different weights to the same source word. Return to the Transformer guide in NLP 07 to compare this learned recurrent alignment with projected query–key attention and encoder–decoder cross-attention.
Problems
Attention does not solve unknown vocabulary by itself, and the recurrent decoder still runs sequentially. A source annotation already includes surrounding context, so its weight cannot be interpreted as the exclusive contribution of one isolated word. Attractive alignment plots are useful diagnostics but not proof of causal explanation or correct translation. Length robustness must also be checked outside the paper’s filtered training and evaluation conditions.
References
- M. Hagiwara, Real-World Natural Language Processing, Manning, 2021, Chapters 6 and 8. Book · Sequence-to-sequence chapter
- S. McRoy, Principles of Natural Language Processing, open textbook, 2021, Chapter 9. Book · Generation chapter
- P. F. Brown, S. A. Della Pietra, V. J. Della Pietra and R. L. Mercer, “The Mathematics of Statistical Machine Translation: Parameter Estimation,” Computational Linguistics 19(2), 1993. Paper
- P. Koehn, F. J. Och and D. Marcu, “Statistical Phrase-Based Translation,” HLT-NAACL, 2003. Paper
- K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. Paper
- I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
- D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. Paper
- A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
- R. Sennrich, B. Haddow and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL, 2016. Paper
- M. Ranzato, S. Chopra, M. Auli and W. Zaremba, “Sequence Level Training with Recurrent Neural Networks,” ICLR, 2016. Paper
- S. Bengio, O. Vinyals, N. Jaitly and N. Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” NeurIPS, 2015. Paper
- Y. Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. Paper
- K. Papineni, S. Roukos, T. Ward and W.-J. Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation,” ACL, 2002. Paper
- M. Post, “A Call for Clarity in Reporting BLEU Scores,” WMT, 2018. Paper
- M. Popović, “chrF: character n-gram F-score for automatic MT evaluation,” WMT, 2015. Paper
- R. Rei, C. Stewart, A. C. Farinha and A. Lavie, “COMET: A Neural Framework for MT Evaluation,” EMNLP, 2020. Paper
- C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” Text Summarization Branches Out, 2004. Paper
- M. Freitag et al., “Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust,” WMT, 2022. Report
- A. Lavie et al., “Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help,” WMT, 2025. Report
- T. Kocmi et al., “Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet,” WMT, 2024. Report
- NLLB Team et al., “Scaling neural machine translation to 200 languages,” Nature 630, 2024. DOI
- F. Stahlberg, “Neural Machine Translation: A Review,” Journal of Artificial Intelligence Research 69, 2020. DOI
- O. Vinyals and Q. V. Le, “A Neural Conversational Model,” ICML Deep Learning Workshop, 2015. Paper
- J. Li, M. Galley, C. Brockett, J. Gao and B. Dolan, “A Diversity-Promoting Objective Function for Neural Conversation Models,” NAACL-HLT, 2016. Paper
- J. Deriu, Á. Rodrigo, A. Otegi, G. Echegoyen, S. Rosset, E. Agirre and M. Cieliebak, “Survey on Evaluation Methods for Dialogue Systems,” Artificial Intelligence Review 54, 2021. Survey
- J. Ni, T. Young, V. Pandelea, F. Xue and E. Cambria, “Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey,” Artificial Intelligence Review 56, 2023. Survey
- Y. Qin et al., “Tool Learning with Foundation Models,” ACM Computing Surveys 57, 2024. Author manuscript · Publication
- P. Schmidtova et al., “Do My Eyes Deceive Me? A Survey of Human Evaluations of Hallucinations in NLG,” INLG, 2025. Survey
- C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016. Paper
- M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan and W. Macherey, “Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation,” Transactions of the Association for Computational Linguistics 9, 2021. DOI