Full search page →

Bahdanau attention — Jointly Learning to Align and Translate

Optional · paper guide 1 of 1 for NLP 08 · Sequence-to-Sequence Models with Transformers (Machine Translation, Chatbots). Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

A fixed-vector encoder–decoder must compress the source before generating any translation. Bahdanau, Cho and Bengio ask whether the decoder can instead consult different source locations at each output step [7]. Their thesis is to learn translation and a soft alignment jointly, using only sentence-pair supervision. Read the chapter’s encoder–decoder setup first. This recurrent attention model explains the motivation for attention even if you already studied the later Transformer in NLP 07.

Method

Original bidirectional source annotations weighted by decoder-conditioned attention
Source: Bahdanau et al., ICLR 2015 conference manuscript, Figure 1, PDF p. 3. Bottom arrows build bidirectional source annotations. The weights above them change with the decoder step, producing the context used to predict the next target word. [7]

A bidirectional recurrent encoder creates an annotation for every source position. Each annotation combines left and right context. Before the decoder predicts a target word, a small learned alignment network scores how useful each annotation is for its previous state:

eij=a(si−1,hj),αij=exp⁡eij∑k=1Txexp⁡eik,ci=∑j=1Txαijhj.e_{ij}=a(s_{i-1},h_j),\qquad \alpha_{ij}=\frac{\exp e_{ij}}{\sum_{k=1}^{T_x}\exp e_{ik}},\qquad c_i=\sum_{j=1}^{T_x}\alpha_{ij}h_j.

Here ii is a target step, jj a source position, TxT_x the source length, si−1s_{i-1} the previous decoder state, and hjh_j a bidirectional source annotation. The learned feed-forward function aa gives score eije_{ij}; αij\alpha_{ij} is its normalized attention weight and cic_i the resulting context vector. The index kk ranges over source positions in the normalization. The decoder uses this context along with its previous state and previous target token.

Figure 1’s arrows make the difference from a fixed-vector model concrete: source annotations remain available throughout generation. The weighted sum is differentiable, so the translation loss can train the alignment mechanism without separately supplied word alignments. It is a soft distribution over positions, not a discrete commitment that each target word translates exactly one source word.

Results

Table 1 reports 17.82 BLEU for RNNencdec-50 and 26.75 for RNNsearch-50 on the full English→French test set. Longer-trained RNNsearch-50 reaches 28.45, while the reported Moses baseline reaches 33.30. On the filtered subset with no unknown words in source or references, the longer-trained attention model reaches 36.15 versus Moses’s 35.63. The filtered evaluation also disallows generated unknown tokens. That result does not establish superiority on the full test distribution.

Ablations

The comparisons cross two model types with maximum training sentence lengths of 30 or 50. Figure 2 plots performance against sentence length and supports the claim that attention handles longer inputs better in this setup. However, the starred Table 1 model was trained longer, so its gain is not an attention-only ablation. Keep architecture, training-length filtering, training duration and unknown-word filtering distinct when reading the evidence.

Conclusion

The key step is replacing one fixed source summary with a decoder-dependent summary at every output position. Explain why two target steps can assign different weights to the same source word. Return to the Transformer guide in NLP 07 to compare this learned recurrent alignment with projected query–key attention and encoder–decoder cross-attention.

Problems

Attention does not solve unknown vocabulary by itself, and the recurrent decoder still runs sequentially. A source annotation already includes surrounding context, so its weight cannot be interpreted as the exclusive contribution of one isolated word. Attractive alignment plots are useful diagnostics but not proof of causal explanation or correct translation. Length robustness must also be checked outside the paper’s filtered training and evaluation conditions.

References

  1. M. Hagiwara, Real-World Natural Language Processing, Manning, 2021, Chapters 6 and 8. Book · Sequence-to-sequence chapter
  2. S. McRoy, Principles of Natural Language Processing, open textbook, 2021, Chapter 9. Book · Generation chapter
  3. P. F. Brown, S. A. Della Pietra, V. J. Della Pietra and R. L. Mercer, “The Mathematics of Statistical Machine Translation: Parameter Estimation,” Computational Linguistics 19(2), 1993. Paper
  4. P. Koehn, F. J. Och and D. Marcu, “Statistical Phrase-Based Translation,” HLT-NAACL, 2003. Paper
  5. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. Paper
  6. I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
  7. D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. Paper
  8. A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  9. R. Sennrich, B. Haddow and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL, 2016. Paper
  10. M. Ranzato, S. Chopra, M. Auli and W. Zaremba, “Sequence Level Training with Recurrent Neural Networks,” ICLR, 2016. Paper
  11. S. Bengio, O. Vinyals, N. Jaitly and N. Shazeer, “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks,” NeurIPS, 2015. Paper
  12. Y. Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. Paper
  13. K. Papineni, S. Roukos, T. Ward and W.-J. Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation,” ACL, 2002. Paper
  14. M. Post, “A Call for Clarity in Reporting BLEU Scores,” WMT, 2018. Paper
  15. M. Popović, “chrF: character n-gram F-score for automatic MT evaluation,” WMT, 2015. Paper
  16. R. Rei, C. Stewart, A. C. Farinha and A. Lavie, “COMET: A Neural Framework for MT Evaluation,” EMNLP, 2020. Paper
  17. C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” Text Summarization Branches Out, 2004. Paper
  18. M. Freitag et al., “Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust,” WMT, 2022. Report
  19. A. Lavie et al., “Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help,” WMT, 2025. Report
  20. T. Kocmi et al., “Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet,” WMT, 2024. Report
  21. NLLB Team et al., “Scaling neural machine translation to 200 languages,” Nature 630, 2024. DOI
  22. F. Stahlberg, “Neural Machine Translation: A Review,” Journal of Artificial Intelligence Research 69, 2020. DOI
  23. O. Vinyals and Q. V. Le, “A Neural Conversational Model,” ICML Deep Learning Workshop, 2015. Paper
  24. J. Li, M. Galley, C. Brockett, J. Gao and B. Dolan, “A Diversity-Promoting Objective Function for Neural Conversation Models,” NAACL-HLT, 2016. Paper
  25. J. Deriu, Á. Rodrigo, A. Otegi, G. Echegoyen, S. Rosset, E. Agirre and M. Cieliebak, “Survey on Evaluation Methods for Dialogue Systems,” Artificial Intelligence Review 54, 2021. Survey
  26. J. Ni, T. Young, V. Pandelea, F. Xue and E. Cambria, “Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey,” Artificial Intelligence Review 56, 2023. Survey
  27. Y. Qin et al., “Tool Learning with Foundation Models,” ACM Computing Surveys 57, 2024. Author manuscript · Publication
  28. P. Schmidtova et al., “Do My Eyes Deceive Me? A Survey of Human Evaluations of Hallucinations in NLG,” INLG, 2025. Survey
  29. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016. Paper
  30. M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan and W. Macherey, “Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation,” Transactions of the Association for Computational Linguistics 9, 2021. DOI