Full search page →

Transformer — Attention Is All You Need

Optional · paper guide 1 of 1 for NLP 07 · Attention Mechanism and Transformers. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

Recurrent translation models pass information through sequential state updates, which restricts parallel training and lengthens some dependency paths. Vaswani et al. replace recurrence with attention and position-wise feed-forward networks [6]. Their thesis is architectural: attention can supply the communication mechanism for an encoder–decoder translation system. First understand this chapter’s queries, keys, values and masks; then read the original diagram as three distinct attention routes.

Method

Original Transformer encoder and decoder stacks with three attention routes
Source: Vaswani et al., NeurIPS 2017, Figure 1, PDF p. 3. Left: encoder self-attention. Right: masked decoder self-attention followed by attention to encoder outputs. Add & Norm follows each sublayer in this original architecture. [6]

For one attention head, the computation can be written with an explicit mask:

Attention⁡(Q,K,V)=softmax⁡(QK⊤dk+M)V.\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V.

QQ contains query vectors, KK key vectors, VV value vectors, and dkd_k is the query/key dimension. The matrix MM is zero for allowed query–key pairs and negative infinity for forbidden pairs; softmax normalizes each query’s row. The scale keeps large dot products from making the softmax too sharply saturated as dimension grows. Multiple learned heads perform this operation in different projected spaces before their outputs are combined.

In encoder self-attention, all three inputs come from encoder states. In decoder self-attention, they come from decoder states, with future target positions masked. In cross-attention, queries come from the decoder while keys and values come from the encoder. Positional information supplies order that attention alone lacks. Residual paths, feed-forward sublayers and normalization complete the block; the original uses normalization after the residual addition.

Teacher-forced training can process target positions together because the shifted correct prefix and causal mask prevent future-label access. Generation still produces tokens autoregressively. Parallel training therefore does not imply simultaneous unrestricted generation of the whole output.

Results

Table 2 reports newstest2014 translation scores: Transformer base reaches 27.3 BLEU on English→German, and big reaches 28.4; big reaches 41.0 on English→French. The reported evaluation uses checkpoint averaging, with five checkpoints for base and twenty for big, and beam search. State these settings when comparing to a single checkpoint or greedy decoding. The paper’s efficiency claim concerns its translation setup, not every possible sequence length or hardware configuration.

Ablations

Table 3 varies heads, dimensions, feed-forward capacity and regularization on English→German newstest2013 development data, without checkpoint averaging. A single attention head performs worse than the best setting, while increasing the head count too far also hurts. These development scores are not the Table 2 test scores. Label smoothing can improve BLEU while worsening perplexity, illustrating why a better training-style likelihood metric need not imply a better decoded translation.

Conclusion

The useful reading skill is to label where each attention block obtains its queries, keys and values, then mark which positions are forbidden. Continue to NLP 08 for full decoding behavior and to NLP 10 for an encoder-only pretraining use of the same building blocks.

Problems

Dense attention creates a quadratic number of token-pair scores. Masks and position handling can silently change the task, and a missing causal mask leaks future targets during training. Attention weights describe a learned computation, not a guaranteed causal explanation. Removing recurrence changes the architecture; it does not remove context limits, data bias or the need to evaluate long-sequence behavior separately.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 8, Manning, 2021. Publisher · Chapter 8
  2. I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
  3. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. Paper
  4. D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. Paper
  5. M.-T. Luong, H. Pham and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” EMNLP, 2015. Paper
  6. A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  7. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
  8. R. Xiong et al., “On Layer Normalization in the Transformer Architecture,” ICML, 2020. Paper
  9. P. Shaw, J. Uszkoreit and A. Vaswani, “Self-Attention with Relative Position Representations,” NAACL, 2018. Paper
  10. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing 568, 2024. DOI
  11. O. Press, N. A. Smith and M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” ICLR, 2022. Paper
  12. I. Beltagy, M. E. Peters and A. Cohan, “Longformer: The Long-Document Transformer,” arXiv:2004.05150, 2020. Paper
  13. A. Katharopoulos, A. Vyas, N. Pappas and F. Fleuret, “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention,” ICML, PMLR 119, 2020. Paper
  14. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
  15. N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need,” arXiv:1911.02150, 2019. Paper
  16. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón and S. Sanghai, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” EMNLP, 2023. Paper
  17. Y. Tay, M. Dehghani, D. Bahri and D. Metzler, “Efficient Transformers: A Survey,” ACM Computing Surveys 55(6), 2022. DOI
  18. T. Lin, Y. Wang, X. Liu and X. Qiu, “A Survey of Transformers,” AI Open 3, 2022. DOI
  19. S. Jain and B. C. Wallace, “Attention is not Explanation,” NAACL, 2019. Paper
  20. S. Wiegreffe and Y. Pinter, “Attention is not not Explanation,” EMNLP-IJCNLP, 2019. Paper
  21. PyTorch, TransformerEncoderLayer documentation. Documentation
  22. PyTorch, scaled_dot_product_attention documentation, version 2.15, accessed 2026-11-05. Documentation
  23. PyTorch, MultiheadAttention documentation, version 2.15, accessed 2026-11-05. Documentation