Transformer — Attention Is All You Need
Optional · paper guide 1 of 1 for NLP 07 · Attention Mechanism and Transformers. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
Recurrent translation models pass information through sequential state updates, which restricts parallel training and lengthens some dependency paths. Vaswani et al. replace recurrence with attention and position-wise feed-forward networks [6]. Their thesis is architectural: attention can supply the communication mechanism for an encoder–decoder translation system. First understand this chapter’s queries, keys, values and masks; then read the original diagram as three distinct attention routes.
Method

For one attention head, the computation can be written with an explicit mask:
contains query vectors, key vectors, value vectors, and is the query/key dimension. The matrix is zero for allowed query–key pairs and negative infinity for forbidden pairs; softmax normalizes each query’s row. The scale keeps large dot products from making the softmax too sharply saturated as dimension grows. Multiple learned heads perform this operation in different projected spaces before their outputs are combined.
In encoder self-attention, all three inputs come from encoder states. In decoder self-attention, they come from decoder states, with future target positions masked. In cross-attention, queries come from the decoder while keys and values come from the encoder. Positional information supplies order that attention alone lacks. Residual paths, feed-forward sublayers and normalization complete the block; the original uses normalization after the residual addition.
Teacher-forced training can process target positions together because the shifted correct prefix and causal mask prevent future-label access. Generation still produces tokens autoregressively. Parallel training therefore does not imply simultaneous unrestricted generation of the whole output.
Results
Table 2 reports newstest2014 translation scores: Transformer base reaches 27.3 BLEU on English→German, and big reaches 28.4; big reaches 41.0 on English→French. The reported evaluation uses checkpoint averaging, with five checkpoints for base and twenty for big, and beam search. State these settings when comparing to a single checkpoint or greedy decoding. The paper’s efficiency claim concerns its translation setup, not every possible sequence length or hardware configuration.
Ablations
Table 3 varies heads, dimensions, feed-forward capacity and regularization on English→German newstest2013 development data, without checkpoint averaging. A single attention head performs worse than the best setting, while increasing the head count too far also hurts. These development scores are not the Table 2 test scores. Label smoothing can improve BLEU while worsening perplexity, illustrating why a better training-style likelihood metric need not imply a better decoded translation.
Conclusion
The useful reading skill is to label where each attention block obtains its queries, keys and values, then mark which positions are forbidden. Continue to NLP 08 for full decoding behavior and to NLP 10 for an encoder-only pretraining use of the same building blocks.
Problems
Dense attention creates a quadratic number of token-pair scores. Masks and position handling can silently change the task, and a missing causal mask leaks future targets during training. Attention weights describe a learned computation, not a guaranteed causal explanation. Removing recurrence changes the architecture; it does not remove context limits, data bias or the need to evaluate long-sequence behavior separately.
References
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 8, Manning, 2021. Publisher · Chapter 8
- I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
- K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. Paper
- D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. Paper
- M.-T. Luong, H. Pham and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” EMNLP, 2015. Paper
- A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
- J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
- R. Xiong et al., “On Layer Normalization in the Transformer Architecture,” ICML, 2020. Paper
- P. Shaw, J. Uszkoreit and A. Vaswani, “Self-Attention with Relative Position Representations,” NAACL, 2018. Paper
- J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing 568, 2024. DOI
- O. Press, N. A. Smith and M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” ICLR, 2022. Paper
- I. Beltagy, M. E. Peters and A. Cohan, “Longformer: The Long-Document Transformer,” arXiv:2004.05150, 2020. Paper
- A. Katharopoulos, A. Vyas, N. Pappas and F. Fleuret, “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention,” ICML, PMLR 119, 2020. Paper
- T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
- N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need,” arXiv:1911.02150, 2019. Paper
- J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón and S. Sanghai, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” EMNLP, 2023. Paper
- Y. Tay, M. Dehghani, D. Bahri and D. Metzler, “Efficient Transformers: A Survey,” ACM Computing Surveys 55(6), 2022. DOI
- T. Lin, Y. Wang, X. Liu and X. Qiu, “A Survey of Transformers,” AI Open 3, 2022. DOI
- S. Jain and B. C. Wallace, “Attention is not Explanation,” NAACL, 2019. Paper
- S. Wiegreffe and Y. Pinter, “Attention is not not Explanation,” EMNLP-IJCNLP, 2019. Paper
- PyTorch,
TransformerEncoderLayerdocumentation. Documentation - PyTorch,
scaled_dot_product_attentiondocumentation, version 2.15, accessed 2026-11-05. Documentation - PyTorch,
MultiheadAttentiondocumentation, version 2.15, accessed 2026-11-05. Documentation