Full search page →

Paper guides

Optional reading guides to the papers behind NLP 07 · Attention Mechanism and Transformers, one page each: the problem, the idea, what it showed and what to check when you read it.

Navigate: this page and this track: Navigate: this page and this track

These optional guides connect the chapter to original papers. Read the challenge and figure first, then unfold the method, evidence and limitations.

  1. Transformer — Attention Is All You Need

    Recurrent translation models pass information through sequential state updates, which restricts parallel training and lengthens some dependency paths.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 8, Manning, 2021. Publisher · Chapter 8
  2. I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
  3. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. Paper
  4. D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. Paper
  5. M.-T. Luong, H. Pham and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” EMNLP, 2015. Paper
  6. A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  7. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
  8. R. Xiong et al., “On Layer Normalization in the Transformer Architecture,” ICML, 2020. Paper
  9. P. Shaw, J. Uszkoreit and A. Vaswani, “Self-Attention with Relative Position Representations,” NAACL, 2018. Paper
  10. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing 568, 2024. DOI
  11. O. Press, N. A. Smith and M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” ICLR, 2022. Paper
  12. I. Beltagy, M. E. Peters and A. Cohan, “Longformer: The Long-Document Transformer,” arXiv:2004.05150, 2020. Paper
  13. A. Katharopoulos, A. Vyas, N. Pappas and F. Fleuret, “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention,” ICML, PMLR 119, 2020. Paper
  14. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
  15. N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need,” arXiv:1911.02150, 2019. Paper
  16. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón and S. Sanghai, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” EMNLP, 2023. Paper
  17. Y. Tay, M. Dehghani, D. Bahri and D. Metzler, “Efficient Transformers: A Survey,” ACM Computing Surveys 55(6), 2022. DOI
  18. T. Lin, Y. Wang, X. Liu and X. Qiu, “A Survey of Transformers,” AI Open 3, 2022. DOI
  19. S. Jain and B. C. Wallace, “Attention is not Explanation,” NAACL, 2019. Paper
  20. S. Wiegreffe and Y. Pinter, “Attention is not not Explanation,” EMNLP-IJCNLP, 2019. Paper
  21. PyTorch, TransformerEncoderLayer documentation. Documentation
  22. PyTorch, scaled_dot_product_attention documentation, version 2.15, accessed 2026-11-05. Documentation
  23. PyTorch, MultiheadAttention documentation, version 2.15, accessed 2026-11-05. Documentation