完整搜尋頁面 →

論文導讀

選讀:NLP 07・注意力機制與 Transformer 所依據論文的導讀,每篇一頁:說明它要解決的問題、核心想法、結果,以及閱讀時該注意什麼。

導覽:本頁與本系列: 導覽:本頁與本系列

以下為選讀導讀,將本章連到原始論文。先讀挑戰與圖,再逐層展開方法、證據及限制。

  1. Transformer — Attention Is All You Need

    循環翻譯模型透過依序更新狀態傳遞資訊,限制訓練平行化,也拉長部分相依路徑。

參考文獻

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 8, Manning, 2021. Publisher · Chapter 8
  2. I. Sutskever, O. Vinyals and Q. V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
  3. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” EMNLP, 2014. Paper
  4. D. Bahdanau, K. Cho and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” ICLR, 2015. Paper
  5. M.-T. Luong, H. Pham and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” EMNLP, 2015. Paper
  6. A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  7. J. L. Ba, J. R. Kiros and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
  8. R. Xiong et al., “On Layer Normalization in the Transformer Architecture,” ICML, 2020. Paper
  9. P. Shaw, J. Uszkoreit and A. Vaswani, “Self-Attention with Relative Position Representations,” NAACL, 2018. Paper
  10. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo and Y. Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing 568, 2024. DOI
  11. O. Press, N. A. Smith and M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” ICLR, 2022. Paper
  12. I. Beltagy, M. E. Peters and A. Cohan, “Longformer: The Long-Document Transformer,” arXiv:2004.05150, 2020. Paper
  13. A. Katharopoulos, A. Vyas, N. Pappas and F. Fleuret, “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention,” ICML, PMLR 119, 2020. Paper
  14. T. Dao, D. Y. Fu, S. Ermon, A. Rudra and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS, 2022. Paper
  15. N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need,” arXiv:1911.02150, 2019. Paper
  16. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón and S. Sanghai, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” EMNLP, 2023. Paper
  17. Y. Tay, M. Dehghani, D. Bahri and D. Metzler, “Efficient Transformers: A Survey,” ACM Computing Surveys 55(6), 2022. DOI
  18. T. Lin, Y. Wang, X. Liu and X. Qiu, “A Survey of Transformers,” AI Open 3, 2022. DOI
  19. S. Jain and B. C. Wallace, “Attention is not Explanation,” NAACL, 2019. Paper
  20. S. Wiegreffe and Y. Pinter, “Attention is not not Explanation,” EMNLP-IJCNLP, 2019. Paper
  21. PyTorch, TransformerEncoderLayer documentation. Documentation
  22. PyTorch, scaled_dot_product_attention 文件,2.15 版,查閱日期 2026-11-05。文件
  23. PyTorch, MultiheadAttention 文件,2.15 版,查閱日期 2026-11-05。文件