完整搜尋頁面 →

BERT — Pre-training of Deep Bidirectional Transformers for Language Understanding

選讀・NLP 10・預訓練模型:BERT 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

挑戰

由左至右的語言模型,在建立目前 token 的表示時不能使用後方詞彙。Devlin 等人提出預訓練深層雙向 Transformer 編碼器,再適配多種理解任務 [1]。論點是遮罩 token 預測能提供雙向訓練,同時避免直接把每個 token 本身當成答案暴露給模型。先理解本章的編碼器自注意力、token 表示與任務輸出頭。

方法

原論文 BERT 預訓練與微調圖,包含 token 與句級輸出頭
來源:Devlin 等人,NAACL 2019,圖 1,PDF 第 3 頁。左側是遮罩語言與下一句目標,右側用任務輸出頭重用編碼器。微調會更新編碼器參數,也會更新新的任務頭。[1]

BERT 選取 15% 的 token 位置進行遮罩語言模型(MLM)訓練。在選中位置內,80% 改成遮罩 token、10% 改成隨機 token、10% 維持原樣。這三個百分比的分母是選中子集,不是整句。目標可簡寫為:

LMLM=−∑i∈Slog⁡pθ(xi∣x~).\mathcal L_{\mathrm{MLM}}=-\sum_{i\in S}\log p_\theta(x_i\mid\tilde x).

xix_i 是位置 ii 的原始 token,SS 是選中的預測位置,x~\tilde x 是擾動後輸入,pθp_\theta 是參數為 θ\theta 的編碼器與預測頭所產生的 token 分佈。損失 LMLM\mathcal L_{\mathrm{MLM}} 監督選中位置,也包含選中但保留原樣的 token;不是要求預測每個未遮罩輸入。

原始流程也使用下一句預測(NSP):一半配對延續同一文件,另一半的第二段隨機取樣。token、區段與位置嵌入共同輸入同一編碼器。微調時,分類頭可讀取特殊的句級表示,token 或跨度輸出頭則讀取 token 狀態。圖 1 表示不同任務共用預訓練結構,不是要求凍結編碼器。

結果

表 5 提供條件明確的一組證據:BERT-base 在開發集的 MNLI 準確率為 84.4、QNLI 準確率 88.4、SQuAD F1 88.5。這些是 base 模型的開發集結果,不是摘要中的 large 模型測試提交。各基準須依自身指標解讀;分類準確率與答案跨度 F1 不是共同的語言理解單位。論文其他任務表格則展示調整輸出頭後的跨任務遷移。

消融與設計證據

表 5 中移除 NSP,QNLI 準確率從 88.4 降至 84.9,SQuAD F1 從 88.5 降至 87.9。改成由左至右預訓練且沒有 NSP,SQuAD F1 為 77.8;微調時加入隨機初始化的雙向 LSTM,也只回復到 84.9。這支持原始流程中雙向預訓練的價值,但不能證明所有後續語料、排程或編碼器設計都必須使用 NSP。

結論

BERT 讓同一預訓練編碼器轉成多種任務預測器。檢查理解時,標出哪些輸入位置被擾動、哪些位置計入 MLM 損失,以及微調時更新哪些參數。接著讀 NLP 11 的 Sentence-BERT 導讀,理解好的 token 表示為何不會自動變成好的句子相似度向量。

問題與限制

MLM 不是由左至右的序列機率分解,不能直接比較一般自迴歸困惑度。隨機負例 NSP 可能獎勵主題線索,而非深入篇章理解。預訓練資料、tokenizer 與上下文長度限制遷移,監督式微調仍需要具代表性的標籤。雙向性讓模型看得到上下文,卻不保證推理、事實正確或跨領域穩健性。

參考文獻

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL, 2019. Paper
  2. Google Research, BERT: original implementation and released models, software, 2018. Encoder and pooler · Tokenizer · Multilingual and Chinese models
  3. Ashish Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  4. Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI technical report, 2018. Paper
  5. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer, “Deep Contextualized Word Representations,” NAACL, 2018. Paper
  6. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space,” ICLR Workshop, 2013. Paper
  7. Yonghui Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. Paper
  8. Dan Hendrycks and Kevin Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016, revised 2023. Paper
  9. Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
  10. Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. Paper
  11. Yinhan Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv:1907.11692, 2019. Paper
  12. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma and Radu Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” ICLR, 2020. Paper
  13. Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing at NeurIPS, 2019. Paper
  14. Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning, “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,” ICLR, 2020. Paper
  15. Telmo Pires, Eva Schlinger and Dan Garrette, “How Multilingual is Multilingual BERT?,” ACL, 2019. Paper
  16. Alexis Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale,” ACL, 2020. Paper
  17. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin and Ziqing Yang, “Pre-Training With Whole Word Masking for Chinese BERT,” IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, 2021. Publisher · Author manuscript
  18. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 9, Manning, 2021. Publisher
  19. Kawin Ethayarajh, “How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings,” EMNLP-IJCNLP, 2019. Paper
  20. Ian Tenney, Dipanjan Das and Ellie Pavlick, “BERT Rediscovers the Classical NLP Pipeline,” ACL, 2019. Paper
  21. John Hewitt and Christopher D. Manning, “A Structural Probe for Finding Syntax in Word Representations,” NAACL, 2019. Paper
  22. John Hewitt and Percy Liang, “Designing and Interpreting Probes with Control Tasks,” EMNLP-IJCNLP, 2019. Paper
  23. Kevin Clark, Urvashi Khandelwal, Omer Levy and Christopher D. Manning, “What Does BERT Look at? An Analysis of BERT’s Attention,” BlackboxNLP Workshop, 2019. Paper
  24. Anna Rogers, Olga Kovaleva and Anna Rumshisky, “A Primer in BERTology: What We Know About How BERT Works,” Transactions of the Association for Computational Linguistics 8, 2020. Paper
  25. Benjamin Warner et al., “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference,” ACL, 2025. Paper
  26. Wayne Xin Zhao et al., “A Survey of Large Language Models,” Frontiers of Computer Science, 2026. Updated survey
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” JMLR 21(140), 2020. Paper