完整搜尋頁面 →

BPE — Neural Machine Translation of Rare Words with Subword Units

選讀・NLP 02・語言學基礎 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

挑戰

固定詞彙表會把罕見複合詞或陌生姓名變成未知詞。Sennrich、Haddow 與 Birch 探討:翻譯模型能否用數量可控的子詞來組合罕見詞 [17]?論點是改變文字表示,就能改善罕見詞翻譯,而不必另外依賴字典回退。先讀本章的斷詞與詞彙表說明;這篇改的是模型接收的單位,不是注意力架構。

方法

原論文的四個 BPE 字元配對合併操作
來源:Sennrich 等人,ACL 2016,圖 1,PDF 第 4 頁。由上往下讀:已合併的符號可以參與後續合併。圓點代表詞尾;這些是學到的操作,不是語言學剖析。[17]

先把每個詞拆成字元並加入詞尾標記。統計訓練詞典內相鄰符號配對的次數,以各詞在語料中的頻率加權。合併最常見的配對、重新計數,重複到設定的合併次數。可簡寫為:

(a∗,b∗)=arg⁡max⁡(a,b)∑w∈Vf(w) na,b(w).(a^*,b^*)=\arg\max_{(a,b)}\sum_{w\in V} f(w)\,n_{a,b}(w).

VV 是訓練詞彙表,f(w)f(w) 是詞 ww 的出現頻率,na,b(w)n_{a,b}(w) 計算其目前切分中符號 a,ba,b 相鄰出現的次數。帶星號的配對就是下一次合併對象。推論時依學得順序套用儲存的操作,不可在測試集重新統計配對。

圖 1 使用「low、lowest、newer、wider」這組詞。學到的片段能把未見過的「lower」切成「low er·」。旁邊的 Algorithm 1 使用另一組示範詞典,因此不能拿它的計數解釋這張圖。合併不跨越預先切好的詞邊界。聯合 BPE 從兩種語言學習共用的合併表,使切分較一致,對需要複製或音譯的姓名尤其有用。

結果

表 2 報告英譯德 newstest2015 結果。聯合 BPE-90k 的八模型集成得到 24.7 BLEU,詞級字典回退系統為 24.2。罕見詞 unigram F1 從 36.8 升至 41.8,但詞彙表外詞的 unigram F1 從 36.8 降至 33.6。因此,「改善罕見詞」不代表所有未見詞都改善。表中的單模型與集成欄位也必須分開比較。

消融與設計證據

表 1–2 呈現詞彙表大小與序列長度的取捨,並比較分開與聯合切分。較小單位能涵蓋更多詞形,但會拉長序列。分開的 BPE-60k 與聯合 BPE-90k 同時改變了詞彙表預算,因此差異無法單獨歸因於共用操作。後續實驗應固定語料、合併預算及翻譯模型,只改變是否共用合併表。

結論

可延伸的核心想法,是讓開放詞彙的組合能力成為輸入表示的一部分。看翻譯分數前,先追蹤一個詞如何套用合併表,並確認能還原原詞。接著讀 NLP 04 的詞向量目標,區分「選擇 token」與「學習 token 向量」。

問題與限制

常見配對不一定是詞素或有意義的詞。這個以字元為起點的版本仍可能遇到未見字元,不能直接視為後來系統的位元組級 tokenizer。較長的子詞序列也會占用更多模型上下文。應分別檢查複合詞、姓名與音譯錯誤,而不是用單一翻譯總分證明語言學上的正確性。

參考文獻

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021. 出版社
  2. Susan McRoy, Principles of Natural Language Processing, University of Wisconsin–Milwaukee open textbook, 2021. Grammars and Syntactic Processing
  3. Catherine Anderson, Bronwyn Bjorkman, Derek Denis, Julianne Doner, Margaret Grant, Nathan Sanders and Ai Taniguchi, Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 開放教科書
  4. Catherine Anderson et al., “3.1 Modality,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  5. Catherine Anderson et al., “4.1 Phonemes and allophones,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  6. Catherine Anderson et al., “3.12 Tone and intonation,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  7. Catherine Anderson et al., “5.1 What is morphology?,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  8. Catherine Anderson et al., “6.4 Identifying phrases: Constituency tests,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  9. Catherine Anderson et al., “8.4 Conversational implicatures,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  10. Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter and Barbara Plank, “Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges,” ACL, pp. 8679–8696, 2025. 論文
  11. Catherine Anderson et al., “10.2 Language varies,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. 章節
  12. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che and Philip S. Yu, “A survey of multilingual large language models,” Patterns, 6(1), 101118, 2025. 論文
  13. Yanru Lu and Stefan Müller, “Deliminative verbal reduplication in Mandarin Chinese,” Journal of Linguistics, 62(3), pp. 595–632, 2026. 論文
  14. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock and Almeida Jacqueline Toribio, “A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies,” ACL-IJCNLP, pp. 1654–1666, 2021. 論文
  15. Genta Winata, Alham Fikri Aji, Zheng Xin Yong and Thamar Solorio, “The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges,” Findings of ACL, pp. 2936–2978, 2023. 論文
  16. M. F. Porter, “An algorithm for suffix stripping,” Program, 14(3), pp. 130–137, 1980. DOI
  17. Rico Sennrich, Barry Haddow and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL, 2016. 論文
  18. Mike Schuster and Kaisuke Nakajima, “Japanese and Korean Voice Search,” ICASSP, pp. 5149–5152, 2012. DOI
  19. Yonghui Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. 論文
  20. Taku Kudo, “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates,” ACL, 2018. 論文
  21. Taku Kudo and John Richardson, “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing,” EMNLP System Demonstrations, 2018. 論文
  22. Sabrina J. Mielke et al., “Between Words and Characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP,” arXiv:2112.10508, 2021. 綜述
  23. Hugging Face, “The Tokenization Pipeline,” Tokenizers documentation, current online edition. 文件
  24. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre and Daniel Zeman, “Universal Dependencies,” Computational Linguistics, 47(2), 2021. 論文
  25. Universal Dependencies contributors, “Universal POS tags,” Universal Dependencies documentation, current online edition. 文件
  26. Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics, 19(2), 1993. 論文
  27. Thomas Emerson, “The Second International Chinese Word Segmentation Bakeoff,” Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, 2005. 論文
  28. Jacob Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, 20(1), pp. 37–46, 1960. DOI
  29. Steven T. Piantadosi, “Zipf’s word frequency law in natural language: A critical review and future directions,” Psychonomic Bulletin & Review, 21(5), pp. 1112–1130, 2014. DOI
  30. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019. 論文