Full search page →

BPE — Neural Machine Translation of Rare Words with Subword Units

Optional · paper guide 1 of 1 for NLP 02 · Linguistic Essentials. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

A fixed word vocabulary turns a rare compound or unfamiliar name into an unknown token. Sennrich, Haddow and Birch ask whether a translation model can instead compose rare words from a manageable inventory of subwords [17]. Their thesis is that changing the representation can improve rare-word translation without a separate dictionary backoff. Read the chapter’s tokenization and vocabulary discussion first; this paper changes the units supplied to a model, not its attention architecture.

Method

Four original BPE character-pair merge operations
Source: Sennrich et al., ACL 2016, Figure 1, PDF p. 4. Read downward: previously merged symbols become candidates in later merges. The dot is the end-of-word marker; these are learned operations, not a linguistic parse. [17]

Start with characters and an end-of-word marker in each word. Count adjacent symbol pairs across the training dictionary, weighting each word by its corpus frequency. Merge the most frequent pair, recount, and repeat for the chosen merge budget. In compact notation,

(a∗,b∗)=arg⁡max⁡(a,b)∑w∈Vf(w) na,b(w).(a^*,b^*)=\arg\max_{(a,b)}\sum_{w\in V} f(w)\,n_{a,b}(w).

Here VV is the training word vocabulary, f(w)f(w) is word ww‘s frequency, and na,b(w)n_{a,b}(w) counts adjacent occurrences of symbols a,ba,b in its current segmentation. The starred pair is the next merge. At inference, apply the saved operations in their learned order; do not recount pairs on the test set.

Figure 1 uses the dictionary “low, lowest, newer, wider.” Its learned pieces allow an unseen “lower” to become “low er·.” The nearby Algorithm 1 uses a different toy dictionary, so its counts should not be read as the provenance of this figure. Merges stay within pre-tokenized words. Joint BPE learns a shared merge inventory from both languages to encourage consistent segmentation, useful for names that must be copied or transliterated.

Results

Table 2 reports English→German newstest2015 results. The eight-model joint-BPE-90k ensemble scores 24.7 BLEU versus 24.2 for the word-level dictionary-backoff system. Rare-word unigram F1 rises from 36.8 to 41.8. However, out-of-vocabulary unigram F1 falls from 36.8 to 33.6. “Better rare-word handling” therefore does not mean that every previously unseen word improves. The table’s single-model and ensemble columns are different comparisons; keep them separate.

Ablations

Tables 1–2 expose the vocabulary-size/sequence-length tradeoff and compare separate with joint segmentation. Smaller units cover more forms but lengthen sequences. Separate BPE-60k and joint BPE-90k also use different vocabulary budgets, so their difference cannot isolate sharing alone. A useful follow-up experiment would fix the training corpus, merge budget and translation model while changing only whether the merge inventory is shared.

Conclusion

The reusable idea is to make open-vocabulary composition part of the input representation. Before reading the translation score, trace one word through the saved merges and check that detokenization reconstructs it. Continue to the word-vector objectives in NLP 04 to distinguish choosing tokens from learning their vectors.

Problems

Frequent pairs need not correspond to morphemes or meaningful words. Unseen characters remain a coverage issue in this character-based formulation; it is not automatically the byte-level tokenizer used by later systems. A longer subword sequence also consumes more model context. Inspect errors by compounds, names and transliteration rather than treating one aggregate translation metric as proof of linguistic correctness.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021. Publisher
  2. Susan McRoy, Principles of Natural Language Processing, University of Wisconsin–Milwaukee open textbook, 2021. Grammars and Syntactic Processing
  3. Catherine Anderson, Bronwyn Bjorkman, Derek Denis, Julianne Doner, Margaret Grant, Nathan Sanders and Ai Taniguchi, Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Open textbook
  4. Catherine Anderson et al., “3.1 Modality,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  5. Catherine Anderson et al., “4.1 Phonemes and allophones,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  6. Catherine Anderson et al., “3.12 Tone and intonation,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  7. Catherine Anderson et al., “5.1 What is morphology?,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  8. Catherine Anderson et al., “6.4 Identifying phrases: Constituency tests,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  9. Catherine Anderson et al., “8.4 Conversational implicatures,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  10. Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter and Barbara Plank, “Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges,” ACL, pp. 8679–8696, 2025. Paper
  11. Catherine Anderson et al., “10.2 Language varies,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
  12. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che and Philip S. Yu, “A survey of multilingual large language models,” Patterns, 6(1), 101118, 2025. Paper
  13. Yanru Lu and Stefan Müller, “Deliminative verbal reduplication in Mandarin Chinese,” Journal of Linguistics, 62(3), pp. 595–632, 2026. Paper
  14. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock and Almeida Jacqueline Toribio, “A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies,” ACL-IJCNLP, pp. 1654–1666, 2021. Paper
  15. Genta Winata, Alham Fikri Aji, Zheng Xin Yong and Thamar Solorio, “The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges,” Findings of ACL, pp. 2936–2978, 2023. Paper
  16. M. F. Porter, “An algorithm for suffix stripping,” Program, 14(3), pp. 130–137, 1980. DOI
  17. Rico Sennrich, Barry Haddow and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL, 2016. Paper
  18. Mike Schuster and Kaisuke Nakajima, “Japanese and Korean Voice Search,” ICASSP, pp. 5149–5152, 2012. DOI
  19. Yonghui Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. Paper
  20. Taku Kudo, “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates,” ACL, 2018. Paper
  21. Taku Kudo and John Richardson, “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing,” EMNLP System Demonstrations, 2018. Paper
  22. Sabrina J. Mielke et al., “Between Words and Characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP,” arXiv:2112.10508, 2021. Review
  23. Hugging Face, “The Tokenization Pipeline,” Tokenizers documentation, current online edition. Documentation
  24. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre and Daniel Zeman, “Universal Dependencies,” Computational Linguistics, 47(2), 2021. Paper
  25. Universal Dependencies contributors, “Universal POS tags,” Universal Dependencies documentation, current online edition. Documentation
  26. Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics, 19(2), 1993. Paper
  27. Thomas Emerson, “The Second International Chinese Word Segmentation Bakeoff,” Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, 2005. Paper
  28. Jacob Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, 20(1), pp. 37–46, 1960. DOI
  29. Steven T. Piantadosi, “Zipf’s word frequency law in natural language: A critical review and future directions,” Psychonomic Bulletin & Review, 21(5), pp. 1112–1130, 2014. DOI
  30. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019. Paper