BPE — Neural Machine Translation of Rare Words with Subword Units
Optional · paper guide 1 of 1 for NLP 02 · Linguistic Essentials. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
A fixed word vocabulary turns a rare compound or unfamiliar name into an unknown token. Sennrich, Haddow and Birch ask whether a translation model can instead compose rare words from a manageable inventory of subwords [17]. Their thesis is that changing the representation can improve rare-word translation without a separate dictionary backoff. Read the chapter’s tokenization and vocabulary discussion first; this paper changes the units supplied to a model, not its attention architecture.
Method

Start with characters and an end-of-word marker in each word. Count adjacent symbol pairs across the training dictionary, weighting each word by its corpus frequency. Merge the most frequent pair, recount, and repeat for the chosen merge budget. In compact notation,
Here is the training word vocabulary, is word ‘s frequency, and counts adjacent occurrences of symbols in its current segmentation. The starred pair is the next merge. At inference, apply the saved operations in their learned order; do not recount pairs on the test set.
Figure 1 uses the dictionary “low, lowest, newer, wider.” Its learned pieces allow an unseen “lower” to become “low er·.” The nearby Algorithm 1 uses a different toy dictionary, so its counts should not be read as the provenance of this figure. Merges stay within pre-tokenized words. Joint BPE learns a shared merge inventory from both languages to encourage consistent segmentation, useful for names that must be copied or transliterated.
Results
Table 2 reports English→German newstest2015 results. The eight-model joint-BPE-90k ensemble scores 24.7 BLEU versus 24.2 for the word-level dictionary-backoff system. Rare-word unigram F1 rises from 36.8 to 41.8. However, out-of-vocabulary unigram F1 falls from 36.8 to 33.6. “Better rare-word handling” therefore does not mean that every previously unseen word improves. The table’s single-model and ensemble columns are different comparisons; keep them separate.
Ablations
Tables 1–2 expose the vocabulary-size/sequence-length tradeoff and compare separate with joint segmentation. Smaller units cover more forms but lengthen sequences. Separate BPE-60k and joint BPE-90k also use different vocabulary budgets, so their difference cannot isolate sharing alone. A useful follow-up experiment would fix the training corpus, merge budget and translation model while changing only whether the merge inventory is shared.
Conclusion
The reusable idea is to make open-vocabulary composition part of the input representation. Before reading the translation score, trace one word through the saved merges and check that detokenization reconstructs it. Continue to the word-vector objectives in NLP 04 to distinguish choosing tokens from learning their vectors.
Problems
Frequent pairs need not correspond to morphemes or meaningful words. Unseen characters remain a coverage issue in this character-based formulation; it is not automatically the byte-level tokenizer used by later systems. A longer subword sequence also consumes more model context. Inspect errors by compounds, names and transliteration rather than treating one aggregate translation metric as proof of linguistic correctness.
References
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021. Publisher
- Susan McRoy, Principles of Natural Language Processing, University of Wisconsin–Milwaukee open textbook, 2021. Grammars and Syntactic Processing
- Catherine Anderson, Bronwyn Bjorkman, Derek Denis, Julianne Doner, Margaret Grant, Nathan Sanders and Ai Taniguchi, Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Open textbook
- Catherine Anderson et al., “3.1 Modality,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Catherine Anderson et al., “4.1 Phonemes and allophones,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Catherine Anderson et al., “3.12 Tone and intonation,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Catherine Anderson et al., “5.1 What is morphology?,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Catherine Anderson et al., “6.4 Identifying phrases: Constituency tests,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Catherine Anderson et al., “8.4 Conversational implicatures,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter and Barbara Plank, “Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges,” ACL, pp. 8679–8696, 2025. Paper
- Catherine Anderson et al., “10.2 Language varies,” Essentials of Linguistics, 2nd ed., eCampusOntario, 2022. Chapter
- Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che and Philip S. Yu, “A survey of multilingual large language models,” Patterns, 6(1), 101118, 2025. Paper
- Yanru Lu and Stefan Müller, “Deliminative verbal reduplication in Mandarin Chinese,” Journal of Linguistics, 62(3), pp. 595–632, 2026. Paper
- A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock and Almeida Jacqueline Toribio, “A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies,” ACL-IJCNLP, pp. 1654–1666, 2021. Paper
- Genta Winata, Alham Fikri Aji, Zheng Xin Yong and Thamar Solorio, “The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges,” Findings of ACL, pp. 2936–2978, 2023. Paper
- M. F. Porter, “An algorithm for suffix stripping,” Program, 14(3), pp. 130–137, 1980. DOI
- Rico Sennrich, Barry Haddow and Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL, 2016. Paper
- Mike Schuster and Kaisuke Nakajima, “Japanese and Korean Voice Search,” ICASSP, pp. 5149–5152, 2012. DOI
- Yonghui Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. Paper
- Taku Kudo, “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates,” ACL, 2018. Paper
- Taku Kudo and John Richardson, “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing,” EMNLP System Demonstrations, 2018. Paper
- Sabrina J. Mielke et al., “Between Words and Characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP,” arXiv:2112.10508, 2021. Review
- Hugging Face, “The Tokenization Pipeline,” Tokenizers documentation, current online edition. Documentation
- Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre and Daniel Zeman, “Universal Dependencies,” Computational Linguistics, 47(2), 2021. Paper
- Universal Dependencies contributors, “Universal POS tags,” Universal Dependencies documentation, current online edition. Documentation
- Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics, 19(2), 1993. Paper
- Thomas Emerson, “The Second International Chinese Word Segmentation Bakeoff,” Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, 2005. Paper
- Jacob Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, 20(1), pp. 37–46, 1960. DOI
- Steven T. Piantadosi, “Zipf’s word frequency law in natural language: A critical review and future directions,” Psychonomic Bulletin & Review, 21(5), pp. 1112–1130, 2014. DOI
- Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019. Paper