Build subwords one merge at a time
BeginnerNLP 01 · Introduction to Natural Language ProcessingInteractive plot
Description
Train byte-pair encoding on a tiny weighted vocabulary and apply the learned merge order to a new word.
Try this: Compare “lower”, “slowest” and “glow”. Which pieces transfer to unseen words, and which merges only compress the training vocabulary?
Hints
The chapter section that uses this lab:
Lab · Build subwords one merge at a time
- Learned merges
- 4
- Training pieces, weighted
- 31
- Input pieces
- 5
Merge counts come only from the fixed training vocabulary. Encoding applies the learned rules in order; it does not refit them to your input.
Try this: Compare “lower”, “slowest” and “glow”. Which pieces transfer to unseen words, and which merges only compress the training vocabulary?
Character-level BPE with an explicit end marker ▁; lexicographic tie breaking. This is not a byte tokenizer or a pretrained vocabulary. New characters remain individual pieces; spaces inside the input are treated as characters.