Full search page →

Build subwords one merge at a time

BeginnerNLP 01 · Introduction to Natural Language ProcessingInteractive plot

Description

Train byte-pair encoding on a tiny weighted vocabulary and apply the learned merge order to a new word.

Try this: Compare “lower”, “slowest” and “glow”. Which pieces transfer to unseen words, and which merges only compress the training vocabulary?

Lab · Build subwords one merge at a time

Learned merges
4
Training pieces, weighted
31
Input pieces
5

Merge counts come only from the fixed training vocabulary. Encoding applies the learned rules in order; it does not refit them to your input.

Try this: Compare “lower”, “slowest” and “glow”. Which pieces transfer to unseen words, and which merges only compress the training vocabulary?

Character-level BPE with an explicit end marker ▁; lexicographic tie breaking. This is not a byte tokenizer or a pretrained vocabulary. New characters remain individual pieces; spaces inside the input are treated as characters.

Method sources

Open in the Playground →