Full search page →

Count the next word

BeginnerNLP 01 · Introduction to Natural Language ProcessingInteractive plot

Description

Build an n-gram distribution from a tiny editable corpus. Inspect counts, sentence boundaries and smoothing.

Try this: Enter an unseen context and set α to zero. Why is the estimate undefined? Increase α and inspect where the probability goes.

Lab · Count the next word

Context count
3
Vocabulary size
8
Probability sum (all tokens)
1

Context: robots. P(w | h) = (count(h,w) + α) / (count(h) + α|V|).

Try this: Enter an unseen context and set α to zero. Why is the estimate undefined? Increase α and inspect where the probability goes.

Each nonempty line is one sentence. Tokens are lowercased and split on whitespace; punctuation stays attached. The first 8,000 characters are used. Vocabulary comes only from the corpus, plus [EOS] and [UNK]. [BOS] pads the start but is never predicted. The last n − 1 query tokens form the context; an empty query means sentence start. This is a counting demonstration, not a Chinese word segmenter.

Method sources

Open in the Playground →