Count the next word
BeginnerNLP 01 · Introduction to Natural Language ProcessingInteractive plot
Description
Build an n-gram distribution from a tiny editable corpus. Inspect counts, sentence boundaries and smoothing.
Try this: Enter an unseen context and set α to zero. Why is the estimate undefined? Increase α and inspect where the probability goes.
Hints
The chapter section that uses this lab:
Lab · Count the next word
- Context count
- 3
- Vocabulary size
- 8
- Probability sum (all tokens)
- 1
Context: robots. P(w | h) = (count(h,w) + α) / (count(h) + α|V|).
Try this: Enter an unseen context and set α to zero. Why is the estimate undefined? Increase α and inspect where the probability goes.
Each nonempty line is one sentence. Tokens are lowercased and split on whitespace; punctuation stays attached. The first 8,000 characters are used. Vocabulary comes only from the corpus, plus [EOS] and [UNK]. [BOS] pads the start but is never predicted. The last n − 1 query tokens form the context; an empty query means sentence start. This is a counting demonstration, not a Chinese word segmenter.