Full search page →

NLP 01 · Introduction to Natural Language Processing

Natural Language ProcessingBeginner22 minUpdated Nov 6, 2026

Needs: Start Here · Learning Paths and Recommended Textbooks

Navigate: this page and this track: Navigate: this page and this track
Explore this chapter 2 interactive experiments

Build an n-gram distribution from a tiny editable corpus. Inspect counts, sentence boundaries and smoothing.

What you’ll learn

  • Explain why language is hard for computers, with an example of ambiguity at each linguistic level.
  • Trace the main eras of NLP — rules, statistical learning, neural networks and pretrained models — through the papers that marked them.
  • Classify common NLP tasks by the shape of their output and find the tutorial that covers each.
  • Recognize the issues specific to Chinese text: word segmentation, and the Traditional and Simplified character sets.
  • Define an NLP task through its input, output and required linguistic evidence, and preserve that evidence through normalization and tokenization.
  • Build data splits and diagnose errors at the stage where they originate.

The big picture

Natural language processing (NLP) is the study of computer systems that analyze, interpret or produce human language. An NLP pipeline converts a particular language problem into a computation. Consider “Pump B: not ready.” A maintenance application might need the equipment name and its stated readiness. A message-routing application might need only a category. A conversational assistant might need to ask what remains unfinished. These are different tasks even though the input string is identical.

The first job is to decide which answer the application needs and what evidence supports it. Then preserve that evidence while turning text into model input. A powerful model cannot recover a distinction that preprocessing has already erased, and a plausible label is useless if it describes the wrong target.

Hagiwara’s application-first treatment and McRoy’s treatment of language and its data structures provide the two anchors for this tutorial [1, 2]. We first look at why language is difficult, how the field arrived at today’s methods, which tasks it solves and what changes for Chinese text. Then we follow one practical route: define the task, preserve text, choose computational units, construct data and evaluate the result. NLP 02 develops the language structures behind those decisions, and the roadmap at the end links the whole track.

1. Why language is hard for computers

Plain version. The same string of characters can mean different things, and what it means depends on structure, context and the speaker’s purpose. A computer sees only the characters unless we give it the rest.

Ambiguity appears at every level of linguistic analysis. Each row below is developed with examples and code in NLP 02:

LevelExampleTwo readings
Word boundaries學生會說話“students can speak” / “the student council speaks”
Word category“the pilots can land” / “the metal can fell”“can” as an auxiliary / as a noun
Word sense“window”an opening in a wall / an area on a screen
Attachment“The technician inspected the panel with the camera.”the camera is the tool / the camera is on the panel
Scope“Every technician uses a camera.”possibly different cameras / one shared camera
Reference“Mei put a camera beside a lamp. It was overheating.”“it” is the camera / the lamp
Intention“Can you lower the gain?”a question about ability / a request to act

People resolve most of these without noticing, using the surrounding sentence, the situation and knowledge of the world. A program must either be given that context or learn to approximate it from data. Three further properties compound the difficulty.

  • Productivity. Speakers build new words and sentences from familiar parts all the time, so no finite list of inputs covers the language.
  • Skewed frequency. A few words are extremely common and most words are rare. NLP 02 measures this with Zipf’s law: in one corpus, a third of all word types occur exactly once. Rare words are where models have the least evidence.
  • Variation. Dialects, registers, spelling conventions, mixed-language messages and writing systems all differ. A system trained on one variety can fail quietly on another.

None of this means NLP is impossible. It means every system makes choices about which distinctions it preserves, and good engineering makes those choices explicit.

2. A short history: rules, statistics, neural networks and pretraining

Plain version. Early systems followed rules that people wrote by hand. Later systems learned probabilities from large collections of text. Neural networks then learned their own features, and today most systems start from a large model pretrained on unlabeled text and adapt it to the task.

Timeline of NLP milestones from ELIZA in 1966 through statistical translation and the Penn Treebank in 1993, neural language models and word2vec, to the Transformer, BERT, GPT-3 and InstructGPT
Figure 1 — Milestones cited in this tutorial, grouped into four approximate eras. Events are equally spaced, not to scale; in practice the approaches overlap.

Rules and hand-built knowledge. Weizenbaum’s ELIZA (1966) responded to typed English by matching keywords and applying decomposition and reassembly rules from a script; its best-known script imitated a psychotherapist’s conversational style [6]. Systems of this era encoded grammars, lexicons and patterns by hand. They could be precise inside their design, but every new phrasing or domain required new rules.

Statistical learning from corpora. Over the following decades, researchers increasingly estimated models from data instead of writing every rule. Brown and colleagues (1993) described a series of statistical translation models whose parameters are estimated from pairs of sentences that are translations of each other [7]. In the same year the Penn Treebank released a large English corpus annotated with part-of-speech tags and syntactic structure [8], giving taggers and parsers shared training data and a shared test. The key change was that the evidence came from counts in corpora, and evaluation on held-out data became standard practice.

Neural networks. Bengio and colleagues (2003) trained a neural language model that learned a distributed representation for each word together with the probability of word sequences [9]. Collobert and colleagues (2011) used one neural architecture for part-of-speech tagging, chunking, named-entity recognition and semantic role labeling, learning internal representations largely from unlabeled text instead of hand-engineered features [10]. Mikolov and colleagues (2013) made word vectors cheap to train on large corpora [11]. Models still tended to be built for one task at a time.

Pretrained models and large language models. The Transformer (2017) replaced recurrence with attention [12]. ELMo (2018) supplied context-dependent word representations from a pretrained bidirectional language model [13], and BERT (2019) pretrained a deep bidirectional Transformer on unlabeled text and then fine-tuned it for each task with a small output layer [14]. GPT-3 (2020) showed that a large autoregressive model could perform many tasks from a few examples in its prompt, without gradient updates [15], and InstructGPT (2022) fine-tuned such a model with human feedback to follow instructions [16]. Zhao and colleagues’ survey organizes the large-language-model literature around pretraining, adaptation, utilization and capacity evaluation [17].

The eras overlap: rule-based components, counting methods and neural models all remain in use. What carries over is the pipeline in Sections 4–9. Whatever the model, someone still has to define the task, preserve the evidence, choose units, build data and evaluate.

3. The main tasks and applications

Plain version. Most NLP tasks can be grouped by what they output: a label for the whole text, a label for each word, a structure, a passage from a collection, or new text.

Output shapeExample tasksApplicationsCovered in
One label per text or text pairtopic classification, sentiment, spam detection, natural language inferencerouting, moderation, review analysisNLP 03, NLP 11
One label per token or spanpart-of-speech tagging, named-entity recognition, information extractionturning reports into recordsNLP 06
A linguistic structureparsing, semantic roles, coreferenceanalysis of who did what to whomNLP 12
Evidence from a collectionsearch, extractive question answeringdocument search, help desksNLP 11, NLP 09
New text conditioned on inputtranslation, summarization, dialoguetranslators, assistants, chatbotsNLP 08, NLP 09
A reusable representationword and sentence embeddings, pretrained encoderssimilarity, clustering, transfer to new tasksNLP 04, NLP 10

The same application often combines several rows. An assistant that answers maintenance questions may classify the question, retrieve passages, extract an equipment name and generate a reply. Each step has its own output and its own errors, which is why the rest of this tutorial treats the task definition as the first design decision.

4. Specify the task before choosing a model

Plain version. “Understand this message” is too broad to train or evaluate. Specify which part of its meaning must become an output.

For the invented message “Pump B: not ready,” three possible outputs are:

TaskOutputEvidence that matters
Topic classificationmaintenanceThe message concerns equipment status
Entity extractionPump B, with its source spanThe exact equipment-name characters
Status extractionEquipment Pump B; readiness falseThe name and the negated readiness statement

The first answer does not establish the third. A classifier can recognize a topic from “pump” while ignoring “not.” A status extractor must preserve that negation. A system that returns equipment names also needs boundary rules: should B be included, and should the colon be excluded? Write these decisions into the annotation instructions.

Now compare “Pump B was not ready yesterday” with “Pump B is ready now.” The equipment is unchanged, but time and polarity differ. A current-status application needs a rule for multiple dated statements; a historical extractor may keep both. The correct answer depends on the task definition and available context, rather than on one universal interpretation of the string.

For “It still isn’t ready,” the equipment is absent from the message. If the application supplies earlier conversation, reference resolution can use it. If it supplies only this isolated sentence, identifying Pump B would add information that was never available. This is an input-design issue, developed further in NLP 12 (coreference) and NLP 08 (dialogue state).

Separate a prediction from an action. Predicting a request to reschedule does not determine which booking to change. Start with an output that can be checked against the text: an intent label, an equipment span or a stated event. Later tutorials add the contextual reasoning needed to resolve missing details.

5. Preserve the original text and the evidence

Plain version. The text used for model input and the text shown as evidence can be different representations of the same message. Keep a way to connect them.

A pipeline can be written as:

r→gx→fθy^.r\xrightarrow{g}x\xrightarrow{f_\theta}\widehat y.

Here rr is the raw input record; gg is the preprocessing and numerical encoding function; xx is model input; fθf_\theta is a predictor with parameters θ\theta; and y^\widehat y is its output. A record can include text and permitted context. Preprocessing includes decisions about normalization, token boundaries and vocabulary IDs.

If two different records become the same xx, a deterministic predictor must return the same result for both:

g(r1)=g(r2)⟹fθ(g(r1))=fθ(g(r2)).g(r_1)=g(r_2)\quad\Longrightarrow\quad f_\theta(g(r_1))=f_\theta(g(r_2)).

Here r1,r2r_1,r_2 are two raw records. This simple identity explains why preprocessing is part of the model. If removing “not” makes affirmative and negative statements identical, changing the neural architecture cannot separate them afterward. The same reasoning applies when lowercasing merges US with us or punctuation removal collapses a decimal number into another string.

Some equivalences are useful. Unicode NFC normalization maps canonically equivalent character sequences to a common form [3]. This is different from choosing to remove accents, punctuation or case. The first concerns a specified character equivalence; the others may remove distinctions that the task needs. Decide what should remain equivalent for this application, rather than treating every transformation as harmless cleaning.

Raw text passes through normalization, tokenization, numerical encoding, prediction and task output
Figure 2 — An illustrative text-processing pipeline. Preserve a connection to the original text, and use the same fitted encoding during training and evaluation.

A worked normalization and span example

The word Café can use a precomposed accented letter or a base e followed by a combining accent. They look similar but occupy different numbers of code points [3]. If an extractor reports character positions, normalization can therefore shift a later span.

A span [a,b)[a,b) means the characters starting at offset aa and ending just before bb in a specified string. Both offsets are nonnegative integers, with a≤ba\leq b. They are not meaningful without knowing which text representation and indexing unit they refer to.

Code · Python · 20 lines
import re
import unicodedata

raw = 'Café Pump B: not ready.'
normalized = unicodedata.normalize('NFC', raw)
needle = 'Pump B'
raw_start = raw.index(needle)
normal_start = normalized.index(needle)
raw_span = (raw_start, raw_start + len(needle))
normal_span = (normal_start, normal_start + len(needle))
tokens = [(m.group(), m.start(), m.end())
          for m in re.finditer(r'\w+|[^\w\s]', normalized)]
assert normalized != raw
assert unicodedata.normalize('NFC', normalized) == normalized
assert raw[raw_span[0]:raw_span[1]] == needle
assert raw[normal_span[0]:normal_span[1]] != needle
assert all(normalized[start:end] == token for token, start, end in tokens)
print('code points:', len(raw), '->', len(normalized))
print('raw / normalized evidence spans:', raw_span, normal_span)
print('normalized tokens:', [token for token, _, _ in tokens])

The code prints 24 -> 23 code points. The raw equipment span is (6, 12) and the normalized span is (5, 11). Both select Pump B in their own strings; applying the second span to the first string selects the wrong characters. The final assertion checks the normalized tokens against their own source offsets.

For extraction, either maintain an alignment from transformed positions back to the original text or retain annotations directly on the raw string. This example demonstrates the failure; it does not implement a general alignment algorithm. NLP 06 explains how word, token and character spans become structured outputs.

6. Decide which units the computation will use

Plain version. A word, a character and a model token answer different questions. Choose computational units without confusing them with all the units of language.

In “a cat and a dog,” whitespace splitting produces five token occurrences and four distinct types: a, cat, and, dog. A vocabulary assigns an ID to each retained type. IDs are names for table entries: ID 7 does not represent a larger meaning than ID 3.

The string “Pump B: not ready.” raises different choices. Whitespace splitting attaches punctuation to words. A punctuation-aware splitter may produce Pump, B, :, not, ready, .. A subword tokenizer can split a word further. All can be valid computational encodings, but their token positions are different. Word labels must be aligned to whichever encoding the model actually uses.

Subword tokenizers learn from data which pieces to keep. NLP 02 trains a small byte-pair encoding (BPE) tokenizer step by step and compares it with WordPiece, the unigram model and SentencePiece. Its central lesson is that learned pieces reflect frequency, not morphology, and that a tokenizer is a complete fitted procedure, not just a vocabulary list.

Vocabulary coverage and unknown inputs

A word vocabulary contains only its retained entries. Map an unseen word to an unknown marker, and all such words share that ID. This can be acceptable when the task depends on familiar surrounding words, but it removes the difference between an unseen equipment name and an unseen account name.

Define an illustrative unknown-token rate:

u=∑i=1N1[ti∉V]N.u=\frac{\sum_{i=1}^{N}\mathbf1[t_i\notin V]}{N}.

Here uu is the unknown-token rate; tit_i is token occurrence ii under a declared tokenizer; VV is its fitted vocabulary; N>0N\gt0 is the number of evaluated occurrences; and 1[⋅]\mathbf1[\cdot] equals one when the condition is true. A repeated unknown word counts repeatedly. This measures missing vocabulary entries, not lost meaning or task accuracy.

7. Chinese text: segmentation and two character sets

Plain version. Chinese is written without spaces between words, so deciding where words begin and end is itself an analysis. Chinese is also written in two standard character sets, Traditional and Simplified, and a model fitted on one sees many unfamiliar characters in the other.

Word segmentation

Chinese text often lacks spaces between words. “學生會說話” permits 學生 / 會 / 說話 (students can speak) or 學生會 / 說話 (the student council speaks); context helps choose the intended reading. A model tokenizer may simply use character or subword pieces. Its segmentation need not be a complete morphological or syntactic analysis. NLP 02 enumerates both readings with a small program and distinguishes those linguistic structures.

Segmentation is also a matter of convention. The Second International Chinese Word Segmentation Bakeoff used four corpora, two in Traditional characters (Academia Sinica in Taiwan and City University of Hong Kong) and two in Simplified characters (Peking University and Microsoft Research), each segmented under its provider’s own standard [18]. When a dataset has word-level labels, the standard it follows is part of the task definition, and a character-based model must align its outputs to those words before evaluation.

Traditional and Simplified characters

Text from Taiwan and Hong Kong is mainly written in Traditional characters and text from mainland China in Simplified characters; the bakeoff corpora above follow that division [18]. Many characters are the same in both, but many are different Unicode code points: Traditional 機 is U+6A5F while its Simplified counterpart is U+673A. Unicode normalization does not merge them, because they are distinct characters, not alternative encodings of one character.

Conversion is not a one-to-one character swap. The Unihan database records Simplified and Traditional variants for each character and notes that context can decide the mapping: when converting Simplified to Traditional, U+810F becomes 臟 when it means “viscera” and 髒 when it means “dirty” [19]. Word choice can also differ between regions, so even a perfect character converter does not turn one variety’s text into the other’s.

The script mismatch shows up directly as vocabulary coverage. The program below fits a character vocabulary on one Traditional sentence and measures the unknown-token rate uu on a new sentence written both ways. The Simplified version is written with escape codes so it is visible which characters differ.

Code · Python · 7 lines
vocab = set("這台相機已經校正好了")              # character vocabulary fitted on Traditional text
tests = {"Traditional": "那台相機已經過熱了",
         # The same sentence in Simplified characters, written as escapes:
         "Simplified": "那台相\u673a已\u7ecf\u8fc7\u70ed了"}
for name, text in tests.items():
    unknown = [f"U+{ord(ch):04X}" for ch in text if ch not in vocab]
    print(f"{name:11s} u = {len(unknown)}/{len(text)}  unknown: {unknown}")

The Traditional sentence has u=3/9u=3/9: 那, 過 and 熱 were not in the tiny training text. The Simplified sentence has u=5/9u=5/9: 那 (identical in both scripts) and the Simplified forms of 過 and 熱 are unknown as before, plus the Simplified counterparts of 機 and 經, which the vocabulary contains only in Traditional form. Real vocabularies are far larger, but the effect is the same in kind: a model trained mainly on one script has less evidence for the other. Decide whether to train on both, convert with a context-aware tool, or evaluate each script separately, and report which you chose.

8. Build examples and labels that answer the task

Plain version. A corpus is a collection of observations. Training examples are constructed from those observations according to a task. The construction determines what the model learns.

For classification, an example might be a whole message paired with one category. For extraction, it might be a message paired with character spans and their labels. For language modeling, the text supplies prediction targets without category annotations. These distinctions explain why the same corpus can support different objectives [1, 2].

Suppose a maintenance message contains “the camera is fine, but the delivery was late.” A product-sentiment label, a delivery-sentiment label and a whole-message label need not agree. Include the target in the instructions. If annotators choose different targets, a disagreement is evidence about the task definition before it is evidence about model quality.

Define the unit of independence for your split. Two paraphrases of one complaint are distinct strings but may describe the same event. If one appears in training and one in testing, the test can reward recognition of that event rather than transfer to a new one. Split related messages, conversations or document revisions together when their shared information would otherwise leak.

There are two kinds of fitted state to track. The predictor learns weights from training labels. A tokenizer or vocabulary may learn from unlabeled training text. Keeping labels out of testing is not enough if the feature pipeline still uses test text to decide its vocabulary. Freeze both according to the experiment’s stated rules.

Use a small baseline to diagnose the representation

Begin with a representation whose behavior is easy to inspect. A count-based message representation preserves which words appear while ignoring their order. “The dog bit the man” and “The man bit the dog” then produce the same counts despite reversing who did what. Every deterministic classifier using only those counts must give the same prediction for both.

This is a precise representation limitation. More examples cannot solve this pair while the feature map remains identical. Local phrase features can separate some order differences; contextual models can use richer input interactions. NLP 03 develops and tests those choices. Choose added structure in response to an observed failure, so the experiment identifies what changed.

9. Evaluate the output and diagnose the cause

Plain version. Evaluation should tell you what went wrong and where to improve it. One aggregate score rarely identifies the cause.

For classification, inspect errors for each category, especially rare ones; accuracy can conceal a category that is never predicted. Precision asks how often returned items are correct, while recall asks how many required items were returned. Both depend on what counts as an item [5]. For extraction, finding the right word with the wrong span is a boundary error. For a status record, the correct equipment paired with the wrong polarity is a meaning error.

Keep a few challenge pairs where one linguistic difference matters: “ready” versus “not ready,” Pump A versus Pump B, and “yesterday” versus “now.” Hold the intended task fixed while changing that difference. A correct answer on an ordinary message does not explain whether the system uses the distinction needed by these cases.

ObservationInspect firstWhat an improvement must establish
Unseen equipment names collapse togetherVocabulary and fallbackNames remain distinguishable in the input
Correct words, incorrect evidence highlightNormalization and offset alignmentReturned offsets refer to the displayed source
Negation reverses the true label but not the predictionFeature map and supervisionThe relevant distinction survives encoding and affects the decision
Strong validation result, weak new-customer resultSplit independence and domain differencesPerformance transfers to the intended population
Good results on one Chinese script, poor on the otherCharacter coverage and conversionBoth scripts are represented, or the gap is reported

Change one stage at a time when possible. If a preprocessing change and a model change happen together, a better score does not tell you which caused the improvement. Inspect transformed inputs before interpreting weights or outputs. This turns error analysis into a testable explanation instead of a list of model names.

10. Roadmap of the NLP track

The track follows the eras in Section 2: language structure first, then neural building blocks, sequence models, attention and Transformers, and finally pretrained models and structured tasks.

TutorialWhat it adds
NLP 01 · Introduction to Natural Language ProcessingWhy language is hard, history, tasks, Chinese issues and the task-to-evaluation pipeline
NLP 02 · Linguistic EssentialsMorphology, tokens and subwords, parts of speech, syntax, meaning, context, corpora and Zipf’s law
NLP 03 · Neural Networks for NLPCount features, TF–IDF, logistic and neural text classifiers
NLP 04 · Lexical Semantics and Word EmbeddingsDistributional meaning, count-based vectors, word2vec
NLP 05 · Recurrent Neural Networks for Languagen-gram language models, perplexity, recurrent language models
NLP 06 · Sequence Labeling with RNNsPart-of-speech tagging, entity recognition and span evaluation
NLP 07 · Attention Mechanism and TransformersAttention, contextual representations and position
NLP 08 · Sequence-to-Sequence Models with TransformersEncoder–decoder models, decoding, machine translation and chatbots
NLP 09 · Language Modeling with TransformersCausal language models, text generation and retrieval-augmented generation
NLP 10 · Pretrained Models: BERTMasked-language pretraining and the BERT encoder
NLP 11 · Classification with BERTSentiment analysis, natural language inference and extractive QA heads
NLP 12 · Syntactic and Semantic TasksParsing, semantic roles, coreference and discourse

Related tutorials on this site cover the general machinery: ML 02 for neural networks, ML 04 for recurrent networks, ML 05 for attention and the Transformer block, and IT 01 for entropy and cross-entropy.

A first route through the papers. Start with the main chapters; these twelve guides are optional checkpoints, not extra prerequisites. Each guide opens with what to know first and ends with a next step. The route groups questions rather than publication dates: within attention, read Seq2Seq → Bahdanau → Transformer to follow the architectural motivation.

Paper route 1 · Units and word vectors

Know first: Tokenization, vector dot products, cross-entropy

Read: BPE → word2vec → GloVe

Check your understanding: What is a token, and what evidence gives its vector meaning?

Paper route 2 · Local and sequential decisions

Know first: Convolution, recurrent states, conditional probability

Read: TextCNN → Seq2Seq → BiLSTM-CRF

Check your understanding: How do local features become a class, a sequence or a constrained label path?

Paper route 3 · Attention and pretraining

Know first: Encoder–decoder flow, softmax, masks

Read: Bahdanau attention → Transformer → BERT

Check your understanding: Which context can the model read, and what is it trained to predict?

Paper route 4 · Task objectives and structure

Know first: Fine-tuning, sentence pairs, span evaluation

Read: Sentence-BERT → InstructGPT → Coreference

Check your understanding: How do similarity, human preference and entity links require different evidence?

For each paper, explain one arrow in its original figure, define the symbols in its objective, and identify the split and model setting behind one reported result. If any step is unclear, return to the linked parent chapter before continuing.

Modern view

Chang and colleagues’ evaluation survey organizes what, where and how to evaluate language models [4]. For a pipeline, that motivates separate checks of encoding, task output and application behavior. A fluent generated explanation is not an alternative measurement of equipment-name recall; the two outputs answer different questions.

Zhao and colleagues’ survey of large language models describes the current default: pretrain one large model on broad text, then adapt it (for example by instruction tuning or alignment with human preferences), use it through prompting, and evaluate its capacities across many tasks [17]. Its organization mirrors the history in Section 2, where each era kept the task definitions of the previous one and changed how the mapping from text to output was obtained.

For current systems, the transferable lesson is to inspect interfaces before comparing model families. A larger input budget is useful only if the relevant text survives truncation. A multilingual tokenizer is useful only if its representations support the distinctions the task needs, including both Chinese character sets when users write in both. A pretrained representation can reduce labeled-data requirements, but the application’s targets still require clear supervision and evaluation. Later tutorials analyze the respective mechanisms rather than treating these as automatic benefits.

Open questions include how to measure linguistic coverage beyond character coverage, how segmentation affects different languages and registers, and how to separate representation failures from insufficient labels or missing context. A useful small study compares two encodings with one fixed task, independently split data and a few diagnostic linguistic cases. Report what each encoding preserves or loses, then explain the observed errors in those terms.

Key takeaways

  • Language is ambiguous at every level, from word boundaries to intention; context and world knowledge resolve what the string alone cannot.
  • NLP moved from hand-written rules to corpus statistics, then to neural networks and pretrained models; the task-definition and evaluation problems carried over unchanged.
  • Group tasks by output shape: text labels, token labels, structures, retrieved evidence, generated text or representations.
  • Chinese adds word segmentation under competing standards and two character sets whose conversion is not one-to-one.
  • Define the desired output and its linguistic evidence before choosing a model, and keep raw text distinct from transformed model input.
  • A predictor cannot recover distinctions erased by its preprocessing function; tokens and vocabulary IDs are computational units, not complete analyses of language.
  • Fit learned representations on permitted training text, keep related observations together in splits, and diagnose vocabulary, boundary, meaning and transfer errors before changing architecture.

Exercises

  1. For “Pump B was not ready yesterday, but it is ready now,” define labels for topic classification, historical status extraction and current-status extraction. Which labels require a time policy?
Hint

The equipment name can stay fixed while status values and their validity times differ. Decide whether you return one state or two dated events.

  1. Replace the decomposed accent in the normalization example with a precomposed one. Explain which offsets change and which assertions remain valid.
Hint

Compare raw and normalized strings separately. An offset is a location in one representation, not in all equivalent-looking strings.

  1. Make a preprocessing rule remove “not.” Construct two messages with different required status labels but identical transformed inputs, and prove why a deterministic predictor cannot separate them.
Hint

Apply the equality involving g(r1)g(r_1) and g(r2)g(r_2). The contradiction arises before training.

  1. In the script-coverage program, refit the vocabulary on the union of the Traditional training sentence and its Simplified version. Recompute uu for both test sentences. Then explain why converting every Simplified test character to Traditional before lookup is not always possible with a character table alone.
Hint

The union removes the script-specific unknowns but not 那, 過 and 熱. For conversion, use the Unihan example: one Simplified code point maps to two Traditional characters depending on meaning.

  1. Group three rewritten messages from one incident into a train/test dataset. Design a split and two diagnostics that distinguish event memorization from transfer to new incidents.
Hint

Split incident groups together, then examine unseen names and unseen wording separately. Keep the label policy fixed.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapters 1–3 and 10–11, Manning, 2021. Publisher
  2. Susan McRoy, Principles of Natural Language Processing, open textbook, 2021. Book
  3. Unicode Consortium, “Unicode Normalization Forms,” Unicode Standard Annex #15, revision 58, 2026. Specification
  4. Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Survey
  5. Scikit-learn developers, “Metrics and Scoring: Quantifying the Quality of Predictions,” documentation, current online edition. Documentation
  6. Joseph Weizenbaum, “ELIZA—A Computer Program for the Study of Natural Language Communication Between Man and Machine,” Communications of the ACM, 9(1), pp. 36–45, 1966. DOI
  7. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra and Robert L. Mercer, “The Mathematics of Statistical Machine Translation: Parameter Estimation,” Computational Linguistics, 19(2), 1993. Paper
  8. Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics, 19(2), 1993. Paper
  9. Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin, “A Neural Probabilistic Language Model,” Journal of Machine Learning Research, 3, pp. 1137–1155, 2003. Paper
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu and Pavel Kuksa, “Natural Language Processing (Almost) from Scratch,” Journal of Machine Learning Research, 12, pp. 2493–2537, 2011. Paper
  11. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space,” arXiv:1301.3781, 2013. Paper
  12. Ashish Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  13. Matthew E. Peters et al., “Deep Contextualized Word Representations,” NAACL-HLT, 2018. Paper
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019. Paper
  15. Tom B. Brown et al., “Language Models are Few-Shot Learners,” NeurIPS, 2020. Paper
  16. Long Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback,” NeurIPS, 2022. Paper
  17. Wayne Xin Zhao et al., “A Survey of Large Language Models,” arXiv:2303.18223, 2023 (continuously updated). Survey
  18. Thomas Emerson, “The Second International Chinese Word Segmentation Bakeoff,” Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, 2005. Paper
  19. Unicode Consortium, “Unicode Han Database (Unihan),” Unicode Standard Annex #38, revision 41, 2026. Specification

Check your understanding

This tutorial has 5 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next NLP 02 · Linguistic Essentials · 37 min