Full search page →

NLP 04 · Lexical Semantics and Word Embeddings

Natural Language ProcessingBeginner36 minUpdated Nov 6, 2026

Needs: NLP 03 · Neural Networks for NLP

Navigate: this page and this track: Navigate: this page and this track
Explore this chapter 2 interactive experiments

Pad a three-word sentence with zero vectors and compare a correct masked mean with a mean that counts the padding in its denominator.

What you’ll learn

  • Distinguish word forms, lemmas and senses, and name the main lexical relations: synonymy, antonymy, hypernymy and meronymy, as organized in WordNet.
  • Explain the distributional hypothesis, what shared contexts can reveal about words and what they cannot establish.
  • Construct a word–context matrix, turn positive PMI into compact vectors, and derive CBOW, Skip-gram, negative sampling, GloVe and fastText objectives.
  • Evaluate embeddings with similarity and analogy tests while recognizing their caveats, and measure an association bias.
  • Pool word vectors into sentence representations, train sentence embeddings contrastively, and explain what changes with contextual embeddings.

The big picture

A word ID identifies a vocabulary entry. Its numerical value says nothing about what the word does in language. An embedding assigns that entry a learned vector, allowing words to share useful statistical patterns. The central question is not simply how to make a vector smaller. It is which evidence and objective should determine its coordinates.

Linguists describe word meaning with lexical semantics: senses, and relations such as synonymy and hypernymy, recorded by hand in resources like WordNet. Distributional methods learn vectors from text instead. This lesson connects the two views. Hagiwara develops word and document embeddings as reusable features for practical applications [1]. McRoy distinguishes lexical similarity from language tasks such as sentiment and entailment [2]. Here we develop the representation itself, and we keep asking which lexical relation a vector geometry actually captures.

NLP 03 learned word vectors as a by-product of a task: an embedding layer inside a classifier, and the table CC inside the neural probabilistic language model of Bengio et al. [3]. Word2Vec makes the vectors themselves the goal and uses nearby words, not a following word, as the prediction target. These are different prediction contracts:

QuestionNeural probabilistic language modelStatic Word2Vec embedding
What is observed?An ordered history of n−1n-1 tokens and the token that followsA center word and selected neighbors on both sides
What is predicted?A normalized distribution over the next tokenA center from neighbors, or individual neighbors from a center
Is left-to-right generation the purpose?Yes; it is a language modelBasic window training is not an ordered generator
What is reused afterward?The whole network; word vectors are a by-productA learned vector table

Continuous in CBOW means that its internal representation has real-valued coordinates. It does not mean that the text is continuous, that words occur without boundaries, or that every word sense receives its own vector. By the end, we will connect the count-based and prediction-based constructions and identify exactly which information each retains. Count-based language models and their evaluation come later, in NLP 05.

Lexical semantics: senses and relations

Plain version. A written word can carry several meanings, and meanings relate to each other in structured ways: some are near-equivalents, some are opposites, some are kinds of others. Embeddings will later blur several of these relations together, so name them first.

Word forms, lemmas and senses

A word form is what appears in text (banks, banked); a lemma is the dictionary entry that groups inflected forms (bank). A sense is one meaning of a lemma. bank has at least two unrelated senses, the slope beside a river and the financial institution; this is homonymy. window has related senses, an opening in a wall and a region of a screen; this is polysemy. NLP 02 discusses how context and composition select a sense. For representation, the key point is that a vocabulary entry is a form or lemma, not a sense: a table with one row per entry must merge every sense of that entry.

Relations between senses

Lexical relations hold between senses, not between spelling strings:

  • Synonymy: two senses that can substitute for each other in many contexts with little change in meaning, such as slope and incline. Exact synonymy is rare; register, collocation and connotation usually differ.
  • Antonymy: senses that are opposite along one scale or dimension, such as hot and cold. Antonyms are otherwise very similar: they describe the same property and occur in the same kinds of sentences.
  • Hyponymy and hypernymy: an is-a relation. poodle is a hyponym of dog, and dog is a hypernym of poodle. The relation is asymmetric and usually transitive, forming a hierarchy.
  • Meronymy: a part–whole relation, such as wheel and car.

Two further notions matter for evaluation. Similarity asks whether two concepts share properties (cup and mug); relatedness or association asks whether they tend to occur together (cup and coffee). A related pair need not be similar at all.

WordNet: a hand-built lexical database

WordNet organizes English nouns, verbs, adjectives and adverbs into synsets, sets of synonymous lemmas that express one concept, each with a short gloss, connected by relations such as hypernymy, meronymy and antonymy [4]. Looking up a lemma returns its synsets, one per sense. Walking hypernym links from a synset reaches increasingly general concepts, which lets a program measure how far apart two senses are in the hierarchy.

The following example builds a miniature database in the same style, by hand, to show the mechanics without downloading WordNet itself. Its entries are invented for this lesson.

Code · Python · 31 lines
# A hand-built miniature lexical database in the style of WordNet (not WordNet data).
synsets = {
    "bank.n.slope":   {"lemmas": ["bank"], "gloss": "sloping land beside a river", "hypernym": "slope.n"},
    "bank.n.finance": {"lemmas": ["bank"], "gloss": "an institution that holds deposits", "hypernym": "institution.n"},
    "slope.n":        {"lemmas": ["slope", "incline"], "gloss": "a surface at an angle", "hypernym": "landform.n"},
    "landform.n":     {"lemmas": ["landform"], "gloss": "a natural feature of land", "hypernym": "entity.n"},
    "institution.n":  {"lemmas": ["institution"], "gloss": "an organization with a public role", "hypernym": "organization.n"},
    "school.n":       {"lemmas": ["school"], "gloss": "an institution for teaching", "hypernym": "institution.n"},
    "organization.n": {"lemmas": ["organization"], "gloss": "a group with a purpose", "hypernym": "entity.n"},
    "entity.n":       {"lemmas": ["entity"], "gloss": "anything that exists", "hypernym": None},
}

def senses(word):
    return [s for s, v in synsets.items() if word in v["lemmas"]]

def path_to_root(s):
    path = [s]
    while synsets[path[-1]]["hypernym"]:
        path.append(synsets[path[-1]]["hypernym"])
    return path

def path_similarity(a, b):            # 1 / (1 + edges on the shortest is-a path)
    pa, pb = path_to_root(a), path_to_root(b)
    common = next(s for s in pa if s in pb)
    return 1 / (1 + pa.index(common) + pb.index(common))

print("senses of bank:", senses("bank"))
print("synonyms in slope.n:", synsets["slope.n"]["lemmas"])
print("hypernym chain:", " -> ".join(path_to_root("bank.n.finance")))
for other in ["school.n", "slope.n"]:
    print(f"path_similarity(bank.n.finance, {other}) = {path_similarity('bank.n.finance', other):.3f}")

The run prints:

Code · 5 lines
senses of bank: ['bank.n.slope', 'bank.n.finance']
synonyms in slope.n: ['slope', 'incline']
hypernym chain: bank.n.finance -> institution.n -> organization.n -> entity.n
path_similarity(bank.n.finance, school.n) = 0.333
path_similarity(bank.n.finance, slope.n) = 0.167

The financial sense of bank is closer to school (both are institutions) than to slope, even though the other sense of bank is a kind of slope. A sense inventory makes that distinction explicit; a single vector for the string bank cannot.

Hand-built resources are precise and interpretable, but they are expensive to build, cover some languages and domains far better than others, lag behind new words and senses, and encode one set of sense distinctions that may not match an application. Distributional vectors make the opposite trade: they are cheap and broad, but their relations are implicit and mixed. Much practical work combines the two, for example by checking embedding neighbors against a lexical resource.

Distributional evidence: contexts are observations

Plain version. Words that appear in similar surroundings can acquire similar vectors. The surroundings provide evidence about usage; they do not directly supply definitions or truth conditions.

Harris argued that differences in meaning correlate with differences in distribution: the environments in which linguistic elements occur [5]. This distributional hypothesis is the foundation of every method in the rest of this lesson.

A word type is a vocabulary entry, while a word token is one occurrence. A static embedding pools evidence across many tokens into one vector for the type. Occurrences of bank beside river and beside account therefore influence the same row. The model can learn that the word has mixed usage, but the lookup operation cannot choose a different row for each sense.

Define context before collecting statistics. A radius-two word window contains up to two tokens on each side of the center, excluding the center itself. A shorter window emphasizes immediate surroundings; a larger one includes more distant topic evidence. If direction matters, left and right context positions can be separate features. Our examples deliberately merge the two sides so the model’s invariance is visible.

The distributional assumption is useful because text provides repeated observations without requiring a semantic label for every word. It is incomplete because shared use includes synonymy, opposition, thematic association and grammatical compatibility. hot and cold can both appear before water; similarity on that evidence does not establish that they express the same property. In the terms of the previous section, distributional similarity mixes synonymy, antonymy, co-hyponymy and relatedness. Turney and Pantel organize these vector-space approaches by the evidence represented in their matrices [6].

Keep corpus, tokenizer and window specification together. Changing word segmentation changes the types whose contexts are counted. An embedding learned on technical incident reports may organize charge differently from one learned on electronics manuals. The coordinates are an outcome of those choices, not language-independent measurements of meaning.

Count-based embeddings: inspect the shared-context assumption

Plain version. Count how often each word appears beside each context, then compare that count with a baseline that treats the word and context as independent. An unusually strong association contributes a positive coordinate.

The distributional survey by Turney and Pantel [6] explains this count-based route. Let CwcC_{wc} count the observed ordered pairs consisting of center word ww and neighboring context word cc. The two roles remain distinct even when their vocabularies contain the same words. Define empirical probabilities from the table:

NC=∑w,cCwc,P(w,c)=CwcNC,P(w)=∑cP(w,c),P(c)=∑wP(w,c).N_C=\sum_{w,c}C_{wc},\qquad P(w,c)=\frac{C_{wc}}{N_C},\qquad P(w)=\sum_cP(w,c),\qquad P(c)=\sum_wP(w,c).

Here NCN_C is the total center–context pair count, CwcC_{wc} is one cell count, P(w,c)P(w,c) is its empirical joint probability, P(w)P(w) is the center marginal, and P(c)P(c) is the context marginal. The sums range over the corresponding vocabularies. A token may contribute several pairs, so NCN_C is not the number of tokens.

Pointwise mutual information (PMI) compares the empirical joint probability with the independence baseline. Positive PMI (PPMI) retains only associations stronger than that baseline:

PMI(w,c)=log⁡P(w,c)P(w)P(c),PPMI(w,c)=max⁡(PMI(w,c),0).\mathrm{PMI}(w,c)=\log\frac{P(w,c)}{P(w)P(c)},\qquad \mathrm{PPMI}(w,c)=\max\bigl(\mathrm{PMI}(w,c),0\bigr).

Here PMI(w,c)\mathrm{PMI}(w,c) is the log association score for one pair, PPMI(w,c)\mathrm{PPMI}(w,c) is its nonnegative version, and log⁡\log is the natural logarithm. The probabilities are those defined above. A pair with zero joint count and positive marginals has PMI tending to negative infinity and PPMI zero; an entirely unobserved row requires a separate policy.

A frequent context does not automatically receive a large association score. It must occur near this center more often than its overall frequency predicts. Conversely, a rare pair can have high PMI on little evidence. Inspect counts alongside scores; removing negative associations and keeping only positive ones also discards information rather than making every remaining association reliable.

The PPMI matrix still has one coordinate per context. A truncated singular value decomposition (SVD) compresses it:

M=AΣB⊤,MD=ADΣDBD⊤,E=ADΣD.M=A\Sigma B^\top,\qquad M_D=A_D\Sigma_DB_D^\top,\qquad E=A_D\Sigma_D.

Here MM is the PPMI matrix, AA and BB contain its left and right singular vectors, Σ\Sigma is the diagonal matrix of singular values, DD is the retained dimension, and the subscript DD selects the first DD singular components. MDM_D is the resulting low-rank reconstruction, and EE supplies the center-word vectors used below. This is the convention implemented in the code; other distributions of singular-value scaling produce different vector geometries.

SVD identifies shared patterns across rows rather than assigning a semantic name to each coordinate. The compression cannot preserve every cell when the original matrix has higher rank than DD. The following synthetic corpus lets us inspect both the construction and a failure of interpreting cosine similarity as synonymy.

Code · Python · 37 lines
import numpy as np

sentences = """the cat drinks milk . the dog drinks water . a cat chases a mouse .
a dog chases a cat . the kitten drinks milk . the puppy drinks water .
a kitten chases a ball . a puppy chases a ball . the car needs fuel .
the truck needs fuel . a car drives on the road . a truck drives on the road .""".split(".")
tokens = [s.split() for s in sentences if s.strip()]
vocab = sorted({w for s in tokens for w in s})
idx = {w: i for i, w in enumerate(vocab)}

# 1) Co-occurrence counts within a +-2 word window.
C = np.zeros((len(vocab), len(vocab)))
for s in tokens:
    for i, w in enumerate(s):
        for j in range(max(0, i - 2), min(len(s), i + 3)):
            if j != i:
                C[idx[w], idx[s[j]]] += 1

# 2) Positive pointwise mutual information.
total = C.sum()
pw, pc = C.sum(1, keepdims=True) / total, C.sum(0, keepdims=True) / total
with np.errstate(divide="ignore"):
    pmi = np.log((C / total) / (pw * pc))
ppmi = np.maximum(pmi, 0)

# 3) Truncated SVD gives dense 8-dimensional embeddings.
U, S, _ = np.linalg.svd(ppmi)
E = U[:, :8] * S[:8]
E /= np.linalg.norm(E, axis=1, keepdims=True)

def nearest(word, k=3):
    sims = E @ E[idx[word]]
    order = [vocab[i] for i in np.argsort(-sims) if vocab[i] != word]
    return [(w, round(float(sims[idx[w]]), 2)) for w in order[:k]]

for w in ["cat", "puppy", "truck", "milk"]:
    print(w, "->", nearest(w))

Output:

Code · 4 lines
cat -> [('kitten', 0.96), ('puppy', 0.88), ('dog', 0.88)]
puppy -> [('dog', 1.0), ('kitten', 0.98), ('cat', 0.88)]
truck -> [('car', 1.0), ('needs', 0.6), ('the', 0.47)]
milk -> [('water', 0.97), ('a', 0.62), ('chases', 0.51)]

Animals cluster with animals and vehicles with vehicles, learned from twelve sentences and no labels. Note “milk” and “water” are nearly identical: both are things that are drunk in this corpus. Embedding similarity measures shared contexts, which often but not always matches similar meaning; antonyms such as “hot” and “cold” also share contexts.

Left: a 2-D projection of the eight-dimensional vectors; cat, kitten, dog and puppy cluster on the left, car and truck on the right, milk and water at the bottom. Right: a cosine-similarity matrix with high values inside the animal block, the vehicle block and the milk–water pair.
Figure 1 — Vectors from the code above. Cosine similarity is high within groups that share contexts.

Word2Vec: learn a word by predicting its neighbors

Plain version. Words used in similar surroundings often serve similar roles. Word2Vec turns that observation into a small prediction problem: learn vectors that help recover a word from its neighbors, or its neighbors from the word. The vectors are learned parameters, not hand-written lists of meanings. The two basic architectures are continuous bag of words (CBOW) and Skip-gram [7].

Turn a sentence into training examples

Start with the invented sentence small red robots move slowly. A context window chooses nearby positions; radius one means the word immediately before and after the center. For this example, use only interior centers, so every window has two neighbors:

CenterContextCBOW training exampleSkip-gram training examples
redsmall, robots[small, robots] → redred → small; red → robots
robotsred, move[red, move] → robotsrobots → red; robots → move
moverobots, slowly[robots, slowly] → movemove → robots; move → slowly

CBOW combines several context words to predict one center word. Skip-gram uses one center word to predict each context word separately. Skip-gram does not generate an ordered sentence, and “skip” does not mean remove random words. The target may come from either side of the center. At sentence boundaries, an implementation can use shorter windows; our code omits those centers to keep the tensor shapes simple.

The training labels come from the text itself. Nobody must annotate robots with a semantic category before these examples can be created. Repeated occurrences create repeated evidence. The chosen window determines what relationships the model is encouraged to capture: changing it changes the training task, even when the vocabulary stays the same.

CBOW averages the input embeddings for red and move to predict robots; Skip-gram looks up the input embedding for robots and uses it in separate predictions for red and move.
Figure 2 — The same sentence creates opposite prediction directions. Both architectures learn input and output embedding tables; the average appears only in the CBOW path shown here.

Two embedding tables and a softmax

A word needs a vector when it is an input and a vector when it is a prediction candidate. Use two learned tables, rather than assuming these roles share weights:

E,U∈RV×D,ew=Ew,:⊤,uw=Uw,:⊤,p(o∣h)=exp⁡(uo⊤h)∑v=1Vexp⁡(uv⊤h).E,U\in\mathbb R^{V\times D},\qquad \mathbf e_w=E_{w,:}^{\top},\qquad \mathbf u_w=U_{w,:}^{\top},\qquad p(o\mid\mathbf h)=\frac{\exp(\mathbf u_o^\top\mathbf h)}{\sum_{v=1}^{V}\exp(\mathbf u_v^\top\mathbf h)}.

Here EE is the input embedding table, UU is the output embedding table, VV is vocabulary size, DD is vector dimension, ww is a word ID, and the colon selects all coordinates in its row. The transpose makes ew\mathbf e_w and uw\mathbf u_w column vectors. The vector h∈RD\mathbf h\in\mathbb R^D represents the input, oo is the target word ID, vv indexes a candidate word, and p(o∣h)p(o\mid\mathbf h) is its softmax probability. Each dot product is a prediction score; the denominator makes probabilities sum to one over the vocabulary.

CBOW forms the input by averaging the context vectors. Its loss asks the model to assign high probability to the observed center:

ht=1∣Ct∣∑j∈Ctewj,LCBOW=−1∣T∣∑t∈Tlog⁡p(wt∣ht).\mathbf h_t=\frac{1}{|C_t|}\sum_{j\in C_t}\mathbf e_{w_j},\qquad \mathcal L_{\mathrm{CBOW}}=-\frac{1}{|\mathcal T|}\sum_{t\in\mathcal T}\log p(w_t\mid\mathbf h_t).

Here wtw_t is the center word at position tt, CtC_t is the set of selected neighboring positions excluding tt, ∣Ct∣|C_t| is its size, ht\mathbf h_t is their mean vector, T\mathcal T contains the valid center positions, and LCBOW\mathcal L_{\mathrm{CBOW}} is the average negative log probability of those centers. The mean ignores context order: exchanging left and right neighbors produces the same input. Some formulations sum context vectors instead; this lesson and its code use the mean.

Skip-gram instead looks up the center vector and applies the same prediction rule to each neighbor:

LSG=−1∣P∣∑(t,j)∈Plog⁡p(wj∣ewt).\mathcal L_{\mathrm{SG}}=-\frac{1}{|\mathcal P|}\sum_{(t,j)\in\mathcal P}\log p(w_j\mid\mathbf e_{w_t}).

Here LSG\mathcal L_{\mathrm{SG}} is the Skip-gram loss, P\mathcal P is the collection of observed center–context position pairs, ∣P∣|\mathcal P| is its size, tt indexes a center position, jj indexes one neighboring position, wtw_t is the input word and wjw_j is its target. Occurrences remain separate even when their word IDs repeat.

A training step increases the observed target’s score relative to competing candidates. It changes both the selected input vectors and the output vectors. Over many examples, reusing the same input row makes the model reconcile that word’s different surroundings. This shared prediction task is the route from discrete word IDs to learned continuous features. Neither architecture needs a deep stack of hidden layers in this basic form, which is the main computational difference from the neural language model in NLP 03.

Negative sampling: compare observed pairs with random draws

A full softmax scores every vocabulary word for every prediction. Negative sampling replaces that vocabulary-wide classification with a smaller binary task: distinguish an observed center–context pair from pairs formed with randomly drawn target words. It was introduced in the Word2Vec follow-up paper [8].

For an observed input word cc and context word oo, one sampled loss is:

Lneg(c,o)=−log⁡σ(uo⊤ec)−∑k=1Klog⁡σ(−unk⊤ec),nk∼q,σ(a)=11+exp⁡(−a).\mathcal L_{\mathrm{neg}}(c,o)=-\log\sigma(\mathbf u_o^\top\mathbf e_c)-\sum_{k=1}^{K}\log\sigma(-\mathbf u_{n_k}^\top\mathbf e_c),\qquad n_k\sim q,\qquad \sigma(a)=\frac{1}{1+\exp(-a)}.

Here Lneg\mathcal L_{\mathrm{neg}} is the negative-sampling loss for one observed pair, cc is the center word ID, oo is its observed context word ID, ec\mathbf e_c is the center’s input vector, uo\mathbf u_o and unk\mathbf u_{n_k} are output vectors, KK is the chosen number of random targets, nkn_k is draw kk from distribution qq, and σ\sigma is the sigmoid applied to a real score aa. The first term rewards a large observed-pair score; the remaining terms reward small random-pair scores. The full training objective averages this expression over observed pairs and sampling randomness. For a CBOW version, use the averaged context vector in place of ec\mathbf e_c.

A common sampling distribution from the original paper is:

q(w)=f(w)3/4∑v=1Vf(v)3/4.q(w)=\frac{f(w)^{3/4}}{\sum_{v=1}^{V}f(v)^{3/4}}.

Here q(w)q(w) is the probability of drawing vocabulary word ww, f(w)f(w) is its corpus occurrence count, vv indexes vocabulary entries, and VV is vocabulary size. The exponent flattens the frequency distribution relative to sampling directly by counts. It is a design choice, not a universal probability law.

For the observed pair robots → move, a random draw might create robots → red. The label “negative” means drawn by the noise process; it does not assert that the two words can never occur together. A draw can be a genuine context elsewhere, or even collide with this target. An implementation must specify whether it retains, skips or redraws collisions; changing that rule changes the sampled objective. The model separates statistical sources of pairs rather than learning a dictionary of absolute incompatibilities.

The binary sigmoid scores do not sum to one over vocabulary words. Negative sampling is a different training objective, not an exact computation of the softmax above, and its loss is not a normalized next-token likelihood. Use it to learn useful representations; evaluate those representations on the intended downstream task.

Why prediction changes vector geometry

The phrase “learn from neighbors” becomes concrete in the gradient. For one full-softmax target, let its score be the dot product between an output vector and the input representation:

sv=uv⊤h,ℓ=−log⁡p(o∣h),∂ℓ∂sv=p(v∣h)−1[v=o],∇hℓ=∑v=1Vp(v∣h)uv−uo.s_v=\mathbf u_v^\top\mathbf h,\qquad \ell=-\log p(o\mid\mathbf h),\qquad \frac{\partial\ell}{\partial s_v}=p(v\mid\mathbf h)-\mathbf1[v=o],\qquad \nabla_{\mathbf h}\ell=\sum_{v=1}^{V}p(v\mid\mathbf h)\mathbf u_v-\mathbf u_o.

Here svs_v is candidate vv‘s score, ℓ\ell is the loss for observed target oo, 1[v=o]\mathbf1[v=o] equals one for that target and zero otherwise, and ∇hℓ\nabla_{\mathbf h}\ell is the gradient with respect to the input representation. The embedding tables, vocabulary size VV and conditional probability are those defined above.

Gradient descent moves the input toward the observed output vector and away from the model’s probability-weighted average of alternatives. A word with many occurrences receives many such updates through the same row. Similar context distributions can therefore give different words compatible predictive roles, without any training label declaring them synonyms.

For mean-based CBOW, every selected context row receives ∇hℓ/∣Ct∣\nabla_{\mathbf h}\ell/|C_t|. The input cannot distinguish two contexts that are permutations of the same words. For Skip-gram, one center row receives updates from multiple neighbor targets. Its basic score has no slot for whether a neighbor was on the left or right. Choosing the window therefore chooses which relations the objective can directly represent; a wider window is not simply a more accurate version of a narrow one.

What negative-sampling scores estimate

We can inspect the sampling objective before imposing the embedding dimension. Fix a center word and imagine that every candidate has an independently adjustable score. Let pop_o be its empirical context probability and qoq_o its noise probability. The expected contribution for that candidate is:

ℓo(so)=−polog⁡σ(so)−Kqolog⁡σ(−so),dℓodso=(po+Kqo)σ(so)−po.\ell_o(s_o)=-p_o\log\sigma(s_o)-Kq_o\log\sigma(-s_o),\qquad \frac{d\ell_o}{ds_o}=(p_o+Kq_o)\sigma(s_o)-p_o.

Here oo is one candidate context, sos_o is its unrestricted real score, pop_o and qoq_o are the two probabilities for the fixed center, KK is the number of independent noise draws per observed pair and σ\sigma is the sigmoid defined above. This calculation retains collisions between noise draws and observed targets.

When both probabilities are positive, the stationary score is:

so∗=log⁡poKqo.s_o^*=\log\frac{p_o}{Kq_o}.

Here so∗s_o^* minimizes that candidate’s expected loss. For po=0.5p_o=0.5, qo=0.25q_o=0.25 and K=2K=2, it is zero: half the relevant probability mass is attributed to each statistical source. If qo=0.05q_o=0.05 instead, the optimum becomes log⁡5≈1.609\log5\approx1.609. The observed context frequency stayed fixed; the noise distribution changed the score. A positive score therefore means “more associated than this sampling baseline,” rather than “the pair is certainly meaningful.” If the observed probability is zero and noise probability is positive, the unrestricted optimum tends toward negative infinity.

When noise is exactly the corpus context marginal, this ratio yields PMI minus log⁡K\log K. Levy and Goldberg analyze the resulting matrix-factorization interpretation [9]. The common three-quarter-power distribution changes the ratio. In an actual embedding model, scores share low-dimensional vectors, so they cannot generally reach every independent optimum. The learned tables compromise among the pair statistics under a rank constraint. This explains both the connection to the count-based construction and why truncating a PPMI matrix and training Word2Vec need not give identical vectors.

Full softmax costs roughly O(VD)O(VD) arithmetic per target, while scoring one observed target and KK draws costs O((K+1)D)O((K+1)D), before sampling and batching overhead. Here DD is embedding dimension. The saving comes with a different objective. Neither the sigmoid value nor the negative-sampling loss is a substitute for the normalized token probability used to report perplexity.

Train CBOW and Skip-gram on CPU

This standalone example learns both input and output tables using full softmax. The vocabulary is tiny, so we can inspect the objective without a sampling approximation. Both models use the same invented corpus, radius-one interior windows, embedding width and training loop. A one-token input average in the Skip-gram path is simply the center vector.

Code · Python · 57 lines
import torch
from torch import nn
from torch.nn import functional as F

torch.manual_seed(7)
torch.set_num_threads(1)
corpus = [
    'small red robots move slowly',
    'small blue robots move quickly',
    'large red sensors work slowly',
    'large blue sensors work quickly',
    'small red sensors work quietly',
    'large blue robots move quietly',
]
vocab = sorted(set(' '.join(corpus).split()))
ids = {word: i for i, word in enumerate(vocab)}
cbow_x, cbow_y, sg_x, sg_y = [], [], [], []
for sentence in corpus:
    tokens = [ids[word] for word in sentence.split()]
    # Use radius 1 and interior centers, so every context has two words.
    for t in range(1, len(tokens) - 1):
        neighbors = [tokens[t - 1], tokens[t + 1]]
        cbow_x.append(neighbors)
        cbow_y.append(tokens[t])
        for neighbor in neighbors:
            sg_x.append([tokens[t]])
            sg_y.append(neighbor)

class Word2Vec(nn.Module):
    def __init__(self, size, width=8):
        super().__init__()
        self.input = nn.Embedding(size, width)
        self.output = nn.Embedding(size, width)
        nn.init.normal_(self.input.weight, std=0.1)
        nn.init.normal_(self.output.weight, std=0.1)

    def forward(self, context):
        h = self.input(context).mean(dim=1)
        return h @ self.output.weight.T

embeddings = {}
for name, inputs, targets in [('CBOW', cbow_x, cbow_y),
                             ('Skip-gram', sg_x, sg_y)]:
    x, y = torch.tensor(inputs), torch.tensor(targets)
    model = Word2Vec(len(vocab))
    optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
    initial = F.cross_entropy(model(x), y).item()
    for step in range(250):
        optimizer.zero_grad()
        loss = F.cross_entropy(model(x), y)
        loss.backward()
        optimizer.step()
    final = F.cross_entropy(model(x), y).item()
    assert final < initial
    embeddings[name] = model.input.weight.detach().clone()
    print(name, len(y), f'{initial:.3f} -> {final:.3f}')
print('vocabulary', len(vocab), 'embedding shape', tuple(embeddings['CBOW'].shape))

The run prints CBOW 18 2.384 -> 0.159, Skip-gram 36 2.402 -> 1.198, and vocabulary 11 embedding shape (11, 8). Losses are mean negative log probabilities in nats. The assertions establish that the implementation fits these training examples. The smaller CBOW loss does not establish that CBOW produces better word vectors: its targets and conditional inputs differ from Skip-gram’s. This corpus is too small to judge semantic neighborhoods or natural-language generalization.

After training, embeddings['CBOW'][ids['robots']] retrieves one learned input vector. The output table was needed to train it but can be omitted when serving a downstream model that only looks up input vectors. Record which table you retain; using input vectors, output vectors or a combination is a representation choice.

Optional: word2vec (in the Paper guides). Follow the original figure, objective, experimental comparisons and limitations.

GloVe: fit log co-occurrence counts directly

Plain version. Instead of streaming through windows one prediction at a time, GloVe first counts how often every pair of words co-occurs, then fits word vectors whose dot products reproduce the logarithms of those counts. Frequent pairs get more weight, but not unlimited weight.

Pennington et al. motivate GloVe with ratios of co-occurrence probabilities [10]. Let XikX_{ik} be the (distance-weighted) number of times word kk occurs in the context of word ii, and P(k∣i)=Xik/∑k′Xik′P(k\mid i)=X_{ik}/\sum_{k'}X_{ik'}. For two words ii and jj, the ratio P(k∣i)/P(k∣j)P(k\mid i)/P(k\mid j) is large for context words kk that belong with ii but not jj, small for the reverse, and near one for words that are equally related or unrelated to both. The ratio discriminates better than either probability alone. The model asks vector differences to encode such ratios, which leads to a weighted least-squares objective over the co-occurrence matrix:

J=∑i,j=1Vf ⁣(Xij)(wi⊤w~j+bi+b~j−log⁡Xij)2,f(x)={(x/xmax⁡)αx<xmax⁡,1otherwise.J=\sum_{i,j=1}^{V}f\!\left(X_{ij}\right)\left(\mathbf w_i^\top\tilde{\mathbf w}_j+b_i+\tilde b_j-\log X_{ij}\right)^2, \qquad f(x)=\begin{cases}(x/x_{\max})^{\alpha}&x\lt x_{\max},\\1&\text{otherwise.}\end{cases}

Here VV is vocabulary size, XijX_{ij} is the co-occurrence count of words ii and jj, wi∈RD\mathbf w_i\in\mathbb R^D is the word vector of ii, w~j∈RD\tilde{\mathbf w}_j\in\mathbb R^D is a separate context vector of jj, bib_i and b~j\tilde b_j are scalar biases, ff is a weighting function, and xmax⁡x_{\max} and α\alpha are its cutoff and exponent. Because f(0)=0f(0)=0, cells with zero count contribute nothing, so log⁡0\log 0 is never needed and the sum runs only over observed pairs. The paper fixes xmax⁡=100x_{\max}=100 and α=3/4\alpha=3/4, uses a co-occurrence weighting in which a pair dd words apart adds 1/d1/d, and reports the sum W+W~W+\tilde W of the two tables as the final word vectors [10].

Three design choices separate GloVe from Word2Vec. It trains on global counts rather than on individual windows, so one pass over the corpus builds XX and training then costs time proportional to the number of nonzero cells. The biases absorb word frequency, so the dot product models what remains. And the weighting function prevents very frequent pairs, such as those involving the, from dominating, while giving rare and noisy pairs little influence. Like negative sampling, the objective factorizes a shifted log-count matrix under a rank constraint, which is why GloVe and Skip-gram vectors often behave similarly in practice.

The following code computes the ingredients on the earlier toy corpus: co-occurrence ratios and the weighting function.

Code · Python · 27 lines
import numpy as np

text = """the cat drinks milk . the dog drinks water . a cat chases a mouse .
a dog chases a cat . the kitten drinks milk . the puppy drinks water .
a kitten chases a ball . a puppy chases a ball . the car needs fuel .
the truck needs fuel . a car drives on the road . a truck drives on the road ."""
sents = [s.split() for s in text.split(".") if s.strip()]
vocab = sorted({w for s in sents for w in s})
idx = {w: i for i, w in enumerate(vocab)}

X = np.zeros((len(vocab), len(vocab)))            # +-2 window, 1/distance weighting
for s in sents:
    for i, w in enumerate(s):
        for j in range(max(0, i - 2), min(len(s), i + 3)):
            if j != i:
                X[idx[w], idx[s[j]]] += 1.0 / abs(i - j)
P = X / X.sum(1, keepdims=True)                   # P(k | i): row-normalized counts

for k in ["drinks", "chases", "needs", "the"]:
    a, b = P[idx["kitten"], idx[k]], P[idx["truck"], idx[k]]
    ratio = "inf" if b == 0 else f"{a / b:.2f}"
    print(f"P({k}|kitten)={a:.3f}  P({k}|truck)={b:.3f}  ratio={ratio}")

def f(x, x_max=100.0, alpha=0.75):                 # GloVe weighting function
    return np.minimum((x / x_max) ** alpha, 1.0)
print("f at 1, 10, 100, 1000:", np.round(f(np.array([1.0, 10.0, 100.0, 1000.0])), 3))
print("nonzero cells:", int((X > 0).sum()), "of", X.size)

The run prints:

Code · 6 lines
P(drinks|kitten)=0.200  P(drinks|truck)=0.000  ratio=inf
P(chases|kitten)=0.200  P(chases|truck)=0.000  ratio=inf
P(needs|kitten)=0.000  P(needs|truck)=0.200  ratio=0.00
P(the|kitten)=0.200  P(the|truck)=0.200  ratio=1.00
f at 1, 10, 100, 1000: [0.032 0.178 1.    1.   ]
nonzero cells: 96 of 361

The ratios separate animal contexts from vehicle contexts, while the gives a ratio of one and therefore no information. The output also exposes a limitation of this tiny corpus: almost three quarters of the matrix is zero, and GloVe ignores zero cells. With so few observed pairs, a model with a few hundred parameters can fit every observed cell exactly, and the resulting geometry depends mainly on initialization. GloVe’s count-based design needs large corpora in which the matrix is well populated; we therefore do not train it here. Exercise 3 asks you to observe this underdetermination yourself.

Optional: GloVe (in the Paper guides). Follow the original figure, objective, experimental comparisons and limitations.

fastText: vectors from character n-grams

Plain version. A word-level table has no vector for a word it never saw, and it treats charger and chargers as unrelated rows. fastText builds each word’s vector from vectors of its character pieces, so related spellings share parameters and unseen words still get a vector.

Bojanowski et al. extend Skip-gram with negative sampling by representing each word as a bag of character n-grams [11]. The word is wrapped in boundary symbols, < and >, so that prefixes and suffixes are distinguishable from the same letters inside a word, and the whole wrapped word is added as one extra unit. In the paper, all n-grams with 3≤n≤63\le n\le 6 are used. The score between a center word ww and a context word cc becomes

s(w,c)=∑g∈Gwzg⊤vc.s(w,c)=\sum_{g\in\mathcal G_w}\mathbf z_g^\top\mathbf v_c.

Here Gw\mathcal G_w is the set of n-grams of ww (including the wrapped word itself), zg∈RD\mathbf z_g\in\mathbb R^D is the learned vector of n-gram gg, and vc∈RD\mathbf v_c\in\mathbb R^D is the context vector of cc. The word’s own vector is ∑g∈Gwzg\sum_{g\in\mathcal G_w}\mathbf z_g. This score replaces uo⊤ec\mathbf u_o^\top\mathbf e_c in the negative-sampling loss above. To bound memory, the paper maps n-grams into a fixed number of buckets with a hash function, so unrelated n-grams can share a vector [11].

Code · Python · 25 lines
import numpy as np

def char_ngrams(word, n_min=3, n_max=6):
    w = f"<{word}>"                       # boundary symbols mark prefixes and suffixes
    grams = {w[i:i + n] for n in range(n_min, n_max + 1) for i in range(len(w) - n + 1)}
    return grams | {w}                    # the whole word is kept as one extra unit

print(sorted(char_ngrams("robot", 3, 3)))
vocab = ["charge", "charger", "recharge", "discharge", "robot", "robots"]
query = "chargers"                        # not in the vocabulary
q = char_ngrams(query)
for word in vocab:
    print(f"{word:10s} shares {len(q & char_ngrams(word)):2d} of {len(q)} units with '{query}'")

# A fastText-style vector is the sum of its unit vectors, so an unseen word still gets one.
rng = np.random.default_rng(0)
table = {}                                # stands in for a trained (hashed) n-gram table
def vector(word, dim=8):
    return sum(table.setdefault(g, rng.normal(size=dim)) for g in sorted(char_ngrams(word)))
def cos(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
for word in vocab:
    vector(word)
print("cos(chargers, charger) =", round(cos(vector("chargers"), vector("charger")), 3))
print("cos(chargers, robots)  =", round(cos(vector("chargers"), vector("robots")), 3))

The run prints:

Code · 9 lines
['<ro', '<robot>', 'bot', 'obo', 'ot>', 'rob']
charge     shares 14 of 27 units with 'chargers'
charger    shares 18 of 27 units with 'chargers'
recharge   shares 10 of 27 units with 'chargers'
discharge  shares 10 of 27 units with 'chargers'
robot      shares  0 of 27 units with 'chargers'
robots     shares  0 of 27 units with 'chargers'
cos(chargers, charger) = 0.804
cos(chargers, robots)  = -0.222

The n-gram vectors here are random, not trained, so the high cosine between chargers and charger comes only from the units they share. That is exactly the mechanism: an unseen inflection inherits most of its vector from known pieces. Training decides whether sharing is helpful. It usually is for inflection and derivation, and it is misleading for words that share letters by accident; discharge and recharge share charge but differ in meaning in ways the pieces do not reveal. Subword vectors help most for morphologically rich languages and for rare words. Note the relation to the subword tokenizers in NLP 02: BPE splits a word into one sequence of pieces before lookup, whereas fastText sums vectors of all overlapping n-grams.

Dense word and document representations

Plain version. A dense vector lets different words share statistical strength. That can connect related phrases, but the vector’s geometry reflects its training data and objective rather than a universal definition of meaning.

Static vectors provide compact reusable features for a classifier, nearest-neighbor word exploration or a pooled document baseline. They are useful when a labeled application has little data but an appropriate text corpus is available. Training the embedding and the downstream classifier are different stages: neighbor prediction never directly learns a billing label or an abstention policy.

A static embedding table assigns a vector to each vocabulary entry:

et=Eit,:,h(d)=1T∑t=1Tet.\mathbf e_t=E_{i_t,:}, \qquad \mathbf h(d)=\frac1T\sum_{t=1}^{T}\mathbf e_t.

Here E∈RV×DE\in\mathbb R^{V\times D} is the embedding table, VV is vocabulary size, DD is embedding dimension, iti_t is the token ID at position tt, et\mathbf e_t is its row vector, TT is document length in tokens and h(d)\mathbf h(d) is the mean-pooled document vector. The colon denotes all coordinates in the selected row.

The tables above supply ways to obtain EE. Once token vectors are available, mean pooling turns a variable-length document into a fixed-width feature. This aggregation step is separate from the objective used to train the table.

Mean pooling is an instructive baseline because it exposes a limitation clearly. Any permutation of a document’s tokens gives the same average. The representation of “the technician helped the customer” is identical to “the customer helped the technician” if the same static token vectors are used. It also treats negation as one contribution among many rather than an operator changing the meaning of a phrase. The word bank retrieves the same static vector in river bank and bank account; more training data does not change this lookup rule.

There are two separate choices: how token vectors are obtained and how they are combined. You can freeze pretrained token vectors and train only a classifier, fine-tune the vectors with the classifier, or train all parameters from scratch. Freezing reduces the number of task-specific parameters. Fine-tuning can adapt meanings to a domain, but small datasets can distort representations or memorize labels.

Do not compare embeddings only by visually attractive clusters. A two-dimensional projection can conceal overlap in the original space. Compare representations under the same task, split, classifier capacity and tuning allowance. If a dense representation changes the downstream head too, the experiment no longer isolates the representation.

Cosine measures direction, not truth

The count-based example compares unit-normalized vectors. Its dot product is then the cosine of the angle between them:

cos⁡(a,b)=a⊤b∥a∥2∥b∥2.\operatorname{cos}(\mathbf a,\mathbf b)= \frac{\mathbf a^\top\mathbf b}{\|\mathbf a\|_2\|\mathbf b\|_2}.

Here a\mathbf a and b\mathbf b are nonzero embedding column vectors, a⊤b\mathbf a^\top\mathbf b is their inner product, and ∥⋅∥2\|\cdot\|_2 is Euclidean length. The cosine lies between −1-1 and 11. A zero vector needs an explicit handling rule because the denominator vanishes.

Cosine removes positive rescaling of either vector. It does not remove every transformation compatible with the training objective. In the two-table model, the predictive scores depend on products of an input and an output vector. Many different tables produce the same products:

E′=EA,U′=UA−⊤,E′U′⊤=EU⊤.E'=EA,\qquad U'=UA^{-\top},\qquad E'U'^\top=EU^\top.

Here EE and UU are the input and output tables with shape V×DV\times D, AA is any invertible D×DD\times D matrix, A−⊤A^{-\top} is the transpose of its inverse, and E′E' and U′U' are the transformed tables. Each cell of EU⊤EU^\top is one center–output prediction score. Thus full-softmax predictions and the negative-sampling objective are unchanged by this paired transformation. The same argument applies to GloVe’s wi⊤w~j\mathbf w_i^\top\tilde{\mathbf w}_j.

An orthogonal change of coordinates preserves input-vector angles, but general invertible changes need not. Consequently, identical prediction loss does not uniquely determine cosine neighborhoods in the retained input table. Optimization, initialization and additional constraints select a particular solution. This is one reason a low training loss alone cannot certify useful downstream similarity.

This standalone numerical check combines three claims: score preservation, order invariance of averaging, and the noise-dependent optimum derived above. The matrices are hand-chosen for this calculation; they are not trained language vectors.

Code · Python · 28 lines
import numpy as np

# Two input rows and three output rows: scores are center-output dot products.
E = np.array([[1., 1.], [1., -1.]])
U = np.array([[1., 0.], [0., 1.], [-1., 1.]])
A = np.diag([2., 0.5])
changed_E = E @ A
changed_U = U @ np.linalg.inv(A).T

def cosine(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

assert np.allclose(E @ U.T, changed_E @ changed_U.T)
assert np.allclose(E[[0, 1]].mean(0), E[[1, 0]].mean(0))
print('scores unchanged:', np.allclose(E @ U.T, changed_E @ changed_U.T))
print('cosine:', f'{cosine(E[0], E[1]):.3f}', '->',
      f'{cosine(changed_E[0], changed_E[1]):.3f}')
print('mean unchanged by order:',
      np.allclose(E[[0, 1]].mean(0), E[[1, 0]].mean(0)))

# The unrestricted negative-sampling optimum depends on the noise baseline.
p, K = 0.5, 2
for q in [0.25, 0.05]:
    score = np.log(p / (K * q))
    sigmoid = 1 / (1 + np.exp(-score))
    gradient = (p + K * q) * sigmoid - p
    assert np.isclose(gradient, 0)
    print('noise', q, 'optimal score', f'{score:.3f}')

The output is:

Code · 5 lines
scores unchanged: True
cosine: 0.000 -> 0.882
mean unchanged by order: True
noise 0.25 optimal score 0.000
noise 0.05 optimal score 1.609

The cosine changed while all center–output scores stayed fixed. The equal means have a different implication: once a representation merges two token orders, no classifier that receives only that mean can recover which order occurred. These are properties of the representation, rather than evidence that one optimizer ran poorly.

Evaluating word embeddings

Plain version. There are two ways to judge embeddings: ask whether their geometry agrees with human judgments about words (intrinsic), or plug them into a task and measure the task (extrinsic). Neither alone tells you whether vectors are “good.”

Similarity: correlate cosines with human ratings

A word-similarity dataset lists word pairs with graded human ratings. The model scores each pair by cosine, and the evaluation measures how well the two rankings agree with Spearman’s rank correlation:

ρ=1−6∑i=1nδi2n(n2−1).\rho=1-\frac{6\sum_{i=1}^{n}\delta_i^2}{n(n^2-1)}.

Here nn is the number of pairs, δi\delta_i is the difference between the rank of pair ii under the model’s cosines and its rank under the human ratings, and ρ∈[−1,1]\rho\in[-1,1]. This formula assumes no ties; with ties, ρ\rho is the Pearson correlation of the ranks. Spearman’s ρ\rho uses only ranks, so it ignores the scale of cosines.

What the ratings mean matters as much as the statistic. WordSim-353 is a widely used set of rated word pairs [12]. Hill et al. point out that its ratings do not separate similarity from association, and built SimLex-999 to quantify similarity explicitly, so that associated but dissimilar pairs receive low ratings [13]. A model trained with broad windows may score well on association-style data and poorly on similarity-style data, for the reason given in the lexical-semantics section: distributional evidence mixes the relations.

Analogy: the vector-offset test and its caveats

Mikolov, Yih and Zweig observed that some relations appear as roughly constant vector offsets [14]. To answer “aa is to a∗a^* as bb is to ?”, the offset method returns

x∗=argmax⁡x∈V∖{a,a∗,b}cos⁡ ⁣(ex, ea∗−ea+eb).x^*=\operatorname*{argmax}_{x\in\mathcal V\setminus\{a,a^*,b\}}\operatorname{cos}\!\left(\mathbf e_x,\ \mathbf e_{a^*}-\mathbf e_a+\mathbf e_b\right).

Here V\mathcal V is the vocabulary, ex\mathbf e_x is the vector of candidate xx, and the three question words are excluded from the candidates. Linzen points out that this exclusion is doing real work: the target point ea∗−ea+eb\mathbf e_{a^*}-\mathbf e_a+\mathbf e_b often lies closest to bb itself, so the method’s success mixes offset consistency with the neighborhood structure around bb [15]. He recommends reporting simple baselines, such as returning the nearest neighbor of bb, alongside analogy accuracy.

The next example evaluates the PPMI vectors from earlier with invented similarity ratings and three analogies. The ratings are made up for this lesson and are not a benchmark.

Code · Python · 41 lines
import numpy as np
from scipy.stats import spearmanr

text = """the cat drinks milk . the dog drinks water . a cat chases a mouse .
a dog chases a cat . the kitten drinks milk . the puppy drinks water .
a kitten chases a ball . a puppy chases a ball . the car needs fuel .
the truck needs fuel . a car drives on the road . a truck drives on the road ."""
sents = [s.split() for s in text.split(".") if s.strip()]
vocab = sorted({w for s in sents for w in s})
idx = {w: i for i, w in enumerate(vocab)}
C = np.zeros((len(vocab), len(vocab)))
for s in sents:
    for i, w in enumerate(s):
        for j in range(max(0, i - 2), min(len(s), i + 3)):
            if j != i:
                C[idx[w], idx[s[j]]] += 1
P = C / C.sum()
with np.errstate(divide="ignore"):
    ppmi = np.maximum(np.log(P / (P.sum(1, keepdims=True) * P.sum(0, keepdims=True))), 0)
U, S, _ = np.linalg.svd(ppmi)
E = U[:, :8] * S[:8]
E /= np.linalg.norm(E, axis=1, keepdims=True)
cos = lambda a, b: float(E[idx[a]] @ E[idx[b]])

# 1) Similarity: correlate model cosines with (invented) graded judgments, 0-10.
judged = [("cat", "kitten", 9), ("dog", "puppy", 9), ("car", "truck", 8), ("milk", "water", 5),
          ("cat", "dog", 6), ("cat", "mouse", 3), ("car", "road", 2), ("kitten", "fuel", 0)]
model = [cos(a, b) for a, b, _ in judged]
rho = spearmanr(model, [h for _, _, h in judged]).statistic
print(f"Spearman rho = {rho:.3f} over {len(judged)} pairs")

# 2) Analogy a:b :: c:? with the offset b - a + c (3CosAdd).
def analogy(a, b, c, exclude_inputs=True):
    target = E[idx[b]] - E[idx[a]] + E[idx[c]]
    scores = E @ (target / np.linalg.norm(target))
    for i in np.argsort(-scores):
        if not exclude_inputs or vocab[i] not in (a, b, c):
            return vocab[i]
for a, b, c in [("cat", "kitten", "dog"), ("car", "fuel", "cat"), ("cat", "milk", "puppy")]:
    print(f"{a}:{b} :: {c}:?  ->", analogy(a, b, c), "| without excluding inputs ->",
          analogy(a, b, c, exclude_inputs=False))

The run prints:

Code · 4 lines
Spearman rho = 0.837 over 8 pairs
cat:kitten :: dog:?  -> puppy | without excluding inputs -> puppy
car:fuel :: cat:?  -> kitten | without excluding inputs -> cat
cat:milk :: puppy:?  -> water | without excluding inputs -> water

Two analogies come out as intended. The middle one shows both caveats at once. Without exclusion, the answer is cat, one of the question words. With exclusion, the answer is kitten, the nearest neighbor of cat, not milk, the intended “what it consumes” answer. The offset did not carry the relation; the neighborhood of bb decided. Eight invented pairs and three analogies are, of course, far too few to support any general claim about these vectors.

Intrinsic versus extrinsic

Schnabel et al. compare evaluation methods and find that different evaluations order embedding methods differently, that word frequency affects results and cosine similarity, and that extrinsic tasks should not be used as a proxy for generic embedding quality [16]. The practical rule is to evaluate on the relation your application needs.

Nearest-neighbor lists can reveal obvious corpus patterns, but they are a diagnostic rather than an application score. Separate lexical questions: is the candidate a synonym, an antonym, a word associated with the topic, or a word that can occupy the same grammatical slot? A single unqualified notion of “related” hides these distinctions.

Use controlled pairs alongside ordinary held-out examples. For a representation used in sentiment, compare a positive claim with its negation. For a role-sensitive task, exchange who acted on whom while holding the word inventory fixed. For a technical corpus, test a term used in its domain-specific and everyday senses. These tests ask whether the representation preserves a required difference; they do not demand that every word pair occupy a predetermined location.

If mean pooling makes a critical pair identical, collecting more classifier labels cannot fix that collision while the representation remains fixed. If the vectors distinguish the pair but the classifier predicts incorrectly, supervision or the decision boundary may be responsible. Locate the failure at the earliest stage that loses the needed evidence.

Bias in embeddings

Plain version. Embeddings learn whatever regularities the corpus contains, including social stereotypes. If a group of names co-occurs with some occupations more than others, the vectors encode that association, and downstream systems can reproduce it.

Bolukbasi et al. showed that word embeddings trained on news text encode gender stereotypes, identified a gender direction in the space, and proposed removing it from words that should be gender-neutral [17]. Caliskan et al. introduced the Word Embedding Association Test (WEAT), modeled on the Implicit Association Test from psychology, and found that embeddings reproduce a range of documented human biases [18]. For a target word ww and two attribute sets AA and BB, the association used by WEAT is

s(w,A,B)=1∣A∣∑a∈Acos⁡(ew,ea)−1∣B∣∑b∈Bcos⁡(ew,eb).s(w,A,B)=\frac{1}{|A|}\sum_{a\in A}\operatorname{cos}(\mathbf e_w,\mathbf e_a)-\frac{1}{|B|}\sum_{b\in B}\operatorname{cos}(\mathbf e_w,\mathbf e_b).

Here ∣A∣|A| and ∣B∣|B| are the set sizes and e\mathbf e denotes word vectors. A positive value means ww is closer on average to set AA. The full test aggregates this score over two sets of target words and compares it with a permutation distribution.

The following invented corpus contains no sentence linking a name to an occupation. Occupation words always appear with their own activity, while invented names appear with activities at skewed rates.

Code · Python · 47 lines
import random
import numpy as np

rng = random.Random(0)
# An invented corpus. Job words always appear with their own activity; invented
# names appear with activities at skewed rates. No sentence says "ana is a pilot".
group_a, group_b = ["ana", "ari"], ["bo", "bea"]
activity = {"pilot": "flies the plane", "baker": "bakes the bread", "clerk": "files the forms"}
share_a = {"pilot": 0.8, "baker": 0.2, "clerk": 0.5}    # P(group A | activity)
sents = []
for _ in range(3000):
    job = rng.choice(list(activity))
    if rng.random() < 0.5:
        sents.append(f"the {job} {activity[job]}".split())
    else:
        name = rng.choice(group_a if rng.random() < share_a[job] else group_b)
        sents.append(f"{name} {activity[job]}".split())
vocab = sorted({w for s in sents for w in s})
idx = {w: i for i, w in enumerate(vocab)}
C = np.zeros((len(vocab), len(vocab)))
for s in sents:
    for i, w in enumerate(s):
        for j in range(max(0, i - 2), min(len(s), i + 3)):
            if j != i:
                C[idx[w], idx[s[j]]] += 1
P = C / C.sum()
with np.errstate(divide="ignore"):
    ppmi = np.maximum(np.log(P / (P.sum(1, keepdims=True) * P.sum(0, keepdims=True))), 0)
U, S, _ = np.linalg.svd(ppmi)
E = U[:, :4] * S[:4]
E /= np.linalg.norm(E, axis=1, keepdims=True)

def association(word, E):            # mean cosine to group A minus mean cosine to group B
    unit = E / np.linalg.norm(E, axis=1, keepdims=True)
    w = unit[idx[word]]
    return np.mean([w @ unit[idx[a]] for a in group_a]) - np.mean([w @ unit[idx[b]] for b in group_b])

for job in activity:
    print(f"{job:6s} association {association(job, E):+.3f}")

# Project out one "group direction" and measure again.
d = E[[idx[a] for a in group_a]].mean(0) - E[[idx[b] for b in group_b]].mean(0)
d /= np.linalg.norm(d)
E_proj = E.copy()
for job in activity:
    E_proj[idx[job]] -= (E_proj[idx[job]] @ d) * d
print("after projection:", [f"{job} {round(association(job, E_proj), 3) + 0.0:+.3f}" for job in activity])

The run prints:

Code · 4 lines
pilot  association +0.795
baker  association -0.811
clerk  association +0.072
after projection: ['pilot +0.000', 'baker +0.000', 'clerk +0.000']

The skew in who performs which activity became a strong association between names and occupation words, even though the two never co-occur in a sentence: both share the activity contexts. Projecting out the group direction sets the measured association to zero, but that result is circular, because the projection removes exactly the direction this score measures. Gonen and Goldberg show that, for published debiasing methods, the bias is largely hidden rather than removed: it remains in the distances between supposedly neutralized words and can be recovered from them [19]. Treat a debiasing step as one intervention to be audited on the downstream task, not as a guarantee, and remember that the source of the association is the data.

From word vectors to sentence embeddings

Plain version. An embedding supplies features; an application still needs a target, supervision and an action rule. When the application compares whole sentences, train the sentence representation for the comparison it will support.

For document classification, mean pooling is a convenient first experiment. Build one vector per training document, fit a classifier on its labels, and apply the same tokenizer, embedding lookup and pooling at evaluation time. Compare it with the sparse features and convolutional encoders developed in NLP 03. Keep the data split and classifier tuning allowance fixed so that a change in representation can be assessed fairly.

A frozen table uses contextual statistics learned before the labeled task. Fine-tuning lets the labels adjust the rows. These policies answer different questions: whether reusable corpus features already distinguish the classes, and whether task-specific changes improve them. With limited labels, fine-tuning can also fit incidental identifiers. Holding out groups such as conversations or repeated document templates helps reveal that behavior.

Decide how to handle uncovered text. Mapping every unknown word to one row merges all such words. Dropping them changes the pooling denominator and may leave an empty document. Returning a zero vector does not create a meaningful cosine direction. Subword vectors reduce but do not remove this problem. Coverage, the number of retained tokens, and the action for an empty representation are part of the model specification.

For word-level tagging, retain the sequence of token vectors rather than pooling it into one document vector. A downstream encoder can then combine local context before predicting a label per position. This does not make the original table contextual: the initial lookup of bank is still identical in both senses. The context-dependent computation happens afterward.

Similarity and contrastive sentence learning

A representation useful for predicting neighboring words may need adaptation before distances between whole sentences become reliable. Sentence-BERT trains paired sentence representations with a shared encoder and task objectives, making sentence comparisons possible without jointly encoding every candidate pair [20]. The computational distinction matters: separately cached document vectors can support retrieval, whereas a model that reads a query and candidate together must run for each pair.

A generic contrastive objective makes the intended geometry explicit:

Lpair=−log⁡exp⁡(cos⁡(q,a+)/τ)∑j=1Mexp⁡(cos⁡(q,aj)/τ).\mathcal L_{\mathrm{pair}}= -\log\frac{\exp(\operatorname{cos}(\mathbf q,\mathbf a^+)/\tau)} {\sum_{j=1}^{M}\exp(\operatorname{cos}(\mathbf q,\mathbf a_j)/\tau)}.

Here Lpair\mathcal L_{\mathrm{pair}} is the loss for one query, q\mathbf q is its vector, a+\mathbf a^+ is the vector of a relevant paired text included among the candidates, aj\mathbf a_j is candidate jj, MM is candidate count and τ>0\tau>0 is a temperature controlling the sharpness of the distribution. This is a generic learning construction, not a claim that every sentence encoder uses this exact objective. It has the same shape as the Word2Vec softmax: one observed pair competes with alternatives.

SimCSE is a widely used instance [21]. Its unsupervised version encodes the same sentence twice; because dropout differs between the two passes, the two vectors differ slightly and form a positive pair, while other sentences in the batch are negatives. The authors report that removing dropout leads to representation collapse. Its supervised version uses natural-language-inference data, taking entailment pairs as positives and contradiction pairs as hard negatives [21].

Negative examples define what “different” means. Two tickets about different invoices might be useful negatives for finding the same case but poor negatives for recognizing the billing category. False negatives can push semantically related examples apart. Before changing an encoder, inspect how positive and negative pairs are sampled.

Similarity also differs from entailment. “The parcel arrived” and “The parcel did not arrive” share topic and many tokens. They may be close in a semantic search space while making opposite claims. Classification, semantic search and inference therefore require different validation examples, even when they share an encoder. NLP 11 treats inference as its own task.

Preview: contextual embeddings

Plain version. A static table gives bank one vector everywhere. A contextual model computes a different vector for each occurrence, using the surrounding words, so the river sense and the money sense can come apart.

Peters et al. compute ELMo representations as functions of the entire input sentence, using the internal states of a bidirectional language model, and add them as features to task models [22]. BERT takes the step further within its pretrained encoder: a token’s final hidden state depends on the other permitted input tokens [23]. The distinction is lookup versus computation. Both systems may begin with an embedding table; static Word2Vec exposes a row directly, while BERT processes rows and positions through a context-dependent network.

Context can improve sense discrimination, but it does not turn every hidden-state cosine into a calibrated similarity measure. A static vector is a type-level summary across many occurrences; a contextual vector is a token-level representation of one occurrence. Which one an application needs depends on whether it asks about a word in general or a word as used here. NLP 07 builds the attention mechanism these models use, and NLP 10 develops masked pretraining and BERT’s contextual representations rather than repeating them here.

Modern view

Turney and Pantel’s survey [6] distinguishes term–document, word–context and pair–pattern matrices. Its taxonomy makes the source of evidence explicit: a document association, a neighboring-word relation and a relational pattern are different observations. The lesson here is to define the matrix before interpreting the vector space.

Apidianaki’s 2023 review [24] follows word meaning from static type-level vectors to contextual token-level representations and back. It covers semantic specialization, sense distinctions, representation interpretation and the derivation of static vectors from contextual encoders. Its central distinction is the level at which evidence is aggregated, rather than a ranking in which a newer architecture automatically solves every lexical task. Methods that inject lexical-resource relations into vectors connect directly to the WordNet section of this lesson.

For sentence- and text-level embeddings, Muennighoff et al.’s MTEB benchmark spans 8 embedding tasks, 58 datasets and 112 languages, and its authors find that no single text-embedding method dominates across all tasks [25]. In 2023–2026, most practical text embeddings come from pretrained Transformer encoders or language models fine-tuned with contrastive objectives of the kind shown above, and benchmark suites in the style of MTEB are the usual way to compare them. Static vectors remain useful as fast, transparent baselines, as features in low-resource settings, and as objects of study.

A useful research lens is therefore: what meaning distinction survives this representation, and how was that checked? Contextual hidden states, static vectors obtained by aggregating them, and task-trained sentence representations answer different questions. An embedding that preserves a broad topic may still collapse a negated claim or a rare sense. A small static table can remain an appropriate baseline when its measurable invariances match the application.

Open questions include reliable geometry across domains, sense separation when evidence is sparse, bias that survives debiasing, and comparisons that hold corpus and evaluation conditions fixed. Multilingual work adds segmentation and uneven coverage: a row is only useful if the tokenizer and training corpus supplied evidence for that type. Report those conditions and challenge examples alongside the final score. This keeps the explanation tied to observed language behavior.

Key takeaways

  • A vocabulary entry is a form or lemma, not a sense; lexical relations such as synonymy, antonymy and hypernymy hold between senses, and WordNet records them by hand.
  • A static embedding aggregates many occurrences into one vector per vocabulary entry; distributional similarity captures shared usage and mixes synonymy, antonymy and relatedness.
  • PPMI compares co-occurrence with independence; SVD, Skip-gram with negative sampling and GloVe all compress association statistics under a rank constraint.
  • CBOW predicts a center from combined neighbors; Skip-gram predicts neighbors separately; negative sampling distinguishes statistical sources of pairs and is not a normalized distribution.
  • fastText builds word vectors from character n-grams, so related spellings share parameters and unseen words get vectors.
  • Similarity and analogy tests depend on what the ratings mean and on details such as excluding the question words; evaluate the relation your application needs.
  • Embeddings absorb corpus associations, including social biases; projection-based debiasing can hide rather than remove them.
  • Mean pooling erases order; sentence embeddings should be trained for the comparison they serve, and contextual models compute a vector per occurrence rather than looking one up.

Exercises

  1. Build the examples. Enumerate radius-one CBOW and Skip-gram examples for robots help people today, using only interior centers. Then include the boundary centers. Which input shapes change?
Hint

Count centers separately from center–context pairs. Variable-length CBOW contexts need a mask or separate processing; Skip-gram still receives one center ID per pair.

  1. Inspect a matrix cell. Construct a corpus in which one context occurs frequently overall but has PMI near zero for a chosen center. Then add observed center–context pairs and explain both the numerator and marginal changes.
Hint

Begin with a joint table close to independence. Recompute all marginals after editing the count; keeping them fixed would analyze a different table.

  1. Train GloVe and watch it wobble. Implement the GloVe objective in PyTorch on the toy corpus, training only on nonzero cells. Train it with three random seeds and compare the nearest neighbors of cat. Then count the parameters and the nonzero cells. What explains the instability, and what would make the problem better determined?
Hint

Two V×DV\times D tables plus two bias vectors can easily outnumber the observed cells. A larger corpus with more varied contexts adds equations; a smaller DD removes unknowns.

  1. Similarity versus relatedness. Write ten word pairs that are related but not similar (such as cup and coffee) and ten that are similar. Using the PPMI code with window radius 1 and then 4 on a corpus you write, compare how the cosine separates the two groups.
Hint

Wider windows collect topical co-occurrence, which favors relatedness; narrow windows emphasize words that fill the same slot. Report the change for both groups, not a single score.

  1. Expose an invariant. Create two meaningful sentences with the same word counts and different participant roles. Show why static mean pooling cannot separate them, even if every word vector is learned perfectly for its training objective.
Hint

Changing order does not change a sum. The limitation holds for every possible static embedding table, not only for the synthetic values in this lesson.

  1. Audit beyond the projection. In the bias example, after projection, compute each occupation word’s nearest names using all remaining dimensions, and train a small logistic classifier to predict group A versus group B from the projected occupation-word neighbors. Does the association still leave a trace?
Hint

The projection removed one direction measured one way. Look for structure the score did not measure, as Gonen and Goldberg did with clustering and classification.

Paper guides

2 paper guides, each on a page of its own.

Open the paper guides →

References

  1. M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 3–4. publisher
  2. S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
  3. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
  4. G. A. Miller, “WordNet: a lexical database for English,” Communications of the ACM 38(11), 1995. paper
  5. Z. S. Harris, “Distributional structure,” Word 10(2–3), 1954. paper
  6. P. D. Turney and P. Pantel, “From frequency to meaning: vector space models of semantics,” Journal of Artificial Intelligence Research 37, 2010. survey
  7. T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space,” ICLR Workshop, 2013. paper
  8. T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26, 2013. paper
  9. O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in Neural Information Processing Systems 27, 2014. paper
  10. J. Pennington, R. Socher and C. D. Manning, “GloVe: global vectors for word representation,” EMNLP, 2014. paper
  11. P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics 5, 2017. paper
  12. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman and E. Ruppin, “Placing search in context: the concept revisited,” ACM Transactions on Information Systems 20(1), 2002. paper
  13. F. Hill, R. Reichart and A. Korhonen, “SimLex-999: evaluating semantic models with (genuine) similarity estimation,” Computational Linguistics 41(4), 2015. paper
  14. T. Mikolov, W.-t. Yih and G. Zweig, “Linguistic regularities in continuous space word representations,” NAACL-HLT, 2013. paper
  15. T. Linzen, “Issues in evaluating semantic spaces using word analogies,” Workshop on Evaluating Vector-Space Representations for NLP (RepEval), 2016. paper
  16. T. Schnabel, I. Labutov, D. Mimno and T. Joachims, “Evaluation methods for unsupervised word embeddings,” EMNLP, 2015. paper
  17. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings,” Advances in Neural Information Processing Systems 29, 2016. paper
  18. A. Caliskan, J. J. Bryson and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science 356(6334), 2017. paper
  19. H. Gonen and Y. Goldberg, “Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” NAACL-HLT, 2019. paper
  20. N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-networks,” EMNLP-IJCNLP, 2019. paper
  21. T. Gao, X. Yao and D. Chen, “SimCSE: simple contrastive learning of sentence embeddings,” EMNLP, 2021. paper
  22. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee and L. Zettlemoyer, “Deep contextualized word representations,” NAACL-HLT, 2018. paper
  23. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019. paper
  24. M. Apidianaki, “From word types to tokens and back: a survey of approaches to word meaning representation and interpretation,” Computational Linguistics 49(2), 2023. survey
  25. N. Muennighoff, N. Tazi, L. Magne and N. Reimers, “MTEB: massive text embedding benchmark,” EACL, 2023. paper

Check your understanding

This tutorial has 6 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next NLP 05 · Recurrent Neural Networks for Language · 31 min