Full search page →

word2vec — Efficient Estimation of Word Representations in Vector Space

Optional · paper guide 1 of 2 for NLP 04 · Lexical Semantics and Word Embeddings. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

A vector for every word is useful only if a training objective gives those coordinates meaning. Mikolov, Chen, Corrado and Dean propose inexpensive predictive objectives that can learn from large text collections [7]. The thesis is that removing the costly nonlinear hidden layer of earlier neural language models permits useful word representations at scale. This guide covers the early CBOW/Skip-gram paper; negative sampling belongs to the later paper [8]. First understand the chapter’s distributional hypothesis and softmax.

Method

Original CBOW context-to-word and Skip-gram word-to-context diagrams
Source: Mikolov et al., 2013, Figure 1, arXiv:1301.3781, PDF p. 5. Left: surrounding words predict the center word. Right: the center word predicts its neighbors. Arrows denote prediction direction, not causal relations between words. [7]

CBOW pools the surrounding embeddings and predicts the missing center word. Pooling makes it inexpensive but removes order within that context. Skip-gram reverses the prediction: use one center word to predict each selected nearby word. Its conceptual objective is

L=−∑t=1T∑j∈Ctlog⁡p(wt+j∣wt).\mathcal L=-\sum_{t=1}^{T}\sum_{j\in C_t}\log p(w_{t+j}\mid w_t).

Here TT is the token count, wtw_t the center token, and CtC_t the selected nonzero relative offsets whose positions remain inside the text. The conditional probability pp predicts a context word from the center embedding; L\mathcal L is the negative log-likelihood to minimize. The paper implements efficient prediction with hierarchical softmax using a Huffman tree. Do not substitute a negative-sampling loss and label it this paper’s algorithm.

For a short sequence “the cat sleeps,” using “cat” as center creates predictions toward its selected neighbors. CBOW instead combines neighbors to predict “cat.” Skip-gram samples a context radius, so distant positions are included less often. Its input and output roles have different parameters; a learned vector is not a one-hot identifier, even though the token lookup begins with a discrete ID.

Results

Table 3 compares models with 640-dimensional vectors, 320 million training words and an 82k vocabulary. CBOW obtains 24% semantic and 64% syntactic analogy accuracy; Skip-gram obtains 55% and 59%. The direction of prediction changes which relationships are captured well. A single “best embedding” label hides this task dependence. Analogy accuracy requires the expected exact word, so an otherwise plausible synonym can still count as an error.

Ablations

Table 2 varies dimensionality and training data, showing that increasing either in isolation eventually yields smaller returns. That study uses an analogy subset restricted to a 30k vocabulary; it should not be compared directly with full-test scores elsewhere. The clean lesson is to balance representation capacity with available evidence, not to treat a larger dimension as a guaranteed improvement. Runtime also depends on the output approximation and vocabulary frequency distribution.

Conclusion

The model learns representations through prediction rather than hand-written semantic coordinates. Read the adjacent GloVe guide next to compare a local predictive objective with weighted fitting of global co-occurrence counts. Before moving on, explain why swapping CBOW’s arrows changes the training examples, rather than merely changing a diagram.

Problems

One static vector must represent a word’s different senses. The model also inherits associations and biases in the corpus, and analogy tests are a narrow proxy for downstream usefulness. Neither a vector offset nor a nearby neighbor proves a universal linguistic rule. Report corpus, vocabulary, objective and evaluation protocol together; “word2vec” alone does not identify a reproducible training setup.

References

  1. M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 3–4. publisher
  2. S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
  3. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
  4. G. A. Miller, “WordNet: a lexical database for English,” Communications of the ACM 38(11), 1995. paper
  5. Z. S. Harris, “Distributional structure,” Word 10(2–3), 1954. paper
  6. P. D. Turney and P. Pantel, “From frequency to meaning: vector space models of semantics,” Journal of Artificial Intelligence Research 37, 2010. survey
  7. T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space,” ICLR Workshop, 2013. paper
  8. T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26, 2013. paper
  9. O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in Neural Information Processing Systems 27, 2014. paper
  10. J. Pennington, R. Socher and C. D. Manning, “GloVe: global vectors for word representation,” EMNLP, 2014. paper
  11. P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics 5, 2017. paper
  12. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman and E. Ruppin, “Placing search in context: the concept revisited,” ACM Transactions on Information Systems 20(1), 2002. paper
  13. F. Hill, R. Reichart and A. Korhonen, “SimLex-999: evaluating semantic models with (genuine) similarity estimation,” Computational Linguistics 41(4), 2015. paper
  14. T. Mikolov, W.-t. Yih and G. Zweig, “Linguistic regularities in continuous space word representations,” NAACL-HLT, 2013. paper
  15. T. Linzen, “Issues in evaluating semantic spaces using word analogies,” Workshop on Evaluating Vector-Space Representations for NLP (RepEval), 2016. paper
  16. T. Schnabel, I. Labutov, D. Mimno and T. Joachims, “Evaluation methods for unsupervised word embeddings,” EMNLP, 2015. paper
  17. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings,” Advances in Neural Information Processing Systems 29, 2016. paper
  18. A. Caliskan, J. J. Bryson and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science 356(6334), 2017. paper
  19. H. Gonen and Y. Goldberg, “Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” NAACL-HLT, 2019. paper
  20. N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-networks,” EMNLP-IJCNLP, 2019. paper
  21. T. Gao, X. Yao and D. Chen, “SimCSE: simple contrastive learning of sentence embeddings,” EMNLP, 2021. paper
  22. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee and L. Zettlemoyer, “Deep contextualized word representations,” NAACL-HLT, 2018. paper
  23. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019. paper
  24. M. Apidianaki, “From word types to tokens and back: a survey of approaches to word meaning representation and interpretation,” Computational Linguistics 49(2), 2023. survey
  25. N. Muennighoff, N. Tazi, L. Magne and N. Reimers, “MTEB: massive text embedding benchmark,” EACL, 2023. paper