Full search page →

GloVe — Global Vectors for Word Representation

Optional · paper guide 2 of 2 for NLP 04 · Lexical Semantics and Word Embeddings. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

Predictive embeddings learn from local windows, while count-based methods explicitly summarize corpus statistics. Pennington, Socher and Manning ask how to retain global co-occurrence information in a trainable low-dimensional representation [10]. Their thesis is that ratios of co-occurrence probabilities reveal useful distinctions, motivating a weighted fit to log counts. Read the chapter’s co-occurrence matrix and the word2vec guide first.

Method

Original GloVe weighting function rising to a capped value of one
Source: Pennington et al., EMNLP 2014, Figure 1, PDF p. 4. Rare co-occurrences receive small weights; the weight saturates at the count threshold. This objective has no necessary encoder-decoder architecture, so its original weighting diagram is the useful figure. [10]

First build a word–context co-occurrence matrix with a chosen context window. The motivation compares how often a probe word appears around two target words: a probability ratio can distinguish a relevant probe from one that is equally associated with both. GloVe turns that motivation into a practical regression objective:

J=∑i,j:Xij>0f(Xij)(wi⊤w~j+bi+b~j−log⁡Xij)2,f(x)=min⁡((x/xmax⁡)α,1).J=\sum_{i,j:X_{ij}>0} f(X_{ij})\left(w_i^\top\tilde w_j+b_i+\tilde b_j-\log X_{ij}\right)^2, \qquad f(x)=\min\left((x/x_{\max})^\alpha,1\right).

XijX_{ij} is the co-occurrence count for target word ii and context word jj. The vectors wi,w~jw_i,\tilde w_j represent their two roles; bi,b~jb_i,\tilde b_j are learned scalar biases. JJ is the training loss, ff the confidence weight, xmax⁡x_{\max} its saturation threshold and α\alpha its exponent. The paper uses xmax⁡=100x_{\max}=100 and α=3/4\alpha=3/4. Only nonzero entries contribute, avoiding an undefined logarithm at zero.

The dot product and biases approximate the log count, not a normalized conditional probability. Large raw counts do not grow the weight without bound. In Figure 1, moving right of the threshold leaves the weight at one, although the regression target still changes with the count. Training therefore combines corpus-wide statistics with repeated updates to compact vectors rather than storing a dense word-by-word matrix as the final representation.

Results

Table 2 reports 71.7% total analogy accuracy for 300-dimensional GloVe trained on 6 billion tokens, compared with 69.1% for the table’s matched 6-billion-token Skip-gram run. GloVe’s 42-billion-token result is 75.0%, but that comparison also changes the corpus and data amount. It is evidence for that larger training setup, not an isolated gain from the objective. Analogy accuracy remains distinct from sentence classification or understanding.

Ablations

Figure 2 varies vector dimension and context-window choices on the same 6-billion-token corpus. These studies help separate capacity and the definition of “context” from simply adding data. The paper also examines training progress and corpus effects. For a careful reproduction, record whether the window is symmetric, how distance affects co-occurrence counts, and how many optimization passes were used; all affect the matrix or the fit.

Conclusion

GloVe makes the relation between an observed statistic and a learned geometry explicit. Ask what happens to one training term when its count doubles: both the log target and, below the threshold, its weight change. Continue to NLP 05 to see how a sequence model adds context-dependent states on top of word inputs.

Problems

A single vector still conflates senses, and missing or rare words have little evidence. Building and storing sparse co-occurrence counts has a cost of its own. Corpus bias and window design shape the geometry. A lower regression loss does not establish better performance on every language task; use downstream evaluation and keep data and vocabulary comparable when testing competing embeddings.

References

  1. M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 3–4. publisher
  2. S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
  3. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
  4. G. A. Miller, “WordNet: a lexical database for English,” Communications of the ACM 38(11), 1995. paper
  5. Z. S. Harris, “Distributional structure,” Word 10(2–3), 1954. paper
  6. P. D. Turney and P. Pantel, “From frequency to meaning: vector space models of semantics,” Journal of Artificial Intelligence Research 37, 2010. survey
  7. T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space,” ICLR Workshop, 2013. paper
  8. T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26, 2013. paper
  9. O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in Neural Information Processing Systems 27, 2014. paper
  10. J. Pennington, R. Socher and C. D. Manning, “GloVe: global vectors for word representation,” EMNLP, 2014. paper
  11. P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics 5, 2017. paper
  12. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman and E. Ruppin, “Placing search in context: the concept revisited,” ACM Transactions on Information Systems 20(1), 2002. paper
  13. F. Hill, R. Reichart and A. Korhonen, “SimLex-999: evaluating semantic models with (genuine) similarity estimation,” Computational Linguistics 41(4), 2015. paper
  14. T. Mikolov, W.-t. Yih and G. Zweig, “Linguistic regularities in continuous space word representations,” NAACL-HLT, 2013. paper
  15. T. Linzen, “Issues in evaluating semantic spaces using word analogies,” Workshop on Evaluating Vector-Space Representations for NLP (RepEval), 2016. paper
  16. T. Schnabel, I. Labutov, D. Mimno and T. Joachims, “Evaluation methods for unsupervised word embeddings,” EMNLP, 2015. paper
  17. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings,” Advances in Neural Information Processing Systems 29, 2016. paper
  18. A. Caliskan, J. J. Bryson and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science 356(6334), 2017. paper
  19. H. Gonen and Y. Goldberg, “Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” NAACL-HLT, 2019. paper
  20. N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-networks,” EMNLP-IJCNLP, 2019. paper
  21. T. Gao, X. Yao and D. Chen, “SimCSE: simple contrastive learning of sentence embeddings,” EMNLP, 2021. paper
  22. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee and L. Zettlemoyer, “Deep contextualized word representations,” NAACL-HLT, 2018. paper
  23. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019. paper
  24. M. Apidianaki, “From word types to tokens and back: a survey of approaches to word meaning representation and interpretation,” Computational Linguistics 49(2), 2023. survey
  25. N. Muennighoff, N. Tazi, L. Magne and N. Reimers, “MTEB: massive text embedding benchmark,” EACL, 2023. paper