GloVe — Global Vectors for Word Representation
Optional · paper guide 2 of 2 for NLP 04 · Lexical Semantics and Word Embeddings. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
Predictive embeddings learn from local windows, while count-based methods explicitly summarize corpus statistics. Pennington, Socher and Manning ask how to retain global co-occurrence information in a trainable low-dimensional representation [10]. Their thesis is that ratios of co-occurrence probabilities reveal useful distinctions, motivating a weighted fit to log counts. Read the chapter’s co-occurrence matrix and the word2vec guide first.
Method

First build a word–context co-occurrence matrix with a chosen context window. The motivation compares how often a probe word appears around two target words: a probability ratio can distinguish a relevant probe from one that is equally associated with both. GloVe turns that motivation into a practical regression objective:
is the co-occurrence count for target word and context word . The vectors represent their two roles; are learned scalar biases. is the training loss, the confidence weight, its saturation threshold and its exponent. The paper uses and . Only nonzero entries contribute, avoiding an undefined logarithm at zero.
The dot product and biases approximate the log count, not a normalized conditional probability. Large raw counts do not grow the weight without bound. In Figure 1, moving right of the threshold leaves the weight at one, although the regression target still changes with the count. Training therefore combines corpus-wide statistics with repeated updates to compact vectors rather than storing a dense word-by-word matrix as the final representation.
Results
Table 2 reports 71.7% total analogy accuracy for 300-dimensional GloVe trained on 6 billion tokens, compared with 69.1% for the table’s matched 6-billion-token Skip-gram run. GloVe’s 42-billion-token result is 75.0%, but that comparison also changes the corpus and data amount. It is evidence for that larger training setup, not an isolated gain from the objective. Analogy accuracy remains distinct from sentence classification or understanding.
Ablations
Figure 2 varies vector dimension and context-window choices on the same 6-billion-token corpus. These studies help separate capacity and the definition of “context” from simply adding data. The paper also examines training progress and corpus effects. For a careful reproduction, record whether the window is symmetric, how distance affects co-occurrence counts, and how many optimization passes were used; all affect the matrix or the fit.
Conclusion
GloVe makes the relation between an observed statistic and a learned geometry explicit. Ask what happens to one training term when its count doubles: both the log target and, below the threshold, its weight change. Continue to NLP 05 to see how a sequence model adds context-dependent states on top of word inputs.
Problems
A single vector still conflates senses, and missing or rare words have little evidence. Building and storing sparse co-occurrence counts has a cost of its own. Corpus bias and window design shape the geometry. A lower regression loss does not establish better performance on every language task; use downstream evaluation and keep data and vocabulary comparable when testing competing embeddings.
References
- M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 3–4. publisher
- S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
- Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
- G. A. Miller, “WordNet: a lexical database for English,” Communications of the ACM 38(11), 1995. paper
- Z. S. Harris, “Distributional structure,” Word 10(2–3), 1954. paper
- P. D. Turney and P. Pantel, “From frequency to meaning: vector space models of semantics,” Journal of Artificial Intelligence Research 37, 2010. survey
- T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space,” ICLR Workshop, 2013. paper
- T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26, 2013. paper
- O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in Neural Information Processing Systems 27, 2014. paper
- J. Pennington, R. Socher and C. D. Manning, “GloVe: global vectors for word representation,” EMNLP, 2014. paper
- P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics 5, 2017. paper
- L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman and E. Ruppin, “Placing search in context: the concept revisited,” ACM Transactions on Information Systems 20(1), 2002. paper
- F. Hill, R. Reichart and A. Korhonen, “SimLex-999: evaluating semantic models with (genuine) similarity estimation,” Computational Linguistics 41(4), 2015. paper
- T. Mikolov, W.-t. Yih and G. Zweig, “Linguistic regularities in continuous space word representations,” NAACL-HLT, 2013. paper
- T. Linzen, “Issues in evaluating semantic spaces using word analogies,” Workshop on Evaluating Vector-Space Representations for NLP (RepEval), 2016. paper
- T. Schnabel, I. Labutov, D. Mimno and T. Joachims, “Evaluation methods for unsupervised word embeddings,” EMNLP, 2015. paper
- T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings,” Advances in Neural Information Processing Systems 29, 2016. paper
- A. Caliskan, J. J. Bryson and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science 356(6334), 2017. paper
- H. Gonen and Y. Goldberg, “Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” NAACL-HLT, 2019. paper
- N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-networks,” EMNLP-IJCNLP, 2019. paper
- T. Gao, X. Yao and D. Chen, “SimCSE: simple contrastive learning of sentence embeddings,” EMNLP, 2021. paper
- M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee and L. Zettlemoyer, “Deep contextualized word representations,” NAACL-HLT, 2018. paper
- J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019. paper
- M. Apidianaki, “From word types to tokens and back: a survey of approaches to word meaning representation and interpretation,” Computational Linguistics 49(2), 2023. survey
- N. Muennighoff, N. Tazi, L. Magne and N. Reimers, “MTEB: massive text embedding benchmark,” EACL, 2023. paper