word2vec — Efficient Estimation of Word Representations in Vector Space
選讀・NLP 04・詞彙語意與詞嵌入 的論文導讀(第 1/2 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
挑戰
替每個詞配置向量,仍需要訓練目標讓座標具有意義。Mikolov、Chen、Corrado 與 Dean 提出成本較低的預測目標,讓模型能從大量文字學習 [7]。論點是移除早期神經語言模型昂貴的非線性隱藏層,就能擴大詞表示的訓練規模。本導讀聚焦早期 CBOW/Skip-gram 論文;負取樣屬於後來的另一篇論文 [8]。先理解本章的分佈假說與 softmax。
方法

CBOW 聚合周圍詞的嵌入,再預測缺少的中心詞。聚合降低成本,也移除了上下文內的順序。Skip-gram 反過來,用中心詞分別預測選到的鄰近詞。其概念性目標是:
是 token 數, 是中心 token, 是選到且不超出文字範圍的非零相對位移集合。條件機率 從中心詞嵌入預測上下文詞; 是要最小化的負對數概似。論文用 Huffman 樹的階層式 softmax 降低預測成本,不能換成負取樣損失後仍稱為這篇的演算法。
以「the cat sleeps」為例,選「cat」作中心,就向選到的鄰近詞建立預測;CBOW 則組合鄰近詞來預測「cat」。Skip-gram 會取樣上下文半徑,因此較遠位置較少被納入。輸入與輸出角色使用不同參數;雖然查表始於離散 ID,學到的向量並不是 one-hot 識別碼。
結果
表 3 比較 640 維向量、3.2 億訓練詞與 82k 詞彙表的模型。CBOW 的語意、句法類比準確率分別為 24%、64%,Skip-gram 則為 55%、59%。預測方向會影響較擅長捕捉哪種關係;只說「最佳嵌入」會掩蓋任務差異。類比準確率要求完全命中預期詞,即使合理的同義詞也可能被判錯。
消融與設計證據
表 2 改變向量維度與訓練資料量,顯示單獨擴大其中一項,收益終究會縮小。這個實驗使用限於 30k 詞彙的類比子集,不能直接與其他完整測試分數相比。可學到的是讓表示容量與資料證據相配,而不是把加大維度當成保證改善。執行時間也受輸出近似方式及詞頻分佈影響。
結論
模型透過預測學習表示,而不是手動替座標指定語意。接著讀旁邊的 GloVe 導讀,比較局部預測目標與全域共現計數的加權擬合。繼續前,先解釋為何反轉 CBOW 的箭頭會改變訓練樣本,而不只是換一種畫法。
問題與限制
一個靜態向量必須兼顧同一詞的不同詞義,也會繼承語料中的關聯與偏差。類比測試只涵蓋下游效用的一小部分;向量位移或近鄰都不能證明普遍語言規則。報告結果時應同時列出語料、詞彙表、目標與評估流程;「word2vec」這個名稱不足以界定可重現的訓練設定。
參考文獻
- M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 3–4. publisher
- S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
- Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
- G. A. Miller, “WordNet: a lexical database for English,” Communications of the ACM 38(11), 1995. paper
- Z. S. Harris, “Distributional structure,” Word 10(2–3), 1954. paper
- P. D. Turney and P. Pantel, “From frequency to meaning: vector space models of semantics,” Journal of Artificial Intelligence Research 37, 2010. survey
- T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space,” ICLR Workshop, 2013. paper
- T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26, 2013. paper
- O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in Neural Information Processing Systems 27, 2014. paper
- J. Pennington, R. Socher and C. D. Manning, “GloVe: global vectors for word representation,” EMNLP, 2014. paper
- P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics 5, 2017. paper
- L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman and E. Ruppin, “Placing search in context: the concept revisited,” ACM Transactions on Information Systems 20(1), 2002. paper
- F. Hill, R. Reichart and A. Korhonen, “SimLex-999: evaluating semantic models with (genuine) similarity estimation,” Computational Linguistics 41(4), 2015. paper
- T. Mikolov, W.-t. Yih and G. Zweig, “Linguistic regularities in continuous space word representations,” NAACL-HLT, 2013. paper
- T. Linzen, “Issues in evaluating semantic spaces using word analogies,” Workshop on Evaluating Vector-Space Representations for NLP (RepEval), 2016. paper
- T. Schnabel, I. Labutov, D. Mimno and T. Joachims, “Evaluation methods for unsupervised word embeddings,” EMNLP, 2015. paper
- T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings,” Advances in Neural Information Processing Systems 29, 2016. paper
- A. Caliskan, J. J. Bryson and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science 356(6334), 2017. paper
- H. Gonen and Y. Goldberg, “Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” NAACL-HLT, 2019. paper
- N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-networks,” EMNLP-IJCNLP, 2019. paper
- T. Gao, X. Yao and D. Chen, “SimCSE: simple contrastive learning of sentence embeddings,” EMNLP, 2021. paper
- M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee and L. Zettlemoyer, “Deep contextualized word representations,” NAACL-HLT, 2018. paper
- J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019. paper
- M. Apidianaki, “From word types to tokens and back: a survey of approaches to word meaning representation and interpretation,” Computational Linguistics 49(2), 2023. survey
- N. Muennighoff, N. Tazi, L. Magne and N. Reimers, “MTEB: massive text embedding benchmark,” EACL, 2023. paper