完整搜尋頁面 →

word2vec — Efficient Estimation of Word Representations in Vector Space

選讀・NLP 04・詞彙語意與詞嵌入 的論文導讀(第 1/2 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

挑戰

替每個詞配置向量,仍需要訓練目標讓座標具有意義。Mikolov、Chen、Corrado 與 Dean 提出成本較低的預測目標,讓模型能從大量文字學習 [7]。論點是移除早期神經語言模型昂貴的非線性隱藏層,就能擴大詞表示的訓練規模。本導讀聚焦早期 CBOW/Skip-gram 論文;負取樣屬於後來的另一篇論文 [8]。先理解本章的分佈假說與 softmax。

方法

原論文 CBOW 由上下文預測詞,以及 Skip-gram 由詞預測上下文的圖
來源:Mikolov 等人,2013,圖 1,arXiv:1301.3781,PDF 第 5 頁。左側以周圍詞預測中心詞,右側以中心詞預測鄰近詞。箭頭表示預測方向,不是詞之間的因果關係。[7]

CBOW 聚合周圍詞的嵌入,再預測缺少的中心詞。聚合降低成本,也移除了上下文內的順序。Skip-gram 反過來,用中心詞分別預測選到的鄰近詞。其概念性目標是:

L=−∑t=1T∑j∈Ctlog⁡p(wt+j∣wt).\mathcal L=-\sum_{t=1}^{T}\sum_{j\in C_t}\log p(w_{t+j}\mid w_t).

TT 是 token 數,wtw_t 是中心 token,CtC_t 是選到且不超出文字範圍的非零相對位移集合。條件機率 pp 從中心詞嵌入預測上下文詞;L\mathcal L 是要最小化的負對數概似。論文用 Huffman 樹的階層式 softmax 降低預測成本,不能換成負取樣損失後仍稱為這篇的演算法。

以「the cat sleeps」為例,選「cat」作中心,就向選到的鄰近詞建立預測;CBOW 則組合鄰近詞來預測「cat」。Skip-gram 會取樣上下文半徑,因此較遠位置較少被納入。輸入與輸出角色使用不同參數;雖然查表始於離散 ID,學到的向量並不是 one-hot 識別碼。

結果

表 3 比較 640 維向量、3.2 億訓練詞與 82k 詞彙表的模型。CBOW 的語意、句法類比準確率分別為 24%、64%,Skip-gram 則為 55%、59%。預測方向會影響較擅長捕捉哪種關係;只說「最佳嵌入」會掩蓋任務差異。類比準確率要求完全命中預期詞,即使合理的同義詞也可能被判錯。

消融與設計證據

表 2 改變向量維度與訓練資料量,顯示單獨擴大其中一項,收益終究會縮小。這個實驗使用限於 30k 詞彙的類比子集,不能直接與其他完整測試分數相比。可學到的是讓表示容量與資料證據相配,而不是把加大維度當成保證改善。執行時間也受輸出近似方式及詞頻分佈影響。

結論

模型透過預測學習表示,而不是手動替座標指定語意。接著讀旁邊的 GloVe 導讀,比較局部預測目標與全域共現計數的加權擬合。繼續前,先解釋為何反轉 CBOW 的箭頭會改變訓練樣本,而不只是換一種畫法。

問題與限制

一個靜態向量必須兼顧同一詞的不同詞義,也會繼承語料中的關聯與偏差。類比測試只涵蓋下游效用的一小部分;向量位移或近鄰都不能證明普遍語言規則。報告結果時應同時列出語料、詞彙表、目標與評估流程;「word2vec」這個名稱不足以界定可重現的訓練設定。

參考文獻

  1. M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 3–4. publisher
  2. S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
  3. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
  4. G. A. Miller, “WordNet: a lexical database for English,” Communications of the ACM 38(11), 1995. paper
  5. Z. S. Harris, “Distributional structure,” Word 10(2–3), 1954. paper
  6. P. D. Turney and P. Pantel, “From frequency to meaning: vector space models of semantics,” Journal of Artificial Intelligence Research 37, 2010. survey
  7. T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space,” ICLR Workshop, 2013. paper
  8. T. Mikolov, I. Sutskever, K. Chen, G. Corrado and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in Neural Information Processing Systems 26, 2013. paper
  9. O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” Advances in Neural Information Processing Systems 27, 2014. paper
  10. J. Pennington, R. Socher and C. D. Manning, “GloVe: global vectors for word representation,” EMNLP, 2014. paper
  11. P. Bojanowski, E. Grave, A. Joulin and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics 5, 2017. paper
  12. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman and E. Ruppin, “Placing search in context: the concept revisited,” ACM Transactions on Information Systems 20(1), 2002. paper
  13. F. Hill, R. Reichart and A. Korhonen, “SimLex-999: evaluating semantic models with (genuine) similarity estimation,” Computational Linguistics 41(4), 2015. paper
  14. T. Mikolov, W.-t. Yih and G. Zweig, “Linguistic regularities in continuous space word representations,” NAACL-HLT, 2013. paper
  15. T. Linzen, “Issues in evaluating semantic spaces using word analogies,” Workshop on Evaluating Vector-Space Representations for NLP (RepEval), 2016. paper
  16. T. Schnabel, I. Labutov, D. Mimno and T. Joachims, “Evaluation methods for unsupervised word embeddings,” EMNLP, 2015. paper
  17. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings,” Advances in Neural Information Processing Systems 29, 2016. paper
  18. A. Caliskan, J. J. Bryson and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science 356(6334), 2017. paper
  19. H. Gonen and Y. Goldberg, “Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” NAACL-HLT, 2019. paper
  20. N. Reimers and I. Gurevych, “Sentence-BERT: sentence embeddings using Siamese BERT-networks,” EMNLP-IJCNLP, 2019. paper
  21. T. Gao, X. Yao and D. Chen, “SimCSE: simple contrastive learning of sentence embeddings,” EMNLP, 2021. paper
  22. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee and L. Zettlemoyer, “Deep contextualized word representations,” NAACL-HLT, 2018. paper
  23. J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019. paper
  24. M. Apidianaki, “From word types to tokens and back: a survey of approaches to word meaning representation and interpretation,” Computational Linguistics 49(2), 2023. survey
  25. N. Muennighoff, N. Tazi, L. Magne and N. Reimers, “MTEB: massive text embedding benchmark,” EACL, 2023. paper