Full search page →

Sentence-BERT — Sentence Embeddings using Siamese BERT-Networks

Optional · paper guide 1 of 1 for NLP 11 · Classification with BERT (Sentiment Analysis, Natural Language Inference). Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

A pairwise cross-encoder must jointly process each candidate sentence pair, making large semantic searches expensive. Reimers and Gurevych ask how to produce independent sentence vectors that can be compared efficiently [20]. Their thesis is that a shared encoder, trained with suitable sentence-pair objectives, yields a more useful cosine-similarity space than naïvely pooling pretrained BERT. First understand this chapter’s bi-encoder/cross-encoder distinction, pooling and cosine similarity.

Method

Original SBERT shared encoders with a training classifier on the left and cosine scoring on the right
Source: Reimers and Gurevych, EMNLP-IJCNLP 2019, Figures 1–2, PDF p. 3. Left: a sentence-pair classification training head. Right: independent sentence vectors compared by cosine at inference; this path also supports similarity-regression training. Both towers share encoder weights. [20]

Encode each sentence with the same BERT parameters and pool its token states, typically by a mean, to obtain one vector. For natural-language-inference training, concatenate the two vectors and their elementwise absolute difference before a softmax classifier. At retrieval time, discard that classification head and compare the vectors directly:

z=[u;v;∣u−v∣],sim⁡(u,v)=u⊤v∥u∥2∥v∥2.z=[u;v;|u-v|],\qquad \operatorname{sim}(u,v)=\frac{u^\top v}{\|u\|_2\|v\|_2}.

Here uu and vv are the pooled sentence embeddings, zz the training classifier’s input, semicolons denote concatenation, and ∣u−v∣|u-v| is the coordinatewise absolute difference. The function sim⁡\operatorname{sim} is cosine similarity, defined for nonzero vectors; ∥⋅∥2\|\cdot\|_2 denotes Euclidean norm. The classifier and cosine scorer are different functions, so a classification probability is not the retrieval score.

Figure 1 explains how supervised sentence-pair labels shape the shared encoder. Figure 2 explains the efficiency benefit: corpus sentence embeddings can be computed and stored in advance. A query needs its own encoding followed by vector search. This avoids a fresh joint encoder pass for every query–candidate pair; it does not make all pairwise comparisons or indexing costs vanish.

Results

Table 1’s NLI-trained SBERT-base obtains 77.03 on STS-B and a 74.89 average across the listed seven semantic-similarity tasks, measured as Spearman correlation multiplied by 100. Mean-pooled BERT obtains 46.35 and 54.81 respectively. The table labels this an “unsupervised” similarity evaluation because it does not train on the evaluated STS datasets; SBERT still uses supervised NLI training. Do not describe these vectors as learned without labels.

Ablations

Table 6 uses STS-B development data and averages ten seeds. In the NLI-trained setting, mean, max and special-token pooling obtain 80.78, 79.07 and 79.80 respectively. Concatenating only the two sentence vectors gives 66.04, versus 80.78 when including their absolute difference. The table’s separately STS-trained regression setting behaves differently, so pooling results must remain attached to their training objective. These development scores are not interchangeable with Table 1’s evaluation scores.

Conclusion

Training a task head can shape a representation that remains useful after the head is removed. For a practical reading check, list which computations can be cached for a fixed document collection and which must run for each new query. Return to this chapter’s retrieval discussion to choose between efficient candidate retrieval and a more expensive joint reranker.

Problems

Cosine similarity is neither a calibrated probability nor a guarantee of entailment or factual agreement. Negation and role reversal can remain difficult despite high lexical overlap. Domain, language and sentence length affect transfer, while truncation can remove decisive evidence. Evaluate retrieval relevance on the intended collection, not only sentence-similarity correlation, and remember that approximate indexes introduce their own recall–latency tradeoff.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapters 9–11, Manning, 2021. Publisher
  2. Susan McRoy, Principles of Natural Language Processing, Chapters 6 and 8, open textbook, 2021. Open textbook
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL-HLT, 2019. Paper
  4. Google Research, BERT: original implementation, software, 2018. Classification, Question answering, Optimizer
  5. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng and Christopher Potts, “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,” EMNLP, 2013. Paper
  6. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng and Christopher Potts, “Learning Word Vectors for Sentiment Analysis,” ACL-HLT, 2011. Paper
  7. Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning, “A large annotated corpus for learning natural language inference,” EMNLP, 2015. Paper
  8. Adina Williams, Nikita Nangia and Samuel R. Bowman, “A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference,” NAACL-HLT, 2018. Paper
  9. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman and Noah A. Smith, “Annotation Artifacts in Natural Language Inference Data,” NAACL-HLT, 2018. Paper
  10. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger and Benjamin Van Durme, “Hypothesis Only Baselines in Natural Language Inference,” *SEM, 2018. Paper
  11. R. Thomas McCoy, Ellie Pavlick and Tal Linzen, “Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference,” ACL, 2019. Paper
  12. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev and Percy Liang, “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” EMNLP, 2016. Paper
  13. Pranav Rajpurkar, Robin Jia and Percy Liang, “Know What You Don’t Know: Unanswerable Questions for SQuAD,” ACL, 2018. Paper
  14. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman, “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding,” ICLR, 2019. Paper
  15. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman, “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems,” NeurIPS, 2019. Paper
  16. Marius Mosbach, Maksym Andriushchenko and Dietrich Klakow, “On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines,” ICLR, 2021. Paper
  17. Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. Paper
  18. Shrey Desai and Greg Durrett, “Calibration of Pre-trained Transformers,” EMNLP, 2020. Paper
  19. scikit-learn developers, “Classification metrics,” scikit-learn documentation, current online edition. Documentation
  20. Nils Reimers and Iryna Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” EMNLP-IJCNLP, 2019. Paper
  21. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu and Jianfeng Gao, “Deep Learning–based Text Classification: A Comprehensive Review,” ACM Computing Surveys 54(3), 2021. Survey
  22. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan and Lidong Bing, “Sentiment Analysis in the Era of Large Language Models: A Reality Check,” Findings of NAACL, 2024. Paper
  23. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini et al., “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference,” ACL, 2025. Paper
  24. Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Survey
  25. Wayne Xin Zhao et al., “A Survey of Large Language Models,” Frontiers of Computer Science, 2026. Updated survey
  26. Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. Paper