BERT — Pre-training of Deep Bidirectional Transformers for Language Understanding
Optional · paper guide 1 of 1 for NLP 10 · Pretrained Models: BERT. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
A left-to-right language model cannot use future words when forming the current token’s representation. Devlin et al. propose pretraining a deeply bidirectional Transformer encoder and adapting it to many understanding tasks [1]. Their thesis is that masked-token prediction supplies bidirectional training without simply exposing each token as its own answer. First understand encoder self-attention, token representations and task heads in this chapter.
Method

BERT selects 15% of token positions for masked language modeling (MLM). Of those selected positions, 80% receive a mask token, 10% a random token and 10% remain unchanged. These percentages describe the selected subset, not three fractions of the whole sentence. A compact form of the objective is
Here is the original token at position , the selected prediction positions, the corrupted input sequence, and the encoder and prediction head’s token distribution with parameters . The loss supervises selected positions, including the selected tokens left unchanged. It does not require predicting every unmasked input token.
The original recipe also uses next-sentence prediction (NSP): half the pairs continue the same document and half use a sampled random second segment. Input token, segment and position embeddings feed the same encoder. During fine-tuning, a classification head can read the special sentence-level representation, while token or span heads read token states. Figure 1 depicts shared pretrained structure with different output tasks, not a requirement to freeze the encoder.
Results
Table 5 provides a useful, tightly identified evidence set: the BERT-base development evaluation reaches 84.4 MNLI accuracy, 88.4 QNLI accuracy and 88.5 SQuAD F1. These are base-model development results, not the headline large-model test submissions. Read each benchmark’s metric separately; classification accuracy and answer-span F1 cannot be interpreted as one common unit of language understanding. The paper’s broader task tables establish transfer across several tasks with adapted output heads.
Ablations
In Table 5, removing NSP lowers QNLI accuracy from 88.4 to 84.9 and SQuAD F1 from 88.5 to 87.9. Switching to left-to-right pretraining without NSP yields 77.8 SQuAD F1; adding a randomly initialized bidirectional LSTM during fine-tuning recovers only part of the difference, to 84.9. These comparisons support bidirectional pretraining in the original recipe. They do not establish that NSP is necessary for every later training corpus, schedule or encoder design.
Conclusion
BERT turns one pretrained encoder into several task-specific predictors. To check understanding, mark which input positions are corrupted, which positions incur MLM loss, and which parameters update at fine-tuning. Continue to the Sentence-BERT guide in NLP 11 to see why useful token representations do not automatically produce good sentence similarity vectors.
Problems
MLM is not a left-to-right sequence-probability factorization, so ordinary autoregressive perplexities are not directly comparable. Random-negative NSP can reward topic cues rather than rich discourse understanding. Pretraining data, tokenization and context limits constrain transfer, while supervised fine-tuning still needs representative labels. Bidirectionality gives access to context; it does not guarantee reasoning, factuality or robustness to a changed domain.
References
- Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” NAACL, 2019. Paper
- Google Research, BERT: original implementation and released models, software, 2018. Encoder and pooler · Tokenizer · Multilingual and Chinese models
- Ashish Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
- Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI technical report, 2018. Paper
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer, “Deep Contextualized Word Representations,” NAACL, 2018. Paper
- Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space,” ICLR Workshop, 2013. Paper
- Yonghui Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” arXiv:1609.08144, 2016. Paper
- Dan Hendrycks and Kevin Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016, revised 2023. Paper
- Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
- Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. Paper
- Yinhan Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv:1907.11692, 2019. Paper
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma and Radu Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” ICLR, 2020. Paper
- Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing at NeurIPS, 2019. Paper
- Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning, “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,” ICLR, 2020. Paper
- Telmo Pires, Eva Schlinger and Dan Garrette, “How Multilingual is Multilingual BERT?,” ACL, 2019. Paper
- Alexis Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale,” ACL, 2020. Paper
- Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin and Ziqing Yang, “Pre-Training With Whole Word Masking for Chinese BERT,” IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, 2021. Publisher · Author manuscript
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapter 9, Manning, 2021. Publisher
- Kawin Ethayarajh, “How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings,” EMNLP-IJCNLP, 2019. Paper
- Ian Tenney, Dipanjan Das and Ellie Pavlick, “BERT Rediscovers the Classical NLP Pipeline,” ACL, 2019. Paper
- John Hewitt and Christopher D. Manning, “A Structural Probe for Finding Syntax in Word Representations,” NAACL, 2019. Paper
- John Hewitt and Percy Liang, “Designing and Interpreting Probes with Control Tasks,” EMNLP-IJCNLP, 2019. Paper
- Kevin Clark, Urvashi Khandelwal, Omer Levy and Christopher D. Manning, “What Does BERT Look at? An Analysis of BERT’s Attention,” BlackboxNLP Workshop, 2019. Paper
- Anna Rogers, Olga Kovaleva and Anna Rumshisky, “A Primer in BERTology: What We Know About How BERT Works,” Transactions of the Association for Computational Linguistics 8, 2020. Paper
- Benjamin Warner et al., “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference,” ACL, 2025. Paper
- Wayne Xin Zhao et al., “A Survey of Large Language Models,” Frontiers of Computer Science, 2026. Updated survey
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” JMLR 21(140), 2020. Paper