BiLSTM-CRF — Bidirectional LSTM-CRF Models for Sequence Tagging
Optional · paper guide 1 of 1 for NLP 06 · Sequence Labeling with RNNs (POS Tagging, Information Extraction). Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
An independently predicted token label may look plausible while the whole label sequence is inconsistent. Huang, Xu and Yu combine context-sensitive recurrent features with a globally normalized sequence model [13]. Their thesis separates two jobs: a bidirectional LSTM reads the sentence, and a CRF scores neighboring label decisions together. Read the chapter’s BIO tags, emissions, transition scores and dynamic programming first. This guide concerns the 2015 manuscript, including its engineered input features.
Method

The forward and backward LSTMs let each token’s representation depend on both sides of the sentence. These representations produce emission scores for candidate labels. A linear-chain CRF adds transition scores and normalizes over complete label paths:
Here is the input sentence, its length, a label path, the emission score for label at position , and a learned transition score from label to . The symbol denotes a start state; boundary terms can be included similarly. is the set of candidate paths and indexes a path in the denominator. Scores are unnormalized real numbers, not token probabilities.
Training maximizes the correct path’s conditional likelihood. A forward dynamic program computes the sum over paths efficiently; Viterbi decoding instead finds the highest-scoring path. Summation and maximization are different operations. A learned transition matrix can discourage invalid BIO transitions but does not guarantee hard validity unless the implementation explicitly masks them.
Results
Table 2’s pretrained Senna-embedding BiLSTM-CRF row reports 97.55 POS accuracy, 94.46 chunking F1 and 88.83 NER F1. The parenthesized 90.10 NER F1 additionally uses gazetteer features. Those two NER numbers are not the same input setting. POS accuracy counts token decisions, whereas chunking and NER F1 evaluate extracted spans; their numerical values should not be ranked as though they measure the same thing.
Ablations
With random embeddings, the corresponding BiLSTM-CRF row obtains 97.43, 94.13 and 84.26. Comparing these rows reveals a much larger pretraining effect for NER in this setup. The architecture comparisons distinguish recurrent directionality from adding a CRF. Feature experiments also matter: removing spelling and context features changes chunking F1 from 94.46 to 94.40. This paper is not evidence that all hand-crafted features have already disappeared from the pipeline.
Conclusion
Use this model to understand why contextual token evidence and structured output constraints are complementary. Given two competing BIO paths, identify which difference comes from emissions and which comes from transitions. Continue to NLP 12 for structures beyond an adjacent label chain, including spans and coreference links.
Problems
Bidirectional features require future input, so this model is not a causal streaming tagger. A linear-chain CRF directly models neighboring labels, not every document-level relation. Features, embeddings and label conventions affect reproducibility, and entity spans can still be wrong even when the label path is formally valid. Evaluate boundary errors and entity-type errors separately before deciding that a larger contextual encoder is the needed fix.
References
- Susan McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapters 3–5 and 8. Open textbook
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapter 5, “Sequential labeling and language modeling.” Chapter
- L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of the IEEE, 77(2), 1989. Paper
- M.-C. de Marneffe, C. D. Manning, J. Nivre and D. Zeman, “Universal Dependencies,” Computational Linguistics, 47(2), 2021. Paper
- J. Lafferty, A. McCallum and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” ICML, 2001. Paper
- Erik F. Tjong Kim Sang and Fien De Meulder, “Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition,” CoNLL, 2003. Paper
- Jing Li, Aixin Sun, Jianglei Han and Chenliang Li, “A Survey on Deep Learning for Named Entity Recognition,” IEEE Transactions on Knowledge and Data Engineering 34(1), 2022. Author-hosted paper
- Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang and Enhong Chen, “Large language models for generative information extraction: a survey,” Frontiers of Computer Science 18(6), 2024. Survey
- Urchade Zaratiana, Nadi Tomeh, Pierre Holat and Thierry Charnois, “GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer,” NAACL, 2024. Paper
- Lance Ramshaw and Mitch Marcus, “Text Chunking using Transformation-Based Learning,” Third Workshop on Very Large Corpora, 1995. Paper
- Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics 19(2), 1993. Paper
- Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu and Pavel Kuksa, “Natural Language Processing (Almost) from Scratch,” Journal of Machine Learning Research 12, 2011. Paper
- Zhiheng Huang, Wei Xu and Kai Yu, “Bidirectional LSTM-CRF Models for Sequence Tagging,” arXiv:1508.01991, 2015. arXiv
- Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami and Chris Dyer, “Neural Architectures for Named Entity Recognition,” NAACL-HLT, 2016. Paper
- Xuezhe Ma and Eduard Hovy, “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF,” ACL, 2016. Paper
- Alan Akbik, Duncan Blythe and Roland Vollgraf, “Contextual String Embeddings for Sequence Labeling,” COLING, 2018. Paper
- Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang and Zhi Zhong, “Towards Robust Linguistic Analysis using OntoNotes,” CoNLL, 2013. Paper
- Leon Derczynski, Eric Nichols, Marieke van Erp and Nut Limsopatham, “Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition,” Workshop on Noisy User-generated Text, 2017. Paper
- Christopher Walker, Stephanie Strassel, Julie Medero and Kazuaki Maeda, ACE 2005 Multilingual Training Corpus, LDC2006T06, Linguistic Data Consortium, 2006. Catalog
- Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli and Christopher D. Manning, “Position-aware Attention and Supervised Data Improve Slot Filling,” EMNLP, 2017. Paper
- Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou and Maosong Sun, “DocRED: A Large-Scale Document-Level Relation Extraction Dataset,” ACL, 2019. Paper
- Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen and Hoifung Poon, “UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition,” ICLR, 2024. arXiv
- Mike Schuster and Kuldip K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing 45(11), 1997. DOI