Full search page →

BiLSTM-CRF — Bidirectional LSTM-CRF Models for Sequence Tagging

Optional · paper guide 1 of 1 for NLP 06 · Sequence Labeling with RNNs (POS Tagging, Information Extraction). Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

An independently predicted token label may look plausible while the whole label sequence is inconsistent. Huang, Xu and Yu combine context-sensitive recurrent features with a globally normalized sequence model [13]. Their thesis separates two jobs: a bidirectional LSTM reads the sentence, and a CRF scores neighboring label decisions together. Read the chapter’s BIO tags, emissions, transition scores and dynamic programming first. This guide concerns the 2015 manuscript, including its engineered input features.

Method

Original bidirectional LSTM features feeding a connected sequence of CRF labels
Source: Huang et al., 2015, Figure 7, arXiv:1508.01991, PDF p. 4. Bottom words feed forward and backward recurrent states; the top chain connects label decisions. Bidirectional input context and label-transition modeling perform different jobs. [13]

The forward and backward LSTMs let each token’s representation depend on both sides of the sentence. These representations produce emission scores for candidate labels. A linear-chain CRF adds transition scores and normalizes over complete label paths:

s(x,y)=∑t=1Tet(yt)+∑t=1TAyt−1,yt,p(y∣x)=exp⁡s(x,y)∑y′∈Y(x)exp⁡s(x,y′).s(x,y)=\sum_{t=1}^{T}e_t(y_t)+\sum_{t=1}^{T}A_{y_{t-1},y_t},\qquad p(y\mid x)=\frac{\exp s(x,y)}{\sum_{y'\in\mathcal Y(x)}\exp s(x,y')}.

Here xx is the input sentence, TT its length, yy a label path, et(k)e_t(k) the emission score for label kk at position tt, and Aa,bA_{a,b} a learned transition score from label aa to bb. The symbol y0y_0 denotes a start state; boundary terms can be included similarly. Y(x)\mathcal Y(x) is the set of candidate paths and y′y' indexes a path in the denominator. Scores are unnormalized real numbers, not token probabilities.

Training maximizes the correct path’s conditional likelihood. A forward dynamic program computes the sum over paths efficiently; Viterbi decoding instead finds the highest-scoring path. Summation and maximization are different operations. A learned transition matrix can discourage invalid BIO transitions but does not guarantee hard validity unless the implementation explicitly masks them.

Results

Table 2’s pretrained Senna-embedding BiLSTM-CRF row reports 97.55 POS accuracy, 94.46 chunking F1 and 88.83 NER F1. The parenthesized 90.10 NER F1 additionally uses gazetteer features. Those two NER numbers are not the same input setting. POS accuracy counts token decisions, whereas chunking and NER F1 evaluate extracted spans; their numerical values should not be ranked as though they measure the same thing.

Ablations

With random embeddings, the corresponding BiLSTM-CRF row obtains 97.43, 94.13 and 84.26. Comparing these rows reveals a much larger pretraining effect for NER in this setup. The architecture comparisons distinguish recurrent directionality from adding a CRF. Feature experiments also matter: removing spelling and context features changes chunking F1 from 94.46 to 94.40. This paper is not evidence that all hand-crafted features have already disappeared from the pipeline.

Conclusion

Use this model to understand why contextual token evidence and structured output constraints are complementary. Given two competing BIO paths, identify which difference comes from emissions and which comes from transitions. Continue to NLP 12 for structures beyond an adjacent label chain, including spans and coreference links.

Problems

Bidirectional features require future input, so this model is not a causal streaming tagger. A linear-chain CRF directly models neighboring labels, not every document-level relation. Features, embeddings and label conventions affect reproducibility, and entity spans can still be wrong even when the label path is formally valid. Evaluate boundary errors and entity-type errors separately before deciding that a larger contextual encoder is the needed fix.

References

  1. Susan McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapters 3–5 and 8. Open textbook
  2. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapter 5, “Sequential labeling and language modeling.” Chapter
  3. L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of the IEEE, 77(2), 1989. Paper
  4. M.-C. de Marneffe, C. D. Manning, J. Nivre and D. Zeman, “Universal Dependencies,” Computational Linguistics, 47(2), 2021. Paper
  5. J. Lafferty, A. McCallum and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” ICML, 2001. Paper
  6. Erik F. Tjong Kim Sang and Fien De Meulder, “Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition,” CoNLL, 2003. Paper
  7. Jing Li, Aixin Sun, Jianglei Han and Chenliang Li, “A Survey on Deep Learning for Named Entity Recognition,” IEEE Transactions on Knowledge and Data Engineering 34(1), 2022. Author-hosted paper
  8. Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang and Enhong Chen, “Large language models for generative information extraction: a survey,” Frontiers of Computer Science 18(6), 2024. Survey
  9. Urchade Zaratiana, Nadi Tomeh, Pierre Holat and Thierry Charnois, “GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer,” NAACL, 2024. Paper
  10. Lance Ramshaw and Mitch Marcus, “Text Chunking using Transformation-Based Learning,” Third Workshop on Very Large Corpora, 1995. Paper
  11. Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics 19(2), 1993. Paper
  12. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu and Pavel Kuksa, “Natural Language Processing (Almost) from Scratch,” Journal of Machine Learning Research 12, 2011. Paper
  13. Zhiheng Huang, Wei Xu and Kai Yu, “Bidirectional LSTM-CRF Models for Sequence Tagging,” arXiv:1508.01991, 2015. arXiv
  14. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami and Chris Dyer, “Neural Architectures for Named Entity Recognition,” NAACL-HLT, 2016. Paper
  15. Xuezhe Ma and Eduard Hovy, “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF,” ACL, 2016. Paper
  16. Alan Akbik, Duncan Blythe and Roland Vollgraf, “Contextual String Embeddings for Sequence Labeling,” COLING, 2018. Paper
  17. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang and Zhi Zhong, “Towards Robust Linguistic Analysis using OntoNotes,” CoNLL, 2013. Paper
  18. Leon Derczynski, Eric Nichols, Marieke van Erp and Nut Limsopatham, “Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition,” Workshop on Noisy User-generated Text, 2017. Paper
  19. Christopher Walker, Stephanie Strassel, Julie Medero and Kazuaki Maeda, ACE 2005 Multilingual Training Corpus, LDC2006T06, Linguistic Data Consortium, 2006. Catalog
  20. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli and Christopher D. Manning, “Position-aware Attention and Supervised Data Improve Slot Filling,” EMNLP, 2017. Paper
  21. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou and Maosong Sun, “DocRED: A Large-Scale Document-Level Relation Extraction Dataset,” ACL, 2019. Paper
  22. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen and Hoifung Poon, “UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition,” ICLR, 2024. arXiv
  23. Mike Schuster and Kuldip K. Paliwal, “Bidirectional Recurrent Neural Networks,” IEEE Transactions on Signal Processing 45(11), 1997. DOI