Full search page →

Coreference — End-to-end Neural Coreference Resolution

Optional · paper guide 1 of 1 for NLP 12 · Syntactic and Semantic Tasks. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

Coreference resolution must decide which text spans refer to the same entity, including spans that were not supplied by a separate mention detector. Lee, He, Lewis and Zettlemoyer propose jointly learning span representations, mention scores and antecedent decisions [19]. Their thesis is that document-level coreference can be trained from clusters while considering candidate spans directly. First understand this chapter’s spans, mentions, antecedents and clustering; an entity label on one token is not yet a coreference link.

Method

Original coreference span representations combining recurrent endpoints and attention-weighted heads
Source: Lee et al., EMNLP 2017, Figure 1, PDF p. 3. Token encodings form candidate span representations, which receive mention scores. Low-scoring spans are pruned before pairwise antecedent scoring. [19]

A span representation combines bidirectional recurrent endpoint states, an attention-weighted representation of words inside the span, and span-width features. The learned attention acts as a soft head; it does not require gold syntactic head annotations. A mention scorer prunes candidates before antecedent pairs are evaluated.

Original coreference antecedent scorer comparing earlier mentions with a dummy antecedent
Source: Lee et al., EMNLP 2017, Figure 2, PDF p. 3. The company can link to an earlier span or the dummy antecedent. The dummy can represent a first mention or a non-mention; it does not always mean the model failed. [19]

For a retained span, score earlier candidates using both mention quality and pair compatibility:

s(i,j)=sm(i)+sm(j)+sa(i,j),P(yi=j∣D)=exp⁡s(i,j)∑k∈Y(i)exp⁡s(i,k).s(i,j)=s_m(i)+s_m(j)+s_a(i,j),\qquad P(y_i=j\mid D)=\frac{\exp s(i,j)}{\sum_{k\in\mathcal Y(i)}\exp s(i,k)}.

Here DD is the document, ii the current span, jj a non-dummy earlier span, sms_m a mention score and sas_a a pairwise antecedent score. yiy_i is the selected antecedent, while Y(i)\mathcal Y(i) contains available earlier candidates plus dummy ϵ\epsilon; kk indexes that set. The dummy uses s(i,ϵ)=0s(i,\epsilon)=0 instead of the additive formula. Its role includes starting a new entity cluster.

Training maximizes the sum of probabilities assigned to all correct antecedents, since a gold cluster can contain several valid earlier mentions. It does not force one arbitrary gold link. If no gold antecedent survives, the dummy is used. At inference, chosen antecedent links connect mentions into clusters; errors in early candidate pruning cannot be repaired by the later scorer.

Results

Table 1 reports English CoNLL-2012 test performance as the average F1 of MUC, B³ and CEAFφ4: 67.2 for the single model and 68.8 for a five-model ensemble. These are cluster metrics, not token accuracy or mention-detection F1. The ensemble and single-model results have different compute costs. Compare under the same evaluation script and mention setting before assigning significance to a small score difference.

Ablations

Table 2 uses development data: the full single model scores 67.7, removing head attention gives 66.4, and removing distance/width features gives 63.9. These drops must use 67.7 as their baseline, not the 67.2 test score. The system also limits span width to 10 words, retains up to 0.4 times the document’s token count in spans, and caps antecedents at 250. Those pruning choices control cost while potentially removing correct mentions or links.

Conclusion

The important connection is between representation learning and a structured decision over candidates. Trace “the company” through span construction, pruning and antecedent scoring in the two figures. Then inspect the chapter’s coreference evaluation discussion: detecting a mention, selecting its antecedent and forming the right cluster are three different places where errors can occur.

Problems

Pruning imposes a recall ceiling, and the dummy combines several situations rather than expressing a complete uncertainty model. Document genre, speaker metadata and annotation conventions affect transfer. A high-scoring link can still be semantically wrong, especially when resolving it requires knowledge absent from the text. End-to-end training removes a separately supervised mention pipeline; it does not remove feature choices, search approximations or dataset assumptions.

References

  1. Susan McRoy, Principles of Natural Language Processing, Chapters 3–5 and 7 and the chart-parsing appendix, open textbook, 2021. Open textbook
  2. Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd ed. draft (August 19, 2026), Chapters 19, 20, 22 and 24. Book site
  3. E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini and T. Strzalkowski, “A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars,” Speech and Natural Language Workshop, 1991. Paper
  4. Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics 19(2), 1993. Paper
  5. Mitchell Stern, Jacob Andreas and Dan Klein, “A Minimal Span-Based Neural Constituency Parser,” ACL, 2017. Paper
  6. Nikita Kitaev and Dan Klein, “Constituency Parsing with a Self-Attentive Encoder,” ACL, 2018. Paper
  7. Nikita Kitaev, Steven Cao and Dan Klein, “Multilingual Constituency Parsing with Self-Attention and Pre-Training,” ACL, 2019. Paper
  8. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre and Daniel Zeman, “Universal Dependencies,” Computational Linguistics 47(2), 2021. Paper
  9. Joakim Nivre, “Algorithms for Deterministic Incremental Dependency Parsing,” Computational Linguistics 34(4), 2008. Paper
  10. Danqi Chen and Christopher Manning, “A Fast and Accurate Dependency Parser using Neural Networks,” EMNLP, 2014. Paper
  11. Ryan McDonald, Fernando Pereira, Kiril Ribarov and Jan Hajič, “Non-Projective Dependency Parsing using Spanning Tree Algorithms,” HLT-EMNLP, 2005. Paper
  12. Eliyahu Kiperwasser and Yoav Goldberg, “Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations,” Transactions of the ACL 4, 2016. Paper
  13. Timothy Dozat and Christopher D. Manning, “Deep Biaffine Attention for Neural Dependency Parsing,” ICLR, 2017. Paper
  14. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre and Slav Petrov, “CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies,” CoNLL Shared Task, 2018. Paper
  15. Daniel Gildea and Daniel Jurafsky, “Automatic Labeling of Semantic Roles,” Computational Linguistics 28(3), 2002. Paper
  16. Martha Palmer, Daniel Gildea and Paul Kingsbury, “The Proposition Bank: An Annotated Corpus of Semantic Roles,” Computational Linguistics 31(1), 2005. Paper
  17. Xavier Carreras and Lluís Màrquez, “Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling,” CoNLL, 2005. Paper
  18. Luheng He, Kenton Lee, Mike Lewis and Luke Zettlemoyer, “Deep Semantic Role Labeling: What Works and What’s Next,” ACL, 2017. Paper
  19. Kenton Lee, Luheng He, Mike Lewis and Luke Zettlemoyer, “End-to-end Neural Coreference Resolution,” EMNLP, 2017. Paper
  20. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly and Lynette Hirschman, “A Model-Theoretic Coreference Scoring Scheme,” MUC-6, 1995. Paper
  21. Xiaoqiang Luo, “On Coreference Resolution Performance Metrics,” HLT-EMNLP, 2005. Paper
  22. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng and Michael Strube, “Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation,” ACL, 2014. Paper
  23. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina and Yuchen Zhang, “CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes,” EMNLP-CoNLL Shared Task, 2012. Paper
  24. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer and Nathan Schneider, “Abstract Meaning Representation for Sembanking,” Linguistic Annotation Workshop, 2013. Paper
  25. Shu Cai and Kevin Knight, “Smatch: an Evaluation Metric for Semantic Feature Structures,” ACL, 2013. Paper
  26. Mandar Joshi, Omer Levy, Luke Zettlemoyer and Daniel Weld, “BERT for Coreference Resolution: Baselines and Analysis,” EMNLP-IJCNLP, 2019. Paper
  27. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer and Omer Levy, “SpanBERT: Improving Pre-training by Representing and Predicting Spans,” Transactions of the ACL 8, 2020. Paper
  28. John Hewitt and Christopher D. Manning, “A Structural Probe for Finding Syntax in Word Representations,” NAACL-HLT, 2019. Paper
  29. Ian Tenney, Dipanjan Das and Ellie Pavlick, “BERT Rediscovers the Classical NLP Pipeline,” ACL, 2019. Paper
  30. Wayne Xin Zhao et al., “A Survey of Large Language Models,” Frontiers of Computer Science, 2026; online manuscript updated March 2026. Survey
  31. Meishan Zhang, “A survey of syntactic-semantic parsing based on constituent and dependency structures,” Science China Technological Sciences 63, 2020. Survey
  32. Massimo Poesio, Juntao Yu, Silviu Paun, Abdulrahman Aloraini, Pengcheng Lu, Janosch Haber and Derya Cokal, “Computational Models of Anaphora,” Annual Review of Linguistics 9, 2023. Survey
  33. Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Survey
  34. Huiyao Chen, Meishan Zhang, Jing Li, Lilja Øvrelid, Jan Hajič, Hao Fei and Min Zhang, “A Systematic Survey of Semantic Role Labeling in the Era of Pretrained Language Models,” arXiv:2502.08660, 2025; accepted at Artificial Intelligence Review. Survey
  35. Yuanhe Tian, Fei Xia and Yan Song, “Large Language Models are No Longer Shallow Parsers,” ACL, 2024. Paper
  36. John T. Hale and Miloš Stanojević, “Do LLMs learn a true syntactic universal?” EMNLP, 2024. Paper
  37. Nghia T. Le and Alan Ritter, “Are Large Language Models Robust Coreference Resolvers?” arXiv:2305.14489, 2023. Paper
  38. Ian Porada, Xiyuan Zou and Jackie Chi Kit Cheung, “A Controlled Reevaluation of Coreference Resolution Models,” LREC-COLING, 2024. Paper
  39. Zhijing Jin, Yuen Chen, Fernando Gonzalez Adauto, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf and Mona Diab, “Analyzing the Role of Semantic Representations in the Era of Large Language Models,” NAACL, 2024. Paper
  40. NLTK contributors, nltk.grammar.PCFG, API documentation, accessed 2026-11-05. Normalization contract