Full search page →

TextCNN — Convolutional Neural Networks for Sentence Classification

Optional · paper guide 1 of 1 for NLP 03 · Neural Networks for NLP. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

Can a sentence classifier learn useful phrase detectors without a full syntactic parser or a recurrent network? Kim’s EMNLP 2014 paper argues that a small convolutional model becomes a strong baseline when initialized with pretrained word vectors [10]. Read embeddings, convolution and classification loss in this chapter first. The key distinction is between learning local features and deciding which local features matter for the whole sentence.

Method

Original two-channel sentence CNN with convolution, max pooling and classifier
Source: Kim, EMNLP 2014, Figure 1, PDF p. 2. Follow the embedding rows through several filter widths, one maximum per filter, then the output classifier. Two embedding channels depict the multichannel variant, not every experimental model. [10]

Stack the word vectors into a sentence matrix. A filter covers several consecutive words and all embedding coordinates. Slide it over the sentence to produce a feature map, then keep its largest response:

ci=f(w⊤xi:i+h−1+b),c^=max⁡ici.c_i=f(w^\top x_{i:i+h-1}+b),\qquad \hat c=\max_i c_i.

Here xi:i+h−1x_{i:i+h-1} concatenates the embeddings of hh words starting at position ii; ww is the matching filter vector, bb its scalar bias, and ff a nonlinear activation. The scalar cic_i is one window’s response and c^\hat c is that filter’s pooled sentence feature. Concatenate the pooled features from many filters, apply dropout during training, and predict the class with a softmax layer.

A width-three filter can detect a local pattern wherever it occurs. Max pooling preserves the strongest evidence, not its location or number of occurrences. In the two-channel model, one embedding table stays fixed and the other is fine-tuned; each filter operates on both channels and adds their responses. The pretrained vectors supply information from external text, so this is not learning everything from the small classification dataset.

Results

Table 2 gives MR accuracy of 76.1 for randomly initialized CNN-rand, 81.0 for pretrained CNN-static and 81.5 for CNN-non-static. These numbers support the value of pretraining and task adaptation in this setup, rather than establishing that convolution alone causes the gain. Dataset protocols differ across the table: some use fixed test splits and others cross-validation. Compare model rows within a dataset before comparing scores across tasks.

Ablations

The static, non-static and multichannel variants test how embeddings are adapted. Multichannel improves SST-2 accuracy from non-static’s 87.2 to 88.1, but lowers TREC accuracy from 93.6 to 92.2. A second channel is therefore not an unconditional upgrade. The paper keeps fold assignments and relevant random initializations consistent within a dataset to make these comparisons more meaningful. Filter-width and regularization choices still affect the baseline you reproduce.

Conclusion

This is a useful first neural text classifier because its information flow is inspectable: word vectors → local responses → pooled evidence → class. Try explaining which three-word window drives a filter, then ask what changes if that phrase appears twice. Continue to NLP 04 for where the pretrained vectors come from.

Problems

Local word order survives inside a filter window, but max pooling discards much of the sentence’s global structure. Negation with a distant scope, several interacting clauses, or counts of repeated evidence can be hard to represent. A high activation is a model response, not a causal explanation of the label. Evaluate against a simple non-neural baseline and control external training data before attributing improvements to architecture.

References

  1. M. Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Manning, 2021; Chapters 2–4, 7 and 10–11. publisher
  2. S. McRoy, Principles of Natural Language Processing, open textbook, 2021; Chapter 6, “Benchmark Tasks for Language Modelling.” chapter
  3. Scikit-learn developers, “Text feature extraction,” scikit-learn documentation, current online edition. documentation
  4. Scikit-learn developers, “Logistic regression,” scikit-learn documentation, current online edition. documentation
  5. Y. Goldberg, “A primer on neural network models for natural language processing,” Journal of Artificial Intelligence Research 57, 2016. paper
  6. M. Iyyer, V. Manjunatha, J. Boyd-Graber and H. Daumé III, “Deep unordered composition rivals syntactic methods for text classification,” ACL-IJCNLP, 2015. paper
  7. A. Joulin, E. Grave, P. Bojanowski and T. Mikolov, “Bag of tricks for efficient text classification,” EACL, 2017. paper
  8. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research 3, 2003. paper
  9. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa, “Natural language processing (almost) from scratch,” Journal of Machine Learning Research 12, 2011. paper
  10. Y. Kim, “Convolutional neural networks for sentence classification,” EMNLP, 2014. paper
  11. Y. Zhang and B. Wallace, “A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification,” IJCNLP, 2017. paper
  12. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research 15, 2014. paper
  13. Scikit-learn developers, “Classification metrics,” scikit-learn documentation, current online edition. documentation
  14. C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On calibration of modern neural networks,” ICML, 2017. paper
  15. S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu and J. Gao, “Deep learning-based text classification: a comprehensive review,” ACM Computing Surveys 54(3), 2021. survey
  16. W. X. Zhao et al., “A survey of large language models,” Frontiers of Computer Science, 2026; online manuscript updated March 2026. survey
  17. Y. Chang et al., “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology, 2024. survey
  18. G. Sperduti and A. Moreo, “Misspellings in natural language processing: a survey,” arXiv:2501.16836, 2025. survey