Feature comparison

Here’s a quick comparison of the functionalities offered by spaCy, NLTK and CoreNLP.

When should I use what?

Natural Language Understanding is an active area of research and development, so there are many different tools or technologies catering to different use-cases. The table below summarizes a few libraries (spaCy, NLTK, AllenNLP, StanfordNLP and TensorFlow) to help you get a feel for things fit together.

Benchmarks

Two peer-reviewed papers in 2015 confirmed that spaCy offers the fastest syntactic parser in the world and that its accuracy is within 1% of the best available. The few systems that are more accurate are 20× slower or more.

Algorithm comparison

In this section, we compare spaCy’s algorithms to recently published systems, using some of the most popular benchmarks. These benchmarks are designed to help isolate the contributions of specific algorithmic decisions, so they promote slightly “idealized” conditions. Specifically, the text comes pre-processed with “gold standard” token and sentence boundaries. The data sets also tend to be fairly small, to help researchers iterate quickly. These conditions mean the models trained on these data sets are not always useful for practical purposes.

Parse accuracy (Penn Treebank / Wall Street Journal)

This is the “classic” evaluation, so it’s the number parsing researchers are most easily able to put in context. However, it’s quite far removed from actual usage: it uses sentences with gold-standard segmentation and tokenization, from a pretty specific type of text (articles from a single newspaper, 1984-1989).

NER accuracy (OntoNotes 5, no pre-process)

This is the evaluation we use to tune spaCy’s parameters to decide which algorithms are better than the others. It’s reasonably close to actual usage, because it requires the parses to be produced from raw text, without any pre-processing.

Model comparison

In this section, we provide benchmark accuracies for the pre-trained model pipelines we distribute with spaCy. Evaluations are conducted end-to-end from raw text, with no “gold standard” pre-processing, over text from a mix of genres where possible.

English

Spanish

Detailed speed comparison

Here we compare the per-document processing time of various spaCy functionalities against other NLP libraries. We show both absolute timings (in ms) and relative performance (normalized to spaCy). Lower is better.