Overview

Top-level Functions

spacy.load function

Load a pipeline using the name of an installed package, a string path or a Path-like object. spaCy will try resolving the load argument in this order. If a pipeline is loaded from a string name, spaCy will assume it’s a Python package and import it and call the package’s own load() method. If a pipeline is loaded from a path, spaCy will assume it’s a data directory, load its config.cfg and use the language and pipeline information to construct the Language class. The data will be loaded in via Language.from_disk.

NameDescription
namePipeline to load, i.e. package name or path. Union[str, Path]
keyword-only
vocabOptional shared vocab to pass in on initialization. If True (default), a new Vocab object will be created. Union[Vocab, bool]
disableNames of pipeline components to disable. Disabled pipes will be loaded but they won’t be run unless you explicitly enable them by calling nlp.enable_pipe. List[str]
exclude v3.0Names of pipeline components to exclude. Excluded components won’t be loaded. List[str]
config v3.0Optional config overrides, either as nested dict or dict keyed by section value in dot notation, e.g. "components.name.value". Union[Dict[str, Any], Config]

Essentially, spacy.load() is a convenience wrapper that reads the pipeline’s config.cfg, uses the language and pipeline information to construct a Language object, loads in the model data and weights, and returns it.

Abstract example

cls = spacy.util.get_lang_class(lang) # 1. Get Language class, e.g. English nlp = cls() # 2. Initialize it for name in pipeline: nlp.add_pipe(name) # 3. Add the component to the pipeline nlp.from_disk(data_path) # 4. Load in the binary data

spacy.blank function

Create a blank pipeline of a given language class. This function is the twin of spacy.load().

NameDescription
nameISO code of the language class to load. str
keyword-only
vocabOptional shared vocab to pass in on initialization. If True (default), a new Vocab object will be created. Union[Vocab, bool]
config v3.0Optional config overrides, either as nested dict or dict keyed by section value in dot notation, e.g. "components.name.value". Union[Dict[str, Any], Config]
metaOptional meta overrides for nlp.meta. Dict[str, Any]

spacy.info function

The same as the info command. Pretty-print information about your installation, installed pipelines and local setup from within spaCy.

NameDescription
modelOptional pipeline, i.e. a package name or path (optional). Optional[str]
keyword-only
markdownPrint information as Markdown. bool
silentDon’t print anything, just return. bool

spacy.explain function

Get a description for a given POS tag, dependency label or entity type. For a list of available terms, see glossary.py.

NameDescription
termTerm to explain. str

spacy.prefer_gpu function

Allocate data and perform operations on GPU, if available. If data has already been allocated on CPU, it will not be moved. Ideally, this function should be called right after importing spaCy and before loading any pipelines.

NameDescription
gpu_idDevice index to select. Defaults to 0. int

spacy.require_gpu function

Allocate data and perform operations on GPU. Will raise an error if no GPU is available. If data has already been allocated on CPU, it will not be moved. Ideally, this function should be called right after importing spaCy and before loading any pipelines.

NameDescription
gpu_idDevice index to select. Defaults to 0. int

spacy.require_cpu functionv3.0.0

Allocate data and perform operations on CPU. If data has already been allocated on GPU, it will not be moved. Ideally, this function should be called right after importing spaCy and before loading any pipelines.

NameDescription

displaCy

As of v2.0, spaCy comes with a built-in visualization suite. For more info and examples, see the usage guide on visualizing spaCy.

displacy.serve method

Serve a dependency parse tree or named entity visualization to view it in your browser. Will run a simple web server.

NameDescription
docsDocument(s) or span(s) to visualize. Union[Iterable[Union[Doc, Span]], Doc, Span]
styleVisualization style, "dep" or "ent". Defaults to "dep". str
pageRender markup as full HTML page. Defaults to True. bool
minifyMinify HTML markup. Defaults to False. bool
optionsVisualizer-specific options, e.g. colors. Dict[str, Any]
manualDon’t parse Doc and instead expect a dict or list of dicts. See here for formats and examples. Defaults to False. bool
portPort to serve visualization. Defaults to 5000. int
hostHost to serve visualization. Defaults to "0.0.0.0". str

displacy.render method

Render a dependency parse tree or named entity visualization.

NameDescription
docsDocument(s) or span(s) to visualize. Union[Iterable[Union[Doc, Span]], Doc, Span]
styleVisualization style, "dep" or "ent". Defaults to "dep". str
pageRender markup as full HTML page. Defaults to True. bool
minifyMinify HTML markup. Defaults to False. bool
optionsVisualizer-specific options, e.g. colors. Dict[str, Any]
manualDon’t parse Doc and instead expect a dict or list of dicts. See here for formats and examples. Defaults to False. bool
jupyterExplicitly enable or disable ”Jupyter mode” to return markup ready to be rendered in a notebook. Detected automatically if None (default). Optional[bool]

Visualizer options

The options argument lets you specify additional settings for each visualizer. If a setting is not present in the options, the default value will be used.

Dependency Visualizer options

NameDescription
fine_grainedUse fine-grained part-of-speech tags (Token.tag_) instead of coarse-grained tags (Token.pos_). Defaults to False. bool
add_lemma Print the lemmas in a separate row below the token texts. Defaults to False. bool
collapse_punctAttach punctuation to tokens. Can make the parse more readable, as it prevents long arcs to attach punctuation. Defaults to True. bool
collapse_phrasesMerge noun phrases into one token. Defaults to False. bool
compact“Compact mode” with square arrows that takes up less space. Defaults to False. bool
colorText color (HEX, RGB or color names). Defaults to "#000000". str
bgBackground color (HEX, RGB or color names). Defaults to "#ffffff". str
fontFont name or font family for all text. Defaults to "Arial". str
offset_xSpacing on left side of the SVG in px. Defaults to 50. int
arrow_strokeWidth of arrow path in px. Defaults to 2. int
arrow_widthWidth of arrow head in px. Defaults to 10 in regular mode and 8 in compact mode. int
arrow_spacingSpacing between arrows in px to avoid overlaps. Defaults to 20 in regular mode and 12 in compact mode. int
word_spacingVertical spacing between words and arcs in px. Defaults to 45. int
distanceDistance between words in px. Defaults to 175 in regular mode and 150 in compact mode. int

Named Entity Visualizer options

NameDescription
entsEntity types to highlight or None for all types (default). Optional[List[str]]
colorsColor overrides. Entity types should be mapped to color names or values. Dict[str, str]
template Optional template to overwrite the HTML used to render entity spans. Should be a format string and can use {bg}, {text} and {label}. See templates.py for examples. Optional[str]

By default, displaCy comes with colors for all entity types used by spaCy’s trained pipelines. If you’re using custom entity types, you can use the colors setting to add your own colors for them. Your application or pipeline package can also expose a spacy_displacy_colors entry point to add custom labels and their colors automatically.

registry v3.0

spaCy’s function registry extends Thinc’s registry and allows you to map strings to functions. You can register functions to create architectures, optimizers, schedules and more, and then refer to them and set their arguments in your config file. Python type hints are used to validate the inputs. See the Thinc docs for details on the registry methods and our helper library catalogue for some background on the concept of function registries. spaCy also uses the function registry for language subclasses, model architecture, lookups and pipeline component factories.

Registry nameDescription
architecturesRegistry for functions that create model architectures. Can be used to register custom model architectures and reference them in the config.cfg.
augmentersRegistry for functions that create data augmentation callbacks for corpora and other training data iterators.
batchersRegistry for training and evaluation data batchers.
callbacksRegistry for custom callbacks to modify the nlp object before training.
displacy_colorsRegistry for custom color scheme for the displacy NER visualizer. Automatically reads from entry points.
factoriesRegistry for functions that create pipeline components. Added automatically when you use the @spacy.component decorator and also reads from entry points.
initializersRegistry for functions that create initializers.
languagesRegistry for language-specific Language subclasses. Automatically reads from entry points.
layersRegistry for functions that create layers.
loggersRegistry for functions that log training results.
lookupsRegistry for large lookup tables available via vocab.lookups.
lossesRegistry for functions that create losses.
miscRegistry for miscellaneous functions that return data assets, knowledge bases or anything else you may need.
optimizersRegistry for functions that create optimizers.
readersRegistry for file and data readers, including training and evaluation data readers like Corpus.
schedulesRegistry for functions that create schedules.
tokenizersRegistry for tokenizer factories. Registered functions should return a callback that receives the nlp object and returns a Tokenizer or a custom callable.

spacy-transformers registry

The following registries are added by the spacy-transformers package. See the Transformer API reference and usage docs for details.

Registry nameDescription
span_gettersRegistry for functions that take a batch of Doc objects and return a list of Span objects to process by the transformer, e.g. sentences.
annotation_settersRegistry for functions that create annotation setters. Annotation setters are functions that take a batch of Doc objects and a FullTransformerBatch and can set additional annotations on the Doc.

Loggers v3.0

A logger records the training results. When a logger is created, two functions are returned: one for logging the information for each training step, and a second function that is called to finalize the logging when the training is finished. To log each training step, a dictionary is passed on from the spacy train, including information such as the training loss and the accuracy scores on the development set.

There are two built-in logging functions: a logger printing results to the console in tabular format (which is the default), and one that also sends the results to a Weights & Biases dashboard. Instead of using one of the built-in loggers listed here, you can also implement your own.

spacy.ConsoleLogger.v1 registered function

Writes the results of a training step to the console in a tabular format.

python -m spacy train config.cfg
ℹ Using CPU
ℹ Loading config and nlp from: config.cfg
ℹ Pipeline: ['tok2vec', 'tagger']
ℹ Start training
ℹ Training. Initial learn rate: 0.0

E     #        LOSS TOK2VEC   LOSS TAGGER   TAG_ACC   SCORE
---   ------   ------------   -----------   -------   ------
  1        0           0.00         86.20      0.22     0.00
  1      200           3.08      18968.78     34.00     0.34
  1      400          31.81      22539.06     33.64     0.34
  1      600          92.13      22794.91     43.80     0.44
  1      800         183.62      21541.39     56.05     0.56
  1     1000         352.49      25461.82     65.15     0.65
  1     1200         422.87      23708.82     71.84     0.72
  1     1400         601.92      24994.79     76.57     0.77
  1     1600         662.57      22268.02     80.20     0.80
  1     1800        1101.50      28413.77     82.56     0.83
  1     2000        1253.43      28736.36     85.00     0.85
  1     2200        1411.02      28237.53     87.42     0.87
  1     2400        1605.35      28439.95     88.70     0.89

Note that the cumulative loss keeps increasing within one epoch, but should start decreasing across epochs.

spacy.WandbLogger.v1 registered function

Built-in logger that sends the results of each training step to the dashboard of the Weights & Biases tool. To use this logger, Weights & Biases should be installed, and you should be logged in. The logger will send the full config file to W&B, as well as various system information such as memory utilization, network traffic, disk IO, GPU statistics, etc. This will also include information such as your hostname and operating system, as well as the location of your Python executable.

NameDescription
project_nameThe name of the project in the Weights & Biases interface. The project will be created automatically if it doesn’t exist yet. str
remove_config_valuesA list of values to include from the config before it is uploaded to W&B (default: empty). List[str]

Readers

File readers v3.0

The following file readers are provided by our serialization library srsly. All registered functions take one argument path, pointing to the file path to load.

NameDescription
srsly.read_json.v1Read data from a JSON file.
srsly.read_jsonl.v1Read data from a JSONL (newline-delimited JSON) file.
srsly.read_yaml.v1Read data from a YAML file.
srsly.read_msgpack.v1Read data from a binary MessagePack file.

spacy.read_labels.v1 registered function

Read a JSON-formatted labels file generated with init labels. Typically used in the [initialize] block of the training config to speed up the model initialization process and provide pre-generated label sets.

NameDescription
pathThe path to the labels file generated with init labels. Path
requireWhether to require the file to exist. If set to False and the labels file doesn’t exist, the loader will return None and the initialize method will extract the labels from the data. Defaults to False. bool

Corpus readers v3.0

Corpus readers are registered functions that load data and return a function that takes the current nlp object and yields Example objects that can be used for training and pretraining. You can replace it with your own registered function in the @readers registry to customize the data loading and streaming.

spacy.Corpus.v1 registered function

The Corpus reader manages annotated corpora and can be used for training and development datasets in the DocBin (.spacy) format. Also see the Corpus class.

NameDescription
pathThe directory or filename to read from. Expects data in spaCy’s binary .spacy format. Union[str, Path]
 gold_preprocWhether to set up the Example object with gold-standard sentences and tokens for the predictions. See Corpus for details. bool
max_lengthMaximum document length. Longer documents will be split into sentences, if sentence boundaries are available. Defaults to 0 for no limit. int
limitLimit corpus to a subset of examples, e.g. for debugging. Defaults to 0 for no limit. int
augmenterApply some simply data augmentation, where we replace tokens with variations. This is especially useful for punctuation and case replacement, to help generalize beyond corpora that don’t have smart-quotes, or only have smart quotes, etc. Defaults to None. Optional[Callable]

spacy.JsonlCorpus.v1 registered function

Create Example objects from a JSONL (newline-delimited JSON) file of texts keyed by "text". Can be used to read the raw text corpus for language model pretraining from a JSONL file. Also see the JsonlCorpus class.

NameDescription
pathThe directory or filename to read from. Expects newline-delimited JSON with a key "text" for each record. Union[str, Path]
min_lengthMinimum document length (in tokens). Shorter documents will be skipped. Defaults to 0, which indicates no limit. int
max_lengthMaximum document length (in tokens). Longer documents will be skipped. Defaults to 0, which indicates no limit. int
limitLimit corpus to a subset of examples, e.g. for debugging. Defaults to 0 for no limit. int

Batchers v3.0

A data batcher implements a batching strategy that essentially turns a stream of items into a stream of batches, with each batch consisting of one item or a list of items. During training, the models update their weights after processing one batch at a time. Typical batching strategies include presenting the training data as a stream of batches with similar sizes, or with increasing batch sizes. See the Thinc documentation on schedules for a few standard examples.

Instead of using one of the built-in batchers listed here, you can also implement your own, which may or may not use a custom schedule.

spacy.batch_by_words.v1 registered function

Create minibatches of roughly a given number of words. If any examples are longer than the specified batch length, they will appear in a batch by themselves, or be discarded if discard_oversize is set to True. The argument docs can be a list of strings, Doc objects or Example objects.

NameDescription
seqsThe sequences to minibatch. Iterable[Any]
sizeThe target number of words per batch. Can also be a block referencing a schedule, e.g. compounding. Union[int, Sequence[int]]
toleranceWhat percentage of the size to allow batches to exceed. float
discard_oversizeWhether to discard sequences that by themselves exceed the tolerated size. bool
get_lengthOptional function that receives a sequence item and returns its length. Defaults to the built-in len() if not set. Optional[Callable[[Any], int]]

spacy.batch_by_sequence.v1 registered function

Create a batcher that creates batches of the specified size.

NameDescription
sizeThe target number of items per batch. Can also be a block referencing a schedule, e.g. compounding. Union[int, Sequence[int]]
get_lengthOptional function that receives a sequence item and returns its length. Defaults to the built-in len() if not set. Optional[Callable[[Any], int]]

spacy.batch_by_padded.v1 registered function

Minibatch a sequence by the size of padded batches that would result, with sequences binned by length within a window. The padded size is defined as the maximum length of sequences within the batch multiplied by the number of sequences in the batch.

NameDescription
sizeThe largest padded size to batch sequences into. Can also be a block referencing a schedule, e.g. compounding. Union[int, Sequence[int]]
bufferThe number of sequences to accumulate before sorting by length. A larger buffer will result in more even sizing, but if the buffer is very large, the iteration order will be less random, which can result in suboptimal training. int
discard_oversizeWhether to discard sequences that are by themselves longer than the largest padded batch size. bool
get_lengthOptional function that receives a sequence item and returns its length. Defaults to the built-in len() if not set. Optional[Callable[[Any], int]]

Augmenters v3.0

Data augmentation is the process of applying small modifications to the training data. It can be especially useful for punctuation and case replacement – for example, if your corpus only uses smart quotes and you want to include variations using regular quotes, or to make the model less sensitive to capitalization by including a mix of capitalized and lowercase examples. See the usage guide for details and examples.

spacy.orth_variants.v1 registered function

Create a data augmentation callback that uses orth-variant replacement. The callback can be added to a corpus or other data iterator during training. It’s especially useful for punctuation and case replacement, to help generalize beyond corpora that don’t have smart quotes, or only have smart quotes etc.

NameDescription
levelThe percentage of texts that will be augmented. float
lowerThe percentage of texts that will be lowercased. float
orth_variantsA dictionary containing the single and paired orth variants. Typically loaded from a JSON file. See en_orth_variants.json for an example. Dict[str, Dict[List[Union[str, List[str]]]]]

spacy.lower_case.v1 registered function

Create a data augmentation callback that lowercases documents. The callback can be added to a corpus or other data iterator during training. It’s especially useful for making the model less sensitive to capitalization.

NameDescription
levelThe percentage of texts that will be augmented. float

Training data and alignment

training.offsets_to_biluo_tags function

Encode labelled spans into per-token tags, using the BILUO scheme (Begin, In, Last, Unit, Out). Returns a list of strings, describing the tags. Each tag string will be in the form of either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". The string "-" is used where the entity offsets don’t align with the tokenization in the Doc object. The training algorithm will view these as missing values. O denotes a non-entity token. B denotes the beginning of a multi-token entity, I the inside of an entity of three or more tokens, and L the end of an entity of two or more tokens. U denotes a single-token entity.

NameDescription
docThe document that the entity offsets refer to. The output tags will refer to the token boundaries within the document. Doc
entitiesA sequence of (start, end, label) triples. start and end should be character-offset integers denoting the slice into the original string. List[Tuple[int, int, Union[str, int]]]
missingThe label used for missing values, e.g. if tokenization doesn’t align with the entity offsets. Defaults to "O". str

training.biluo_tags_to_offsets function

Encode per-token tags following the BILUO scheme into entity offsets.

NameDescription
docThe document that the BILUO tags refer to. Doc
entitiesA sequence of BILUO tags with each tag describing one token. Each tag string will be of the form of either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". List[str]

training.biluo_tags_to_spans function

Encode per-token tags following the BILUO scheme into Span objects. This can be used to create entity spans from token-based tags, e.g. to overwrite the doc.ents.

NameDescription
docThe document that the BILUO tags refer to. Doc
entitiesA sequence of BILUO tags with each tag describing one token. Each tag string will be of the form of either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". List[str]

Utility functions

spaCy comes with a small collection of utility functions located in spacy/util.py. Because utility functions are mostly intended for internal use within spaCy, their behavior may change with future releases. The functions documented on this page should be safe to use and we’ll try to ensure backwards compatibility. However, we recommend having additional tests in place if your application depends on any of spaCy’s utilities.

util.get_lang_class function

Import and load a Language class. Allows lazy-loading language data and importing languages using the two-letter language code. To add a language code for a custom language class, you can register it using the @registry.languages decorator.

NameDescription
langTwo-letter language code, e.g. "en". str

util.lang_class_is_loaded function

Check whether a Language subclass is already loaded. Language subclasses are loaded lazily to avoid expensive setup code associated with the language data.

NameDescription
nameTwo-letter language code, e.g. "en". str

util.load_model function

Load a pipeline from a package or data path. If called with a string name, spaCy will assume the pipeline is a Python package and import and call its load() method. If called with a path, spaCy will assume it’s a data directory, read the language and pipeline settings from the config.cfg and create a Language object. The model data will then be loaded in via Language.from_disk.

NameDescription
namePackage name or path. str
keyword-only
vocabOptional shared vocab to pass in on initialization. If True (default), a new Vocab object will be created. Union[Vocab, bool]
disableNames of pipeline components to disable. Disabled pipes will be loaded but they won’t be run unless you explicitly enable them by calling nlp.enable_pipe. List[str]
exclude v3.0Names of pipeline components to exclude. Excluded components won’t be loaded. List[str]
config v3.0Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. "nlp.pipeline". Union[Dict[str, Any], Config]

util.load_model_from_init_py function

A helper function to use in the load() method of a pipeline package’s __init__.py.

NameDescription
init_filePath to package’s __init__.py, i.e. __file__. Union[str, Path]
keyword-only
vocab v3.0Optional shared vocab to pass in on initialization. If True (default), a new Vocab object will be created. Union[Vocab, bool]
disableNames of pipeline components to disable. Disabled pipes will be loaded but they won’t be run unless you explicitly enable them by calling nlp.enable_pipe. List[str]
exclude v3.0Names of pipeline components to exclude. Excluded components won’t be loaded. List[str]
config v3.0Config overrides as nested dict or flat dict keyed by section values in dot notation, e.g. "nlp.pipeline". Union[Dict[str, Any], Config]

util.load_config functionv3.0

Load a pipeline’s config.cfg from a file path. The config typically includes details about the components and how they’re created, as well as all training settings and hyperparameters.

NameDescription
pathPath to the pipeline’s config.cfg. Union[str, Path]
overridesOptional config overrides to replace in loaded config. Can be provided as nested dict, or as flat dict with keys in dot notation, e.g. "nlp.pipeline". Dict[str, Any]
interpolateWhether to interpolate the config and replace variables like ${paths.train} with their values. Defaults to False. bool

util.load_meta functionv3.0

Get a pipeline’s meta.json from a file path and validate its contents. The meta typically includes details about author, licensing, data sources and version.

NameDescription
pathPath to the pipeline’s meta.json. Union[str, Path]

util.get_installed_models functionv3.0

List all pipeline packages installed in the current environment. This will include any spaCy pipeline that was packaged with spacy package. Under the hood, pipeline packages expose a Python entry point that spaCy can check, without having to load the nlp object.

NameDescription

util.is_package function

Check if string maps to a package installed via pip. Mainly used to validate pipeline packages.

NameDescription
nameName of package. str

util.get_package_path function

Get path to an installed package. Mainly used to resolve the location of pipeline packages. Currently imports the package to find its path.

NameDescription
package_nameName of installed package. str

util.is_in_jupyter function

Check if user is running spaCy from a Jupyter notebook by detecting the IPython kernel. Mainly used for the displacy visualizer.

NameDescription

util.compile_prefix_regex function

Compile a sequence of prefix rules into a regex object.

NameDescription
entriesThe prefix rules, e.g. lang.punctuation.TOKENIZER_PREFIXES. Iterable[Union[str, Pattern]]

util.compile_suffix_regex function

Compile a sequence of suffix rules into a regex object.

NameDescription
entriesThe suffix rules, e.g. lang.punctuation.TOKENIZER_SUFFIXES. Iterable[Union[str, Pattern]]

util.compile_infix_regex function

Compile a sequence of infix rules into a regex object.

NameDescription
entriesThe infix rules, e.g. lang.punctuation.TOKENIZER_INFIXES. Iterable[Union[str, Pattern]]

util.minibatch function

Iterate over batches of items. size may be an iterator, so that batch-size can vary on each step.

NameDescription
itemsThe items to batch up. Iterable[Any]
sizeThe batch size(s). Union[int, Sequence[int]]

util.filter_spans function

Filter a sequence of Span objects and remove duplicates or overlaps. Useful for creating named entities (where one token can only be part of one entity) or when merging spans with Retokenizer.merge. When spans overlap, the (first) longest span is preferred over shorter spans.

NameDescription
spansThe spans to filter. Iterable[Span]

util.get_words_and_spaces functionv3.0

Given a list of words and a text, reconstruct the original tokens and return a list of words and spaces that can be used to create a Doc. This can help recover destructive tokenization that didn’t preserve any whitespace information.

NameDescription
wordsThe list of words. Iterable[str]
textThe original text. str