spacy.load function

Load a pipeline using the name of an installed package, a string path or a Path-like object. spaCy will try resolving the load argument in this order. If a pipeline is loaded from a string name, spaCy will assume it’s a Python package and import it and call the package’s own load() method. If a pipeline is loaded from a path, spaCy will assume it’s a data directory, load its config.cfg and use the language and pipeline information to construct the Language class. The data will be loaded in via Language.from_disk. Loading a pipeline from a package will also import any custom code, if present, whereas loading from a directory does not. For these cases, you need to manually import your custom code.

Essentially, spacy.load() is a convenience wrapper that reads the pipeline’s config.cfg, uses the language and pipeline information to construct a Language object, loads in the model data and weights, and returns it.

Abstract example

spacy.blank function

Create a blank pipeline of a given language class. This function is the twin of spacy.load().

spacy.info function

The same as the info command. Pretty-print information about your installation, installed pipelines and local setup from within spaCy.

spacy.explain function

Get a description for a given POS tag, dependency label or entity type. For a list of available terms, see glossary.py.

spacy.prefer_gpu function

Allocate data and perform operations on GPU, if available. If data has already been allocated on CPU, it will not be moved. Ideally, this function should be called right after importing spaCy and before loading any pipelines.

spacy.require_gpu function

Allocate data and perform operations on GPU. Will raise an error if no GPU is available. If data has already been allocated on CPU, it will not be moved. Ideally, this function should be called right after importing spaCy and before loading any pipelines.

spacy.require_cpu functionv3.0.0

Allocate data and perform operations on CPU. If data has already been allocated on GPU, it will not be moved. Ideally, this function should be called right after importing spaCy and before loading any pipelines.

displaCy

Source

As of v2.0, spaCy comes with a built-in visualization suite. For more info and examples, see the usage guide on visualizing spaCy.

displacy.serve method

Serve a dependency parse tree or named entity visualization to view it in your browser. Will run a simple web server.

displacy.render method

Render a dependency parse tree or named entity visualization.

displacy.parse_deps method

Generate dependency parse in {'words': [], 'arcs': []} format. For use with the manual=True argument in displacy.render.

displacy.parse_ents method

Generate named entities in [{start: i, end: i, label: 'label'}] format. For use with the manual=True argument in displacy.render.

displacy.parse_spans method

Generate spans in [{start_token: i, end_token: i, label: 'label'}] format. For use with the manual=True argument in displacy.render.

Visualizer data structures

You can use displaCy’s data format to manually render data. This can be useful if you want to visualize output from other libraries. You can find examples of displaCy’s different data formats below.

Dependency Visualizer data structure

Named Entity Recognition data structure

Span Classification data structure

Visualizer options

The options argument lets you specify additional settings for each visualizer. If a setting is not present in the options, the default value will be used.

Dependency Visualizer options

Named Entity Visualizer options

Span Visualizer options

By default, displaCy comes with colors for all entity types used by spaCy’s trained pipelines for both entity and span visualizer. If you’re using custom entity types, you can use the colors setting to add your own colors for them. Your application or pipeline package can also expose a spacy_displacy_colors entry point to add custom labels and their colors automatically.

By default, displaCy links to # for entities without a kb_id set on their span. If you wish to link an entity to their URL then consider using the kb_url_template option from above. For example if the kb_id on a span is Q95 and this is a Wikidata identifier then this option can be set to https://www.wikidata.org/wiki/{}. Clicking on your entity in the rendered HTML should redirect you to their Wikidata page, in this case https://www.wikidata.org/wiki/Q95.

registry v3.0

Source

spaCy’s function registry extends Thinc’s registry and allows you to map strings to functions. You can register functions to create architectures, optimizers, schedules and more, and then refer to them and set their arguments in your config file. Python type hints are used to validate the inputs. See the Thinc docs for details on the registry methods and our helper library catalogue for some background on the concept of function registries. spaCy also uses the function registry for language subclasses, model architecture, lookups and pipeline component factories.

spacy-transformers registry

The following registries are added by the spacy-transformers package. See the Transformer API reference and usage docs for details.

Loggers v3.0

Source

A logger records the training results. When a logger is created, two functions are returned: one for logging the information for each training step, and a second function that is called to finalize the logging when the training is finished. To log each training step, a dictionary is passed on from the spacy train, including information such as the training loss and the accuracy scores on the development set.

The built-in, default logger is the ConsoleLogger, which prints results to the console in tabular format and saves them to a jsonl file. The spacy-loggers package, included as a dependency of spaCy, enables other loggers, such as one that sends results to a Weights & Biases dashboard.

Instead of using one of the built-in loggers, you can implement your own.

spacy.ConsoleLogger.v2 registered function

Writes the results of a training step to the console in a tabular format and saves them to a jsonl file.

spacy.ConsoleLogger.v3 registered function

Writes the results of a training step to the console in a tabular format and optionally saves them to a jsonl file.

Readers

File readers v3.0

Source

The following file readers are provided by our serialization library srsly. All registered functions take one argument path, pointing to the file path to load.

spacy.read_labels.v1 registered function

Read a JSON-formatted labels file generated with init labels. Typically used in the [initialize] block of the training config to speed up the model initialization process and provide pre-generated label sets.

Corpus readers v3.0

Source

Corpus readers are registered functions that load data and return a function that takes the current nlp object and yields Example objects that can be used for training and pretraining. You can replace it with your own registered function in the @readers registry to customize the data loading and streaming.

spacy.Corpus.v1 registered function

The Corpus reader manages annotated corpora and can be used for training and development datasets in the DocBin (.spacy) format. Also see the Corpus class.

spacy.JsonlCorpus.v1 registered function

Create Example objects from a JSONL (newline-delimited JSON) file of texts keyed by "text". Can be used to read the raw text corpus for language model pretraining from a JSONL file. Also see the JsonlCorpus class.

Batchers v3.0

Source

A data batcher implements a batching strategy that essentially turns a stream of items into a stream of batches, with each batch consisting of one item or a list of items. During training, the models update their weights after processing one batch at a time. Typical batching strategies include presenting the training data as a stream of batches with similar sizes, or with increasing batch sizes. See the Thinc documentation on schedules for a few standard examples.

Instead of using one of the built-in batchers listed here, you can also implement your own, which may or may not use a custom schedule.

spacy.batch_by_words.v1 registered function

Create minibatches of roughly a given number of words. If any examples are longer than the specified batch length, they will appear in a batch by themselves, or be discarded if discard_oversize is set to True. The argument docs can be a list of strings, Doc objects or Example objects.

spacy.batch_by_sequence.v1 registered function

Create a batcher that creates batches of the specified size.

spacy.batch_by_padded.v1 registered function

Minibatch a sequence by the size of padded batches that would result, with sequences binned by length within a window. The padded size is defined as the maximum length of sequences within the batch multiplied by the number of sequences in the batch.

Augmenters v3.0

Source

Data augmentation is the process of applying small modifications to the training data. It can be especially useful for punctuation and case replacement – for example, if your corpus only uses smart quotes and you want to include variations using regular quotes, or to make the model less sensitive to capitalization by including a mix of capitalized and lowercase examples. See the usage guide for details and examples.

spacy.orth_variants.v1 registered function

Create a data augmentation callback that uses orth-variant replacement. The callback can be added to a corpus or other data iterator during training. It’s especially useful for punctuation and case replacement, to help generalize beyond corpora that don’t have smart quotes, or only have smart quotes etc.

spacy.lower_case.v1 registered function

Create a data augmentation callback that lowercases documents. The callback can be added to a corpus or other data iterator during training. It’s especially useful for making the model less sensitive to capitalization.

Callbacks v3.0

Source

The config supports callbacks at several points in the lifecycle that can be used modify the nlp object.

spacy.copy_from_base_model.v1 registered function

Copy the tokenizer and/or vocab from the specified models. It’s similar to the v2 base model option and useful in combination with sourced components when fine-tuning an existing pipeline. The vocab includes the lookups and the vectors from the specified model. Intended for use in [initialize.before_init].

spacy.models_with_nvtx_range.v1 registered function

Recursively wrap the models in each pipe using NVTX range markers. These markers aid in GPU profiling by attributing specific operations to a Model’s forward or backprop passes.

spacy.models_and_pipes_with_nvtx_range.v1 registered functionv3.4

Recursively wrap both the models and methods of each pipe using NVTX range markers. By default, the following methods are wrapped: pipe, predict, set_annotations, update, rehearse, get_loss, initialize, begin_update, finish_update, update.

Training data and alignment

Source

training.offsets_to_biluo_tags function

Encode labelled spans into per-token tags, using the BILUO scheme (Begin, In, Last, Unit, Out). Returns a list of strings, describing the tags. Each tag string will be in the form of either "", "O" or "{action}-{label}", where action is one of "B", "I", "L", "U". The string "-" is used where the entity offsets don’t align with the tokenization in the Doc object. The training algorithm will view these as missing values. O denotes a non-entity token. B denotes the beginning of a multi-token entity, I the inside of an entity of three or more tokens, and L the end of an entity of two or more tokens. U denotes a single-token entity.

training.biluo_tags_to_offsets function

Encode per-token tags following the BILUO scheme into entity offsets.

training.biluo_tags_to_spans function

Encode per-token tags following the BILUO scheme into Span objects. This can be used to create entity spans from token-based tags, e.g. to overwrite the doc.ents.

training.biluo_to_iob function

Convert a sequence of BILUO tags to IOB tags. This is useful if you want use the BILUO tags with a model that only supports IOB tags.

training.iob_to_biluo function

Convert a sequence of IOB tags to BILUO tags. This is useful if you want use the IOB tags with a model that only supports BILUO tags.

training.biluo_to_iob function

Convert a sequence of BILUO tags to IOB tags. This is useful if you want use the BILUO tags with a model that only supports IOB tags.

training.iob_to_biluo function

Convert a sequence of IOB tags to BILUO tags. This is useful if you want use the IOB tags with a model that only supports BILUO tags.

Utility functions

Source

spaCy comes with a small collection of utility functions located in spacy/util.py. Because utility functions are mostly intended for internal use within spaCy, their behavior may change with future releases. The functions documented on this page should be safe to use and we’ll try to ensure backwards compatibility. However, we recommend having additional tests in place if your application depends on any of spaCy’s utilities.

util.get_lang_class function

Import and load a Language class. Allows lazy-loading language data and importing languages using the two-letter language code. To add a language code for a custom language class, you can register it using the @registry.languages decorator.

util.lang_class_is_loaded function

Check whether a Language subclass is already loaded. Language subclasses are loaded lazily to avoid expensive setup code associated with the language data.

util.load_model function

Load a pipeline from a package or data path. If called with a string name, spaCy will assume the pipeline is a Python package and import and call its load() method. If called with a path, spaCy will assume it’s a data directory, read the language and pipeline settings from the config.cfg and create a Language object. The model data will then be loaded in via Language.from_disk.

util.load_model_from_init_py function

A helper function to use in the load() method of a pipeline package’s __init__.py.

util.load_config functionv3.0

Load a pipeline’s config.cfg from a file path. The config typically includes details about the components and how they’re created, as well as all training settings and hyperparameters.

util.load_meta functionv3.0

Get a pipeline’s meta.json from a file path and validate its contents. The meta typically includes details about author, licensing, data sources and version.

util.get_installed_models functionv3.0

List all pipeline packages installed in the current environment. This will include any spaCy pipeline that was packaged with spacy package. Under the hood, pipeline packages expose a Python entry point that spaCy can check, without having to load the nlp object.

util.is_package function

Check if string maps to a package installed via pip. Mainly used to validate pipeline packages.

util.get_package_path function

Get path to an installed package. Mainly used to resolve the location of pipeline packages. Currently imports the package to find its path.

util.is_in_jupyter function

Check if user is running spaCy from a Jupyter notebook by detecting the IPython kernel. Mainly used for the displacy visualizer.

util.compile_prefix_regex function

Compile a sequence of prefix rules into a regex object.

util.compile_suffix_regex function

Compile a sequence of suffix rules into a regex object.

util.compile_infix_regex function

Compile a sequence of infix rules into a regex object.

util.minibatch function

Iterate over batches of items. size may be an iterator, so that batch-size can vary on each step.

util.filter_spans function

Filter a sequence of Span objects and remove duplicates or overlaps. Useful for creating named entities (where one token can only be part of one entity) or when merging spans with Retokenizer.merge. When spans overlap, the (first) longest span is preferred over shorter spans.

util.get_words_and_spaces functionv3.0

Given a list of words and a text, reconstruct the original tokens and return a list of words and spaces that can be used to create a Doc. This can help recover destructive tokenization that didn’t preserve any whitespace information.