Models & Languages
spaCy’s trained pipelines can be installed as Python packages. This means
that they’re a component of your application, just like any other module.
They’re versioned and can be defined as a dependency in your requirements.txt
.
Trained pipelines can be installed from a download URL or a local directory,
manually or via pip. Their data can be
located anywhere on your file system.
Quickstart
Install a default trained pipeline package, get the code to load it from within spaCy and an example to test it. For more options, see the section on available packages below.
python -m spacy download en_core_web_smimport spacynlp = spacy.load("en_core_web_sm")import en_core_web_smnlp = en_core_web_sm.load()doc = nlp("This is a sentence.")print([(w.text, w.pos_) for w in doc])
Usage note
If a trained pipeline is available for a language, you can download it using the
spacy download
command as shown above. In order to use
languages that don’t yet come with a trained pipeline, you have to import them
directly, or use spacy.blank
:
A blank pipeline is typically just a tokenizer. You might want to create a blank
pipeline when you only need a tokenizer, when you want to add more components
from scratch, or for testing purposes. Initializing the language object directly
yields the same result as generating it using spacy.blank()
. In both cases the
default configuration for the chosen language is loaded, and no pretrained
components will be available.
Language support
spaCy currently provides support for the following languages. You can help by improving the existing language data and extending the tokenization patterns. See here for details on how to contribute to development. Also see the training documentation for how to train your own pipelines on your data.
Language | Code | Language Data | Pipelines |
---|---|---|---|
Catalan | ca | lang/ca | 4 packages |
Chinese | zh | lang/zh | 4 packages |
Croatian | hr | lang/hr | 3 packages |
Danish | da | lang/da | 4 packages |
Dutch | nl | lang/nl | 3 packages |
English | en | lang/en | 4 packages |
Finnish | fi | lang/fi | 3 packages |
French | fr | lang/fr | 4 packages |
German | de | lang/de | 4 packages |
Greek | el | lang/el | 3 packages |
Italian | it | lang/it | 3 packages |
Japanese | ja | lang/ja | 4 packages |
Korean | ko | lang/ko | 3 packages |
Lithuanian | lt | lang/lt | 3 packages |
Macedonian | mk | lang/mk | 3 packages |
Multi-language | xx | lang/xx | 2 packages |
Norwegian Bokmål | nb | lang/nb | 3 packages |
Polish | pl | lang/pl | 3 packages |
Portuguese | pt | lang/pt | 3 packages |
Romanian | ro | lang/ro | 3 packages |
Russian | ru | lang/ru | 3 packages |
Slovenian | sl | lang/sl | 4 packages |
Spanish | es | lang/es | 4 packages |
Swedish | sv | lang/sv | 3 packages |
Ukrainian | uk | lang/uk | 4 packages |
Afrikaans | af | lang/af | none yet |
Albanian | sq | lang/sq | none yet |
Amharic | am | lang/am | none yet |
Ancient Greek | grc | lang/grc | none yet |
Arabic | ar | lang/ar | none yet |
Armenian | hy | lang/hy | none yet |
Azerbaijani | az | lang/az | none yet |
Basque | eu | lang/eu | none yet |
Bengali | bn | lang/bn | none yet |
Bulgarian | bg | lang/bg | none yet |
Czech | cs | lang/cs | none yet |
Estonian | et | lang/et | none yet |
Faroese | fo | lang/fo | none yet |
Gujarati | gu | lang/gu | none yet |
Hebrew | he | lang/he | none yet |
Hindi | hi | lang/hi | none yet |
Hungarian | hu | lang/hu | none yet |
Icelandic | is | lang/is | none yet |
Indonesian | id | lang/id | none yet |
Irish | ga | lang/ga | none yet |
Kannada | kn | lang/kn | none yet |
Kyrgyz | ky | lang/ky | none yet |
Latin | la | lang/la | none yet |
Latvian | lv | lang/lv | none yet |
Ligurian | lij | lang/lij | none yet |
Lower Sorbian | dsb | lang/dsb | none yet |
Luganda | lg | lang/lg | none yet |
Luxembourgish | lb | lang/lb | none yet |
Malay | ms | lang/ms | none yet |
Malayalam | ml | lang/ml | none yet |
Marathi | mr | lang/mr | none yet |
Nepali | ne | lang/ne | none yet |
Norwegian Nynorsk | nn | lang/nn | none yet |
Persian | fa | lang/fa | none yet |
Sanskrit | sa | lang/sa | none yet |
Serbian | sr | lang/sr | none yet |
Setswana | tn | lang/tn | none yet |
Sinhala | si | lang/si | none yet |
Slovak | sk | lang/sk | none yet |
Tagalog | tl | lang/tl | none yet |
Tamil | ta | lang/ta | none yet |
Tatar | tt | lang/tt | none yet |
Telugu | te | lang/te | none yet |
Thai | th | lang/th | none yet |
Tigrinya | ti | lang/ti | none yet |
Turkish | tr | lang/tr | none yet |
Upper Sorbian | hsb | lang/hsb | none yet |
Urdu | ur | lang/ur | none yet |
Vietnamese | vi | lang/vi | none yet |
Yoruba | yo | lang/yo | none yet |
Multi-language support
spaCy also supports pipelines trained on more than one language. This is
especially useful for named entity recognition. The language ID used for
multi-language or language-neutral pipelines is xx
. The language class, a
generic subclass containing only the base language data, can be found in
lang/xx
.
To train a pipeline using the neutral multi-language class, you can set
lang = "xx"
in your training config. You can also
\import the MultiLanguage
class directly, or call
spacy.blank("xx")
for lazy-loading.
Chinese language support
The Chinese language class supports three word segmentation options, char
,
jieba
and pkuseg
.
config.cfg
Segmenter | Description |
---|---|
char | Character segmentation: Character segmentation is the default segmentation option. It’s enabled when you create a new Chinese language class or call spacy.blank("zh") . |
jieba | Jieba: to use Jieba for word segmentation, you can set the option segmenter to "jieba" . |
pkuseg | PKUSeg: As of spaCy v2.3.0, support for PKUSeg has been added to support better segmentation for Chinese OntoNotes and the provided Chinese pipelines. Enable PKUSeg by setting tokenizer option segmenter to "pkuseg" . |
The initialize
method for the Chinese tokenizer class supports the following
config settings for loading pkuseg
models:
Name | Description |
---|---|
pkuseg_model | Name of a model provided by spacy-pkuseg or the path to a local model directory. str |
pkuseg_user_dict | Optional path to a file with one word per line which overrides the default pkuseg user dictionary. Defaults to "default" , the default provided dictionary. str |
The initialization settings are typically provided in the training config and the data is loaded in before training and serialized with the model. This allows you to load the data from a local path and save out your pipeline and config, without requiring the same local path at runtime. See the usage guide on the config lifecycle for more background on this.
config.cfg
You can also initialize the tokenizer for a blank language class by calling its
initialize
method:
Examples
You can also modify the user dictionary on-the-fly:
The Chinese pipelines provided by spaCy include a custom pkuseg
model trained only on
Chinese OntoNotes 5.0, since the
models provided by pkuseg
include data restricted to research use. For
research use, pkuseg
provides models for several different domains ("mixed"
(equivalent to "default"
from pkuseg
packages), "news"
"web"
,
"medicine"
, "tourism"
) and for other uses, pkuseg
provides a simple
training API:
Japanese language support
The Japanese language class uses
SudachiPy for word
segmentation and part-of-speech tagging. The default Japanese language class and
the provided Japanese pipelines use SudachiPy split mode A
. The tokenizer
config can be used to configure the split mode to A
, B
or C
.
config.cfg
Extra information, such as reading, inflection form, and the SudachiPy
normalized form, is available in Token.morph
. For B
or C
split modes,
subtokens are stored in Doc.user_data["sub_tokens"]
.
Korean language support
The default MeCab-based Korean tokenizer requires:
For some Korean datasets and tasks, the rule-based tokenizer is better-suited than MeCab. To configure a Korean pipeline with the rule-based tokenizer:
config.cfg
Installing and using trained pipelines
The easiest way to download a trained pipeline is via spaCy’s
download
command. It takes care of finding the
best-matching package compatible with your spaCy installation.
The download command will install the package via
pip and place the package in your site-packages
directory.
If you’re in a Jupyter notebook or similar environment, you can use the !
prefix to
execute commands.
Make sure to restart your kernel or runtime after installation (just like
you would when installing other Python packages) to make sure that the installed
pipeline package can be found.
Installation via pip
To download a trained pipeline directly using
pip, point pip install
to the URL or local
path of the wheel file or archive. Installing the wheel is usually more
efficient.
By default, this will install the pipeline package into your site-packages
directory. You can then use spacy.load
to load it via its package name or
import it explicitly as a module. If you need to download
pipeline packages as part of an automated process, we recommend using pip with a
direct link, instead of relying on spaCy’s download
command.
You can also add the direct download link to your application’s
requirements.txt
. For more details, see the section on
working with pipeline packages in production.
Manual download and installation
In some cases, you might prefer downloading the data manually, for example to place it into a custom directory. You can download the package via your browser from the latest releases, or configure your own download script using the URL of the archive file. The archive consists of a package directory that contains another directory with the pipeline data.
Directory structure
You can place the pipeline package directory anywhere on your local file system.
Installation from Python
Since the spacy download
command installs the pipeline as
a Python package, we always recommend running it from the command line, just
like you install other Python packages with pip install
. However, if you need
to, or if you want to integrate the download process into another CLI command,
you can also import and call the download
function used by the CLI via Python.
Using trained pipelines with spaCy
To load a pipeline package, use spacy.load
with
the package name or a path to the data directory:
Importing pipeline packages as modules
If you’ve installed a trained pipeline via spacy download
or directly via pip, you can also import
it and then call its load()
method
with no arguments:
Editable Code
How you choose to load your trained pipelines ultimately depends on personal
preference. However, for larger code bases, we usually recommend native
imports, as this will make it easier to integrate pipeline packages with your
existing build process, continuous integration workflow and testing framework.
It’ll also prevent you from ever trying to load a package that is not installed,
as your code will raise an ImportError
immediately, instead of failing
somewhere down the line when calling spacy.load()
. For more details, see the
section on working with pipeline packages in production.
Using trained pipelines in production
If your application depends on one or more trained pipeline packages, you’ll usually want to integrate them into your continuous integration workflow and build process. While spaCy provides a range of useful helpers for downloading and loading pipeline packages, the underlying functionality is entirely based on native Python packaging. This allows your application to handle a spaCy pipeline like any other package dependency.
Downloading and requiring package dependencies
spaCy’s built-in download
command is mostly intended as a
convenient, interactive wrapper. It performs compatibility checks and prints
detailed error messages and warnings. However, if you’re downloading pipeline
packages as part of an automated build process, this only adds an unnecessary
layer of complexity. If you know which packages your application needs, you
should be specifying them directly.
Because pipeline packages are valid Python packages, you can add them to your
application’s requirements.txt
. If you’re running your own internal PyPi
installation, you can upload the pipeline packages there. pip’s
requirements file format
supports both package names to download via a PyPi server, as well as
direct URLs. For instance, you can specify the
en_core_web_sm
model for spaCy 3.7.x as follows:
requirements.txt
See the list of models for model download links for the current spaCy version.
All pipeline packages are versioned and specify their spaCy dependency. This
ensures cross-compatibility and lets you specify exact version requirements for
each pipeline. If you’ve trained your own pipeline, you can
use the spacy package
command to generate the required
meta data and turn it into a loadable package.
Loading and testing pipeline packages
Pipeline packages are regular Python packages, so you can also import them as a
package using Python’s native import
syntax, and then call the load
method
to load the data and return an nlp
object:
In general, this approach is recommended for larger code bases, as it’s more
“native”, and doesn’t rely on spaCy’s loader to resolve string names to
packages. If a package can’t be imported, Python will raise an ImportError
immediately. And if a package is imported but not used, any linter will catch
that.
Similarly, it’ll give you more flexibility when writing tests that require
loading pipelines. For example, instead of writing your own try
and except
logic around spaCy’s loader, you can use
pytest’s
importorskip()
method to only run a test if a specific pipeline package or version is
installed. Each pipeline package exposes a __version__
attribute which you can
also use to perform your own version compatibility checks before loading it.