spaCy’s CLI provides a range of helpful commands for downloading and training pipelines, converting data and debugging your config, data and installation. For a list of available commands, you can type python -m spacy --help. You can also add the --help flag to any command or subcommand to see the description, available arguments and usage.

download command

Download trained pipelines for spaCy. The downloader finds the best-matching compatible version and uses pip install to download the Python package. Direct downloads don’t perform any compatibility checks and require the pipeline name to be specified with its version (e.g. en_core_web_sm-3.0.0).

info command

Print information about your spaCy installation, trained pipelines and local setup, and generate Markdown-formatted markup to copy-paste into GitHub issues.

validate command

Find all trained pipeline packages installed in the current environment and check whether they are compatible with the currently installed version of spaCy. Should be run after upgrading spaCy via pip install -U spacy to ensure that all installed packages can be used with the new version. It will show a list of packages and their installed versions. If any package is out of date, the latest compatible versions and command for updating are shown.

init v3.0

The spacy init CLI includes helpful commands for initializing training config files and pipeline directories.

init config commandv3.0

Initialize and save a config.cfg file using the recommended settings for your use case. It works just like the quickstart widget, only that it also auto-fills all default values and exports a training-ready config. The settings you specify will impact the suggested model architectures and pipeline setup, as well as the hyperparameters. You can also adjust and customize those settings in your config file later.

init fill-config v3.0

Auto-fill a partial .cfg file with all default values, e.g. a config generated with the quickstart widget. Config files used for training should always be complete and not contain any hidden defaults or missing values, so this command helps you create your final training config. In order to find the available settings and defaults, all functions referenced in the config will be created, and their signatures are used to find the defaults. If your config contains a problem that can’t be resolved automatically, spaCy will show you a validation error with more details.

init fill-curated-transformer commandv3.7

Auto-fill the Hugging Face model hyperpameters and loader parameters of a Curated Transformer pipeline component in a .cfg file. The name and revision of the Hugging Face model can either be passed as command-line arguments or read from the initialize.components.transformer.encoder_loader config section.

init vectors commandv3.0

Convert word vectors for use with spaCy. Will export an nlp object that you can use in the [initialize] block of your config to initialize a model with vectors. See the usage guide on static vectors for details on how to use vectors in your model.

init labels commandv3.0

Generate JSON files for the labels in the data. This helps speed up the training process, since spaCy won’t have to preprocess the data to extract the labels. After generating the labels, you can provide them to components that accept a labels argument on initialization via the [initialize] block of your config.

find-function commandv3.7

Find the module, path and line number to the file for a given registered function. This functionality is helpful to understand where registered functions, as used in the config file, are defined.

convert command

Convert files into spaCy’s binary training data format, a serialized DocBin, for use with the train command and other experiment management functions. The converter can be specified on the command line, or chosen based on the file extension of the input file.

Converters

debug v3.0

The spacy debug CLI includes helpful commands for debugging and profiling your configs, data and implementations.

debug config commandv3.0

Debug a config.cfg file and show validation errors. The command will create all objects in the tree and validate them. Note that some config validation errors are blocking and will prevent the rest of the config from being resolved. This means that you may not see all validation errors at once and some issues are only shown once previous errors have been fixed. To auto-fill a partial config and save the result, you can use the init fill-config command.

debug data command

Analyze, debug and validate your training and development data. Get useful stats, and find problems like invalid entity annotations, cyclic dependencies, low data labels and more.

debug diff-config command

Show a diff of a config file with respect to spaCy’s defaults or another config file. If additional settings were used in the creation of the config file, then you must supply these as extra parameters to the command when comparing to the default settings. The generated diff can also be used when posting to the discussion forum to provide more information for the maintainers.

debug profile command

Profile which functions take the most time in a spaCy pipeline. Input should be formatted as one JSON object per line with a key "text". It can either be provided as a JSONL file, or be read from sys.sytdin. If no input file is specified, the IMDB dataset is loaded via ml_datasets.

debug model commandv3.0

Debug a Thinc Model by running it on a sample text and checking how it updates its internal weights and parameters.

debug pieces commandv3.7

Analyze word- or sentencepiece stats.

train command

Train a pipeline. Expects data in spaCy’s binary format and a config file with all settings and hyperparameters. Will save out the best model from all epochs, as well as the final pipeline. The --code argument can be used to provide a Python file that’s imported before the training process starts. This lets you register custom functions and architectures and refer to them in your config, all while still using spaCy’s built-in train workflow. If you need to manage complex multi-step training workflows, check out the new spaCy projects.

Calling the training function from Python v3.2

The training CLI exposes a train helper function that lets you run the training just like spacy train. Usually it’s easier to use the command line directly, but if you need to kick off training from code this is how to do it.

pretrain commandexperimental

Pretrain the “token to vector” (Tok2vec) layer of pipeline components on raw text, using an approximate language-modeling objective. Specifically, we load pretrained vectors, and train a component like a CNN, BiLSTM, etc to predict vectors which match the pretrained ones. The weights are saved to a directory after each epoch. You can then include a path to one of these pretrained weights files in your training config as the init_tok2vec setting when you train your pipeline. This technique may be especially helpful if you have little labelled data. See the usage docs on pretraining for more info. To read the raw text, a JsonlCorpus is typically used.

evaluate command

The evaluate subcommand is superseded by spacy benchmark accuracy. evaluate is provided as an alias to benchmark accuracy for compatibility.

benchmark v3.5

The spacy benchmark CLI includes commands for benchmarking the accuracy and speed of your spaCy pipelines.

accuracy commandv3.5

Evaluate the accuracy of a trained pipeline. Expects a loadable spaCy pipeline (package name or path) and evaluation data in the binary .spacy format. The --gold-preproc option sets up the evaluation examples with gold-standard sentences and tokens for the predictions. Gold preprocessing helps the annotations align to the tokenization, and may result in sequences of more consistent length. However, it may reduce runtime accuracy due to train/test skew. To render a sample of dependency parses in a HTML file using the displaCy visualizations, set as output directory as the --displacy-path argument.

speed commandv3.5

Benchmark the speed of a trained pipeline with a 95% confidence interval. Expects a loadable spaCy pipeline (package name or path) and benchmark data in the binary .spacy format. The pipeline is warmed up before any measurements are taken.

apply commandv3.5

Applies a trained pipeline to data and stores the resulting annotated documents in a DocBin. The input can be a single file or a directory. The recognized input formats are:

1. .spacy
2. .jsonl containing a user specified text_key
3. Files with any other extension are assumed to be plain text files containing a single document.

When a directory is provided it is traversed recursively to collect all files.

When loading a .spacy file, any potential annotations stored on the Doc that are not overwritten by the pipeline will be preserved. If you want to evaluate the pipeline on raw text only, make sure that the .spacy file does not contain any annotations.

find-threshold commandv3.5

Runs prediction trials for a trained model with varying thresholds to maximize the specified metric. The search space for the threshold is traversed linearly from 0 to 1 in n_trials steps. Results are displayed in a table on stdout (the corresponding API call to spacy.cli.find_threshold.find_threshold() returns all results).

This is applicable only for components whose predictions are influenced by thresholds - e.g. textcat_multilabel and spancat, but not textcat. Note that the full path to the corresponding threshold attribute in the config has to be provided.

assemble command

Assemble a pipeline from a config file without additional training. Expects a config file with all settings and hyperparameters. The --code argument can be used to import a Python file that lets you register custom functions and refer to them in your config.

package command

Generate an installable Python package from an existing pipeline data directory. All data files are copied over. If additional code files are provided (e.g. Python files containing custom registered functions like pipeline components), they are copied into the package and imported in the __init__.py. If the path to a meta.json is supplied, or a meta.json is found in the input directory, this file is used. Otherwise, the data can be entered directly from the command line. spaCy will then create a build artifact that you can distribute and install with pip install. As of v3.1, the package command will also create a formatted README.md based on the pipeline information defined in the meta.json. If a README.md is already present in the source directory, it will be used instead.

project v3.0

The spacy project CLI includes subcommands for working with spaCy projects, end-to-end workflows for building and deploying custom spaCy pipelines.

project clone command

Clone a project template from a Git repository. Calls into git under the hood and can use the sparse checkout feature if available, so you’re only downloading what you need. By default, spaCy’s project templates repo is used, but you can provide any other repo (public or private) that you have access to using the --repo option.

project assets command

Fetch project assets like datasets and pretrained weights. Assets are defined in the assets section of the project.yml. If a checksum is provided, the file is only downloaded if no local file with the same checksum exists and spaCy will show an error if the checksum of the downloaded file doesn’t match. If assets don’t specify a url they’re considered “private” and you have to take care of putting them into the destination directory yourself. If a local path is provided, the asset is copied into the current project.

project run command

Run a named command or workflow defined in the project.yml. If a workflow name is specified, all commands in the workflow are run, in order. If commands define dependencies or outputs, they will only be re-run if state has changed. For example, if the input dataset changes, a preprocessing command that depends on those files will be re-run.

project push command

Upload all available files or directories listed as in the outputs section of commands to a remote storage. Outputs are archived and compressed prior to upload, and addressed in the remote storage using the output’s relative path (URL encoded), a hash of its command string and dependencies, and a hash of its file contents. This means push should never overwrite a file in your remote. If all the hashes match, the contents are the same and nothing happens. If the contents are different, the new version of the file is uploaded. Deleting obsolete files is left up to you.

Remotes can be defined in the remotes section of the project.yml. Under the hood, spaCy uses cloudpathlib to communicate with the remote storages, so you can use any protocol that cloudpathlib supports, including S3, Google Cloud Storage, and the local filesystem, although you may need to install extra dependencies to use certain protocols.

project pull command

Download all files or directories listed as outputs for commands, unless they are already present locally. When searching for files in the remote, pull won’t just look at the output path, but will also consider the command string and the hashes of the dependencies. For instance, let’s say you’ve previously pushed a checkpoint to the remote, but now you’ve changed some hyper-parameters. Because you’ve changed the inputs to the command, if you run pull, you won’t retrieve the stale result. If you train your pipeline and push the outputs to the remote, the outputs will be saved alongside the prior outputs, so if you change the config back, you’ll be able to fetch back the result.

Remotes can be defined in the remotes section of the project.yml. Under the hood, spaCy uses Pathy to communicate with the remote storages, so you can use any protocol that Pathy supports, including S3, Google Cloud Storage, and the local filesystem, although you may need to install extra dependencies to use certain protocols.

project document command

Auto-generate a pretty Markdown-formatted README for your project, based on its project.yml. Will create sections that document the available commands, workflows and assets. The auto-generated content will be placed between two hidden markers, so you can add your own custom content before or after the auto-generated documentation. When you re-run the project document command, only the auto-generated part is replaced.

Example output

For more examples, see the templates in our projects repo.

project dvc command

Auto-generate Data Version Control (DVC) config file. Calls dvc run with --no-exec under the hood to generate the dvc.yaml. A DVC project can only define one pipeline, so you need to specify one workflow defined in the project.yml. If no workflow is specified, the first defined workflow is used. The DVC config will only be updated if the project.yml changed. For details, see the DVC integration docs.

huggingface-hub v3.1

The spacy huggingface-cli CLI includes commands for uploading your trained spaCy pipelines to the Hugging Face Hub.

huggingface-hub push command

Push a spaCy pipeline to the Hugging Face Hub. Expects a .whl file packaged with spacy package and --build wheel. For more details, see the spaCy project integration.