Overview

Command Line Interface

Download, train and package models, and debug spaCy

As of v1.7.0, spaCy comes with new command line helpers to download and link models and show useful debugging information. For a list of available commands, type spacy --help.

Download

Download models for spaCy. The downloader finds the best-matching compatible version, uses pip to download the model as a package and automatically creates a shortcut link to load the model by name. Direct downloads don’t perform any compatibility checks and require the model name to be specified with its version (e.g. en_core_web_sm-2.0.0).

python -m spacy download [model] [--direct]
ArgumentTypeDescription
modelpositionalModel name or shortcut (en, de, en_core_web_sm).
--direct, -dflagForce direct download of exact model version.
other v2.1-Additional installation options to be passed to pip install when installing the model package. For example, --user to install to the user home directory.
--help, -hflagShow help message and available arguments.

Info

Print information about your spaCy installation, models and local setup, and generate Markdown-formatted markup to copy-paste into GitHub issues.

python -m spacy info [--markdown]
python -m spacy info [model] [--markdown]
ArgumentTypeDescription
modelpositionalA model, i.e. shortcut link, package name or path (optional).
--markdown, -mdflagPrint information as Markdown.
--silent, -s v2.0.12flagDon’t print anything, just return the values.
--help, -hflagShow help message and available arguments.

Validate v2.0

Find all models installed in the current environment (both packages and shortcut links) and check whether they are compatible with the currently installed version of spaCy. Should be run after upgrading spaCy via pip install -U spacy to ensure that all installed models are can be used with the new version. The command is also useful to detect out-of-sync model links resulting from links created in different virtual environments. It will a list of models, the installed versions, the latest compatible version (if out of date) and the commands for updating.

python -m spacy validate
ArgumentTypeDescription

Convert

Convert files into spaCy’s JSON format for use with the train command and other experiment management functions. The converter can be specified on the command line, or chosen based on the file extension of the input file.

python -m spacy convert [input_file] [output_dir] [--file-type] [--converter]
[--n-sents] [--morphology] [--lang]
ArgumentTypeDescription
input_filepositionalInput file.
output_dirpositionalOutput directory for converted file. Defaults to "-", meaning data will be written to stdout.
--file-type, -t v2.1optionType of file to create (see below).
--converter, -c v2.0optionName of converter to use (see below).
--n-sents, -noptionNumber of sentences per document.
--morphology, -moptionEnable appending morphology to tags.
--lang, -l v2.1optionLanguage code (if tokenizer required).
--help, -hflagShow help message and available arguments.

Output file types v2.1

All output files generated by this command are compatible with spacy train.

IDDescription
jsonlNewline-delimited JSON (default).
jsonRegular JSON.
msgBinary MessagePack format.

Converter options

IDDescription
autoAutomatically pick converter based on file extension (default).
conll, conllu, conllubioUniversal Dependencies .conllu or .conll format.
nerTab-based named entity recognition format.
iobIOB or IOB2 named entity recognition format.

Train

Train a model. Expects data in spaCy’s JSON format. On each epoch, a model will be saved out to the directory. Accuracy scores and model details will be added to a meta.json to allow packaging the model using the package command.

python -m spacy train [lang] [output_path] [train_path] [dev_path]
[--base-model] [--pipeline] [--vectors] [--n-iter] [--n-examples] [--use-gpu]
[--version] [--meta-path] [--init-tok2vec] [--parser-multitasks]
[--entity-multitasks] [--gold-preproc] [--noise-level] [--learn-tokens]
[--verbose]
ArgumentTypeDescription
langpositionalModel language.
output_pathpositionalDirectory to store model in. Will be created if it doesn’t exist.
train_pathpositionalLocation of JSON-formatted training data. Can be a file or a directory of files.
dev_pathpositionalLocation of JSON-formatted development data for evaluation. Can be a file or a directory of files.
--base-model, -boptionOptional name of base model to update. Can be any loadable spaCy model.
--pipeline, -p v2.1optionComma-separated names of pipeline components to train. Defaults to 'tagger,parser,ner'.
--vectors, -voptionModel to load vectors from.
--n-iter, -noptionNumber of iterations (default: 30).
--n-examples, -nsoptionNumber of examples to use (defaults to 0 for all examples).
--use-gpu, -goptionWhether to use GPU. Can be either 0, 1 or -1.
--version, -VoptionModel version. Will be written out to the model’s meta.json after training.
--meta-path, -m v2.0optionOptional path to model meta.json. All relevant properties like lang, pipeline and spacy_version will be overwritten.
--init-tok2vec, -t2v v2.1optionPath to pretrained weights for the token-to-vector parts of the models. See spacy pretrain. Experimental.
--parser-multitasks, -ptoptionSide objectives for parser CNN, e.g. 'dep' or 'dep,tag'
--entity-multitasks, -etoptionSide objectives for NER CNN, e.g. 'dep' or 'dep,tag'
--noise-level, -nloptionFloat indicating the amount of corruption for data augmentation.
--gold-preproc, -GflagUse gold preprocessing.
--learn-tokens, -TflagMake parser learn gold-standard tokenization by merging ] subtokens. Typically used for languages like Chinese.
--verbose, -VV v2.0.13flagShow more detailed messages during training.
--help, -hflagShow help message and available arguments.

Environment variables for hyperparameters v2.0

spaCy lets you set hyperparameters for training via environment variables. For example:

token_vector_width=256 learn_rate=0.0001 spacy train [...]
NameDescriptionDefault
dropout_fromInitial dropout rate.0.2
dropout_toFinal dropout rate.0.2
dropout_decayRate of dropout change.0.0
batch_fromInitial batch size.1
batch_toFinal batch size.64
batch_compoundRate of batch size acceleration.1.001
token_vector_widthWidth of embedding tables and convolutional layers.128
embed_sizeNumber of rows in embedding tables.7500
hidden_widthSize of the parser’s and NER’s hidden layers.128
learn_rateLearning rate.0.001
optimizer_B1Momentum for the Adam solver.0.9
optimizer_B2Adagrad-momentum for the Adam solver.0.999
optimizer_epsEpsilon value for the Adam solver.1e-08
L2_penaltyL2 regularization penalty.1e-06
grad_norm_clipGradient L2 norm constraint.1.0

Pretrain experimentalv2.1

Pre-train the “token to vector” (tok2vec) layer of pipeline components, using an approximate language-modeling objective. Specifically, we load pre-trained vectors, and train a component like a CNN, BiLSTM, etc to predict vectors which match the pre-trained ones. The weights are saved to a directory after each epoch. You can then pass a path to one of these pre-trained weights files to the spacy train command.

This technique may be especially helpful if you have little labelled data. However, it’s still quite experimental, so your mileage may vary. To load the weights back in during spacy train, you need to ensure all settings are the same between pretraining and training. The API and errors around this need some improvement.

python -m spacy pretrain [texts_loc] [vectors_model] [output_dir] [--width]
[--depth] [--embed-rows] [--dropout] [--seed] [--n-iter] [--use-vectors]
ArgumentTypeDescription
texts_locpositionalPath to JSONL file with raw texts to learn from, with text provided as the key "text". See here for details.
vectors_modelpositionalName or path to spaCy model with vectors to learn from.
output_dirpositionalDirectory to write models to on each epoch.
--width, -cwoptionWidth of CNN layers.
--depth, -cdoptionDepth of CNN layers.
--embed-rows, -eroptionNumber of embedding rows.
--dropout, -doptionDropout rate.
--batch-size, -bsoptionNumber of words per training batch.
--max-length, -xwoptionMaximum words per example. Longer examples are discarded.
--min-length, -nwoptionMinimum words per example. Shorter examples are discarded.
--seed, -soptionSeed for random number generators.
--n-iter, -ioptionNumber of iterations to pretrain.
--use-vectors, -uvflagWhether to use the static vectors as input features.

JSONL format for raw text

Raw text can be provided as a .jsonl (newline-delimited JSON) file containing one input text per line (roughly paragraph length is good). Optionally, custom tokenization can be provided.

KeyTypeDescription
textunicodeThe raw input text.
tokenslistOptional tokenization, one string per token.

Example

{"text": "Can I ask where you work now and what you do, and if you enjoy it?"} {"text": "They may just pull out of the Seattle market completely, at least until they have autonomous vehicles."} {"text": "My cynical view on this is that it will never be free to the public. Reason: what would be the draw of joining the military? Right now their selling point is free Healthcare and Education. Ironically both are run horribly and most, that I've talked to, come out wishing they never went in."}

Init Model v2.0

Create a new model directory from raw data, like word frequencies, Brown clusters and word vectors. This command is similar to the spacy model command in v1.x.

python -m spacy init-model [lang] [output_dir] [--jsonl-loc] [--vectors-loc]
[--prune-vectors]
ArgumentTypeDescription
langpositionalModel language ISO code, e.g. en.
output_dirpositionalModel output directory. Will be created if it doesn’t exist.
--jsonl-loc, -joptionOptional location of JSONL-formatted vocabulary file with lexical attributes.
--vectors-loc, -voptionOptional location of vectors file. Should be a tab-separated file in Word2Vec format where the first column contains the word and the remaining columns the values. File can be provided in .txt format or as a zipped text file in .zip or .tar.gz format.
--prune-vectors, -VflagNumber of vectors to prune the vocabulary to. Defaults to -1 for no pruning.

Evaluate v2.0

Evaluate a model’s accuracy and speed on JSON-formatted annotated data. Will print the results and optionally export displaCy visualizations of a sample set of parses to .html files. Visualizations for the dependency parse and NER will be exported as separate files if the respective component is present in the model’s pipeline.

python -m spacy evaluate [model] [data_path] [--displacy-path] [--displacy-limit]
[--gpu-id] [--gold-preproc]
ArgumentTypeDescription
modelpositionalModel to evaluate. Can be a package or shortcut link name, or a path to a model data directory.
data_pathpositionalLocation of JSON-formatted evaluation data.
--displacy-path, -dpoptionDirectory to output rendered parses as HTML. If not set, no visualizations will be generated.
--displacy-limit, -dloptionNumber of parses to generate per file. Defaults to 25. Keep in mind that a significantly higher number might cause the .html files to render slowly.
--gpu-id, -goptionGPU to use, if any. Defaults to -1 for CPU.
--gold-preproc, -GflagUse gold preprocessing.

Package

Generate a model Python package from an existing model data directory. All data files are copied over. If the path to a meta.json is supplied, or a meta.json is found in the input directory, this file is used. Otherwise, the data can be entered directly from the command line. After packaging, you can run python setup.py sdist from the newly created directory to turn your model into an installable archive file.

python -m spacy package [input_dir] [output_dir] [--meta-path] [--create-meta] [--force]

Example

python -m spacy package /input /output cd /output/en_model-0.0.0 python setup.py sdist pip install dist/en_model-0.0.0.tar.gz
ArgumentTypeDescription
input_dirpositionalPath to directory containing model data.
output_dirpositionalDirectory to create package folder in.
--meta-path, -m v2.0optionPath to meta.json file (optional).
--create-meta, -c v2.0flagCreate a meta.json file on the command line, even if one already exists in the directory. If an existing file is found, its entries will be shown as the defaults in the command line prompt.
--help, -hflagShow help message and available arguments.