Component for assigning base forms to tokens using rules based on part-of-speech tags, or lookup tables. Different Language subclasses can implement their own lemmatizer components via language-specific factories. The default data used is provided by the spacy-lookups-data extension package.

For a trainable lemmatizer, see EditTreeLemmatizer.

Assigned Attributes

Lemmas generated by rules or predicted will be saved to Token.lemma.

Config and implementation

The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the config argument on nlp.add_pipe or in your config.cfg for training. For examples of the lookups data format used by the lookup and rule-based lemmatizers, see spacy-lookups-data.

Many languages specify a default lemmatizer mode other than lookup if a better lemmatizer is available. The lemmatizer modes rule and pos_lookup require token.pos from a previous pipeline component (see example pipeline configurations in the pretrained pipeline design details) or rely on third-party libraries (pymorphy3).

explosion/spaCy/master/spacy/pipeline/lemmatizer.py

Lemmatizer.__init__ method

Create a new pipeline instance. In your application, you would normally use a shortcut for this and instantiate the component using its string name and nlp.add_pipe.

Lemmatizer.__call__ method

Apply the pipe to one document. The document is modified in place, and returned. This usually happens under the hood when the nlp object is called on a text and all pipeline components are applied to the Doc in order.

Lemmatizer.pipe method

Apply the pipe to a stream of documents. This usually happens under the hood when the nlp object is called on a text and all pipeline components are applied to the Doc in order.

Lemmatizer.initialize method

Initialize the lemmatizer and load any data resources. This method is typically called by Language.initialize and lets you customize arguments it receives via the [initialize.components] block in the config. The loading only happens during initialization, typically before training. At runtime, all data is loaded from disk.

Lemmatizer.lookup_lemmatize method

Lemmatize a token using a lookup-based approach. If no lemma is found, the original string is returned.

Lemmatizer.rule_lemmatize method

Lemmatize a token using a rule-based approach. Typically relies on POS tags.

Lemmatizer.is_base_form method

Check whether we’re dealing with an uninflected paradigm, so we can avoid lemmatization entirely.

Lemmatizer.get_lookups_config classmethod

Returns the lookups configuration settings for a given mode for use in Lemmatizer.load_lookups.

Lemmatizer.to_disk method

Serialize the pipe to disk.

Lemmatizer.from_disk method

Load the pipe from disk. Modifies the object in place and returns it.

Lemmatizer.to_bytes method

Serialize the pipe to a bytestring.

Lemmatizer.from_bytes method

Load the pipe from a bytestring. Modifies the object in place and returns it.

Attributes

Serialization fields

During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from serialization by passing in the string names via the exclude argument.