Match sequences of tokens, based on pattern rules

The Matcher lets you find words and phrases using rules describing their token attributes. Rules can refer to token annotations (like the text or part-of-speech tags), as well as lexical attributes like Token.is_punct. Applying the matcher to a Doc gives you access to the matched tokens in context. For in-depth examples and workflows for combining rules and statistical models, see the usage guide on rule-based matching.

Pattern format

A pattern added to the Matcher consists of a list of dictionaries. Each dictionary describes one token and its attributes. The available token pattern keys correspond to a number of Token attributes. The supported attributes for rule-based matching are:

Attribute Description
ORTHThe exact verbatim text of a token. str
TEXT The exact verbatim text of a token. str
LOWERThe lowercase form of the token text. str
 LENGTHThe length of the token text. int
 IS_ALPHA, IS_ASCII, IS_DIGITToken text consists of alphabetic characters, ASCII characters, digits. bool
 IS_LOWER, IS_UPPER, IS_TITLEToken text is in lowercase, uppercase, titlecase. bool
 IS_PUNCT, IS_SPACE, IS_STOPToken is punctuation, whitespace, stop word. bool
 IS_SENT_STARTToken is start of sentence. bool
 LIKE_NUM, LIKE_URL, LIKE_EMAILToken text resembles a number, URL, email. bool
SPACYToken has a trailing space. bool
 POS, TAG, MORPH, DEP, LEMMA, SHAPEThe token’s simple and extended part-of-speech tag, morphological analysis, dependency label, lemma, shape. str
ENT_TYPEThe token’s entity label. str
_ Properties in custom extension attributes. Dict[str, Any]
OPOperator or quantifier to determine how often to match a token pattern. str

Operators and quantifiers define how often a token pattern should be matched:

!Negate the pattern, by requiring it to match exactly 0 times.
?Make the pattern optional, by allowing it to match 0 or 1 times.
+Require the pattern to match 1 or more times.
*Allow the pattern to match 0 or more times.

Token patterns can also map to a dictionary of properties instead of a single value to indicate whether the expected value is a member of a list or how it compares to another value.

INAttribute value is member of a list. Any
NOT_INAttribute value is not member of a list. Any
ISSUBSETAttribute values (for MORPH) are a subset of a list. Any
ISSUPERSETAttribute values (for MORPH) are a superset of a list. Any
==, >=, <=, >, <Attribute value is equal, greater or equal, smaller or equal, greater or smaller. Union[int, float]

Matcher.__init__ method

Create the rule-based Matcher. If validate=True is set, all patterns added to the matcher will be validated against a JSON schema and a MatchPatternError is raised if problems are found. Those can include incorrect types (e.g. a string where an integer is expected) or unexpected property names.

vocabThe vocabulary object, which must be shared with the documents the matcher will operate on. Vocab
validate Validate all patterns added to this matcher. bool

Matcher.__call__ method

Find all token sequences matching the supplied patterns on the Doc or Span.

doclikeThe Doc or Span to match over. Union[Doc, Span]
as_spans v3.0Instead of tuples, return a list of Span objects of the matches, with the match_id assigned as the span label. Defaults to False. bool
allow_missing v3.0Whether to skip checks for missing annotation for attributes included in patterns. Defaults to False. bool

Matcher.__len__ method

Get the number of rules added to the matcher. Note that this only returns the number of rules (identical with the number of IDs), not the number of individual patterns.


Matcher.__contains__ method

Check whether the matcher contains rules for a match ID.

keyThe match ID. str

Matcher.add method

Add a rule to the matcher, consisting of an ID key, one or more patterns, and an optional callback function to act on the matches. The callback function will receive the arguments matcher, doc, i and matches. If a pattern already exists for the given ID, the patterns will be extended. An on_match callback will be overwritten.

match_idAn ID for the thing you’re matching. str
patternsMatch pattern. A pattern consists of a list of dicts, where each dict describes a token. List[List[Dict[str, Any]]]
on_matchCallback function to act on matches. Takes the arguments matcher, doc, i and matches. Optional[Callable[[Matcher, Doc, int, List[tuple], Any]]
greedy v3.0Optional filter for greedy matches. Can either be "FIRST" or "LONGEST". Optional[str]

Matcher.remove method

Remove a rule from the matcher. A KeyError is raised if the match ID does not exist.

keyThe ID of the match rule. str

Matcher.get method

Retrieve the pattern stored for a key. Returns the rule as an (on_match, patterns) tuple containing the callback and available patterns.

keyThe ID of the match rule. str