Other

Vocab

class
A storage class for vocabulary and other data shared across a language

The Vocab object provides a lookup table that allows you to access Lexeme objects, as well as the StringStore. It also owns underlying C-data that is shared between Doc objects.

Vocab.__init__ method

Create the vocabulary.

NameDescription
lex_attr_gettersA dictionary mapping attribute IDs to functions to compute them. Defaults to None. Optional[Dict[str, Callable[[str], Any]]]
stringsA StringStore that maps strings to hash values, and vice versa, or a list of strings. Union[List[str],StringStore]
lookupsA Lookups that stores the lexeme_norm and other large lookup tables. Defaults to None. Optional[Lookups]
oov_probThe default OOV probability. Defaults to -20.0. float
vectors_nameA name to identify the vectors table. str
writing_systemA dictionary describing the language’s writing system. Typically provided by Language.Defaults. Dict[str, Any]
get_noun_chunksA function that yields base noun phrases used for Doc.noun_chunks. Optional[Callable[[Union[Doc,Span], Iterator[Tuple[int, int, int]]]]]

Vocab.__len__ method

Get the current number of lexemes in the vocabulary.

NameDescription

Vocab.__getitem__ method

Retrieve a lexeme, given an int ID or a string. If a previously unseen string is given, a new lexeme is created and stored.

NameDescription
id_or_stringThe hash value of a word, or its string. Union[int, str]

Vocab.__iter__ method

Iterate over the lexemes in the vocabulary.

NameDescription

Vocab.__contains__ method

Check whether the string has an entry in the vocabulary. To get the ID for a given string, you need to look it up in vocab.strings.

NameDescription
stringThe ID string. str

Vocab.add_flag method

Set a new boolean flag to words in the vocabulary. The flag_getter function will be called over the words currently in the vocab, and then applied to new words as they occur. You’ll then be able to access the flag value on each token, using token.check_flag(flag_id).

NameDescription
flag_getterA function that takes the lexeme text and returns the boolean flag value. Callable[[str], bool]
flag_idAn integer between 1 and 63 (inclusive), specifying the bit at which the flag will be stored. If -1, the lowest available bit will be chosen. int

Vocab.reset_vectors method

Drop the current vector table. Because all vectors must be the same width, you have to call this to change the size of the vectors. Only one of the width and shape keyword arguments can be specified.

NameDescription
keyword-only
widthThe new width. int
shapeThe new shape. int

Vocab.prune_vectors method

Reduce the current vector table to nr_row unique entries. Words mapped to the discarded vectors will be remapped to the closest vector among those remaining. For example, suppose the original table had vectors for the words: ['sat', 'cat', 'feline', 'reclined']. If we prune the vector table to, two rows, we would discard the vectors for “feline” and “reclined”. These words would then be remapped to the closest remaining vector – so “feline” would have the same vector as “cat”, and “reclined” would have the same vector as “sat”. The similarities are judged by cosine. The original vectors may be large, so the cosines are calculated in minibatches to reduce memory usage.

NameDescription
nr_rowThe number of rows to keep in the vector table. int
batch_sizeBatch of vectors for calculating the similarities. Larger batch sizes might be faster, while temporarily requiring more memory. int

Vocab.deduplicate_vectors methodv3.3

Remove any duplicate rows from the current vector table, maintaining the mappings for all words in the vectors.

Vocab.get_vector method

Retrieve a vector for a word in the vocabulary. Words can be looked up by string or hash value. If the current vectors do not contain an entry for the word, a 0-vector with the same number of dimensions (Vocab.vectors_length) as the current vectors is returned.

NameDescription
orthThe hash value of a word, or its unicode string. Union[int, str]

Vocab.set_vector method

Set a vector for a word in the vocabulary. Words can be referenced by string or hash value.

NameDescription
orthThe hash value of a word, or its unicode string. Union[int, str]
vectorThe vector to set. numpy.ndarray[ndim=1, dtype=float32]

Vocab.has_vector method

Check whether a word has a vector. Returns False if no vectors are loaded. Words can be looked up by string or hash value.

NameDescription
orthThe hash value of a word, or its unicode string. Union[int, str]

Vocab.to_disk method

Save the current state to a directory.

NameDescription
pathA path to a directory, which will be created if it doesn’t exist. Paths may be either strings or Path-like objects. Union[str,Path]
keyword-only
excludeString names of serialization fields to exclude. Iterable[str]

Vocab.from_disk method

Loads state from a directory. Modifies the object in place and returns it.

NameDescription
pathA path to a directory. Paths may be either strings or Path-like objects. Union[str,Path]
keyword-only
excludeString names of serialization fields to exclude. Iterable[str]

Vocab.to_bytes method

Serialize the current state to a binary string.

NameDescription
keyword-only
excludeString names of serialization fields to exclude. Iterable[str]

Vocab.from_bytes method

Load state from a binary string.

NameDescription
bytes_dataThe data to load from. bytes
keyword-only
excludeString names of serialization fields to exclude. Iterable[str]

Attributes

NameDescription
stringsA table managing the string-to-int mapping. StringStore
vectorsA table associating word IDs to word vectors. Vectors
vectors_lengthNumber of dimensions for each word vector. int
lookupsThe available lookup tables in this vocab. Lookups
writing_systemA dict with information about the language’s writing system. Dict[str, Any]
get_noun_chunks v3.0A function that yields base noun phrases used for Doc.noun_chunks. Optional[Callable[[Union[Doc,Span], Iterator[Tuple[int, int, int]]]]]

Serialization fields

During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from serialization by passing in the string names via the exclude argument.

NameDescription
stringsThe strings in the StringStore.
vectorsThe word vectors, if available.
lookupsThe lookup tables, if available.