Pack Doc objects for binary serialization

The DocBin class lets you efficiently serialize the information from a collection of Doc objects. You can control which information is serialized by passing a list of attribute IDs, and optionally also specify whether the user data is serialized. The DocBin is faster and produces smaller data sizes than pickle, and allows you to deserialize without executing arbitrary Python code. A notable downside to this format is that you can’t easily extract just one document from the DocBin. The serialization format is gzipped msgpack, where the msgpack object has the following structure:

msgpack object strcutrue

{ "attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE] "tokens": bytes, # Serialized numpy uint64 array with the token data "spaces": bytes, # Serialized numpy boolean array with spaces data "lengths": bytes, # Serialized numpy int32 array with the doc lengths "strings": List[unicode] # List of unique strings in the token data }

Strings for the words, tags, labels etc are represented by 64-bit hashes in the token data, and every string that occurs at least once is passed via the strings object. This means the storage is more efficient if you pack more documents together, because you have less duplication in the strings. For usage examples, see the docs on serializing Doc objects.

DocBin.__init__ method

Create a DocBin object to hold serialized annotations.

attrslistList of attributes to serialize. orth (hash of token text) and spacy (whether the token is followed by whitespace) are always serialized, so they’re not required. Defaults to None.
store_user_databoolWhether to include the Doc.user_data and the values of custom extension attributes. Defaults to False.

DocBin._len__ method

Get the number of Doc objects that were added to the DocBin.


DocBin.add method

Add a Doc’s annotations to the DocBin for serialization.

docDocThe Doc object to add.

DocBin.get_docs method

Recover Doc objects from the annotations, using the given vocab.

vocabVocabThe shared vocab.

DocBin.merge method

Extend the annotations of this DocBin with the annotations from another. Will raise an error if the pre-defined attrs of the two DocBins don’t match.

otherDocBinThe DocBin to merge into the current bin.

DocBin.to_bytes method

Serialize the DocBin’s annotations to a bytestring.


DocBin.from_bytes method

Deserialize the DocBin’s annotations from a bytestring.

bytes_databytesThe data to load from.