Large Language Models
The spacy-llm package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks, no training data required.
Config and implementation
An LLM component is implemented through the LLMWrapper
class. It is accessible
through a generic llm
component factory
as well as through task-specific component factories: llm_ner
, llm_spancat
, llm_rel
,
llm_textcat
, llm_sentiment
and llm_summarization
.
LLMWrapper.__init__ method
Create a new pipeline instance. In your application, you would normally use a
shortcut for this and instantiate the component using its string name and
nlp.add_pipe
.
Name | Description |
---|---|
name | String name of the component instance. llm by default. str |
keyword-only | |
vocab | The shared vocabulary. Vocab |
task | An LLM Task can generate prompts and parse LLM responses. LLMTask |
model | The LLM Model queries a specific LLM API.. Callable[[Iterable[Any]], Iterable[Any]] |
cache | Cache to use for caching prompts and responses per doc. Cache |
save_io | Whether to save LLM I/O (prompts and responses) in the Doc._.llm_io custom attribute. bool |
LLMWrapper.__call__ method
Apply the pipe to one document. The document is modified in place and returned.
This usually happens under the hood when the nlp
object is called on a text
and all pipeline components are applied to the Doc
in order.
Name | Description |
---|---|
doc | The document to process. Doc |
RETURNS | The processed document. Doc |
LLMWrapper.pipe method
Apply the pipe to a stream of documents. This usually happens under the hood
when the nlp
object is called on a text and all pipeline components are
applied to the Doc
in order.
Name | Description |
---|---|
docs | A stream of documents. Iterable[Doc] |
keyword-only | |
batch_size | The number of documents to buffer. Defaults to 128 . int |
YIELDS | The processed documents in order. Doc |
LLMWrapper.add_label method
Add a new label to the pipe’s task. Alternatively, provide the labels upon the
task definition, or through the [initialize]
block of the
config.
Name | Description |
---|---|
label | The label to add. str |
RETURNS | 0 if the label is already present, otherwise 1 . int |
LLMWrapper.to_disk method
Serialize the pipe to disk.
Name | Description |
---|---|
path | A path to a directory, which will be created if it doesn’t exist. Paths may be either strings or Path -like objects. Union[str,Path] |
keyword-only | |
exclude | String names of serialization fields to exclude. Iterable[str] |
LLMWrapper.from_disk method
Load the pipe from disk. Modifies the object in place and returns it.
Name | Description |
---|---|
path | A path to a directory. Paths may be either strings or Path -like objects. Union[str,Path] |
keyword-only | |
exclude | String names of serialization fields to exclude. Iterable[str] |
RETURNS | The modified LLMWrapper object. LLMWrapper |
LLMWrapper.to_bytes method
Serialize the pipe to a bytestring.
Name | Description |
---|---|
keyword-only | |
exclude | String names of serialization fields to exclude. Iterable[str] |
RETURNS | The serialized form of the LLMWrapper object. bytes |
LLMWrapper.from_bytes method
Load the pipe from a bytestring. Modifies the object in place and returns it.
Name | Description |
---|---|
bytes_data | The data to load from. bytes |
keyword-only | |
exclude | String names of serialization fields to exclude. Iterable[str] |
RETURNS | The LLMWrapper object. LLMWrapper |
LLMWrapper.labels property
The labels currently added to the component. Empty tuple if the LLM’s task does not require labels.
Name | Description |
---|---|
RETURNS | The labels added to the component. Tuple[str, …] |
Tasks
Task implementation
A task defines an NLP problem or question, that will be sent to the LLM via a
prompt. Further, the task defines how to parse the LLM’s responses back into
structured information. All tasks are registered in the llm_tasks
registry.
task.generate_prompts
Takes a collection of documents, and returns a collection of “prompts”, which
can be of type Any
. Often, prompts are of type str
- but this is not
enforced to allow for maximum flexibility in the framework.
Argument | Description |
---|---|
docs | The input documents. Iterable[Doc] |
RETURNS | The generated prompts. Iterable[Any] |
task.parse_responses
Takes a collection of LLM responses and the original documents, parses the
responses into structured information, and sets the annotations on the
documents. The parse_responses
function is free to set the annotations in any
way, including Doc
fields like ents
, spans
or cats
, or using custom
defined fields.
The responses
are of type Iterable[Any]
, though they will often be str
objects. This depends on the return type of the model.
Argument | Description |
---|---|
docs | The input documents. Iterable[Doc] |
responses | The generated prompts. Iterable[Any] |
RETURNS | The annotated documents. Iterable[Doc] |
Summarization
A summarization task takes a document as input and generates a summary that is stored in an extension attribute.
spacy.Summarization.v1
The spacy.Summarization.v1
task supports both zero-shot and few-shot
prompting.
Argument | Description |
---|---|
template | Custom prompt template to send to LLM model. Defaults to summarization.v1.jinja. str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
max_n_words | Maximum number of words to be used in summary. Note that this should not expected to work exactly. Defaults to None . Optional[int] |
field | Name of extension attribute to store summary in (i. e. the summary will be available in doc._.{field} ). Defaults to summary . str |
The summarization task prompts the model for a concise summary of the provided
text. It optionally allows to limit the response to a certain number of tokens -
note that this requirement will be included in the prompt, but the task doesn’t
perform a hard cut-off. It’s hence possible that your summary exceeds
max_n_words
.
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
NER
The NER task identifies non-overlapping entities in text.
spacy.NER.v3
Version 3 is fundamentally different to v1 and v2, as it implements Chain-of-Thought prompting, based on the PromptNER paper by Ashok and Lipton (2023). On an internal use-case, we have found this implementation to obtain significant better accuracy - with an increase of F-score of up to 15 percentage points.
When no examples are specified, the v3 implementation will use a dummy example in the prompt. Technically this means that the task will always perform few-shot prompting under the hood.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
label_definitions | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to None . Optional[Dict[str, str]] |
template | Custom prompt template to send to LLM model. Defaults to ner.v3.jinja. str |
description (NEW) | A description of what to recognize or not recognize as entities. str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , defaults to spacy.LowercaseNormalizer.v1 . Defaults to None . Optional[Callable[[str], str]] |
alignment_mode | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are "strict" , "contract" or "expand" . Defaults to "contract" . str |
case_sensitive_matching | Whether to search without case sensitivity. Defaults to False . bool |
Note that the single_match
parameter, used in v1 and v2, is not supported
anymore, as the CoT parsing algorithm takes care of this automatically.
New to v3 is the fact that you can provide an explicit description of what
entities should look like. You can use this feature in addition to
label_definitions
.
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
While not required, this task works best when both positive and negative
examples are provided. The format is different than the files required for v1
and v2, as additional fields such as is_entity
and reason
should now be
provided.
For a fully working example, see this usage example.
spacy.NER.v2
This version supports explicitly defining the provided labels with custom descriptions, and further supports zero-shot and few-shot prompting just like v1.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
label_definitions (NEW) | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to None . Optional[Dict[str, str]] |
template (NEW) | Custom prompt template to send to LLM model. Defaults to ner.v2.jinja. str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , defaults to spacy.LowercaseNormalizer.v1 . Defaults to None . Optional[Callable[[str], str]] |
alignment_mode | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are "strict" , "contract" or "expand" . Defaults to "contract" . str |
case_sensitive_matching | Whether to search without case sensitivity. Defaults to False . bool |
single_match | Whether to match an entity in the LLM’s response only once (the first hit) or multiple times. Defaults to False . bool |
The parameters alignment_mode
, case_sensitive_matching
and single_match
are identical to the v1 implementation. The format of few-shot
examples are also the same.
New to v2 is the fact that you can write definitions for each label and provide
them via the label_definitions
argument. This lets you tell the LLM exactly
what you’re looking for rather than relying on the LLM to interpret its task
given just the label name. Label descriptions are freeform so you can write
whatever you want here, but a brief description along with some examples and
counter examples seems to work quite well.
For a fully working example, see this usage example.
spacy.NER.v1
The original version of the built-in NER task supports both zero-shot and few-shot prompting.
Argument | Description |
---|---|
labels | Comma-separated list of labels. str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , defaults to spacy.LowercaseNormalizer.v1 . Optional[Callable[[str], str]] |
alignment_mode | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are "strict" , "contract" or "expand" . Defaults to "contract" . str |
case_sensitive_matching | Whether to search without case sensitivity. Defaults to False . bool |
single_match | Whether to match an entity in the LLM’s response only once (the first hit) or multiple times. Defaults to False . bool |
The NER task implementation doesn’t currently ask the LLM for specific offsets, but simply expects a list of strings that represent the enties in the document. This means that a form of string matching is required. This can be configured by the following parameters:
- The
single_match
parameter is typically set toFalse
to allow for multiple matches. For instance, the response from the LLM might only mention the entity “Paris” once, but you’d still want to mark it every time it occurs in the document. - The case-sensitive matching is typically set to
False
to be robust against case variances in the LLM’s output. - The
alignment_mode
argument is used to match entities as returned by the LLM to the tokens from the originalDoc
- specifically it’s used as argument in the call todoc.char_span()
. The"strict"
mode will only keep spans that strictly adhere to the given token boundaries."contract"
will only keep those tokens that are fully within the given range, e.g. reducing"New Y"
to"New"
. Finally,"expand"
will expand the span to the next token boundaries, e.g. expanding"New Y"
out to"New York"
.
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
SpanCat
The SpanCat task identifies potentially overlapping entities in text.
spacy.SpanCat.v3
The built-in SpanCat v3 task is a simple adaptation of the NER v3 task to
support overlapping entities and store its annotations in doc.spans
.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
label_definitions | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to None . Optional[Dict[str, str]] |
template | Custom prompt template to send to LLM model. Defaults to spancat.v3.jinja . str |
description (NEW) | A description of what to recognize or not recognize as entities. str |
spans_key | Key of the Doc.spans dict to save the spans under. Defaults to "sc" . str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , defaults to spacy.LowercaseNormalizer.v1 . Optional[Callable[[str], str]] |
alignment_mode | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are "strict" , "contract" or "expand" . Defaults to "contract" . str |
case_sensitive_matching | Whether to search without case sensitivity. Defaults to False . bool |
Note that the single_match
parameter, used in v1 and v2, is not supported
anymore, as the CoT parsing algorithm takes care of this automatically.
spacy.SpanCat.v2
The built-in SpanCat v2 task is a simple adaptation of the NER v2 task to
support overlapping entities and store its annotations in doc.spans
.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
label_definitions (NEW) | Optional dict mapping a label to a description of that label. These descriptions are added to the prompt to help instruct the LLM on what to extract. Defaults to None . Optional[Dict[str, str]] |
template (NEW) | Custom prompt template to send to LLM model. Defaults to spancat.v2.jinja . str |
spans_key | Key of the Doc.spans dict to save the spans under. Defaults to "sc" . str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , defaults to spacy.LowercaseNormalizer.v1 . Optional[Callable[[str], str]] |
alignment_mode | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are "strict" , "contract" or "expand" . Defaults to "contract" . str |
case_sensitive_matching | Whether to search without case sensitivity. Defaults to False . bool |
single_match | Whether to match an entity in the LLM’s response only once (the first hit) or multiple times. Defaults to False . bool |
Except for the spans_key
parameter, the SpanCat v2 task reuses the
configuration from the NER v2 task. Refer to its documentation for
more insight.
spacy.SpanCat.v1
The original version of the built-in SpanCat task is a simple adaptation of the
v1 NER task to support overlapping entities and store its annotations in
doc.spans
.
Argument | Description |
---|---|
labels | Comma-separated list of labels. str |
spans_key | Key of the Doc.spans dict to save the spans under. Defaults to "sc" . str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , defaults to spacy.LowercaseNormalizer.v1 . Optional[Callable[[str], str]] |
alignment_mode | Alignment mode in case the LLM returns entities that do not align with token boundaries. Options are "strict" , "contract" or "expand" . Defaults to "contract" . str |
case_sensitive_matching | Whether to search without case sensitivity. Defaults to False . bool |
single_match | Whether to match an entity in the LLM’s response only once (the first hit) or multiple times. Defaults to False . bool |
Except for the spans_key
parameter, the SpanCat v1 task reuses the
configuration from the NER v1 task. Refer to its documentation for
more insight.
TextCat
The TextCat task labels documents with relevant categories.
spacy.TextCat.v3
On top of the functionality from v2, version 3 of the built-in TextCat tasks allows setting definitions of labels. Those definitions are included in the prompt.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
label_definitions (NEW) | Dictionary of label definitions. Included in the prompt, if set. Defaults to None . Optional[Dict[str, str]] |
template | Custom prompt template to send to LLM model. Defaults to textcat.v3.jinja . str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , falls back to spacy.LowercaseNormalizer.v1 . Defaults to None . Optional[Callable[[str], str]] |
exclusive_classes | If set to True , only one label per document should be valid. If set to False , one document can have multiple labels. Defaults to False . bool |
allow_none | When set to True , allows the LLM to not return any of the given label. The resulting dict in doc.cats will have 0.0 scores for all labels. Defaults to True . bool |
verbose | If set to True , warnings will be generated when the LLM returns invalid responses. Defaults to False . bool |
The formatting of few-shot examples is the same as those for the v1 implementation.
spacy.TextCat.v2
V2 includes all v1 functionality, with an improved prompt template.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
template (NEW) | Custom prompt template to send to LLM model. Defaults to textcat.v2.jinja . str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , falls back to spacy.LowercaseNormalizer.v1 . Optional[Callable[[str], str]] |
exclusive_classes | If set to True , only one label per document should be valid. If set to False , one document can have multiple labels. Defaults to False . bool |
allow_none | When set to True , allows the LLM to not return any of the given label. The resulting dict in doc.cats will have 0.0 scores for all labels. Defaults to True . bool |
verbose | If set to True , warnings will be generated when the LLM returns invalid responses. Defaults to False . bool |
The formatting of few-shot examples is the same as those for the v1 implementation.
spacy.TextCat.v1
Version 1 of the built-in TextCat task supports both zero-shot and few-shot prompting.
Argument | Description |
---|---|
labels | Comma-separated list of labels. str |
examples | Optional function that generates examples for few-shot learning. Deafults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , falls back to spacy.LowercaseNormalizer.v1 . Optional[Callable[[str], str]] |
exclusive_classes | If set to True , only one label per document should be valid. If set to False , one document can have multiple labels. Deafults to False . bool |
allow_none | When set to True , allows the LLM to not return any of the given label. The resulting dict in doc.cats will have 0.0 scores for all labels. Deafults to True . bool |
verbose | If set to True , warnings will be generated when the LLM returns invalid responses. Deafults to False . bool |
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
REL
The REL task extracts relations between named entities.
spacy.REL.v1
The built-in REL task supports both zero-shot and few-shot prompting. It relies on an upstream NER component for entities extraction.
Argument | Description |
---|---|
labels | List of labels or str of comma-separated list of labels. Union[List[str], str] |
template | Custom prompt template to send to LLM model. Defaults to rel.v3.jinja . str |
label_definitions | Dictionary providing a description for each relation label. Defaults to None . Optional[Dict[str, str]] |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
normalizer | Function that normalizes the labels as returned by the LLM. If None , falls back to spacy.LowercaseNormalizer.v1 . Defaults to None . Optional[Callable[[str], str]] |
verbose | If set to True , warnings will be generated when the LLM returns invalid responses. Defaults to False . bool |
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
Note: the REL task relies on pre-extracted entities to make its prediction.
Hence, you’ll need to add a component that populates doc.ents
with recognized
spans to your spaCy pipeline and put it before the REL component.
For a fully working example, see this usage example.
Lemma
The Lemma task lemmatizes the provided text and updates the lemma_
attribute
in the doc’s tokens accordingly.
spacy.Lemma.v1
This task supports both zero-shot and few-shot prompting.
Argument | Description |
---|---|
template | Custom prompt template to send to LLM model. Defaults to lemma.v1.jinja. str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
The task prompts the LLM to lemmatize the passed text and return the lemmatized
version as a list of tokens and their corresponding lemma. E. g. the text
I'm buying ice cream for my friends
should invoke the response
If for any given text/doc instance the number of lemmas returned by the LLM
doesn’t match the number of tokens from the pipeline’s tokenizer, no lemmas are
stored in the corresponding doc’s tokens. Otherwise the tokens .lemma_
property is updated with the lemma suggested by the LLM.
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
Sentiment
Performs sentiment analysis on provided texts. Scores between 0 and 1 are stored
in Doc._.sentiment
- the higher, the more positive. Note in cases of parsing
issues (e. g. in case of unexpected LLM responses) the value might be None
.
spacy.Sentiment.v1
This task supports both zero-shot and few-shot prompting.
Argument | Description |
---|---|
template | Custom prompt template to send to LLM model. Defaults to sentiment.v1.jinja. str |
examples | Optional function that generates examples for few-shot learning. Defaults to None . Optional[Callable[[], Iterable[Any]]] |
field | Name of extension attribute to store summary in (i. e. the summary will be available in doc._.{field} ). Defaults to sentiment . str |
To perform few-shot learning,
you can write down a few examples in a separate file, and provide these to be
injected into the prompt to the LLM. The default reader spacy.FewShotReader.v1
supports .yml
, .yaml
, .json
and .jsonl
.
NoOp
This task is only useful for testing - it tells the LLM to do nothing, and does
not set any fields on the docs
.
spacy.NoOp.v1
This task needs no further configuration.
Models
A model defines which LLM model to query, and how to query it. It can be a
simple function taking a collection of prompts (consistent with the output type
of task.generate_prompts()
) and returning a collection of responses
(consistent with the expected input of parse_responses
). Generally speaking,
it’s a function of type Callable[[Iterable[Any]], Iterable[Any]]
, but specific
implementations can have other signatures, like
Callable[[Iterable[str]], Iterable[str]]
.
Models via REST API
These models all take the same parameters, but note that the config
should
contain provider-specific keys and values, as it will be passed onwards to the
provider’s API.
Argument | Description |
---|---|
name | Model name, i. e. any supported variant for this particular model. Default depends on the specific model (cf. below) str |
config | Further configuration passed on to the model. Default depends on the specific model (cf. below). Dict[Any, Any] |
strict | If True , raises an error if the LLM API returns a malformed response. Otherwise, return the error responses as is. Defaults to True . bool |
max_tries | Max. number of tries for API request. Defaults to 5 . int |
max_request_time | Max. time (in seconds) to wait for request to terminate before raising an exception. Defaults to 30.0 . float |
interval | Time interval (in seconds) for API retries in seconds. Defaults to 1.0 . float |
Currently, these models are provided as part of the core library:
Model | Provider | Supported names | Default name | Default config |
---|---|---|---|---|
spacy.GPT-4.v1 | OpenAI | ["gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314"] | "gpt-4" | {} |
spacy.GPT-4.v2 | OpenAI | ["gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314"] | "gpt-4" | {temperature=0.0} |
spacy.GPT-3-5.v1 | OpenAI | ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-0613-16k"] | "gpt-3.5-turbo" | {} |
spacy.GPT-3-5.v2 | OpenAI | ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-0613-16k"] | "gpt-3.5-turbo" | {temperature=0.0} |
spacy.Davinci.v1 | OpenAI | ["davinci"] | "davinci" | {} |
spacy.Davinci.v2 | OpenAI | ["davinci"] | "davinci" | {temperature=0.0, max_tokens=500} |
spacy.Text-Davinci.v1 | OpenAI | ["text-davinci-003", "text-davinci-002"] | "text-davinci-003" | {} |
spacy.Text-Davinci.v2 | OpenAI | ["text-davinci-003", "text-davinci-002"] | "text-davinci-003" | {temperature=0.0, max_tokens=1000} |
spacy.Code-Davinci.v1 | OpenAI | ["code-davinci-002"] | "code-davinci-002" | {} |
spacy.Code-Davinci.v2 | OpenAI | ["code-davinci-002"] | "code-davinci-002" | {temperature=0.0, max_tokens=500} |
spacy.Curie.v1 | OpenAI | ["curie"] | "curie" | {} |
spacy.Curie.v2 | OpenAI | ["curie"] | "curie" | {temperature=0.0, max_tokens=500} |
spacy.Text-Curie.v1 | OpenAI | ["text-curie-001"] | "text-curie-001" | {} |
spacy.Text-Curie.v2 | OpenAI | ["text-curie-001"] | "text-curie-001" | {temperature=0.0, max_tokens=500} |
spacy.Babbage.v1 | OpenAI | ["babbage"] | "babbage" | {} |
spacy.Babbage.v2 | OpenAI | ["babbage"] | "babbage" | {temperature=0.0, max_tokens=500} |
spacy.Text-Babbage.v1 | OpenAI | ["text-babbage-001"] | "text-babbage-001" | {} |
spacy.Text-Babbage.v2 | OpenAI | ["text-babbage-001"] | "text-babbage-001" | {temperature=0.0, max_tokens=500} |
spacy.Ada.v1 | OpenAI | ["ada"] | "ada" | {} |
spacy.Ada.v2 | OpenAI | ["ada"] | "ada" | {temperature=0.0, max_tokens=500} |
spacy.Text-Ada.v1 | OpenAI | ["text-ada-001"] | "text-ada-001" | {} |
spacy.Text-Ada.v2 | OpenAI | ["text-ada-001"] | "text-ada-001" | {temperature=0.0, max_tokens=500} |
spacy.Command.v1 | Cohere | ["command", "command-light", "command-light-nightly", "command-nightly"] | "command" | {} |
spacy.Claude-2.v1 | Anthropic | ["claude-2", "claude-2-100k"] | "claude-2" | {} |
spacy.Claude-1.v1 | Anthropic | ["claude-1", "claude-1-100k"] | "claude-1" | {} |
spacy.Claude-1-0.v1 | Anthropic | ["claude-1.0"] | "claude-1.0" | {} |
spacy.Claude-1-2.v1 | Anthropic | ["claude-1.2"] | "claude-1.2" | {} |
spacy.Claude-1-3.v1 | Anthropic | ["claude-1.3", "claude-1.3-100k"] | "claude-1.3" | {} |
spacy.Claude-instant-1.v1 | Anthropic | ["claude-instant-1", "claude-instant-1-100k"] | "claude-instant-1" | {} |
spacy.Claude-instant-1-1.v1 | Anthropic | ["claude-instant-1.1", "claude-instant-1.1-100k"] | "claude-instant-1.1" | {} |
To use these models, make sure that you’ve set the relevant API keys as environment variables.
API Keys
Note that when using hosted services, you have to ensure that the proper API keys are set as environment variables as described by the corresponding provider’s documentation.
E. g. when using OpenAI, you have to get an API key from openai.com, and ensure that the keys are set as environmental variables:
For Cohere:
For Anthropic:
Models via HuggingFace
These models all take the same parameters:
Argument | Description |
---|---|
name | Model name, i. e. any supported variant for this particular model. str |
config_init | Further configuration passed on to the construction of the model with transformers.pipeline() . Defaults to {} . Dict[str, Any] |
config_run | Further configuration used during model inference. Defaults to {} . Dict[str, Any] |
Currently, these models are provided as part of the core library:
Model | Provider | Supported names | HF directory |
---|---|---|---|
spacy.Dolly.v1 | Databricks | ["dolly-v2-3b", "dolly-v2-7b", "dolly-v2-12b"] | https://huggingface.co/databricks |
spacy.Llama2.v1 | Meta AI | ["Llama-2-7b-hf", "Llama-2-13b-hf", "Llama-2-70b-hf"] | https://huggingface.co/meta-llama |
spacy.Falcon.v1 | TII | ["falcon-rw-1b", "falcon-7b", "falcon-7b-instruct", "falcon-40b-instruct"] | https://huggingface.co/tiiuae |
spacy.StableLM.v1 | Stability AI | ["stablelm-base-alpha-3b", "stablelm-base-alpha-7b", "stablelm-tuned-alpha-3b", "stablelm-tuned-alpha-7b"] | https://huggingface.co/stabilityai |
spacy.OpenLLaMA.v1 | OpenLM Research | ["open_llama_3b", "open_llama_7b", "open_llama_7b_v2", "open_llama_13b"] | https://huggingface.co/openlm-research |
Note that Hugging Face will download the model the first time you use it - you
can
define the cached directory
by setting the environmental variable HF_HOME
.
Installation with HuggingFace
To use models from HuggingFace, ideally you have a GPU enabled and have
installed transformers
, torch
and CUDA in your virtual environment. This
allows you to have the setting device=cuda:0
in your config, which ensures
that the model is loaded entirely on the GPU (and fails otherwise).
You can do so with
If you don’t have access to a GPU, you can install accelerate
and
setdevice_map=auto
instead, but be aware that this may result in some layers
getting distributed to the CPU or even the hard drive, which may ultimately
result in extremely slow queries.
LangChain models
To use LangChain for the API retrieval part, make sure you have installed it first:
Note that LangChain currently only supports Python 3.9 and beyond.
LangChain models in spacy-llm
work slightly differently. langchain
’s models
are parsed automatically, each LLM class in langchain
has one entry in
spacy-llm
’s registry. As langchain
’s design has one class per API and not
per model, this results in registry entries like langchain.OpenAI.v1
- i. e.
there is one registry entry per API and not per model (family), as for the REST-
and HuggingFace-based entries.
The name of the model to be used has to be passed in via the name
attribute.
Argument | Description |
---|---|
name | The name of a mdodel supported by LangChain for this API. str |
config | Configuration passed on to the LangChain model. Defaults to {} . Dict[Any, Any] |
query | Function that executes the prompts. If None , defaults to spacy.CallLangChain.v1 . Optional[Callable[[“langchain.llms.BaseLLM”, Iterable[Any]], Iterable[Any]]] |
The default query
(spacy.CallLangChain.v1
) executes the prompts by running
model(text)
for each given textual prompt.
Cache
Interacting with LLMs, either through an external API or a local instance, is
costly. Since developing an NLP pipeline generally means a lot of exploration
and prototyping, spacy-llm
implements a built-in cache to avoid reprocessing
the same documents at each run that keeps batches of documents stored on disk.
Argument | Description |
---|---|
path | Cache directory. If None , no caching is performed, and this component will act as a NoOp. Defaults to None . Optional[Union[str,Path]] |
batch_size | Number of docs in one batch (file). Once a batch is full, it will be peristed to disk. Defaults to 64. int |
max_batches_in_mem | Max. number of batches to hold in memory. Allows you to limit the effect on your memory if you’re handling a lot of docs. Defaults to 4. int |
When retrieving a document, the BatchCache
will first figure out what batch
the document belongs to. If the batch isn’t in memory it will try to load the
batch from disk and then move it into memory.
Note that since the cache is generated by a registered function, you can also
provide your own registered function returning your own cache implementation. If
you wish to do so, ensure that your cache object adheres to the Protocol
defined in spacy_llm.ty.Cache
.
Various functions
spacy.FewShotReader.v1
This function is registered in spaCy’s misc
registry, and reads in examples
from a .yml
, .yaml
, .json
or .jsonl
file. It uses
srsly
to read in these files and parses
them depending on the file extension.
Argument | Description |
---|---|
path | Path to an examples file with suffix .yml , .yaml , .json or .jsonl . Union[str,Path] |
spacy.FileReader.v1
This function is registered in spaCy’s misc
registry, and reads a file
provided to the path
to return a str
representation of its contents. This
function is typically used to read
Jinja files containing the prompt
template.
Argument | Description |
---|---|
path | Path to the file to be read. Union[str,Path] |
Normalizer functions
These functions provide simple normalizations for string comparisons, e.g.
between a list of specified labels and a label given in the raw text of the LLM
response. They are registered in spaCy’s misc
registry and have the signature
Callable[[str], str]
.
spacy.StripNormalizer.v1
: only applytext.strip()
spacy.LowercaseNormalizer.v1
: appliestext.strip().lower()
to compare strings in a case-insensitive way.