best_engine_ai_helper.llm module

llm — pluggable local AND cloud model backend for best-engine-ai-helper.

Provides two public functions, chat and embed, that route requests to the backend named by a resolved engine descriptor (preferred) or the SPREZZATURE_LLM_BACKEND environment variable (legacy path). Callers use only these two functions; the transport details (Ollama JSON API vs OpenAI-compatible REST vs Anthropic/Gemini’s own formats vs LangChain) stay invisible to them.

Supported backends

ollama: Default. POSTs to {SPREZZATURE_LLM_BASE_URL}/api/generate. Works offline once the model is pulled.
openai: Any OpenAI-compatible server: vLLM, llama.cpp, LM Studio, Text Generation Inference, OpenAI itself, Mistral, OpenRouter, Together, Azure OpenAI. POSTs to {base_url}/v1/chat/completions.
anthropic: Claude’s own Messages API (POST {base_url}/v1/messages).
gemini: Google’s own generateContent API (POST {base_url}/v1beta/models/{model}:generateContent).
langchain: Thin wrapper over ChatOllama or ChatOpenAI from LangChain. Only useful if you need LangChain retrievers or agent abstractions.
lightweight: Layer streaming, for a model too large to fit in memory: no server at all, the model runs in this process and reads one layer at a time off disk (see best_engine_ai_helper.offload). Text only, no JSON schema, and seconds per token rather than tokens per second. Reached when an engine descriptor names it — never from the BEST_LLM_BACKEND env path, because a 141 GB download is not something an environment variable should start.

A cloud engine descriptor (engine.resolve with mode: cloud) carries an embedded local fallback; chat() tries the cloud primary first and falls over to the local fallback on failure (paid -> local, the safe direction) — see engine= below and best_engine_ai_helper.engine.

Environment variables

BEST_LLM_BACKEND (legacy alias: SPREZZATURE_LLM_BACKEND): ollama | openai | anthropic | gemini | langchain. When unset, falls back to the selection persisted by pull in ~/.best-engine-ai-helper/config.json, then to ollama. Resolved by config.backend(). Only consulted on the legacy no-engine path.
BEST_LLM_BASE_URL (legacy alias: SPREZZATURE_LLM_BASE_URL): Base URL of the server. Same precedence as the backend; defaults to http://localhost:11434. Resolved by config.base_url().
BEST_LLM_TEXT (legacy alias: SPREZZATURE_LLM_TEXT): Model tag for text-only prompts. When unset, falls back to the selection persisted by pull in ~/.best-engine-ai-helper/config.json, then to the qwen3:8b default. Resolved by config.text_model().
BEST_LLM_VISION (legacy alias: SPREZZATURE_LLM_VISION): Model tag for prompts that include images. Same precedence as the text model; resolved by config.vision_model().
SPREZZATURE_LLM_API_KEY: API key for servers that require one on the legacy path. Empty string by default (most local servers do not require authentication). On the engine= path, the key comes from the env var NAMED in the engine’s api_key_env field instead (never the key value itself, never persisted) — see _cloud_api_key().

Observability

Every chat() call fans a small event dict out to any observer registered via add_observer() (backend, model, kind, char counts, real token counts when the provider reports them, latency, success/error). No observer is registered by default. best_engine_ai_helper.observe provides a SQLite-backed sink (call observe.enable()) that turns this into a queryable local activity/cost ledger, surfaced by the activity CLI command and the /api/activity endpoint.

Privacy and safety

chat(..., pseudonymize=True) scrubs personal data from the prompt with a local LLM before it reaches a cloud provider, and restores it in the response — see best_engine_ai_helper.privacy. Cloud-only (a no-op on a local engine — there is nowhere for personal data to leak to).

chat(..., safety=...) scans the prompt/images before sending and the response after receiving for NSFW/policy violations — see best_engine_ai_helper.safety. On by default for EVERY engine, local or cloud (content policy is independent of who is billed); pass safety=False to opt out.

Author

Warith Harchaoui <warithmetics@deraison.ai>

best_engine_ai_helper.llm.add_observer(fn)[source]

Register a per-call observer; it receives the event dict chat() emits.

Parameters:: fn (callable) – Called with one event dict after every chat() call, success or failure. Must not raise — an exception is caught and logged, never propagated, so a broken observer can’t take down inference.
Return type:: None

best_engine_ai_helper.llm.chat(prompt, *, system=None, images=None, json_schema=None, model=None, temperature=0.2, engine=None, kind=None, cache=False, retries=0, pseudonymize=False, safety=None)[source]

Send a prompt to the configured model (local or cloud) and return the response.

The backend and model come from one of two sources. When engine is given (the suite’s preferred path), it is read from a resolved engine descriptor — the gitignored llm.engine.yaml a repo gets from best-engine-ai-helper resolve. A cloud engine’s embedded local fallback is tried after the cloud primary on failure (paid -> local); pass a list of descriptors to define your own failover order instead. Otherwise the legacy env path applies: the backend is SPREZZATURE_LLM_BACKEND and the model resolves via env / persisted config.

Parameters:

prompt (str) – User-facing prompt text.
system (str or None) – System-level instructions sent before the user prompt. Use for persona, output format constraints, or house style rules.
images (list[bytes] or None) – Raw image bytes (PNG or JPEG). When provided, the vision model is used unless model is specified explicitly.
json_schema (dict or None) – When provided, the response is constrained to this JSON Schema where the backend supports grammar-constrained output (Ollama, Gemini); other backends (OpenAI-compatible via response_format, Anthropic via a prompt instruction) parse best-effort, same as every backend’s fallback when the model still returns non-JSON.
model (str or None) – Override the model tag/id. Wins over both the engine descriptor and the env default. When absent with no engine, defaults to the vision model when images are present, else the text model.
temperature (float) – Sampling temperature. Lower values are more deterministic. Defaults to 0.2 because structured extraction tasks benefit from low variance.
engine (dict | str | list | None) – A resolved engine descriptor (dict from engine.resolve() / engine.ensure(), or a path to llm.engine.yaml), or an explicit list of descriptors to try in order. When given, its backend / base_url and the per-kind model drive the request.
kind ({'llm', 'vlm'} or None) – Which model to use from engine. Defaults to vlm when images are present, else llm. Ignored when engine is None.
cache (bool) – Memoize identical calls (same backend/model/prompt/schema/images) via wallet-helper (the [cloud] extra) so a repeated call never pays for the same cloud request twice. A no-op with a warning if the extra is not installed. Ignored on the legacy env path (no engine to key on).
retries (int) – Retry a transient transport failure this many times (exponential backoff via tenacity when installed, immediate retry otherwise) before moving to the next engine in the failover chain.
pseudonymize (bool) – Scrub personal data from prompt with a local LLM before it reaches a cloud engine, and restore it in the response — see best_engine_ai_helper.privacy. No-op on a local engine, or when the cloud engine has no local fallback to do the scrubbing with (warns in that case rather than silently skipping).
safety (bool or None) – Scan the prompt/images before sending and the response after receiving for policy violations — see best_engine_ai_helper.safety. None (the default) resolves to True for every engine, local or cloud: NSFW/policy content is a content-policy concern independent of who is billed, not a cloud-only risk. Pass False to opt out explicitly.

Returns:

When json_schema is provided and the model returns valid JSON, the result is parsed and returned as a dict. Otherwise a plain string.

Return type:

str or dict

Raises:

RuntimeError – If every engine in the failover chain fails, or (with no engine) the single legacy-path backend is unreachable or returns a malformed response.
ValueError – If a backend name (env path) or transport (engine path) is unrecognised.

Examples

>>> # Text prompt (no model running needed for this docstring to parse)
>>> # result = chat("Summarise this paper in one sentence.")
>>> # Vision prompt
>>> # with open("chart.png", "rb") as f:
>>> #     result = chat("Describe the chart.", images=[f.read()])

best_engine_ai_helper.llm.chat_stream(prompt, *, system=None, model=None, temperature=0.2, engine=None, kind=None, base_url=None, safety=None)[source]

Stream a model’s answer, a token at a time, as it is generated.

chat() returns a finished string, which is the right shape for almost everything this suite does: a caption, an alt text, a summary — all of them are used whole, so producing them whole costs nothing.

A spoken conversation is the exception, and it is a sharp one. There, the number that matters is not how long the answer takes but how long the silence before it lasts, and the published budget allows roughly 800 ms between the user’s last word and the agent’s first sound. Generating a whole answer first spends the entire generation inside that budget; streaming spends only the first sentence’s share, because the sentence can be spoken while the rest is still being written. On a local model, that is frequently the difference between a conversation and a wait.

Scope, stated plainly

Ollama only. It is the suite’s default backend and the one every local-first deployment runs; the cloud transports stream too, but each with its own event framing, and shipping one that works beats shipping four that are hard to test. Any other backend raises rather than silently falling back to chat() — a “stream” that yields the whole answer in one piece would make a caller’s latency measurement lie about which mode it was in.

No images, no schema. A grammar-constrained generation is consumed whole by definition, and a vision prompt has no first-sentence advantage worth the extra surface.

The safety scan, and what streaming does to it

chat() scans the prompt before sending and the reply before returning. Here the prompt is scanned identically — nothing is lost on that side — but the reply cannot be scanned before the caller sees it, because seeing it as it arrives is the entire point.

So the reply is scanned when the stream ends, and that is weaker in a specific, stateable way: a caller that buffers the chunks and uses them afterwards gets the same protection as chat(), while a caller that speaks each chunk as it arrives has already said it. There is no way around that which is not simply chat(). It is written here rather than discovered, and the check still runs: finding out afterwards is worth more than not finding out.

param prompt:: User-facing prompt text.
type prompt:: str
param system:: System-level instructions sent before the user prompt.
type system:: str or None
param model:: Override the resolved model tag.
type model:: str or None
param temperature:: Sampling temperature.
type temperature:: float
param engine:: A resolved engine descriptor, as chat() takes.
type engine:: dict, str, list or None
param kind:: "llm" or "vlm"; defaults to "llm".
type kind:: str or None
param base_url:: Override the Ollama endpoint.
type base_url:: str or None
Yields:: str – Each chunk of text as the model produces it. Chunks are whatever the backend emits — usually sub-word — so a caller that needs sentences must accumulate and split them itself.
raises RuntimeError:: When the resolved backend is not Ollama, or the request fails.

Examples

>>> # Accumulate into sentences, which is what a speech caller wants:
>>> # pending = ""
>>> # for piece in chat_stream("Explique la marée."):
>>> #     pending += piece

Parameters:

prompt (str)
system (str | None)
model (str | None)
temperature (float)
engine (dict[str, Any] | str | list[Any] | None)
kind (str | None)
base_url (str | None)
safety (bool | None)

Return type:

Iterator[str]

best_engine_ai_helper.llm.clear_observers()[source]

Remove all registered observers (chiefly for tests, or to disable).

Return type:: None

best_engine_ai_helper.llm.embed(text, model=None)[source]

Return an embedding vector for the given text.

Only the Ollama backend is supported for embeddings. The OpenAI-compatible embedding endpoint (/v1/embeddings) is not yet implemented because the retrieval use case is not yet in scope.

Which model, and why the default is probably not the one you want

With no model, this uses the configured text model — the chat LLM. Ollama will embed with it, and the result is far worse for retrieval than a purpose-built embedder would give. The catalog ships four (nomic-embed-text, mxbai-embed-large, snowflake-arctic-embed2, bge-m3), and usages.resolve_usage() already picks between them on their MTEB score and this machine’s memory — but nothing wires that choice to here, so the caller has to pass it.

The default is left alone deliberately rather than quietly improved. Vectors from two different models are not comparable: swapping the model under an existing store silently poisons every similarity it computes, and a corruption nobody is told about is worse than a weak default that is written down. Pass model explicitly — ideally the embedder named by a resolved usage brief.

param text:: Input text to embed.
type text:: str
param model:: Embedding model tag. Defaults to the configured text model; see above for why that default is a caveat rather than a recommendation.
type model:: str or None
returns:: Dense embedding vector from the Ollama /api/embeddings endpoint.
rtype:: list[float]
raises RuntimeError:: If the Ollama request fails or the response lacks an embedding field.
raises NotImplementedError:: If the active backend is not ollama.

Examples

>>> # vec = embed("hello world")  # requires Ollama running
>>> # len(vec) > 0
>>> True
True

Parameters:

text (str)
model (str | None)

Return type:

list[float]

best_engine_ai_helper.llm.remove_observer(fn)[source]

Unregister one observer, leaving every other one in place.

Exists so a caller can listen for the span of a single operation and then stop — best_engine_ai_helper.throughput meters one gate run this way. clear_observers() would also detach the SQLite ledger the user switched on, which is somebody else’s subscription to drop.

Parameters:: fn (callable) – The observer to remove. Not being registered is not an error, so this is safe to call from a finally block that may run twice.
Return type:: None