best_engine_ai_helper.engine module

engine — resolve a repo’s LLM/VLM usage brief into a concrete serving engine.

The suite’s model-selection contract has two YAML files per consumer repo:

  1. an input brief (committed, hardware-independent) describing what the repo needs from an LLM/VLM — the kinds, memory headroom, comfort floor, and a free-text task; and

  2. an output engine file (gitignored, machine-specific) this module writes: the backend chosen for the current machine plus the concrete model per kind, sized realistically for that backend.

Backend rule (resolve(..., backend="auto")): vLLM only on Linux with a real discrete GPU (NVIDIA/AMD); Ollama everywhere else — macOS (vLLM has no native macOS runtime), Windows (no native vLLM wheel, WSL-only), CPU-only machines (vLLM’s CPU-only path is its own weak spot, notably on Linux), and Intel iGPUs all use Ollama; only a Linux CUDA/ROCm box gets vLLM, chiefly for the request-batching/concurrency handling it offers under real production load, which Ollama does not match. This keeps picks realistic. When vLLM gains a first-class runtime elsewhere, widening this rule is the only change needed.

No DEFAULT_MODEL constant lives in any consumer: the model is always read from the resolved engine file. ensure() is the missing-file policy — a missing engine file is auto-resolved from the brief; a missing brief is a hard error, because the brief is committed and its absence is a real bug.

Author

Warith Harchaoui <warithmetics@deraison.ai>

best_engine_ai_helper.engine.default_backend()[source]

Return the backend for the current machine.

A three-branch decision tree, matching what each backend actually supports well rather than what it merely runs on:

  • macOS -> Ollama. vLLM ships no native macOS runtime.

  • CPU-only (Linux or macOS) -> Ollama. vLLM’s CPU-only path — notably on Linux — is its own reported weak spot; Ollama’s CPU path is the well-trodden one.

  • discrete GPU (NVIDIA/AMD) on Linux — the real production case -> vLLM, chiefly for the request-batching/concurrency handling it offers under real load, which Ollama does not match.

Windows falls back to Ollama too (vLLM has no native Windows wheel, only a WSL path), so in practice: vLLM only on Linux with a real discrete GPU; Ollama everywhere else. Endgame: when vLLM gains a first-class runtime on Mac, Windows, or CPU-only Linux, replace this whole body with return "vllm" and the suite is fully on vLLM.

Return type:

str

best_engine_ai_helper.engine.diagnose(directory='.', *, engine='llm.engine.yaml', timeout=1.5)[source]

Compare what the engine file resolved against what the daemon can serve.

These are two different claims and they drift apart quietly. An engine file is written once, on the machine and catalog of that moment; the daemon it names is mutable, and ollama rm leaves no trace in the descriptor. Today the gap surfaces at the first inference call, which is typically in the middle of a job rather than at resolution — the worst possible moment to learn that a model name was never pullable.

So this asks both sides and reports the difference. It diagnoses only: a command that silently re-resolved would hide the very drift it exists to show.

Parameters:

  • directory (str | Path) – Repo directory holding the engine file.

  • engine (str) – Engine filename within directory.

  • timeout (float) – Seconds to wait on the daemon. Short: an unreachable daemon is a finding here, not a reason to hang.

Returns:

{"engine_path", "backend", "daemon_reachable", "served", "findings": [{kind, model, message}], "ok"}. ok is True only when every model the file names can be served right now.

Return type:

dict

best_engine_ai_helper.engine.ensure(directory='.', *, brief='llm.brief.yaml', engine='llm.engine.yaml', backend='auto', endpoint=None, write=True)[source]

Return the engine descriptor for a repo, resolving it on first use.

Missing-file policy (the suite contract):

  • the engine file exists -> load and return it (fast path, no detection);

  • it is missing but the brief exists -> resolve from the brief, write the engine file (unless write=False), and return it;

  • both are missing -> raise. A committed brief is mandatory; its absence is a real bug, not a machine that has not run detection yet.

Parameters:

  • directory (str | Path) – Repo directory holding the two contract files.

  • brief (str) – Filenames within directory (default to the suite canonical names).

  • engine (str) – Filenames within directory (default to the suite canonical names).

  • backend (see resolve().)

  • endpoint (see resolve().)

  • write (bool) – Persist a freshly resolved engine file. False resolves in-memory only.

Return type:

dict[str, Any]

best_engine_ai_helper.engine.load_brief(brief)[source]

Return the brief as a dict, whether given inline or as a YAML path.

Parameters:

brief (str | Path | dict[str, Any])

Return type:

dict[str, Any]

best_engine_ai_helper.engine.load_engine(path)[source]

Read an engine descriptor written by write_engine().

Parameters:

path (str | Path)

Return type:

dict[str, Any]

best_engine_ai_helper.engine.model_for(engine, kind)[source]

Return (backend, base_url, model) for kind from an engine descriptor.

Raises KeyError if the descriptor has no entry for kind (e.g. asking for a VLM from an engine resolved for an llm-only brief).

Examples

>>> engine = {"backend": "ollama", "base_url": "http://localhost:11434",
...           "llm": {"model": "qwen3:14b"}}
>>> model_for(engine, "llm")
('ollama', 'http://localhost:11434', 'qwen3:14b')
>>> # A kind that fell back to streaming carries its own backend.
>>> engine["vlm"] = {"model": "Qwen/Qwen2.5-VL-72B-Instruct", "backend": "lightweight"}
>>> model_for(engine, "vlm")
('lightweight', '', 'Qwen/Qwen2.5-VL-72B-Instruct')
Parameters:
Return type:

tuple[str, str, str]

best_engine_ai_helper.engine.resolve(brief, *, backend='auto', endpoint=None, catalog=None, hw=None, compute=None, cloud_catalog=None, disk=None)[source]

Resolve a usage brief into a concrete engine descriptor.

The brief’s mode selects local vs cloud (default local):

  • local (default) -> a hardware-specific descriptor: the backend chosen for this machine (Ollama/vLLM) plus the model per kind.

  • cloud -> a provider descriptor (provider, model, optional base_url/api_key_env) plus a local fallback resolved from the SAME brief, so a failed paid call degrades to the always-available local model (paid -> local, the safe direction). model is OPTIONAL: when omitted, the best paid model is auto-picked from the bundled pricing.yaml catalog on a quality-vs-price trade-off (see _resolve_cloud() and best_engine_ai_helper.cloud_catalog).

In local mode there is one escalation: when the conventional backend resolves nothing that FITS for a kind the brief needs, resolution consults offload.decide() and, if this machine can stream, rescues that kind alone with a layer-streamed model (backend: lightweight inside the kind’s own section — see model_for()). The descriptor records the decision and its reason under lightweight, including when the answer was no. Set lightweight: never in the brief, or BEST_ENGINE_LIGHTWEIGHT=never, to switch the escalation off.

Parameters:

  • brief (str | Path | dict) – The input brief (path to llm.brief.yaml or an already-loaded dict). Keys: mode (local/cloud, default local), kind (llm/vlm/both), headroom, min_tps, structured_output, task (free text); cloud-only: provider, model (both optional -> auto-pick), vlm_model, base_url, api_key_env, quality_vs_cost (0..1, default cloud_catalog.DEFAULT_QUALITY_VS_COST, only used when auto-picking); streaming-only: lightweight (auto/always/never, default auto) and lightweight (auto/always/never).

  • backend ({'auto', 'ollama', 'vllm', 'lightweight'}) – auto picks per default_backend() — which never returns lightweight: layer streaming is reached automatically only as a rescue (see below), and otherwise by naming it here. Local mode only.

  • endpoint (str or None) – Override the server base URL (defaults to the local endpoint for the backend, or the provider’s default API root for cloud mode).

  • catalog (optional) – Injectable for tests; default to the live local catalog and detected hardware.

  • hw (optional) – Injectable for tests; default to the live local catalog and detected hardware.

  • compute (optional) – Injectable for tests; default to the live local catalog and detected hardware.

  • cloud_catalog (optional) – Injectable for tests; defaults to the live pricing.yaml catalog. Cloud mode only, and only consulted when the brief omits model.

  • disk (optional) – Injectable storage profile (detect.disk_profile()). Read only on the layer-streaming path, where the drive bounds throughput.

Returns:

The engine descriptor (see write_engine() for the on-disk shape).

Return type:

dict

best_engine_ai_helper.engine.write_engine(engine, path)[source]

Write an engine descriptor to path as YAML with a do-not-commit header.

The file is machine-specific (it encodes the chosen backend and models for this hardware), so it belongs in .gitignore, not in version control.

Parameters:
Return type:

Path