best_engine_ai_helper.score module

score — select the best model for the current hardware.

Given the detected memory pool and the merged catalog, this module picks the highest-scoring model that fits within a safety headroom. The algorithm is intentionally simple: filter, then sort by benchmark score, then take the max.

Memory priority mirrors what Ollama uses at runtime:

unified_gb (Apple Silicon) > vram_gb (discrete GPU) > ram_gb * 0.5 (CPU)

The 0.5 factor for CPU RAM is conservative; a model loader competes with the OS, background daemons, and the inference server itself for RAM.

Author

Warith Harchaoui <warithmetics@deraison.ai>

best_engine_ai_helper.score.EMBED_SCORE_KEY = 'mteb'

Benchmark axis an embedding model is judged on. Embedders are not scored on the general/vision axes the other kinds use, so anything ranking them has to reach for this one by name.

best_engine_ai_helper.score.benchmark_score_or_none(entry, kind, application=None)[source]

Same lookup as _benchmark_score(), but None when no axis is populated.

The two answers _benchmark_score cannot tell apart are “measured at zero” and “nobody measured it”, and it reports both as 0.0. Sorting by raw score does not care – an unscored model goes last either way. A min-max normalisation does: a single unscored candidate pulls the bottom of the quality range down to zero, compressing every real score toward the top and quietly handing the decision to whatever axis it is blended with (see cloud_catalog.rank_cloud(), where that axis is price).

So the caller that normalises asks this one, and leaves the unscored out of the range rather than at the floor of it.

Parameters:

  • entry (dict[str, Any]) – A catalog entry dict.

  • kind ({'llm', 'vlm'}) – The inference kind being selected.

  • application (str or None) – Optional use-case keyword; resolved exactly as in _benchmark_score().

Returns:

The score, or None when every axis in the priority list is null.

Return type:

float or None

Examples

>>> benchmark_score_or_none({"benchmarks": {"general": 0.0}}, "llm")
0.0
>>> benchmark_score_or_none({"benchmarks": {"general": None}}, "llm") is None
True
>>> benchmark_score_or_none({"benchmarks": {"arena_code": 96.1}}, "llm") is None
True
best_engine_ai_helper.score.effective_budget(hw, headroom=0.5, load=None)[source]

Compute the memory budget (GB) a model may occupy at run time.

The budget is the accelerator’s usable memory pool, scaled by an extra headroom margin left for the operating system, your own application, and KV-cache growth as context fills. On Apple Silicon the usable pool is not the whole unified memory: Metal caps GPU allocations at about 66% of the pool at or below 36 GB and about 75% above it, beyond which inference spills to CPU. Compare a catalog entry’s ram_gb (already a peak-inference estimate, weights plus a moderate KV cache) against this budget.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory(). Expected keys: unified_gb, vram_gb, ram_gb.

  • headroom (float) – Extra safety fraction applied on top of the accelerator cap, reserving room for the OS, the caller’s workload, and KV growth. Defaults to MAX_HEADROOM (0.5) and is clamped down to it — a larger value is never honoured, to keep picks realistic.

  • load (dict or None) – Live server state from detect.server_load() (available_ram_gb, cpu_percent, disk_free_gb, …). When given, the theoretical accelerator budget is additionally capped at what is ACTUALLY free right now (another process, or an already-running engine, holds memory the static hardware totals in hw know nothing about), and further derated when the CPU is already saturated or the disk is nearly full. None (the default) reproduces the pre-existing, load-blind behaviour exactly.

Returns:

Effective memory budget in GB.

Return type:

float

Examples

>>> effective_budget({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0})
36.0
>>> effective_budget({'unified_gb': None, 'vram_gb': 24.0, 'ram_gb': 64.0})
11.04
>>> # headroom above the 0.5 ceiling is clamped, not honoured
>>> effective_budget({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}, headroom=0.85)
36.0
>>> # a busy machine gets a smaller budget than an idle one with the same hardware
>>> hw = {'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}
>>> effective_budget(hw, load={'available_ram_gb': 10.0})
10.0
best_engine_ai_helper.score.estimated_tokens_per_second(entry, bandwidth_gbs, backend='ollama', calibrated=True, *, disk_read_gbs=None)[source]

Estimate local decode throughput (tokens/s) for a model on this machine.

Token generation is memory-bandwidth bound: each new token requires reading the model’s active weights from memory once, so the ceiling is bandwidth / model_bytes. The estimate derates that ceiling by a backend-specific decode efficiency (see _DECODE_EFFICIENCY_BY_BACKEND) to reflect KV-cache reads, kernel overhead, and sampling. It describes steady-state generation, not the compute-bound prefill of a long prompt.

Parameters:

  • entry (dict[str, Any]) – Catalog entry; its model_footprint_gb() is the active-model size.

  • bandwidth_gbs (float or None) – Memory bandwidth in GB/s from detect.compute_profile(). When None (unknown hardware, e.g. an unrecognised discrete-GPU model) the estimate is not computable and None is returned.

  • backend ({'ollama', 'vllm', 'lightweight'}) – Serving backend. 'lightweight' switches the whole model: throughput is bounded by disk read bandwidth rather than memory bandwidth, and disk_read_gbs is the input that matters (see offload.estimated_tokens_per_second()). For the other two it affects both the size proxy (heavier FP16 weights under vLLM decode more slowly than the Q4 Ollama figure) AND the decode efficiency (vLLM’s PagedAttention + CUDA-graph decode tracks closer to the bandwidth ceiling than llama.cpp’s — confirmed to differ on identical Ubuntu + discrete-GPU hardware, not just a cross-platform artifact).

  • calibrated (bool) – Apply this machine’s measured correction factor when one exists (see throughput.calibration_factor()). On by default: an estimate that has been corrected against a real measurement on this hardware is strictly better than one that has not. Pass False to see the raw bandwidth model, which is what the calibration itself is derived against – leaving it True there would feed the correction back into its own input. Ignored under backend='lightweight': the calibration was measured against a memory-bound decode and does not describe a disk-bound one.

  • disk_read_gbs (float or None) – Sequential disk read bandwidth in GB/s, from detect.disk_profile(). Only read under backend='lightweight'; None lets that branch look it up from the storage class.

Returns:

Estimated tokens/s, rounded to one decimal, or None when bandwidth or model size is unavailable. Under backend='lightweight' the figure is rounded to three decimals instead, because the honest answer there is usually a fraction of a token per second.

Return type:

float or None

Examples

>>> entry = {'id': 'a', 'ram_gb': 9.9, 'size_b': 14}
>>> estimated_tokens_per_second(entry, 400.0, calibrated=False)
26.3
>>> estimated_tokens_per_second(entry, None) is None
True
>>> # Same model, streamed off a 4 GB/s drive: a thousand times slower.
>>> estimated_tokens_per_second(entry, 400.0, 'lightweight', disk_read_gbs=4.0)
0.121
best_engine_ai_helper.score.is_sizeable(entry, backend='ollama')[source]

True when entry carries enough information to be sized on backend.

A model that cannot be sized is not a candidate: the whole job here is comparing a footprint against this machine’s memory, and an entry that cannot supply one has nothing to compare. Separate from model_footprint_gb() so the call sites that filter read as filters rather than as float comparisons against infinity.

Parameters:

  • entry (dict[str, Any]) – Catalog entry.

  • backend ({'ollama', 'vllm'}) – Serving backend the footprint would be estimated for.

Returns:

False when neither ram_gb nor size_b yields a usable footprint.

Return type:

bool

best_engine_ai_helper.score.model_footprint_gb(entry, backend='ollama')[source]

Estimate a model’s peak inference memory (GB) on a given serving backend.

Three backends, three different physical situations, and sizing one against another’s figure is how a picker comes to recommend a model that will not load. The catalog’s ram_gb is an Ollama Q4 estimate. vLLM instead loads the full FP16/BF16 HuggingFace weights (~2 bytes/param) plus KV cache and runtime buffers, so the same model is markedly heavier there. Sizing a vLLM pick against ram_gb would over-promise and pick a model that will not actually fit — the “unrealistic recommendation” this guards against.

Parameters:

  • entry (dict[str, Any]) – Catalog entry; uses ram_gb (Ollama) or size_b (vLLM).

  • backend ({'ollama', 'vllm', 'lightweight'}) – Serving backend the footprint is estimated for. Unknown values are treated as 'ollama' (the conservative catalog figure). 'lightweight' streams layers off disk and so reports the resident size of ONE layer rather than of the model — see offload.footprint_gb().

Returns:

Estimated peak memory in GB.

Return type:

float

Examples

>>> model_footprint_gb({'id': 'a', 'ram_gb': 9.9, 'size_b': 14})
9.9
>>> model_footprint_gb({'id': 'a', 'ram_gb': 9.9, 'size_b': 14}, 'vllm')
32.2
>>> # Streaming sizes one layer, not the model: a 70B becomes laptop-sized.
>>> model_footprint_gb({'id': 'b', 'size_b': 70, 'layers': 80}, 'lightweight')
6.1
best_engine_ai_helper.score.rank(hw, catalog, kind, headroom=0.5, application=None, backend='ollama', load=None, bandwidth_gbs=None, min_tps=None, installed=None, require_structured=True, disk_read_gbs=None)[source]

Return all candidates sorted by benchmark score, annotated with fit status.

Each entry in the result gets a _fits key (bool) indicating whether it fits within the effective budget. The top entry is identical to what select() would return.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory().

  • catalog (list[dict[str, Any]]) – Merged model entries from catalog.load_catalog().

  • kind ({'llm', 'vlm'}) – The inference kind to filter and rank by.

  • headroom (float) – Extra safety on top of the accelerator cap. Clamped to MAX_HEADROOM (0.5); default 0.5.

  • application (str or None) – Optional use-case keyword ("code", "math", "ocr", "vision", "chat", "generalist"). Drives which benchmark column is used for ranking.

  • backend ({'ollama', 'vllm'}) – Serving backend, so _fits reflects the footprint that actually loads (FP16 for vLLM, Q4 ram_gb for Ollama).

  • load (dict or None) – Live server state from detect.server_load(), forwarded to effective_budget(). None reproduces the load-blind behaviour.

  • bandwidth_gbs (float or None) – Memory bandwidth from detect.compute_profile(), used to estimate throughput for candidates with no measurement. None leaves the throughput annotations unpopulated unless a measurement exists.

  • disk_read_gbs (float or None) – Sequential disk read bandwidth from detect.disk_profile(), used only under backend='lightweight', where throughput is bounded by the drive rather than the memory bus.

  • min_tps (float or None) – Throughput floor in tokens/s. A candidate measured below it on this machine is demoted out of auto-selection, in the same way and for the same reason as one that does not fit memory. A candidate merely estimated below it is annotated and left in place: the estimator was measured here to run optimistic by a wide margin, so letting it disqualify anything would reject models that run perfectly well. None disables the check entirely.

  • require_structured (bool) – Whether schema-constrained output is a requirement of the job. True (the default) demotes a model that cannot honour Ollama structured output below one that can, however well it scores. False drops that key entirely, for a caller whose brief says it does not need schemas.

  • installed (set of str or None) – Model tags already present on the machine. Used only to break ties between candidates of comparable quality (within QUALITY_TIE_BAND), never to promote a worse model over a better one: at equal quality a model already on disk beats a 20 GB download, and at unequal quality the better model still wins. Passed in rather than probed here so that ranking stays a pure function and no network call hides inside a sort.

Returns:

Candidates sorted descending by benchmark score, each annotated with _fits, and with _tps / _tps_basis / _comfy when a throughput figure could be obtained.

Return type:

list[dict[str, Any]]

Examples

>>> catalog = [{'id': 'v', 'kind': 'vlm', 'ram_gb': 9.0,
...             'benchmarks': {'vision': 80}}]
>>> rank({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}, catalog, 'vlm')[0]['_fits']
True
best_engine_ai_helper.score.resolve_benchmark(entry, kind, application=None)[source]

Which axis supplies this entry’s score, and what it says.

The single implementation behind _benchmark_score() and benchmark_score_or_none() — and the one a caller asks when the axis matters as much as the number. It often does: a priority list falls through until something is populated, so two candidates in the same ranking can be scored on different measurements. 88 from an MMLU-style suite and 86 from a human-preference leaderboard are both “out of 100” and are not the same quantity, and a table that prints only the numbers invites a comparison nobody verified.

Parameters:

  • entry (dict[str, Any]) – A catalog entry dict.

  • kind ({'llm', 'vlm'}) – The inference kind being selected.

  • application (str or None) – Optional use-case keyword; None resolves through _DEFAULT_APPLICATION.

Returns:

(axis name, score), or None when every axis in the list is null.

Return type:

tuple[str, float] or None

Examples

>>> resolve_benchmark({"benchmarks": {"general": 88}}, "llm")
('general', 88.0)
>>> resolve_benchmark({"benchmarks": {"general": None, "arena": 85.2}}, "llm")
('arena', 85.2)
>>> resolve_benchmark({"benchmarks": {}}, "llm") is None
True
best_engine_ai_helper.score.select(hw, catalog, kind, headroom=0.5, application=None, backend='ollama', load=None, require_structured=True)[source]

Pick the best-scoring model that fits in available memory.

Candidates for a ‘vlm’ selection include both VLMs and LLMs with vision capability (kind == ‘vlm’). Candidates for a ‘llm’ selection include text-only LLMs and VLMs (since a VLM handles text-only prompts equally).

Selection order:

  1. Filter: keep entries whose ram_gb fits within the effective budget.

  2. Rank: structured-output capability first (a model that cannot honour Ollama structured output is never chosen over one that can), then benchmark score (application-specific if given, else vision for VLM or general for LLM). This matches rank(), so rank(...)[0] and select(...) agree.

  3. Last resort: if nothing fits, return the smallest model in the catalog rather than raising; the caller decides whether to warn the user.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory().

  • catalog (list[dict[str, Any]]) – Merged model entries from catalog.load_catalog().

  • kind ({'llm', 'vlm'}) – The type of model to select.

  • headroom (float) – Extra safety on top of the accelerator cap. Clamped to MAX_HEADROOM (0.5); default 0.5.

  • application (str or None) – Optional use-case keyword ("code", "math", "ocr", "vision", "chat", "generalist"). Selects the benchmark axis used for scoring. None uses the default kind-based rule.

  • backend ({'ollama', 'vllm'}) – Serving backend, so memory fit is checked against the footprint that actually loads (FP16 for vLLM, Q4 ram_gb for Ollama).

  • load (dict or None) – Live server state from detect.server_load(), forwarded to effective_budget(). None reproduces the load-blind behaviour.

  • require_structured (bool)

Returns:

The selected catalog entry.

Return type:

dict[str, Any]

Raises:

ValueError – If the catalog is empty.

Examples

>>> hw = {'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}
>>> catalog = [{'id': 'v', 'kind': 'vlm', 'ram_gb': 9.0,
...             'benchmarks': {'vision': 80}}]
>>> select(hw, catalog, kind='vlm')['id']
'v'
best_engine_ai_helper.score.sizeable_only(entries, backend='ollama', where='selection')[source]

Drop entries that cannot be sized, saying how many and from where.

Announced rather than silent, by the same rule the backend gate follows: a pool quietly shrinking is how a caller concludes their catalog was considered when half of it was skipped.

Parameters:

  • entries (list of dict) – Candidate catalog entries.

  • backend ({'ollama', 'vllm'}) – Serving backend the footprint would be estimated for.

  • where (str) – Caller name, used in the warning so the message names the decision the entries were dropped from.

Returns:

Only the entries that can be sized.

Return type:

list of dict

best_engine_ai_helper.score.usable_pool_gb(hw)[source]

The accelerator’s usable memory pool (GB), before any safety headroom.

The first half of effective_budget(), split out because one backend needs it on its own. On Apple Silicon the usable pool is NOT the whole unified memory — Metal caps GPU allocations at about 66% of a pool up to 36 GB and 75% above it, beyond which inference spills to CPU. A discrete GPU’s VRAM is nearly all usable; a CPU-only machine is held to half its RAM.

Every caller sizing a model that STAYS in memory wants effective_budget() instead: the extra headroom is what leaves room for the OS, the caller’s own workload and KV-cache growth. The exception is layer streaming, where what occupies memory is one transient layer that the runtime frees per layer, and where the KV cache is sized explicitly rather than absorbed into a margin (see offload.footprint_gb()) — applying the headroom there would reserve the same memory twice and reject the models the backend exists to run.

Parameters:

hw (dict[str, float | None]) – Output of detect.available_memory().

Returns:

Usable pool in GB.

Return type:

float

Examples

>>> usable_pool_gb({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0})
72.0
>>> usable_pool_gb({'unified_gb': 8.0, 'vram_gb': None, 'ram_gb': 8.0})
5.28
>>> usable_pool_gb({'unified_gb': None, 'vram_gb': 24.0, 'ram_gb': 64.0})
22.08
>>> usable_pool_gb({'unified_gb': None, 'vram_gb': None, 'ram_gb': 32.0})
16.0