best_engine_ai_helper.throughput module

throughput — measure decode speed from work the Ralph gates already do.

This package ranks models partly on how fast they decode, and until now that figure was always estimated: memory bandwidth times a community constant. For a project whose argument is that you should measure rather than assume, that was the one place it assumed. This module closes it.

The measurement costs no extra inference. pull already pulls a candidate and puts it through two Ralph gates, which means the tokens have already been generated and the only thing missing was a stopwatch. Meter wraps the chat callable the gates are handed, counts what passes through, and leaves the gates themselves untouched — they do not need to know they are being timed.

Two clocks, and the difference matters

Ollama reports eval_duration: the decode phase alone, as timed by the daemon. Wall-clock time around the same call also contains prompt processing, HTTP, and any queueing behind another request. Dividing tokens by wall time therefore understates decode speed, sometimes by a lot on a short reply, and understating is exactly what turns a throughput floor into a gate that rejects models which would have run fine.

So a measurement records which clock produced it. basis="eval_duration" is trustworthy enough to disqualify a model; basis="wall_clock" is a hint and is labelled as one. A caller that refuses to act on a hint can check.

Why the figure is stored per machine, and why the best sample wins

Tokens per second is a property of a model and the machine under it, so a measurement is keyed by both. Measuring on a new machine adds a row, and nothing is ever averaged across hardware that has nothing in common.

Re-measuring the same pair keeps the fastest sample rather than the most recent. This is not optimism, it is what the figure means: the quantity being calibrated is a memory-bandwidth ceiling, and contention can only push an observation below the hardware’s real capability, never above it. Measured during development, the same model on the same machine gave 15.2 tok/s idle and 8.7 tok/s with three test suites running — so taking the latest sample would let whatever happened to be running at the time poison every estimate on the machine. record(..., replace=True) overrules this for the case where the hardware genuinely did slow down.

Author

Warith Harchaoui <warithmetics@deraison.ai>

best_engine_ai_helper.throughput.MIN_TOKENS_FOR_CONFIDENCE = 40

Below this many sampled output tokens a throughput figure is noise rather than a measurement. A handful of tokens is dominated by the first-token latency the decode clock is supposed to exclude, and a model should never be disqualified on that. Deliberately low: the prose gate’s fixtures are short rewrites, so demanding hundreds of tokens would mean never measuring at all.

class best_engine_ai_helper.throughput.Meter

Bases: object

Count tokens and time for every chat call that passes through it.

Wrap a chat callable with wrap() and hand the wrapper to whatever already generates tokens — the Ralph gates, in this package’s case. The meter accumulates across every call, so a gate that makes six requests yields one measurement over all six rather than six noisy ones.

tokens

Output tokens seen so far, as reported by the backend.

Type:

int

gen_ms

Decode-only milliseconds, summed. Zero when no backend reported any.

Type:

float

wall_ms

Wall-clock milliseconds, summed. Always populated.

Type:

float

calls

Calls that yielded a usable output-token count — not calls made. A request that raised, or one whose backend reports no token counts, contributes nothing here, because neither produced anything to divide by. So calls is the size of the sample, which is what a reader of a measurement wants to know.

Type:

int

Examples

A callable that reports no usage cannot be measured, and the meter says so rather than counting the call and inventing a rate for it:

>>> def fake_chat(prompt, **kw):
...     return "ok"
>>> meter = Meter()
>>> meter.wrap(fake_chat)("hello")      # the reply is passed straight through
'ok'
>>> meter.calls                         # ...but nothing was measurable
0
>>> meter.measurement() is None
True

measurement()

Summarise what was metered, or None when there is nothing to say.

Returns None rather than a zero or a guess in every case where a number would be made up: no calls, no token counts, no elapsed time, or too few tokens to be more than first-token latency. A caller can then treat “unmeasured” and “measured slow” as the different things they are.

Returns:

{"tokens_per_second", "basis", "tokens", "calls", "measured_at", "measured_on"}.

Return type:

dict or None

wrap(chat)

Return chat with metering attached, same signature and return value.

The wrapper is transparent: it forwards every argument, returns the callee’s result untouched, and lets exceptions through. Metering must never change what the gates see, or the gate result stops meaning what it meant before.

Parameters:

chat (callable) – A llm.chat-compatible callable.

Returns:

The metered wrapper.

Return type:

callable

best_engine_ai_helper.throughput.calibration_factor(machine=None, store_path=None)

How wrong the bandwidth estimate runs on this machine, and on how much evidence.

The bandwidth model predicts decode speed from memory bandwidth and a community-derived efficiency constant. Those constants are averages over other people’s hardware, so on any particular machine the prediction has a systematic bias. That bias is a property of the machine far more than of the model, which is what makes it transferable: measuring one model improves the prediction for every model in the catalog, including the ones that were never pulled.

How much it transfers, measured rather than assumed

scripts/validate_calibration.py runs the leave-one-out version of that claim — calibrate on A, predict B — and on an Apple M2 Max it held in every direction tried: the error fell from 64-118% to 17-25%.

So the claim is true and it is not precision. Across models and across repeated runs of the same model the ratio ranged roughly 0.46 to 0.61, which is why this returns a median over several measurements rather than trusting one, and why a real measurement still outranks a calibrated estimate wherever one exists. Calibration moves an estimate from wrong by a factor of two to wrong by about a fifth; it does not make it right.

Derived from stored measurements rather than kept as separate state, so it cannot drift out of agreement with the measurements it summarises. Only figures from the serving daemon’s own decode clock count — a wall-clock ratio would bake prompt-processing time into a decode correction.

param machine:

Machine label; defaults to this machine.

type machine:

str or None

param store_path:

Override the measurement store location; for tests.

type store_path:

Path or None

returns:

The multiplier to apply to an estimate and how many measurements back it, or None when there is no usable evidence or the ratio is so far out that the inputs are more likely wrong than the hardware.

rtype:

(factor, samples) or None

Parameters:

• machine (str | None)

• store_path (Path | None)

Return type:

tuple[float, int] | None

best_engine_ai_helper.throughput.load_measurements(store_path=None)

Read the measurement store, keyed "<model>@<machine>".

A missing or unreadable store is an empty dict, not an error: no measurement yet is the normal state on a fresh machine, and every caller here has an estimate to fall back on.

Parameters:

store_path (Path or None) – Override the default location; for tests.

Returns:

Mapping of key to measurement dict.

Return type:

dict

best_engine_ai_helper.throughput.machine_id()

Name the machine a measurement belongs to.

The CPU model is the coarsest label that still separates hardware whose decode speed has nothing in common (“Apple M2 Max” vs “Apple M1”), while staying stable across reboots and OS updates — unlike a hostname, which a user can change, or a MAC address, which has no business in a file like this.

Returns:

A short hardware label, or "unknown" if it cannot be determined.

Return type:

str

best_engine_ai_helper.throughput.measured_context_length(model, store_path=None)

The recorded context window for model on this machine, if one was read.

Parameters:

• model (str) – Model tag.

• store_path (Path or None) – Override the default location; for tests.

Returns:

The window, or None when nothing has been recorded for this pair.

Return type:

int or None

Examples

>>> import tempfile, pathlib
>>> with tempfile.TemporaryDirectory() as d:
...     store = pathlib.Path(d) / "m.yaml"
...     _ = record_context_length("qwen3:8b", 32768, store)
...     measured_context_length("qwen3:8b", store)
32768
>>> import tempfile, pathlib
>>> with tempfile.TemporaryDirectory() as d:
...     measured_context_length("never-pulled", pathlib.Path(d) / "m.yaml") is None
True

best_engine_ai_helper.throughput.measured_tps(model, machine=None, store_path=None, trusted_only=False)

Look up the measured decode speed for model on this machine.

Parameters:

• model (str) – Model tag to look up.

• machine (str or None) – Machine label; defaults to this machine.

• store_path (Path or None) – Override the default location; for tests.

• trusted_only (bool) – When True, ignore a figure derived from the wall clock. Pass this from anywhere a measurement would disqualify a model, since wall time understates decode speed and would reject candidates unfairly.

Returns:

Tokens per second, or None when there is no applicable measurement.

Return type:

float or None

best_engine_ai_helper.throughput.record(model, measurement, store_path=None, estimated_tps=None, replace=False)

Persist one measurement, keeping the best sample for the (model, machine) pair.

Parameters:

• model (str) – The model tag that was measured.

• measurement (dict) – As returned by Meter.measurement().

• store_path (Path or None) – Override the default location; for tests.

• estimated_tps (float or None) – What the bandwidth model predicted for this model on this machine. Stored next to the measurement so calibration_factor() can derive the ratio later without having to re-enter the estimator — which is also what keeps this module from importing score, the module that imports it.

• replace (bool) – Overwrite a faster stored sample with this one. Off by default, because a slower sample almost always means the machine was busy rather than that the model slowed down. Pass True when the hardware really did change and the old ceiling is no longer reachable.

Returns:

The file written.

Return type:

Path

best_engine_ai_helper.throughput.record_context_length(model, window, store_path=None)

Persist the context window a daemon reported for model on this machine.

Deliberately NOT routed through record(), and the reason is the rule that function exists to enforce. record keeps the best sample for a pair because throughput is an observation of a ceiling, and a slower run says more about what else was running than about the model. A context window is not an observation of anything: it is a constant of the checkpoint, read out of its own metadata. So it always overwrites, and a re-pull at a different quantisation is allowed to change it — where routing it through record would have silently dropped the new figure whenever the run that carried it happened to be the slower one.

Parameters:

• model (str) – The model tag the window belongs to.

• window (int) – Context length in tokens, as read from the daemon.

• store_path (Path or None) – Override the default location; for tests.

Returns:

The file written, or None when window is not a usable figure — the ordinary case for a daemon that is absent or a checkpoint that declares nothing, neither of which is worth a row.

Return type:

Path or None

Examples

>>> import tempfile, pathlib
>>> with tempfile.TemporaryDirectory() as d:
...     store = pathlib.Path(d) / "m.yaml"
...     _ = record_context_length("gemma3:12b", 131072, store)
...     rows = load_measurements(store)
...     next(iter(rows.values()))["context_length"]
131072

Nothing to record is not an error:

>>> record_context_length("gemma3:12b", 0) is None
True