best_engine_ai_helper.detect module

detect — turn raw hardware facts into AI-inference throughput estimates.

Raw probing (subprocess calls to nvidia-smi / rocm-smi / system_profiler / lspci, CPU/RAM introspection) is NOT done here: it lives in os_helper.hardware_utils, a generic cross-platform hardware probe shared by every repo in the AI Helpers suite. This module is the thin, AI-domain layer on top of it — it takes the vendor + chip/GPU name + memory size that os_helper reports and turns them into what a local model picker actually needs: an accelerator kind, a memory-bandwidth estimate, and the memory pool available for inference.

Memory bandwidth is the ceiling on decode throughput (token generation reads the whole active model from memory once per token), so it — not raw core count — is what compute_profile() reports and what score.estimated_tokens_per_second keys its tokens/s estimate on. Apple Silicon publishes per-chip bandwidth; NVIDIA/AMD publish it per GPU model. Both are tabulated below by substring match on the chip/GPU name os_helper reports; an unrecognised model degrades gracefully to “throughput not estimated” rather than a wrong number.

Author

Warith Harchaoui <warithmetics@deraison.ai>

best_engine_ai_helper.detect.available_memory()[source]

Detect available memory for model inference.

Reads three pools from os_helper’s hardware facts:

  1. Apple Silicon unified memory (macOS with Apple chip)

  2. NVIDIA/AMD VRAM (summed across all visible GPUs)

  3. System RAM (always populated)

Returns:

A dict with the following keys:

unified_gbfloat or None

Apple Silicon unified memory pool, in GB.

vram_gbfloat or None

Discrete GPU VRAM (sum of all visible GPUs), in GB.

ram_gbfloat

Total system RAM in GB. Always a positive float.

Return type:

dict[str, float | None]

Examples

>>> mem = available_memory()
>>> set(mem.keys()) == {'unified_gb', 'vram_gb', 'ram_gb'}
True
>>> mem['ram_gb'] > 0
True
best_engine_ai_helper.detect.chip_name()[source]

Return the Apple Silicon chip name (e.g. "Apple M2 Max"), or None.

Delegates to os_helper.apple_chip_name(); None on non-macOS platforms or when the chip line is absent (old Intel Macs).

Return type:

str | None

best_engine_ai_helper.detect.chip_vendor()[source]

Identify the primary compute vendor for model inference.

Returns:

One of: ‘apple’, ‘nvidia’, ‘amd’, ‘intel’, ‘cpu’. Delegates to os_helper.gpu_vendor().

Return type:

str

Examples

>>> chip_vendor() in ('apple', 'nvidia', 'amd', 'intel', 'cpu')
True
best_engine_ai_helper.detect.compute_profile()[source]

Describe the machine’s inference accelerator and memory bandwidth.

Returns a dict with:

  • accelerator: "gpu-metal" (Apple Silicon), "gpu-cuda" (NVIDIA), "gpu-rocm" (AMD), or "cpu" (no discrete accelerator detected).

  • chip: the chip / GPU name when known, else None.

  • bandwidth_gbs: memory bandwidth in GB/s when the chip/GPU matches a known model in _APPLE_BANDWIDTH_GBS / _NVIDIA_BANDWIDTH_GBS / _AMD_BANDWIDTH_GBS, else None. This is the ceiling on decode throughput; token generation reads the whole active model from memory once per token, so tokens/s scales with it.

Bandwidth is tabulated for Apple Silicon (per-chip) and for discrete NVIDIA/AMD GPUs (per-board, matched on the model name os_helper reports). An unrecognised GPU model — a new SKU not yet in the table, or a multi-GPU box where the name string is ambiguous — degrades to bandwidth_gbs: None rather than a fabricated number; callers treat that as “throughput not estimated”, never as zero.

Return type:

dict[str, Any]

best_engine_ai_helper.detect.disk_profile(measure=False, directory=None)[source]

Describe the storage this machine would stream model layers from.

The counterpart of compute_profile() for a disk-bound backend. Where memory bandwidth bounds an in-memory decode, disk bandwidth bounds a layer-streaming one (offload/lightweight), which reads every layer back per token.

Parameters:

  • measure (bool) – Run measure_disk_read_gbs() instead of reading the class table. Off by default: it writes a few hundred MB and takes a second or two, and a recommendation should stay a deterministic function of the hardware unless the caller asks for a measurement.

  • directory (str or None) – Volume to measure, when measure is True. None uses the temp directory.

Returns:

storagestr

Class from storage_class().

read_gbsfloat or None

Sequential read bandwidth in GB/s; None when the class is unknown and no measurement succeeded.

basis{‘measured’, ‘declared’, ‘unknown’}

Where read_gbs came from — the same honesty rule the throughput estimates follow: a measured number is never presented as a table lookup, nor the reverse.

free_gbfloat or None

Free space on the volume holding the home directory, where both the HuggingFace cache lands.

Return type:

dict[str, Any]

Examples

>>> p = disk_profile()
>>> p['basis'] in ('measured', 'declared', 'unknown')
True
>>> p['read_gbs'] is None or p['read_gbs'] > 0
True
best_engine_ai_helper.detect.measure_disk_read_gbs(total_bytes=268435456, directory=None)[source]

Measure sequential read bandwidth (GB/s), defeating the page cache.

Writes a scratch file, flushes it to the device, then reads it back with caching disabled for that descriptor (F_NOCACHE on macOS, O_DIRECT on Linux). Without that step the read is served from RAM and reports the memory bus, not the drive — a number three times too high, which would make layer streaming look comfortable when it is not.

Returns None rather than a cached-read figure on any platform that offers no way to bypass the cache (Windows), so an unmeasurable machine falls back to the declared class figure instead of to a flattering one.

Parameters:

  • total_bytes (int) – Bytes to write and read back. Rounded down to a multiple of 4 MiB, the alignment O_DIRECT requires. Defaults to 256 MiB.

  • directory (str or None) – Where to put the scratch file. None uses the system temp directory. Point it at the volume holding the model cache when that differs.

Returns:

Measured GB/s (rounded to two decimals), or None when the measurement is unavailable or the platform cannot bypass its cache.

Return type:

float or None

Examples

>>> v = measure_disk_read_gbs(total_bytes=4 * 1024 * 1024)
>>> v is None or v > 0
True
best_engine_ai_helper.detect.platform_name()[source]

Return the current OS as a short lowercase string.

Returns:

One of: ‘darwin’, ‘linux’, ‘windows’. Delegates to os_helper.platform_name().

Return type:

str

Examples

>>> platform_name() in ('darwin', 'linux', 'windows')
True
best_engine_ai_helper.detect.server_load()[source]

Snapshot the machine’s CURRENT load — as opposed to its static capacity.

Requires an os-helper release whose hardware_info() includes the live fields cpu.percent / available_ram_gb / disk / gpu_utilization_percent — available since os-helper 2.3.0, the floor pinned in pyproject.toml (os-helper>=2.3.0,<3), so any install of this package already has them.

Feeds score.effective_budget()’s optional load parameter, so a recommendation reflects what else is happening on this machine right now: another process (or an already-running engine) can hold memory the static hardware totals from available_memory() know nothing about.

Returns:

available_ram_gbfloat

Free system RAM right now.

cpu_percentfloat

Instantaneous CPU utilization, 0-100.

gpu_percentfloat or None

Live discrete-GPU utilization, 0-100; None on Apple Silicon, CPU-only machines, or when the vendor CLI is unavailable.

disk_free_gbfloat

Free space on the disk holding the home directory (where model caches live).

disk_percent_usedfloat

0-100.

running_enginesint

Best-effort count of already-loaded Ollama models plus running vLLM server processes.

Return type:

dict[str, Any]

Examples

>>> load = server_load()
>>> load["available_ram_gb"] >= 0
True
>>> 0 <= load["cpu_percent"] <= 100
True
>>> load["running_engines"] >= 0
True
best_engine_ai_helper.detect.storage_class()[source]

Classify the drive that holds the model cache.

Layer streaming reads the weights off this drive once per token, so its class — not the memory bus — is what bounds throughput. Classification is deliberately coarse (four classes, see _DISK_READ_GBS): the goal is the right order of magnitude for an estimate, not a device database.

Returns:

One of 'apple-internal', 'nvme', 'ssd-sata', 'hdd' or 'unknown'. 'unknown' is returned rather than a guess when the platform exposes no rotational/transport hint, so callers can say “not estimated” instead of inventing a speed.

Return type:

str

Examples

>>> storage_class() in ('apple-internal', 'nvme', 'ssd-sata', 'hdd', 'unknown')
True