best_engine_ai_helper.sources.arena module

sources.arena — conversational-preference ratings from the LM Arena family.

Arena ratings come from humans comparing two anonymous answers and picking one. That makes them the best public measure of conversational preference and a poor measure of almost everything else: a model that writes agreeable prose outranks one that emits a schema-valid SQL query, because nobody voting was asked to run the query. This adapter therefore contributes one axis, arena, and the ranking layer decides what that axis is worth per use case (see score._APP_BENCH_PRIORITY, where an axis’s weight is its position in the list).

Two problems had to be solved before the numbers could enter the catalog.

Scale. An Elo rating lives around 1000-1500; every other axis in the catalog is 0-100. The ranking layer takes the first non-null axis in a priority list, so two candidates scored on different axes would be compared on incompatible scales. Ratings are therefore converted at ingestion to the expected win rate against a fixed reference rating – see elo_to_score().

Staleness. Arena data is not freely and freshly available. As of 2026-10-19 the site answers 403 behind a bot wall, one community mirror is emitting empty snapshots, and the other two stopped updating in 2025. A rating fourteen months old would rank last year’s models above this year’s, which is strictly worse than having no rating at all. So every source reports the date of the snapshot it returned, and anything older than MAX_SNAPSHOT_AGE_DAYS is refused with the reason rather than quietly merged.

Not supported, deliberately: the commercial scraping APIs that do have fresh data. They require a paid token and route the query through a third party, which is the opposite of what this suite promises. If you have such a feed, pass it as a custom source rather than having this module reach for one.

Author

Warith Harchaoui <warithmetics@deraison.ai>

best_engine_ai_helper.sources.arena.ELO_SCALE = 400.0

1 expected win ratio.

Type:

The Elo scale constant. A 400-point gap is a 10

best_engine_ai_helper.sources.arena.HISTORY_URL = 'https://raw.githubusercontent.com/nakasyou/lmarena-history/main/output/scores.json'

Community mirrors, tried in order. Arena publishes no API, so every entry here is someone else’s scrape and every one of them can rot; the adapter reports which answered and how old it was rather than assuming.

best_engine_ai_helper.sources.arena.MAX_SNAPSHOT_AGE_DAYS = 120

Oldest snapshot still worth merging. Beyond this the ratings describe a different generation of models and would actively mislead the ranker.

best_engine_ai_helper.sources.arena.REFERENCE_ELO = 1200.0

Reference rating for the 0-100 conversion. Roughly a competent mid-board model, so a score of 50 means “wins about half its comparisons against a solid-but-unremarkable opponent”. Fixed rather than derived from the board so the conversion does not rescale every time the leaderboard moves.

best_engine_ai_helper.sources.arena.SNAPSHOT_DIR_URL = 'https://api.github.com/repos/oolong-tea-2026/arena-ai-leaderboards/contents/data'

Directory listing for the dated snapshots. The mirror runs daily and keeps every day it has ever produced, so the newest date is not necessarily the newest date with data — see _from_snapshots().

best_engine_ai_helper.sources.arena.SOURCES = (('arena-ai-leaderboards', <function _from_snapshots>), ('lmarena-history', <function _from_history>))

Mirrors in preference order. The structured one first because its shape is unambiguous; the bulk history as the fallback.

best_engine_ai_helper.sources.arena.elo_to_score(elo, reference=1200.0)

Convert an Elo rating to a 0-100 score comparable with the other axes.

Uses the Elo expected-score formula: the probability that a model rated elo beats one rated reference, as a percentage.

Chosen over min-max normalisation for two reasons. A min-max rescales every time the leaderboard moves, so yesterday’s 100 is today’s 94 and a cached catalog silently disagrees with a fresh one. And Elo is an interval scale with no meaningful zero, so the floor of a min-max would be an artefact of whoever happened to be last that week.

Parameters:

• elo (float) – The rating, typically 800-1600.

• reference (float) – The rating that maps to 50. Defaults to REFERENCE_ELO.

Returns:

Expected win rate against the reference, 0-100, rounded to one decimal.

Return type:

float

Examples

>>> elo_to_score(1200)
50.0
>>> elo_to_score(1600) > 90
True
>>> elo_to_score(800) < 10
True

best_engine_ai_helper.sources.arena.fetch_arena_ratings(session=None, timeout=30.0, max_age_days=120, today=None)

Fetch Arena ratings from whichever mirror is alive and fresh enough.

Parameters:

• session (requests.Session or None) – Optional shared session.

• timeout (float) – Per-request timeout in seconds.

• max_age_days (int) – Refuse a snapshot older than this. Set to 0 to disable the check, which you should only do to inspect an old snapshot deliberately – merging one into a live catalog ranks a previous generation of models first.

• today (datetime.date or None) – Injectable for tests.

Returns:

{"source": str, "snapshot": str, "age_days": int | None, "ratings": {axis: {model: elo}}, "scores": {axis: {model: 0-100}}, "stale": bool, "note": str}. ratings and scores are empty when nothing usable was found; note always says why.

Return type:

dict

Examples

>>> result = fetch_arena_ratings()
>>> result["source"], result["age_days"]
('lmarena-history', 515)

best_engine_ai_helper.sources.arena.match_model(catalog_id, ratings)

Find the Arena entry for a catalog id, or None.

Arena names a model the way its provider announced it (qwen3-235b-a22b-instruct-2507); this catalog names it the way the runtime pulls it (qwen3:32b). No shared identifier exists, so the match is heuristic and deliberately conservative: a wrong match attributes another model’s reputation to this one, which is worse than no match.

Passes, strictest first: exact; exact after normalising separators; exact after dropping a quantization suffix the leaderboard never carries (qwen3:32b-q4_k_m is Arena’s qwen3-32b); a unique prefix match, which catches the release suffixes a leaderboard adds (-instruct, -preview); then the reasoning-effort suffixes (_SERVING_MODES), where one billable id is rated several times and the lowest of those ratings is the one this takes; and finally an alias resolved to the newest dated release of its family, which is what the alias means rather than a heuristic.

A prefix matching several entries is rejected rather than guessed: attributing one model’s reputation to another is worse than leaving the axis empty, because the ranker would act on it.

Parameters:

• catalog_id (str) – An id as models.yaml spells it, e.g. "qwen3-vl:8b".

• ratings (dict) – {arena model name: rating}.

Returns:

The matching Arena key, or None when no confident match exists.

Return type:

str or None

Examples

>>> match_model("qwen3:32b", {"qwen3-32b": 1300.0})
'qwen3-32b'
>>> match_model("qwen3:32b-q4_k_m", {"qwen3-32b": 1300.0})
'qwen3-32b'
>>> match_model("qwen3:32b", {"qwen3-32b-instruct": 1300.0})
'qwen3-32b-instruct'
>>> match_model("qwen3:32b", {"qwen3-32b-a": 1.0, "qwen3-32b-b": 2.0}) is None
True
>>> match_model("mistral-large-latest",
...             {"mistral-large-2407": 1.0, "mistral-large-2411": 2.0})
'mistral-large-2411'
>>> match_model("claude-3-5-sonnet",
...             {"claude-3-5-sonnet-20240620": 1.0,
...              "claude-3-5-sonnet-20241022": 2.0})
'claude-3-5-sonnet-20241022'
>>> match_model("claude-3-5-sonnet",
...             {"claude-3-5-sonnet-20250219-thinking-32k": 1.0}) is None
True

One model, several reasoning efforts, one price per token: the floor is taken, because the higher rating describes a mode the caller has to ask for.

>>> match_model("claude-opus-5-5",
...             {"claude-opus-5.5-max": 97.2, "claude-opus-5.5-high": 85.2})
'claude-opus-5.5-high'

Unless the board also rates the bare id, which IS the default mode:

>>> match_model("claude-opus-4-6",
...             {"claude-opus-4-6": 84.7, "claude-opus-4-6-high": 85.3})
'claude-opus-4-6'

best_engine_ai_helper.sources.arena.merge_into(entries, result)

Add Arena axes to catalog entries, and report what was matched.

Entries are copied, not mutated: a refresh that half-applies and then fails should leave the caller’s catalog as it found it.

Returns:

report carries matched, unmatched and the source note, so a caller can show which models got a rating and which the heuristic declined to guess at.

Return type:

(entries, report)

Parameters:

• entries (list[dict[str, Any]])

• result (dict[str, Any])

Examples

A matched entry carries the snapshot it came from and the board name behind each axis, so the rating can be traced back without re-running the fetch:

>>> result = {"snapshot": "2026-10-07",
...           "scores": {"arena": {"claude-opus-5.5-high": 85.2}}}
>>> merged, report = merge_into([{"id": "claude-opus-5-5"}], result)
>>> merged[0]["benchmarks"]["arena"], merged[0]["arena_snapshot"]
(85.2, '2026-10-07')
>>> merged[0]["arena_matched"]
{'arena': 'claude-opus-5.5-high'}
>>> report["matched"]
['claude-opus-5-5']

best_engine_ai_helper.sources.arena.snapshot_age_days(snapshot, today=None)

Age in days of a snapshot identifier, or None if it carries no date.

Parameters:

• snapshot (str)

• today (date | None)

Return type:

int | None