Methodology

BenchLeader is an aggregator. It does not run models, write prompts or grade answers. Once a day it fetches the published results of independent evaluators and arenas, reconciles the different names they use for the same model, and lays the results side by side. The name says what it is: every benchmark, one leaderboard. No single board tells you where a model stands; the picture only becomes clear when all of them are read together.

What counts as a result

• Published by an independent evaluator or a crowdsourced arena. Numbers from a model’s own launch post are not used.
• Machine-readable at the source: a CSV, JSON or dataset the publisher maintains. We do not screen-scrape charts.
• Attributable: every value on this site links to the page it was taken from, with the publisher’s licence noted on the sources list.

One model, many names

GPT-5 (high), openai/gpt-5, gpt-5-2025-08-07 and gpt-5_high are one model evaluated in different configurations. Labels are reduced to a slug, and tokens that describe a run rather than a different model (reasoning effort, thinking budget, dated snapshot, scaffold, quantisation) are split off as the run’s variant. The slug is then matched against a curated registry of aliases. Anything unrecognised becomes a provisional model automatically, tagged “auto-detected” until reviewed, so a new release shows up on the day a leaderboard adds it.

When a source lists several runs of the same model, the model page shows the best result and lists the others. For a quality benchmark that is the highest score; for price and speed it is the first-party provider where one exists, otherwise the median across providers.

The BenchLeader Index

Benchmarks report on incompatible scales: percentages, Elo-style ratings, indices with different ceilings. To combine them, each benchmark’s scores are standardised across the models it has evaluated: the population mean becomes 50 and one standard deviation becomes 15 points, clipped at ±3 standard deviations. Only benchmarks with at least five evaluated models are standardised.

Standardised scores are averaged within each category using the weights below, and the BenchLeader Index is the average of a configuration’s category scores, pulled toward 50 by n/(n+3) where n is the number of results behind it. Six results keep two-thirds of their distance from average, twelve keep four-fifths, thirty keep more than nine-tenths. This stops a configuration that was only run on a handful of favourable boards from outranking one measured everywhere. A configuration needs results on at least five benchmarks across at least two categories to receive an index. Weights halve benchmarks that are saturated, narrow, or overlap heavily with another board.

Known limitations. Standardising within each benchmark means a model’s score depends on who else was evaluated there: a board that only tests frontier models compresses the differences between them. A model evaluated on few, easy benchmarks can look better than it is; the coverage column and the minimum of five results exist to make that visible. The index is a summary for orientation. The individual benchmark tables are the evidence.

Reasoning effort

The same model is evaluated at different reasoning-effort settings (none, minimal, low, medium, high, xhigh, max, or an unspecified “thinking” mode), and results can differ by ten points or more. Each setting is therefore its own row, scored only on the benchmarks run at that setting. “Default” means the publisher did not say. Prices and context windows apply to every setting of a model; measured speed and time to first answer stay with the setting they were measured at. Tables show each model’s best-scoring setting by default, with a switch to see them all.

What the charts show by default

A scatter of every configuration is unreadable, so charts open on the frontier set: the models no other model beats on both axes, the top eight by index, and the best model in each band of the horizontal axis, drawn from configurations released in the last eighteen months with at least eight results. The dashed line joins the frontier. “Top 50” and “All” are one click away.

Speed, latency and price

List prices come from models.dev and the LiteLLM price table, cross-checked against Artificial Analysis and OpenRouter’s marketplace. The blended price assumes three input tokens for every output token. Output speed and time to first token combine Artificial Analysis’ measured medians with OpenRouter’s medians across healthy providers over the last 30 minutes of live traffic; where both exist we take the median of the observations. They vary through the day and by provider, so treat them as a snapshot, not a guarantee.

Per-task estimates

Five representative workloads turn per-million prices into numbers you can budget with:

• Chat reply: A short assistant turn: 400 tokens of context in, 300 tokens out.
• Summarise a 30-page report: 12,000 tokens of document in, a 600-token summary out.
• Code edit: 6,000 tokens of files and instructions in, a 1,500-token diff out.
• Agentic coding session: One long tool-using loop: 60,000 tokens in across turns, 4,000 tokens out.
• Structured extraction: 2,000 tokens of source text in, 200 tokens of JSON out.

Cost = input tokens × input price + output tokens × output price. Time = time to first answer (first token where that is all we have) + output tokens ÷ output speed. “With caching” assumes three-quarters of input tokens are served from the prompt cache at the provider’s cache-read price, a typical ratio for agent loops that resend the same context. Reasoning tokens and batch pricing are not modelled.

Method version 1.1

Any change to weights, eligibility or the formula reorders history, so the method is versioned. Trend lines are recomputed from every daily snapshot whenever the version changes, so a chart never mixes two methods.

• v1.1 (2026-09-09): Rows became model × reasoning-effort configurations. Index shrunk toward 50 by n/(n+3). Added Artificial Analysis, Vals AI, MathArena, Scale SEAL, Terminal-Bench, EQ-Bench, MMMU and Kagi benchmarks with their weights.
• v1.0 (2026-09-08): First release: per-benchmark z-scores mapped to 50 ± 15, averaged per category then across categories; minimum five results in two categories.

Freshness and failure

The pipeline runs every morning. If a source cannot be fetched, its previous values are kept and flagged as stale on the sources list rather than silently disappearing. Every day’s snapshot is versioned, which is what powers the trend lines and the daily digest.