LLM benchmarks explained: what GPQA, HLE, SWE-bench, ARC-AGI and the rest actually measure
A plain-language guide to the benchmarks behind every AI model ranking: what each one tests, who publishes it, and the traps to watch for.
A benchmark score is only meaningful if you know what the benchmark asks. This is the field guide we wish existed when we started reconciling forty of them. Each entry links to the live leaderboard for that benchmark on BenchLeader, where every number links to its publisher.
Reasoning and knowledge
GPQA Diamond (Epoch AI runs it). 198 graduate-level science questions written so that a smart person with Google cannot answer them. Near saturation at the top, still useful in the middle of the table.
Humanity's Last Exam (Scale AI and CAIS). 2,500 expert-written questions across a hundred subjects, built to be the hardest closed-ended test that exists. Frontier scores are still below fifty percent. Watch the contamination warnings: models released after the questions were published may have seen them.
ARC-AGI (ARC Prize). Visual grid puzzles that require inferring a rule from a few examples. Cheap for humans, expensive for models; the board reports cost per task next to accuracy for that reason. Version 2 is the current one; version 3 is interactive.
SimpleBench and Kagi's benchmark are trick-question sets that humans find easy. They catch models that pattern-match instead of reading.
MMLU-Pro (Vals AI runs it) is the harder successor to MMLU: ten answer choices, more reasoning, less recall. SimpleQA Verified measures factual recall and, more usefully, refusal to make things up.
Coding
SWE-bench Verified: 500 real GitHub issues, human-validated, fixed end to end and checked by the repository's own tests. We show the minimal bash-only harness so the model is what varies. SWE-Bench Pro (Scale) uses longer tasks in private repositories that cannot have leaked into training data.
LiveCodeBench takes competitive-programming problems published after each model's cut-off. Aider Polyglot runs 225 Exercism exercises through a real coding agent and scores the edits. WebDev Arena has humans vote between two generated web apps.
Agents and tools
Terminal-Bench puts a model in a shell with a task and checks the result. τ²-bench simulates a customer-service conversation with tools and a user who does not cooperate. BFCL measures function-calling accuracy, including when no tool should be called. MCP Atlas tests tool use through the Model Context Protocol. OSWorld hands the model a real desktop.
Maths
FrontierMath (Epoch AI) is unpublished research-level mathematics; Tier 4 problems take professional mathematicians days. MathArena evaluates each new competition (AIME, HMMT, IMO) on the day it is released, before anyone can train on it. ProofBench grades full written proofs.
Human preference
LMArena shows two anonymous answers to a real user and asks which is better; thousands of votes become a Bradley-Terry rating. It measures what people like, which is not the same as what is correct, and it is the only large-scale signal of the former. Style control removes the advantage of longer, prettier answers.
Long context and multimodal
AA-LCR and Fiction.LiveBench test whether a model still reasons correctly at 100,000 tokens of context, which the advertised window does not tell you. MMMU and VISTA cover images.
The traps
- Self-reported scores. A launch post is not a measurement. BenchLeader only takes numbers from independent evaluators and arenas.
- Reasoning effort. The same model scores differently at different efforts. A score without an effort is half a number. Our rows are model-plus-effort for that reason.
- Contamination. Benchmarks with public questions decay as they leak into training data. Prefer boards that refresh (LiveBench, MathArena, LiveCodeBench) or stay private (FrontierMath, SWE-Bench Pro).
- Saturation. When the top ten are within two points of the ceiling, the benchmark has stopped discriminating. We halve the weight of saturated boards in the index.
- Scaffold effects. On agentic boards the harness can matter as much as the model. Where a board mixes scaffolds, we say so on the benchmark page.
The full list of what we track, with weights, is on the benchmarks page; how they combine is on the methodology page.