Reasoning effort explained: why the same model gets two different scores
Low, medium, high, xhigh, max: what the reasoning-effort setting does to quality, speed and cost, and why a benchmark score without it is half a number.
Modern models think before they answer, and most let you choose how much. That one knob changes the score, the latency and the bill more than the choice between two neighbouring models does. Yet most leaderboards print a single number per model. This page explains what the knob does, and why BenchLeader treats each setting as its own row.
What the setting is
Reasoning models generate hidden tokens before the visible answer. The effort setting (OpenAI calls the levels low, medium, high and xhigh; Anthropic exposes a budget and named levels up to max; others use a thinking toggle) caps or encourages that hidden work. More effort means more reasoning tokens: better answers on hard problems, longer waits, higher cost, since reasoning tokens are billed as output.
What it does to the numbers
Open any frontier model's page and look at the configuration table. As of 9 Sept 2026, GPT-6 Astra (max) tops the index; the same model at its lowest effort usually sits several points lower and answers in a fraction of the time. On agentic and maths benchmarks the spread between low and max can exceed ten points. On knowledge benchmarks it is often small: recall does not improve with thinking.
Speed moves the other way. Output tokens per second barely change with effort, but time to first answer explodes, because the visible answer only starts once the thinking ends. A configuration that tops SWE-bench at max effort can take minutes before it types anything.
Why we split rows
A leaderboard that shows one number per model has to choose: the best effort (flattering, slow, expensive) or the default (whatever the API does when you do not ask). Neither is what you will deploy without knowing it. So on BenchLeader a row is a model at one effort, scored only on benchmarks that were run at that effort, with its own speed and its own latency. Prices are shared, because the list price does not depend on effort; the bill does, through reasoning tokens, which is why the per-task costs are floors.
Tables show each model's best-scoring effort by default. One click shows them all, and the compare page can put a model's own efforts side by side, which is often the most useful comparison on the site.
"Default" means the publisher did not say
Some boards do not record the effort they ran. Those results go to a configuration called default. It is not a claim that the run used the API default; it is a statement that we do not know. Where a board does record effort (Artificial Analysis, Vals AI, Terminal-Bench, MathArena, LMArena for most models), the row is precise.
How to use this
- Reading a score in a launch post? Ask which effort. If the post does not say, assume the highest.
- Deploying interactively? Pick the effort by time to first answer first, then check what quality you gave up on the model page.
- Running batch or agentic jobs? Higher effort is often worth it; check the per-task cost boards with caching turned on.