Dossier entry
How to read an AI benchmark claim
The plain answer
A benchmark score shows how a system performed on one fixed set of tasks, under specific conditions, at one time. Before trusting it, check what the test measures, what the score is compared against, how the test was run, whether the test data could have leaked into training, and whether anyone independent got the same result.
"New model scores 92% on a reasoning benchmark" sounds decisive. It might be impressive. It might also mean much less than it seems. A benchmark is a measuring instrument, and every instrument has limits.
What is a benchmark?
A benchmark is a fixed collection of tasks, such as questions, problems, or coding exercises, with a scoring method. Running different systems on the same benchmark makes them comparable. That is genuinely useful. But a benchmark samples a skill; it does not capture all of it.
Six questions to ask about a score
1. What does the test actually measure?
Read the benchmark's own description. Multiple-choice knowledge questions, competition maths, and real software tasks test very different things. A name like "reasoning" or "intelligence" is a label, not a guarantee.
2. Compared with what?
A number alone means little. Look for:
- Earlier versions of the same system
- Competing systems run under the same conditions
- Human performance, and which humans (experts, crowd workers, students)
- Simple baselines, such as random guessing on a multiple-choice test
3. Under what conditions?
Small changes can move scores a lot: how many attempts were allowed, whether examples were included in the prompt, whether tools such as code execution were available, and how much computing time was spent per question. Fair comparisons use the same settings.
4. Could the answers have leaked?
If benchmark questions appear in the training data, a model may partly remember answers rather than solve problems. This is called contamination. Good reports describe how they checked for it; newer or private test sets reduce the risk.
5. Is the benchmark saturated?
When top systems all score near 100%, small differences are mostly noise, and further gains say little. Benchmark makers keep releasing harder tests for this reason.
6. Has anyone else reproduced it?
Results run by the developer are a start. Independent evaluations, or open benchmarks anyone can run, are much stronger evidence.
Red flags in benchmark headlines
| Headline pattern | Why to be careful |
|---|---|
| A single score with no comparison | No way to judge if it is good |
| "Beats humans" without saying which humans | Expert and average human baselines differ hugely |
| Many benchmarks reported, only the wins highlighted | Selective reporting |
| Results only in a chart, no paper or report | Method cannot be checked |
| Large jump on an old, public benchmark | Contamination is more likely |
Benchmarks are not the whole story
Real-world use involves messy inputs, long tasks, and costs of error that benchmarks rarely capture. Frameworks such as the NIST AI Risk Management Framework stress evaluating systems in the context where they will actually be used. See Demo vs deployment.
Limitations
This page describes general good practice; specific benchmarks have their own documentation and known weaknesses. Use the hype check to apply these questions to a specific claim.
Sources and further reading
We link to primary or official sources wherever we can. Guidance changes, so check the source before acting on anything important.