We're building new features right now. The site is being worked on live, so things may move around or look unfinished. Nothing is broken — thanks for your patience.

Short Work Tools
Sign In

AGI Benchmarks Explained

A benchmark score is a number over a fixed set of questions, and almost everything that decides what it means is in the design rather than the number. The table is the designs.

Benchmark Built by, and when Size What it tests Human baseline, and how it was obtained
MMLU Hendrycks et al., arXiv:2009.03300, 2020 15,908 test questions, 57 subjects Multiple-choice recall and reasoning across school and professional subjects Estimated expert level around 90 per cent; random guessing scores 25 per cent
GPQA Rein et al., arXiv:2311.12022, 2023 448 questions; the Diamond subset is 198 Graduate-level biology, physics and chemistry, written to be hard to look up PhD-level experts in their own field about 65 per cent; skilled non-experts with unrestricted web access about 34 per cent
SWE-bench Jimenez et al., arXiv:2310.06770, 2023 2,294 real GitHub issues; the Verified subset is 500 Produce a patch that makes a real repository's own tests pass None published; the pass criterion is the repository's test suite, not a person
ARC-AGI-1 François Chollet, arXiv:1911.01547, 2019 400 public training, 400 public evaluation, 100 private tasks Infer a transformation rule from two or three examples and apply it to a new grid Built so that people solve the tasks without instruction; the prize target was set at 85 per cent
ARC-AGI-2 ARC Prize, March 2025 A new task set, with a semi-private and a private split The same idea, with the tasks that 2024 systems had learned to brute-force removed Every task solved by at least two people in two attempts or fewer during live testing
Humanity's Last Exam CAIS and Scale AI, January 2025 2,500 questions, over 100 subjects Expert-written questions at the edge of published knowledge Written and reviewed by domain experts; no aggregate human score is claimed
Long-task horizon METR, March 2025 A suite of software and research tasks The length of task, measured in human expert minutes, a system finishes half the time The unit IS the human: a task's difficulty is how long an expert takes

Read the last column first. It is the only one that tells you what a score is being compared against, and it is the column most often missing when a number is quoted. GPQA is the clearest case: its Diamond subset was built so that PhD-level experts in the relevant field score around 65 per cent while skilled non-experts with a search engine and no time limit score around 34 per cent. That gap is the whole design — it is what makes the benchmark resistant to a system that is merely very good at retrieval. SWE-bench has no human column at all, because its pass criterion is a repository's own test suite, which makes it the most objective of the seven and also the one most sensitive to a flaky test. ARC-AGI's baseline is a claim about solvability rather than an average: the tasks were validated by having people solve them, not by measuring how many people fail.

Why a benchmark dies

Every benchmark on that list is a response to the previous one running out of room. MMLU was published in 2020 with 15,908 questions and a 25 per cent guessing floor, and it discriminated well for about three years. Once frontier systems were scoring in the high eighties, the remaining headroom was smaller than the benchmark's own label noise, and a two-point difference between two systems stopped meaning anything. GPQA was written to have headroom in a place retrieval could not reach. Humanity's Last Exam was written when GPQA started to fill up.

A benchmark dies in one of four ways, and it is worth being able to name which:

  • Saturation. Scores approach the ceiling and the remaining gap is mostly errors in the test itself. A benchmark with known label noise cannot discriminate below that noise floor, however many decimal places a leaderboard prints.
  • Contamination. The questions, or close paraphrases, end up in training data. This is why ARC-AGI and later benchmarks keep a private split that is never published, and why a public-set score and a private-set score are different claims.
  • Shortcutting. The intended skill is not the cheapest route to the answer. ARC-AGI-2 exists because search over program space had become an effective route through ARC-AGI-1 without the reasoning the benchmark was written to test.
  • Drift in the baseline. The human number was measured once, under conditions nobody repeats. GPQA's non-expert baseline allowed unrestricted web access and no time limit; a casual re-run without those conditions is not comparable.

Three questions, in the order that decides how much a number is worth. The split comes first because a score on a public set cannot separate a system that solved the task from one that had seen it, and no amount of care further down the chain repairs that. The number of attempts comes second: a best-of-many score is a different measurement from a single attempt, and both are legitimate as long as the label says which. Compute comes last and is the one most often left out — the same system scored 75.7 per cent and 87.5 per cent on ARC-AGI-1's semi-private set in December 2024, differing only in how much compute it was allowed, which is why ARC Prize reported both numbers rather than the higher one.

The arithmetic of a small benchmark, worked through

Benchmarks are smaller than the precision people quote them with, and the arithmetic takes ten seconds.

SWE-bench Verified has 500 issues. One issue is therefore worth:

1 ÷ 500 = 0.002, which is 0.2 percentage points

A system reported at 62.4 per cent solved 312 issues. One reported at 64.0 per cent solved 320. The gap between the two headlines is 1.6 points, which is eight issues. On a benchmark whose pass criterion is a real repository's own test suite, eight issues is within the range a single environment change or a flaky test can move.

GPQA Diamond is smaller again, at 198 questions:

1 ÷ 198 = 0.00505, which is about 0.5 percentage points

So a system quoted at 70 per cent answered 139 of 198 correctly, and the PhD-expert baseline of about 65 per cent is 129. The difference between "matches experts" and "beats experts" on that benchmark is ten questions.

If X then Y

  • If the benchmark has fewer than about 500 items, treat differences under two percentage points as noise unless a confidence interval is published.
  • If the score is on a public split, treat it as an upper bound, not a measurement.
  • If the compute budget is not stated, the score is not reproducible even in principle.
  • If the human baseline was measured under conditions the write-up does not describe, the comparison is decorative.

What a rising score does not tell you

It does not tell you the capability generalises. A benchmark is a fixed task set, which makes it narrow by construction. That is a design property, not a criticism; it becomes a mistake only when a narrow score is quoted as evidence about general capability. The framing that keeps the two apart is in the sibling page, what is AGI.

It does not tell you anything about length. Every benchmark in the table except the last scores a question, and questions are short. METR's March 2025 work measures the other axis — how long a task a system finishes half the time, in human expert minutes — precisely because a system can be excellent at hard short questions and unable to hold a long one together.

It does not tell you what it costs. Two of the numbers most often quoted from December 2024 are the same system at two compute budgets. A score with no cost attached is half a result.

It does not survive the benchmark being retired. MMLU scores from 2021 and 2026 are not comparable, because the population of systems, the prompting and the evaluation harnesses all changed underneath the questions. A score is a measurement of a system, a harness and a moment together.

Key Takeaways

  • A benchmark's design outlives its leaderboard: who built it, how many items, and how the human baseline was obtained are the facts worth learning.
  • GPQA Diamond is 198 questions; PhD-level experts in their own field score about 65 per cent and skilled non-experts with unrestricted web access about 34 per cent (arXiv:2311.12022).
  • SWE-bench Verified is 500 issues, so one issue is 0.2 points and a 1.6-point gap is eight issues.
  • MMLU is 15,908 questions across 57 subjects with a 25 per cent guessing floor, and it saturated — which is what benchmarks do (arXiv:2009.03300).
  • ARC-AGI-2 exists because search over program space had become a route through ARC-AGI-1; in live testing every ARC-AGI-2 task was solved by at least two people in two attempts or fewer.
  • The same system scored 75.7 and 87.5 per cent on ARC-AGI-1's semi-private set in December 2024, differing only in compute. A score without a compute budget is half a result.
  • Three questions settle how much a quoted number is worth: which split, how many attempts, and at what compute.
Back to the guide

Common questions

What does ARC-AGI measure that MMLU does not?

MMLU measures recall and reasoning over subjects that are written down somewhere, so a system that has read enough can score well. ARC-AGI gives two or three worked examples of a transformation and asks for it to be applied to a new grid, with the tasks built so that the rule is not something that could have been memorised. Chollet s 2019 paper frames that difference as skill versus skill-acquisition efficiency.

Why is there a private split?

Because a public question can end up in training data, by deliberate inclusion or by being scraped from a page that discusses it. A score on a public set therefore cannot separate a system that solved the task from one that had seen the answer. A held-out split that is never published is the only way to make the two distinguishable, which is why ARC-AGI keeps one and reports against it.

Is SWE-bench Verified easier than SWE-bench?

It is cleaner rather than easier. The original set is 2,294 real GitHub issues, and some of them are unsolvable from the information given or have test suites that do not actually check the fix. The Verified subset is 500 issues that human annotators confirmed were well specified and correctly tested, so a failure on Verified is more likely to be the system s fault than the task s.

How much of a difference between two scores is real?

Divide one by the number of items to get what a single question is worth, then read the gap in questions rather than percentage points. On the 198-question GPQA Diamond a single question is about half a point; on 500-issue SWE-bench Verified it is 0.2 points. Without a published confidence interval, a gap of a handful of items is not a result.

Do benchmark scores predict whether a system is useful?

They predict something narrow and measurable, which is not the same thing. Every benchmark in the table except the last scores a short question, and usefulness usually depends on holding a long task together. That is the gap METR s March 2025 work measures directly, by reporting the task length in human expert minutes at which a system succeeds half the time.