AI benchmark: what it is, explained simply

In plain words

When a company says its new model is better, how do we know? We give every model the same exam. That exam is a benchmark: a fixed set of questions or tasks, with a fixed way of marking the answers.

Some benchmarks test school and university knowledge with multiple-choice questions. Others ask a model to fix real bugs in code, solve maths problems, or complete tasks on a website. Each gives a score, often a percentage, and the scores go on public leaderboards.

Benchmarks have limits. Once models score near 100%, a test can no longer tell them apart, so researchers write harder ones. And if test questions leak onto the internet and end up in the training data, a model may remember answers instead of working them out. A high score is a clue, not proof that a model will be good at your task.

Everyday example

A common board exam

Students from hundreds of schools sit the same board exam on the same day, marked by the same scheme. That is what makes a 92 in one school comparable to a 92 in another. But everyone also knows the limits: a student who memorised past papers can score well without deep understanding, and the marks say little about whether someone will be good at a job.

Where the example breaks: Board exams are kept secret until exam day. Many AI benchmarks are public for years, so the questions can seep into what models learn from. Models also do not get nervous or tired.

How it actually works

A benchmark is a dataset plus a scoring rule plus a protocol for running it.

Fixed inputs, fixed scoring

Each item has a known correct answer or a checker (a test suite for code, a grader for maths). The model's outputs are scored automatically, and results are averaged into a single number.

The protocol matters

Scores change with how a test is run: how many example questions are shown in the prompt, whether the model may think step by step, how many tries it gets. Fair comparisons use the same settings for every model.

Saturation

Benchmarks get used up. MMLU (2020), with questions across 57 subjects, was hard for models of its time. By 2025, top models scored over 90%, and new tests such as Humanity's Last Exam were built to be harder.

Contamination

If test items appear in training data, scores rise without real skill. A 2024 study wrote fresh look-alike maths questions and found some model families scored up to 8% lower on them, a sign of memorisation.

Beyond the leaderboard

Because of these problems, labs also use private held-out tests, human preference comparisons and task-specific evaluations (often called evals). Good practice is to test a model on examples from your own work.

Classic early benchmark

ImageNet: the 2012 contest where the AlexNet image model cut the error rate sharply

MMLU

Multiple-choice questions on 57 subjects, from maths to law. Published September 2020 by Hendrycks and colleagues

Humanity's Last Exam

2,500 expert-written questions, released January 2025 because popular tests had saturated

Questions people ask

What is a benchmark in AI?

A standard test used to measure and compare AI models. It is a fixed set of questions or tasks with a fixed scoring method, so that every model is judged the same way. Results are usually a percentage score and often appear on public leaderboards.

Can AI benchmarks be trusted?

Partly. They are useful for comparing models under the same conditions, but scores can be inflated if test questions leaked into training data, and a test may not resemble real work. Treat a score as one piece of evidence and try models on your own tasks.

What is the difference between a benchmark and an eval?

The words overlap. A benchmark usually means a public, standard test that many models are compared on. An eval (evaluation) is any structured test of a model, including private ones a company builds for its own product or for safety checks.

Why do AI companies keep making new benchmarks?

Because old ones stop being informative. When most top models score above 90%, small differences are mostly noise. New benchmarks add harder questions or more realistic tasks, such as fixing real software bugs, so progress can still be measured.

What is benchmark contamination?

It is when test questions, or their answers, end up in the data a model was trained on. The model can then recognise items instead of solving them, which inflates its score. Researchers detect it by writing fresh, similar questions and checking whether scores drop.