Evals in AI: what they are, explained simply

In plain words

Normal software is checked with tests: give it an input, check the output is exactly right. AI is harder, because the same question can get different answers, and "right" is often a matter of quality, not one exact answer.

Evals are how builders test AI anyway. You collect a set of real questions or tasks, decide what a good answer looks like for each, then run the AI on all of them and score the results. A score of 41 out of 50 tells you far more than "it seemed fine when I tried it".

The real value comes later. Change the prompt, switch to a new model or add a feature, then run the same evals again. If the score drops, you have caught a problem before your users do. A benchmark is one kind of eval, shared publicly to compare models; most evals are private, written for one app's own job.

Everyday example

A cook's tasting checklist

A cloud kitchen tests every new cook with the same ten dishes and a checklist: salt right, rice not sticky, served hot, portion correct. One taster marks each plate. When the kitchen switches to a new oil brand, the same ten dishes get marked again. If biryani scores drop, they know the change caused it, before the reviews come in.

Where the example breaks: A dish tasted twice tastes the same. An AI answer can vary between runs, so good evals run each task several times and look at how often it passes.

How it actually works

Anthropic's engineers describe an eval simply: give an AI system an input, then apply grading logic to its output to measure success.

Collect real tasks

Start small with real examples, especially past failures. Anthropic suggests 20 to 50 simple tasks drawn from real failures rather than waiting to build hundreds. OpenAI's guide says to log everything and mine the logs for good cases.

Choose how to grade

Code checks are fast and repeatable: exact match, does the JSON parse, does the code pass its tests. Human reviewers are the gold standard but slow and expensive. A model can grade too, using a rubric ("LLM as a judge"), but it needs checking against human grades first.

Run several trials

Because outputs vary, run each task more than once. Anthropic's example: an agent that succeeds 75% of the time on a task has only about a 42% chance of succeeding three times in a row, which matters for customer-facing tools.

Keep two kinds

Capability evals ask what the system can do and start with a low pass rate to aim for. Regression evals check it still does what it used to, and should stay near 100% to catch backsliding after changes.

OpenAI Evals

An open-source framework and registry of tests, released with GPT-4 on 14 March 2023

Grader types

Code-based, model-based (LLM as a judge) and human

Common mistake

"Vibe-based" evaluation: OpenAI's guide warns against judging by "it seems like it's working"

Tip

Anthropic's docs suggest grading with a different model from the one that wrote the answer

Evals measure what you choose to measure. A system can pass every eval and still fail at things nobody wrote a test for, so evals sit alongside red teaming and human review, not instead of them.

Questions people ask

What is the difference between evals and benchmarks?

A benchmark is a public, fixed test used to compare models across companies. Evals is the broader word for any test of an AI system, and usually means your own tests for your own task, like answering your customers' questions correctly.

What is LLM as a judge?

Using one AI model to grade another's answers against a written rubric. It is cheaper and faster than human graders and handles open-ended answers, but it can be inconsistent, so teams check it against human scores before trusting it.

How many examples do I need for an eval?

Fewer than people think to start. Anthropic suggests 20 to 50 real tasks, especially ones the system got wrong. Grow the set as you find new failures.

Why do AI evals give different scores each run?

Because AI output varies from run to run, especially at higher temperature settings. That is why good evals run tasks several times and report pass rates, not single results.