Evals
A customer complains that the assistant is too chatty. Someone adds "be concise" to the prompt, checks three conversations, and ships. A week later three different complaints arrive: refund answers now skip the timeline, citations have vanished, and Hindi questions get English replies. One sentence changed behaviour in four places, and nobody could see it, because there was nothing to look with.
An eval is how you look. It is the closest thing AI engineering has to a test suite, and the teams that ship good LLM products are, nearly without exception, the teams that built evals early.
You try a new prompt on 20 examples. The old prompt passed 16, the new one passes 18. Is the new prompt better? How confident are you?
What an eval is#
Three parts:
- a dataset: inputs, and where possible the expected output or the facts a good answer must contain
- a grader: a way to score each output
- a number you track over time, for every prompt, model and retrieval change
It is a unit test with one big difference. Model output varies, so an eval does not pass or fail. It gives a rate, and rates come with uncertainty.
Where the examples come from#
Not from your imagination. The documents your team wrote produce questions your team would ask. Take them from real traffic instead.
- Logs. Sample real questions, especially the common ones.
- Every bug. When a customer reports a bad answer, it becomes an eval case before it is fixed. Over a year this becomes your most valuable dataset.
- Edge cases. Empty input, two questions in one, the wrong language, an order id that does not exist.
- Attacks. Inputs that try to override instructions or extract data, so a safety regression shows up as a number.
Fifty to a hundred good examples beat a thousand generated ones. Start there.
Three ways to grade#
Code. Is the JSON valid? Is the category one of the allowed six? Does the answer cite at least one source? Does it contain the order id? Code checks are free, fast and exact. Use them for everything they can check.
A model as the judge. For "is this answer correct and supported by the sources?", give a strong model the question, the sources, the answer and a clear rubric, and ask for a score with a reason. This scales well and has known biases: judges tend to favour longer answers, favour whichever answer is shown first in a side by side comparison, and favour text written by their own model family. Ask yes or no questions about specific criteria rather than a vague 1 to 10, swap positions when comparing two answers, and check the judge against people.
People. The gold standard and the most expensive. Use human labels on a sample to calibrate the model judge, and for decisions that really matter.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Code checks | Free, instant, exact and never drifts. | Can only check form and simple facts, not whether the answer is right. | Always, for everything code can verify. |
| Model as judge | Grades meaning at scale, in minutes, for little money. | Biased toward long answers and its own style, and wrong in ways that look confident. | Correctness, faithfulness to sources, tone. Calibrated against people. |
| Human review | Catches what nothing else catches, and defines what good means. | Slow and expensive, and reviewers disagree with each other. | Building the rubric, calibrating judges, high stakes launches. |
How many examples is enough?#
Back to the question at the top. Every pass rate measured on a sample has a margin of error, and on small samples it is enormous.
On 20 examples, 16 against 18 is noise: the ranges overlap almost completely. Four times as many examples halves the margin. When two prompts are close, run both on the same examples and count where they disagree, which is more sensitive than comparing two pass rates.
Before shipping and after#
Offline evals run before a change ships, ideally in CI, the same way tests do. A prompt or model change that drops the score does not merge.
Online evals watch production: a model judge scoring a sample of real conversations every day, thumbs up and down from users, and how often users rephrase or give up. Production is where you find the cases your dataset is missing, and they flow back into it.
You cannot evaluate what you did not record#
For anything with more than one step, keep a trace of every request: the full prompt, the retrieved chunks with their scores, every tool call with arguments and results, the output, the tokens and the time each step took. When an agent fails on step seven, the trace is the only way to see that step three returned an empty result. Several open source and commercial tools exist for this. The important decision is to record traces from day one, not which tool shows them.
An agent that says "I fixed the failing test" has written a sentence. Grade outcomes, not claims: run the test, query the booking table, check the refund ledger. Agents grading themselves report success far more often than they achieve it.
Checkpoint#
1. Old prompt 16 of 20, new prompt 18 of 20. What can you conclude?
2. Your model judge rates long answers higher, even when they are wrong. What helps most?
3. Where do the most valuable eval examples usually come from?
- An eval is a dataset, a grader and a number tracked over time.
- 20 examples at 80% is plus or minus about 17 points. 4x the examples halves the margin.
- Code checks first, model judges for meaning, people to calibrate.
- Every reported failure becomes an eval case. Grade outcomes, not the agent's claims.
An eval is a dataset of real inputs, a grader, and a pass rate I track for every prompt, model and retrieval change. The examples come from production logs and every reported failure, not from the team's imagination, and 50 to 100 good ones are a fine start. I grade with code wherever possible, use a model as a judge for meaning with a specific rubric while watching for its bias toward long answers, and calibrate it against human labels. Small samples are noisy, 20 examples at 80 percent is about plus or minus 17 points, so I compare changes on the same examples. Offline evals gate changes in CI, online evals sample production, and I record full traces from day one so failures can actually be debugged.