Evaluating RAG
"The bot gave a wrong answer" is the least useful bug report in AI engineering, because a RAG answer can be wrong in at least four different places: the search found the wrong text, the search missed the right text, the model ignored good text, or the model invented something the text never said. Each one is fixed somewhere different. An evaluation that produces a single accuracy number cannot tell you which.
This chapter splits RAG quality into the handful of measurements that point at the broken stage. It builds on the general evals chapter, which covers datasets, graders and sample sizes.
Your RAG assistant's answers got worse after you changed the chunk size. What two things would you measure separately to see whether retrieval or generation got worse?
Four numbers, two stages#
Context precision: of the chunks retrieved, how many were relevant? Low precision means noise in the prompt, which costs tokens and can distract the model.
Context recall: of the facts needed to answer, how many were retrieved? Low recall means the model never had a chance. This is usually the most important number in the whole system.
Faithfulness (also called groundedness): of the claims in the answer, how many are supported by the retrieved chunks? Low faithfulness means the model is inventing, which is the failure customers screenshot.
Answer relevance: does the answer actually address the question? A perfectly grounded answer about the wrong thing still fails.
Try four cases that each look like "wrong answer" from the outside.
Question: My order was 50 minutes late and I paid by UPI. Do I get a refund, and when?
- relevantLate delivery: over 45 minutes qualifies for a full refund
- relevantUPI refunds reach the bank within 48 hours of approval
- noiseWallet credit expires after 90 days
- supportedA 50 minute delay qualifies for a full refund.
- supportedIt goes back to your UPI account within about 48 hours.
These four were popularised by the open source Ragas library from 2023, and most evaluation tools now offer some version of them. They are usually computed by a model acting as a judge, which is why the evals chapter insists on checking the judge against human labels.
Building the dataset#
For each question in your eval set, store three things:
- the question, ideally from real traffic, including follow ups and messy phrasing
- the ids of the chunks or documents that contain the answer, so retrieval can be scored exactly with recall at k, without any judge
- a reference answer, or a short list of facts a correct answer must contain
Retrieval scored against labelled ids is cheap, exact and fast enough to run on every change. Generation needs a judge and costs more, so run it on a sample and before releases.
Labelling hundreds of questions by hand is slow. A common shortcut: have a model write questions from random chunks of your documents, so the source chunk is the label, then have a person review and discard the unrealistic ones, and mix in real questions from logs. Synthetic questions alone are too clean and too easy.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Recall at k on labelled chunks | Exact, cheap, no judge, runs on every change. | Needs labelled questions, and says nothing about the answer. | Every chunking, embedding, indexing or query transformation change. |
| Model judged faithfulness and relevance | Grades the answer itself, at scale. | Judge cost and bias, so it must be calibrated against people. | Prompt and model changes, and a daily sample of production traffic. |
| Human review | Catches what metrics miss, defines what good means. | Slow and expensive. | Building the rubric, calibrating judges, before big launches. |
Teams that ship good RAG products tend to put recall at k in continuous integration, so a change to chunking or embeddings fails the build like a broken test, and run the judged generation metrics nightly on a few hundred sampled production conversations.
Checkpoint#
1. Context recall is 50% and faithfulness is 100%. What is happening?
2. Which metric catches a confident, invented UPI refund timeline?
3. Why store the ids of the chunks that answer each eval question?
- Retrieval: context precision and recall. Generation: faithfulness and answer relevance.
- Recall is the ceiling. Missing facts cannot be answered.
- Label chunk ids so retrieval is scored exactly, on every change.
- Judges for generation, calibrated against people.
A wrong RAG answer can come from four places, so I measure them separately. For retrieval, context precision says how much of what was fetched was relevant and context recall says how much of what was needed was fetched, which is the ceiling for everything downstream. For generation, faithfulness checks that every claim is supported by the retrieved text and answer relevance checks that the answer addresses the question. I label the chunk ids for each eval question so recall at k is exact and runs on every change, use a calibrated model judge for the generation metrics on samples and before releases, and keep feeding real failures from production into the dataset.