RAG from scratch
A bank's assistant tells a customer that prepaying their home loan carries no charge. That was true when the model was trained. The policy changed in March. The customer prepays, gets charged, and screenshots the conversation. The model did its job perfectly: it produced the most likely answer from what it had read. It had simply never read March.
Retrieval augmented generation, RAG, is the fix, and it is a short idea: before the model answers, look up the relevant parts of your own documents and put them in the prompt. The model stops answering from memory and starts answering from what you handed it.
Why not fine-tune the model on all your help articles once, instead of looking articles up on every single question? Write down one reason each way.
Two pipelines#
A RAG system is really two systems that meet in a vector index. One runs ahead of time and prepares your documents. The other runs on every question.
Indexing (top row). Load the documents and strip the junk: navigation menus, cookie banners, page footers repeated on every page. Cut them into chunks of a few hundred tokens, because you want to retrieve the paragraph that answers the question, not a 40 page PDF. Embed each chunk, and store the vector with metadata: the source document, its title, the date, and who is allowed to see it.
Querying (bottom row). Embed the question with the same model, find the nearest chunks, put the best few into the prompt, and ask the model to answer from them.
Step through one real question.
The prompt that answers from sources#
The retrieval gets the attention, but this prompt does half the work.
Answer the customer's question using only the sources below.
Each source has an id. After each claim, cite the id like [2].
If the sources do not contain the answer, say "I don't have that
information" and suggest contacting support. Do not use outside
knowledge about banking.
<sources>
<source id="1" title="Home loan charges, updated 1 March">...</source>
<source id="2" title="Prepayment FAQ">...</source>
</sources>
<question>Is there a charge if I prepay my home loan?</question>"Only the sources below" and "say you do not know" turn the model from an oracle into a reader. Citations make every claim checkable, by the customer and by your evals.
Why not fine-tune instead?#
Fine-tuning means training the model further on your own examples. It is very good at teaching behaviour: a format, a tone, a narrow task done the same way every time. It is bad at teaching facts you need to be exactly right, for four reasons:
- facts change, and retraining every time a policy changes is slow and expensive
- a fine-tuned model still produces likely text, so it can blend old and new facts
- it cannot cite where an answer came from
- it cannot know which customer is allowed to see which document
Retrieval handles all four. The working rule: retrieve knowledge, fine-tune behaviour.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| RAG | Answers from current documents, with citations, filtered per user. | Two pipelines to build and run, and answers are only as good as retrieval. | Any product that answers questions about your own changing information. |
| Paste everything into a long context | No retrieval to build. Works for a handful of documents. | Slow and expensive on every call, and facts in the middle get used poorly. | A few documents that fit comfortably, or a prototype. |
| Fine-tuning | Consistent style and format, and shorter prompts. | Expensive to update, no citations, and facts still blur. | Teaching a behaviour, alongside RAG for the facts. |
Where RAG goes wrong#
When a RAG answer is wrong, the first question is always: did retrieval fetch the right text? Most failures are retrieval failures wearing an answer's clothes.
"The answer was wrong" tells you nothing about where to look. Keep a set of real questions with the document that should answer each. Measure retrieval on its own (was the right chunk in the top k?) and generation on its own (given the right chunk, was the answer correct and supported by it?). The evals chapter builds this.
Checkpoint#
1. Your refund policy changes monthly. Why is RAG a better fit than fine-tuning for answering refund questions?
2. The assistant answers wrongly. Logs show the correct chunk was in the prompt. Which stage do you fix?
3. Where should "customer A must not see customer B's documents" be enforced?
- Retrieve knowledge, fine-tune behaviour.
- Two pipelines: indexing when documents change, querying on every question.
- Permissions are a retrieval filter, never a prompt instruction.
- Debug retrieval first. Was the right chunk in the prompt?
RAG means looking up the relevant parts of your own documents and putting them in the prompt, so the model answers from current sources instead of memory. It is two pipelines. Indexing cleans documents, cuts them into chunks of a few hundred tokens, embeds them and stores them with metadata like source, date and permissions. Querying embeds the question with the same model, filters by permission and version, takes the top few chunks, and asks the model to answer only from them with citations, and to say it does not know otherwise. I would use RAG for facts and fine-tuning only for behaviour. When it fails, I check retrieval first, and I measure retrieval and generation separately.