RAG vs fine-tuning vs long context
A fintech app wants a support assistant that answers from its 3,000 help articles, its fee schedule and each customer's own transactions. Someone in the first meeting always says "let's fine-tune a model on our docs". It is usually the wrong first move, and knowing why is one of the most common AI engineering interview questions.
There are three ways to get your data into an answer. Long context puts the data in the prompt, every time. Retrieval augmented generation (RAG) searches the data and puts only the relevant part in the prompt. Fine-tuning trains the model further so the change lives in its weights. They are not three strengths of the same thing. They change different parts of the system, and they are good at different jobs.
The fee schedule changes every quarter, and each answer must cite the article it came from. A teammate wants to fine-tune a model on the help centre. What goes wrong, and what would you build instead?
Knowledge or behaviour#
The question that sorts almost every case: are you trying to give the model knowledge (facts it does not have) or behaviour (a format, a tone, a skill, a way of reasoning it does not do reliably)?
- Knowledge belongs in the prompt. The model reads it fresh on every request, so it is current, it can be cited, it can be filtered per user, and deleting it is deleting a row. Long context and RAG are both ways to put it there, and the choice between them is mostly about size and cost.
- Behaviour is what fine-tuning is for. Training on a few thousand examples of "input, the output we want" teaches a model to always answer in your JSON schema, write like your support team, classify tickets into your 40 categories, or do a narrow task that a smaller, cheaper model then handles as well as a large one.
Fine-tuning on documents to teach facts is the classic mistake. The model learns the style of your docs far more reliably than their contents, it cannot tell you which document a claim came from, and when the fee schedule changes next quarter you retrain. A 2024 Google study went further: fine-tuning a model on facts it did not already know made it hallucinate more, because it learned to answer confidently past the edge of what it knew.