Chunking and retrieval quality
The demo went well. Twenty questions, written by the team that wrote the documents, and the assistant got nineteen right. Then real customers arrived. They use different words, ask two things at once, paste order numbers, and follow up with "what about for UPI?". Accuracy drops to two in three, and every wrong answer traces back to the same place: the right text never made it into the prompt.
Retrieval is where RAG systems are won or lost. This chapter covers the four levers that move it: how you cut documents, how you search, how you rerank, and how you measure.
Smaller chunks or bigger chunks: which gives you more precise retrieval, and what do you lose by going small?
Chunk size is a trade#
Small chunks are precise. A 100 token chunk about the late delivery policy matches a late delivery question closely, because it is about nothing else. But a small chunk can cut an answer in half. Big chunks keep answers whole, but a 1,500 token chunk is about six things, so its vector is a blur of all six and it matches nothing strongly.
Try it on a real policy. The answer needs two sentences.
Question: Do I get a refund if my order is more than 45 minutes late, and how is it paid?
C1Refunds and cancellations. You can cancel an order free of charge until the restaurant accepts it. After the restaurant accepts, an order can only be cancelled if it has not been picked up, and a fee of 30 percent applies. C2If an item is missing or wrong, report it within 2 hours of delivery from the order page and attach a photo of what you received. Our team reviews most reports within one hour. Approved reports are refunded for the C3affected items only, not the delivery fee. Delivery times shown at checkout are estimates. Orders delayed by more than 45 minutes past the promised time qualify for compensation. Compensation is a full refund of the order value, paid to the C4original payment method within 5 to 7 working days. Cash on delivery orders receive wallet credit instead. Delays caused by heavy rain or road closures are excluded. Wallet credit expires after 90 days. Refunds for UPI payments usually reach your C5bank within 48 hours of approval. If a refund has not arrived after 7 working days, contact support with your order ID.
Shaded: the chunk retrieval would return. Bold: the answer. A wavy line marks answer words the model never sees.
At 40 words with no overlap, the cut lands between the two sentences and the model gets half an answer. Ten words of overlap rescues it. Push the size past 100 and the answer fits again, but now it shares a chunk with cancellation fees and wallet expiry.
Starting rules that hold up in practice:
- Cut on structure, not on a word count. Headings, then paragraphs, then sentences. A chunk that ends mid sentence is a bug.
- Start around 200 to 500 tokens and adjust using measurements, not instinct.
- Overlap by 10 to 20 percent so an answer near a boundary survives.
- Give every chunk its context. Prefix it with the document title and section path: "Refund policy > Late deliveries > ...". A chunk that says "this applies within 7 days" is useless on its own. With its heading attached, it is findable and readable.
Hybrid search#
The embeddings chapter showed that vectors understand meaning and miss exact identifiers. Classic keyword search, usually an algorithm called BM25, has the opposite strengths: it nails "E1043" and "order 44821" and knows nothing about meaning. Run both and merge.
The two searches score on different scales, so you cannot just add their scores. The common merge, reciprocal rank fusion, ignores scores and uses positions. Each document gets 1 divided by (60 plus its rank) from each list, and the totals decide the new order.
A small worked example. The refund article is 1st in keyword search and 4th in vector search: 1/61 + 1/64, about 0.0320. An article that is 2nd in vector search and missing from keyword search entirely scores 1/62, about 0.0161. Showing up near the top of both lists wins comfortably, which is exactly the behaviour you want.
Reranking#
Embedding search is fast because it compares vectors that were computed ahead of time. The question and the chunk never actually meet. A reranker (also called a cross encoder) reads the question and one chunk together and scores how well that chunk answers that question. It is much more accurate and much slower, so you only run it on a short list.
The standard shape: retrieve 50 to 100 candidates cheaply, rerank them, keep the top 3 to 5. It typically adds tens to a couple of hundred milliseconds. For most question answering products, it is the single biggest accuracy gain after fixing chunking.
Fix the question, not just the search#
In a conversation, the question is often not searchable on its own. "And for UPI?" means nothing to a vector index. Before searching, ask a small, fast model to rewrite the latest message into a standalone query using the conversation: "Refund timeline for UPI payments on late orders". For questions that bundle two asks, have it produce two queries and retrieve for both.
Measure, or you are guessing#
Build a test set: 50 to 200 real questions (from logs, not from the documentation team), each with the chunk or document that should answer it. Then measure retrieval on its own:
Recall at k: for what share of questions does the right chunk appear in the top k? If recall at 5 is 70%, the model is answering three questions in ten without the facts, and no prompt will fix that.
Mean reciprocal rank: how high the right chunk sits, on average. It rewards putting the answer first rather than fifth.
Change one thing (chunk size, overlap, hybrid on, reranker on), rerun, and keep the change only if the numbers move.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Smaller chunks | Precise matches, and fewer wasted tokens in the prompt. | Answers split across chunks, and chunks that make no sense alone. | FAQ style content with short, self contained answers. |
| Hybrid search | Catches exact identifiers and meaning, and fails less on odd queries. | Two indexes to keep in sync and a fusion step. | Almost always, and certainly when users paste ids, codes or names. |
| Reranker | The largest accuracy gain for the effort in most systems. | Added latency and cost per query. | When the right chunk is usually in the top 50 but not in the top 5. |
| Query rewriting | Follow up questions become searchable. | An extra model call before retrieval starts. | Any chat interface, where follow ups are most of the traffic. |
Anthropic published a technique in 2024 called contextual retrieval: before embedding each chunk, a model writes a sentence or two explaining where the chunk sits in its document, and that is embedded along with it. Combined with hybrid search and reranking, they reported large reductions in failed retrievals. It is the "give every chunk its context" rule above, automated.
Checkpoint#
1. Why can a very large chunk lose to a smaller, less relevant chunk in vector search?
2. Why merge keyword and vector results with reciprocal rank fusion rather than adding their scores?
3. Recall at 50 is 95% but recall at 5 is 60%. What is the most targeted fix?
- Chunk on structure, 200 to 500 tokens, 10 to 20% overlap, with the heading path attached.
- Hybrid search, fused by rank: 1 / (60 + rank) from each list.
- Retrieve 50, rerank, keep 5.
- Recall at k is the number to watch. If the right chunk is not there, nothing downstream can fix it.
Retrieval quality decides whether RAG works. I would chunk on document structure rather than a fixed count, start around 200 to 500 tokens with 10 to 20 percent overlap, and prefix each chunk with its title and section so it stands alone. I would run keyword search and vector search together, because BM25 catches identifiers and embeddings catch meaning, and merge them with reciprocal rank fusion since their scores are not comparable. Then retrieve about 50, rerank with a cross encoder, and keep the top 5. In chat, I would rewrite follow ups into standalone queries first. And I would measure recall at k on a set of real questions, changing one thing at a time.