Advanced indexing
Query transformation fixes what goes into the search. This chapter fixes what the search runs against. The basic pipeline cuts documents into chunks and stores one vector per chunk, and that one decision quietly limits everything: a chunk small enough to match precisely is often too small to answer from, and one vector cannot capture every way a chunk might be asked about.
The techniques here change what you index and how, so the right text is both findable and useful once found.
Small chunks match questions precisely but often lack the surrounding context needed to answer. Big chunks have context but match poorly. Can you think of a way to get both?
Give every chunk its context#
A chunk that says "This applies within 7 days of delivery" is nearly useless on its own. What applies? To which product? Contextual retrieval, published by Anthropic in 2024, fixes it at index time: before embedding each chunk, a model reads the whole document and writes a sentence or two placing the chunk in it. That context is embedded and keyword indexed along with the chunk.
Step through the indexing pipeline.
Anthropic reported that this cut failed retrievals roughly in half, and by about two thirds when combined with a reranker. The cost is one cheap model call per chunk at index time, paid once.
Search small, answer big#
The answer to the question at the top: index small, return big. Several techniques share that shape.
Better vectors#
Late interaction (ColBERT). A normal embedding squashes a whole chunk into one vector. ColBERT, from 2020 research at Stanford, keeps one vector per token and scores a match by comparing every query token with its best matching document token. It is much more precise for detailed questions and needs far more storage. It is often used as a reranker over a cheaper first pass.
Matryoshka embeddings. Many recent embedding models are trained so the first 256 or 512 numbers of a 1,536 number vector already work on their own. Store short vectors for a fast first search, then compare full length vectors on the shortlist.
Quantization. Store each number in 8 bits, or even 1 bit, instead of 32. Memory drops by 4 to 32 times, with a modest accuracy cost that a rescoring step on the shortlist mostly wins back.
Metadata is retrieval too#
Every chunk should carry structured fields: source, section, product, language, date, version, and who may see it. Filtering on them is exact, cheap and explainable, and it beats asking vectors to understand "only the 2026 policy". The query transformation step from the previous chapter can extract these filters from the question.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Contextual chunk headers | Chunks become findable by what they are about, not only by the words they contain. | One model call per chunk at index time. | Documents where chunks rely on surrounding context. Most of them. |
| Parent document retrieval | Precise matching with enough context to answer. | More tokens per answer, and a second store for parent sections. | Long structured documents: policies, manuals, contracts. |
| Late interaction | Fine grained matching on detailed questions. | Many vectors per chunk, so much more storage and compute. | As a reranker, or on modest corpora where precision matters most. |
| Quantized and shortened vectors | Several times less memory and faster search. | Some accuracy, mostly recovered by rescoring. | Large corpora where memory is the bill. |
Checkpoint#
1. What does contextual retrieval add to each chunk before indexing?
2. Parent document retrieval matches on small chunks and returns what?
3. Why is ColBERT often used as a reranker rather than the first search?
- Contextual retrieval: a model written context sentence per chunk. Reported to halve failed retrievals.
- Search small, answer big: parent document and sentence window retrieval.
- Late interaction keeps a vector per token. Precise, heavy, great for reranking.
- Metadata filters are exact. Use them before asking vectors.
Indexing decides what retrieval can find. I would give every chunk its context by having a small model write a sentence placing it in its document, which Anthropic reported halves failed retrievals. I would index small units for precise matching but return the parent section so the model can answer, and for messy vocabularies store extra vectors per chunk, such as generated questions. For precision, late interaction models like ColBERT work well as rerankers, and for scale, shorter and quantized vectors cut memory with rescoring to recover accuracy. And every chunk carries metadata like product, version, language and permissions, because exact filters beat asking embeddings to understand them.