GraphRAG
Ask a vector RAG system "what are the main complaints customers raised this quarter?" and it retrieves the five chunks most similar to that sentence, which are probably five complaints. Not the main themes across forty thousand tickets. The question is about the whole collection, and similarity search can only ever return a few pieces of it.
GraphRAG adds structure that vectors do not have: the entities in your documents (people, products, places, policies), the relationships between them, and summaries of clusters of related things. Microsoft Research published the best known version in 2024, and the idea has spread into many tools since. It is powerful for a specific kind of question, and expensive enough that you should know exactly when you need it.
Why can't a vector search over chunks answer 'what are the five biggest themes across all our support tickets this quarter?', no matter how good the embeddings are?
How the index is built#
Two kinds of question, two kinds of search#
That answers the question at the top. Vector search returns the few chunks most similar to the question. "Biggest themes" is not similar to any chunk. It is a property of all of them, which only a precomputed summary of the whole collection can answer.
The cost, honestly#
Building the index means a model call for every chunk to extract entities, plus summaries for every community, plus rebuilding parts of it as documents change. For a large corpus that can be many times the cost of embedding it. Lighter variants have appeared since 2024 (LightRAG is a well known one) that cut the extraction and update cost, and some vector databases and graph databases now offer graph retrieval built in.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Vector RAG | Cheap to build and update, fast, great for specific questions. | Cannot answer questions about the whole collection, misses connections between dissimilar text. | Most question answering. The default. |
| GraphRAG | Answers whole-collection questions and connects facts across documents. | Expensive indexing, slower global queries, more moving parts to keep fresh. | Analysis over large corpora: research, investigations, ticket and feedback themes. |
| Hybrid: vector first, graph for some | Cheap path for specific questions, graph only where it earns its cost. | Two indexes and a router. | Products that mix lookup questions with analysis questions. |
GraphRAG is a tool for a question type, not an upgrade to RAG in general. Build vector RAG first, log which questions fail, and add a graph only if many failures are whole-collection or relationship questions. Otherwise you have built an expensive index that makes FAQ answers no better.
Checkpoint#
1. Why does vector RAG fail on "what are the main themes across all tickets"?
2. Where does most of the GraphRAG indexing cost come from?
3. Which search mode answers "which suppliers are linked to the delayed shipments"?
- GraphRAG: extract entities and relations, cluster into communities, summarise them.
- Local search walks from an entity. Global search maps over community summaries.
- It answers whole-collection questions vector RAG cannot.
- Indexing is expensive. Add it only for question types that need it.
Vector RAG returns the few chunks most similar to a question, so it cannot answer questions about a whole collection, like the main themes across all tickets. GraphRAG builds a knowledge graph: a model extracts entities and relationships from every chunk, they are merged, clustered into communities, and each community is summarised. Local search starts from an entity and walks its relationships, which connects facts across documents. Global search asks the question against every community summary and merges the partial answers. The indexing is expensive, a model call per chunk plus summaries, so I would start with vector RAG and add a graph only for analysis and relationship questions that keep failing.