Query transformation

The basic RAG pipeline searches with exactly what the user typed. That works in a demo, where questions are complete sentences written by the people who wrote the documents. Real users type "and if I paid by UPI?", "refund??", or a question that bundles three things together. The search engine can only find what the query describes, so when the query is poor, retrieval fails before it starts.

Query transformation means using a model to turn what the user typed into one or more better searches before retrieval runs. It is the cheapest, highest payoff upgrade in this whole section, and most production RAG systems do some version of it.

Five techniques, one question#

A customer asked about a late order refund, got an answer, and then followed up. Two documents in the help centre are needed to answer. Try each technique.

Rewrite. Turn the latest message into a standalone question using the conversation so far. One small model call, and it fixes follow ups, pronouns ("it", "that one") and typos. If you add only one technique, add this one.

Multi-query. Ask for three to five differently worded searches, run them in parallel, and merge the results with reciprocal rank fusion (from the retrieval quality chapter). It catches vocabulary you did not anticipate, at the cost of several searches per question.

HyDE, hypothetical document embeddings, from a 2022 paper. The model writes a plausible answer and you search with that, because an answer looks far more like your documents than a short question does. Strong on vague, short questions. Weak when the invented answer contains a wrong detail that drags in the wrong documents, so never show the fake answer to anyone.

Decomposition. Split a compound or multi-hop question into sub-questions, retrieve for each, then answer with everything. "Compare the refund policies for late and cancelled orders" becomes two searches.

Step back. For very specific questions, also search a more general version ("what is the refund policy?") so the model gets the principle as well as the detail.

What the code looks like#

A rewrite step is a single small model call with structured output. Here it is as a function you put in front of any retriever.

from anthropic import Anthropic
 
client = Anthropic()
 
REWRITE = """Rewrite the customer's last message as one standalone search
query for a help centre. Use the conversation for context. Return only
the query, no explanation."""
 
def rewrite_query(history: list[dict], last_message: str) -> str:
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in history)
    response = client.messages.create(
        model="claude-haiku-5-5",  # a small, fast model is enough here
        max_tokens=256,
        system=REWRITE,
        messages=[{"role": "user", "content": f"{convo}\nuser: {last_message}"}],
    )
    return next(b.text for b in response.content if b.type == "text").strip()

The pattern is the same with any provider or framework: a cheap model, a narrow instruction, and the original message kept as a fallback if anything goes wrong.

Routing the query#

Not every question should go to the same index. A rewrite step can also classify: is this about an order (use the orders tool, not the help centre), a policy (search the policy index), or small talk (search nothing)? Routing is the routing pattern applied to retrieval, and it often saves a search entirely.

It can also extract filters: "UPI refunds in the last month" becomes a search for "UPI refund" with a date filter, which vector similarity alone handles badly.

Checkpoint#