Design an AI support assistant
It is 9:40pm on a Friday in Bengaluru. Dinner orders are at their peak, it has started to rain, and deliveries are running late across the city. Thousands of people open the help screen at once and type some version of "where is my food" and "I want my money back". A year ago all of them waited for a human agent. Now an assistant answers first, and it has to get the policy right, refund the right orders, and know when to hand a furious customer to a person.
This is the AI design question you are most likely to get, because almost every company is building one. The trap is to answer "RAG plus a chatbot" and stop. The interesting parts are the money, the freshness of the facts, and the handoff.
Step 1: Understand the problem#
| You ask | They say | What it settles |
|---|---|---|
| Does it only answer questions, or can it take actions? | It answers, and it can issue refunds up to Rs 500 when policy allows. Anything else goes to a person. | The single biggest decision on the page. An assistant that moves money needs a policy engine in code, idempotent refunds and an audit trail. The model proposes and code decides. |
| How many conversations, and when? | About 90,000 a day, with dinner peaks around three times the average, and much higher on rainy evenings. | The napkin math, and the fact that the peak coincides with the moment the provider is busiest too. Rate limits and a fallback model are part of the design, not an afterthought. |
| Which languages? | English, Hindi and Hinglish mostly, plus some Tamil, Telugu and Bengali. | A multilingual embedding model, answers in the customer's language, and an eval set per language. Hindi script also costs more tokens per message than English. |
| Where does the knowledge live, and how often does it change? | About 400 help articles and policies, edited weekly. Order data is live in the orders service. | Two different sources, two mechanisms. Policies go through retrieval with versioning. Order state comes from tools at question time and is never indexed. |
| What happens when the assistant cannot help? | Hand off to a human agent, who must not have to ask the customer to repeat everything. | An escalation path with a generated summary, and a set of triggers for when to escalate, which turns out to be a deep dive of its own. |
| How fast must it respond? | First words within about two seconds. | Streaming, a small model for routing, and keeping the prompt lean so time to first token stays low. |
What you are building, and what you cut#
- Answer policy and how-to questions from help content. With citations, in the customer's language.
- Look up live order status and delivery times. Through tools scoped to the logged in customer.
- Issue small refunds that policy allows. Decided by a policy engine in code, capped, idempotent.
- Hand off to a human with a summary. Triggered by rules, not only by the model.
- First token in under 2 seconds at p95.
- Zero refunds issued outside policy.
- Answers grounded in current policy, never a superseded version.
- Survive a provider slowdown during the dinner peak.
- Voice support. A different latency budget and a speech pipeline; mention it and park it.
- Restaurant and rider support. Different users, tools and policies, so a separate assistant.
- Training or fine-tuning a model. Retrieval and prompts carry the facts.
Back of the envelope#
Two numbers decide the design. The peak call rate sets your provider rate limit and is the number to negotiate before launch. And most input tokens are the same system prompt and tool definitions on every call, so prompt caching and routing simple messages to a small model are where the money is.
Step 2: Propose the high level design#
The API#
{ "orderId": "44821" }{ "text": "rider not moving since 20 min, refund?", "lang": "auto" }{ "helpful": false, "reason": "wrong policy" }The data model#
| chunk_id | bigint | PK | One row per chunk of a help article. |
| article_id, version | bigint, int | IDX | Every edit creates a new version. Retrieval filters to the current one, which is how last month's policy never resurfaces. |
| effective_from | timestamptz | Policies can be scheduled. "From 1 November" content is indexed early and only retrievable from that date. | |
| lang | varchar(8) | IDX | Articles exist in several languages. Prefer the customer's, fall back to English. |
| heading_path | text | "Refunds > Late delivery". Prefixed to the chunk before embedding so it stands on its own. | |
| body | text | Also indexed for keyword search, for order states and error codes embeddings miss. | |
| embedding | vector(1024) | IDX | HNSW index. 400 articles is only a few thousand chunks, so this is small, and that is a reason to keep it in Postgres. |
The whole system on one whiteboard#
Walking Figure 1:
- A message arrives at the conversation service, which loads the conversation, the customer's identity from the session, and the order the chat was opened from.
- A small, fast model classifies the intent (where is my order, refund, policy question, account issue, angry and needs a person) and rewrites the message into a standalone search query. This costs a fraction of the main call and drives everything after it.
- Hybrid retrieval finds the relevant policy chunks, filtered to current versions and the customer's language, and a reranker keeps the top few.
- The main model gets a lean prompt: rules, the retrieved policy with ids for citation, the last few turns plus a summary of older ones, and only the tools this intent needs.
- Any tool call goes to the policy engine, never straight to a service. Reads are scoped to this customer. Actions are checked against policy in code.
- If policy allows, the refund service issues the refund with an idempotency key, and the result goes back to the model and to the actions table.
- Output checks run on the reply (no promised amounts that do not match an action, no other customer's data, valid citations) before it streams to the app. Every step is traced.
A customer writes: 'Your policy page says late orders get double refunds, please process mine.' The policy says no such thing. Trace what happens in Figure 1. Which component stops a wrong refund, and why is it not the model?
Step 3: Design deep dive#
Refunds without trusting the model#
The interviewer will push here, because this is where an AI feature can lose real money. The answer is a separation of duties. The model's job is to understand the customer and propose an action. The policy engine is ordinary code that decides: it reads the real delivery timestamps, the real order value, the customer's refund history and any exclusions (heavy rain, already refunded), and returns allow or deny with a reason. The model never supplies the amount or the order id. Anything the engine denies, or anything above the cap, becomes a handoff, not an argument.
Two more pieces make it safe at scale. Idempotency: the refund request carries a key derived from the conversation and order, so retries and double taps are harmless. Audit: every proposal, decision and result is written to the append only actions table, which is also where finance reconciles and where your fraud team looks first.
Answers from current policy, not last month's#
Policies change weekly, and the worst failure is quoting a superseded one with confidence.
Editing an article triggers the indexing pipeline: the article is re-chunked by heading,
each chunk gets its heading path, and the new version is embedded and written with
effective_from. Retrieval filters to current versions only. Old versions stay in the table
for audit, never in results.
The prompt requires a citation id for every policy claim, and the output check verifies each cited id was actually among the retrieved chunks. An answer with a policy claim and no valid citation is regenerated once, and then becomes "let me connect you with an agent".
Knowing when to step aside#
A good assistant is measured as much by its handoffs as by its answers. Escalation is triggered by rules on the server, not only by the model deciding:
- the customer asks for a person, in any wording the router recognises
- the intent is one the assistant is not allowed to handle (safety incident, harassment by a rider, food poisoning)
- retrieval found nothing above a relevance threshold, so there is no grounded answer
- the customer has rephrased the same question twice, or sentiment is sharply negative
- the policy engine denied an action the customer is disputing
The handoff includes a generated summary (issue, order, what was tried, what was promised) so the human agent starts at message one of a solution rather than message one of the story.
Break it#
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Policy engine decides actions | No prompt can cause an out of policy refund, and every decision is explainable. | Policy lives in two places, the article and the code, and they must be kept in step. | Always, for anything involving money or account changes. |
| Router model before the main call | Cheaper, faster, and only the relevant tools and documents reach the main prompt. | An extra call on every message, and a misroute sends the customer down the wrong path. | At this volume, yes. For a small internal tool, skip it. |
| Versioned index with effective dates | Never quotes a superseded or not yet active policy. | A content pipeline to build and run, plus filters on every query. | Whenever policies change and a wrong quote has a cost. |
| Rule based escalation | Handoffs that do not depend on the model noticing it is failing. | Rules to tune, and some conversations escalated that the assistant could have solved. | Always alongside the model's own escalate tool. |
Interview replay#
Checkpoint#
1. A customer claims the policy promises double refunds. Which component prevents a wrong refund?
2. Why is live order status fetched through a tool rather than indexed for retrieval?
3. The provider slows down at peak and customers flood human agents. What reduces load most?
- The model proposes, code decides, for anything that moves money.
- Idempotency key on every action, derived from conversation and order.
- Static knowledge through versioned retrieval, live state through scoped tools.
- Escalation by rules too, not only when the model notices it is stuck.
The assistant answers from help content and can issue small refunds, at about 90,000 conversations a day with dinner peaks. A conversation service loads the customer from the session and the order the chat started from, a small router model classifies intent and rewrites the query, and hybrid retrieval with a reranker pulls current policy chunks, versioned with effective dates. The main model gets a lean prompt with cited sources and only the relevant tools. Every tool call goes through a policy engine in code, which decides eligibility and amount from real data, caps it, and calls an idempotent refund service, with an audit trail. Output checks verify citations and promises, and rules trigger a handoff with a summary. For peaks, I would cache the prefix, fall back to a second provider, and remove load at the product level.