Latency and cost in production
The feature works. People like it. Usage grows ten times in two months, and two graphs start to worry everyone: the model bill is now the largest line in the infrastructure budget, and the slowest 5% of requests take nine seconds. Nothing is broken. The prototype was simply never built for its own success.
This chapter is the toolkit for that moment. The good news is that most of these fixes make the feature faster and cheaper at the same time.
Name three changes that would make an LLM feature both faster and cheaper, without making answers worse.
Measure per feature, per request#
You cannot cut what you cannot see. For every LLM feature, log on every request: input tokens, output tokens, cached tokens, time to first token, total time, model, and cost. Then look at the totals by feature. It is very common to find one forgotten background job, or one prompt with a 6,000 token example block, behind half the bill.
Stream, always#
Streaming does not make a request cheaper or shorter. It makes the wait feel much shorter, because the first words appear after a second instead of after ten. For anything a person reads, it is the first change to make and the cheapest.
Prompt caching#
Most requests in a feature start with the same long prefix: system prompt, tool definitions, reference documents. Providers can keep the processed form of a recent prefix and reuse it, which makes those tokens much cheaper and cuts time to first token, often substantially. The exact discount and how long the cache lives depend on the provider, so check yours.
The rule that makes it work: stable content first, changing content last. One timestamp or user name near the top of the prompt changes the prefix on every call and silently disables the cache for everything after it.
The hit rate is the lever you control. Keep the prefix byte for byte identical, put anything per user or per request after it, and keep traffic flowing so the cache does not expire between calls.
Caching answers#
Exact caching stores the answer for an exact input and is safe for anything deterministic: the same document summarised twice, the same product description generated twice.
Semantic caching returns a stored answer when a new question is similar enough to an old one, using embeddings. It can remove a large share of calls for generic questions ("what are your delivery hours?"). It is also dangerous: "refund for UPI payments" and "refund for card payments" are very similar sentences with different answers. Use it only for questions whose answer does not depend on the user, keep the similarity threshold strict, and key the cache by tenant and language.
The right model for each request#
The biggest model is not needed for most traffic. Two patterns:
Routing. A small, fast model classifies the request and sends easy ones to a small model and hard ones to a large one. Most traffic is easy.
Cascades. Try the small model first. If a check fails (invalid output, low confidence, a failed validation), retry with the large model. You pay for two calls on the minority that needed it and one cheap call on everything else.
Both need an eval, because "the small model is good enough" is a claim you have to measure, per feature.
Work that can wait#
Not everything is interactive. Nightly classification, bulk summaries and backfills can go through the batch APIs many providers offer: you submit a file of requests and get results within hours, often at around half the price. Moving background jobs off the interactive path also frees your rate limit for the people waiting on a response.
Fewer output tokens#
Output is the slow, expensive part. Ask for concise answers. Use structured output instead of prose when code is the reader. Do not ask for reasoning on tasks that do not need it. A feature that answers in 150 tokens instead of 500 is more than three times faster to finish.
When the provider says no#
Providers enforce rate limits and have outages. On a sale day, your feature will meet both.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Prompt caching | Large savings and faster first tokens on repeated prefixes, with no quality change. | Requires discipline about prompt order, and caches expire. | Any feature with a long stable system prompt or shared documents. Nearly all of them. |
| Semantic answer cache | Skips the model entirely for repeated generic questions. | Can return a confidently wrong answer to a question that only looked similar. | Generic FAQ answers that do not depend on who is asking. |
| Routing and cascades | Most traffic runs on a cheaper, faster model. | A classifier or check to build, and evals to prove quality held. | Mixed traffic where most requests are easy. |
| Batch API | Often around half price, and it keeps rate limits free for interactive users. | Results arrive in hours, not seconds. | Background and bulk work. |
Every client library makes retrying easy, and immediate retries against a rate limited provider multiply the load that caused the limit. Retry with exponential backoff and jitter, cap the attempts, put a timeout on every call, and fall back or fail fast instead of queueing forever. The same rules from Design a rate limiter apply here, with you as the client.
Checkpoint#
1. Prompt caching is enabled but the hit rate is near zero. What is the most likely cause?
2. Why is a semantic cache risky for "refund status for my UPI payment"?
3. During a traffic spike, the provider returns 429s and your error rate keeps climbing. What should the client do?
- Log tokens, cache hits, latency and cost per request, per feature.
- Stable prefix first or prompt caching silently stops working.
- Route easy traffic to small models, and prove it with an eval.
- Backoff with jitter, cap retries, have a fallback. Your retries can be the outage.
To make an LLM feature faster and cheaper, I would first measure tokens, cache hits, latency and cost per request for each feature. Then stream every user facing response, turn on prompt caching with stable content first so the prefix actually matches, and cut output tokens by asking for concise or structured answers. I would route easy requests to a small model or use a cascade, proven with evals, use semantic caching only for generic answers, and move background work to batch APIs. For reliability, retries back off with jitter and are capped, load is shed when the queue is full, and a circuit breaker fails over to another provider.