Design an LLM inference platform
Read firstLoad balancingTokens, context and cost
A user types a question and waits. The first word has to appear within about a second or the product feels broken, and the next four hundred have to arrive faster than they can read. Behind that is a machine that costs more per hour than the engineer who provisioned it, and which sits idle through most of the generation unless you do something unusual with it.
Send. The client sends a prompt and keeps the connection open, because the answer will stream back a token at a time for several seconds.
A few words come up on every line of this page, so here they are in plain terms first:
- A token is a chunk of text, roughly three quarters of an English word. Models read and write tokens, not letters. (How LLMs work covers this from the start.)
- The weights are the model itself: billions of numbers learned in training. A 70 billion parameter model stored as 16 bit numbers is 140 GB of weights.
- A GPU is the chip that runs the model. Its own fast memory holds the weights and everything the model is working on right now. A replica is one full copy of the model, spread over as many GPUs as it needs to fit.
- The KV cache is the model's working notes on a conversation. Every token it has read or written leaves a small record behind, so it never has to reread the conversation from the start. Those notes live in GPU memory for as long as the request is alive.
This question is newer than the rest of the course and it is being asked constantly. It is also the one where normal web intuitions are wrong. The unit of capacity is not requests per second. The bottleneck is not CPU. And a bigger batch makes the answer stream more slowly, which surprises almost everybody the first time.
Step 1: Understand the problem#
Six questions turn "serve a model" into a design. Two of them (prompt length and hardware supply) decide what kind of problem this is.
| You ask | They say | What it settles |
|---|---|---|
| Interactive chat, or batch jobs? | Interactive, streaming tokens as they are produced. | Two separate latency numbers, time to first token and the gap between tokens after that. One number called "latency" hides the only interesting failure in this system. |
| How long are prompts, and how long are answers? | Prompts from a hundred tokens to a hundred thousand. Answers a few hundred. | The most important number on the page. Reading the prompt and writing the answer are different kinds of work on the same chip, and the ratio between them decides how you schedule. |
| Can we add GPUs when traffic rises? | Not quickly. The fleet is fixed and supply constrained. | This is a scheduling problem under scarcity, not an autoscaling problem. Admission control and queueing become product decisions rather than safety valves. |
| One model or many? | A few sizes, plus customer fine tunes. | Routing by model, and a hard constraint: loading a hundred gigabytes of weights takes minutes, so you cannot swap models on demand. Either a replica is dedicated, or variants share a base model through adapters. |
| Do requests from one customer share anything? | Yes. Every request carries the same 2,000 token system prompt. | Prefix caching, which can remove most of the prompt reading work for most requests. It is the cheapest large win available and it is invisible unless you ask this question. |
| Under overload, do we go slow or refuse? | Refuse clearly, rather than making everybody slow. | Bounded queues and an explicit 429. An unbounded queue in front of a GPU means every user gets a bad experience instead of most users getting a good one. |
What you are building, and what you cut#
- Serve streaming token responses. Held connections, cancellable, with per tenant priority.
- Batch many requests onto one replica. Continuously, not in fixed groups. This is where the throughput comes from.
- Manage KV cache memory. It is the real capacity limit, and it moves with context length.
- Queue, prioritise and shed. On hardware you cannot buy more of this week.
- First token in under 1 second at p95.
- Under 50ms between tokens at p95, which is faster than anyone reads.
- GPUs busy with useful work above 80% of the time.
- One 100,000 token prompt must not stall everybody else on that replica.
- Training and fine tuning. A different fleet, a different failure model, and nothing here is on its path.
- The model itself: quality, evaluations, and safety behaviour. Assume the weights are given.
- Retrieval and tool calling. Those are services that call this one, and treating them as part of it is how the design loses focus.
- Guardrail classifiers. Name them as a sidecar on the way in and the way out, with their own latency budget.
Back of the envelope#
Before the sliders, one worked example, because the napkin uses a number that looks arbitrary until you see where it comes from.
Take a 70 billion parameter model with 80 layers, where each layer keeps 8 key value heads of 128 numbers each, all stored as 16 bit (2 byte) numbers. Each token leaves behind a key and a value per head per layer:
2 (key and value) x 80 layers x 8 heads x 128 numbers x 2 bytes = 327,680 bytes, about 320 KB
So one 8,000 token conversation holds 8,000 x 320 KB, about 2.5 GB of GPU memory, for as long as it is running. Now the sliders. (The method behind every row is in back of the envelope.) Drag the context length last, and watch what it does to everything else.
The 320 KB per token is the worked example above, and it is the number worth memorising. Now drag the context length to 128,000: concurrency collapses, because each sequence is holding tens of gigabytes of cache. Long context is not a feature you switch on, it is capacity you spend, and the person who adds it to the product is spending most of your fleet.
Almost everyone sizes this in requests per second. Requests per second is an output of this system, not an input to it, because a request holds memory for its whole lifetime and that memory scales with how much text is in it. The second mistake is trusting GPU utilisation: a chip can report 100% while doing almost no useful arithmetic, because generating one token at a time is limited by how fast weights can be read out of memory rather than by how fast they can be multiplied. Think of a chef who must walk to the storeroom for every ingredient: the kitchen is busy all day, and most of that is walking.