Tokens, context and cost
A team ships a support chatbot. They estimate the bill from the average message: about 60 tokens in, 200 out, a hundred thousand conversations a month. The first invoice is nine times the estimate. Nothing was broken. They had priced the message, and the model is paid for the whole request, which also carries a 3,000 token system prompt and the entire conversation so far, sent again on every single turn.
This chapter is the arithmetic that stops that from happening to you.
A conversation runs for 20 turns. Does the total number of tokens you pay for across the whole conversation grow in a straight line with the number of turns, or faster? Why?
The context window#
The context window is the most text a model can look at in one call, counted in tokens. Today's models offer anything from about 128,000 tokens to over a million. That sounds like an amount you will never fill. In a real product it fills up faster than you expect, because the user's message is the smallest thing in it.
On every call, the window holds all of this:
- the system prompt: your instructions, tone, rules and examples
- the tool definitions: a description of every tool the model may call
- the conversation so far: every earlier message and answer
- any retrieved documents you looked up for this question
- the new message
- and room for the answer, since output counts against the same window
Build one request and watch where the space goes.
Two things are worth noticing. The fixed parts, the system prompt and the tool definitions, are paid for on every call, so a 4,000 token block of instructions is often the largest line on the whole bill. And a fuller window does not make a smarter model. A 2023 study called "Lost in the Middle" showed that models use facts at the start and end of a long context much better than facts buried in the middle. Stuffing more in can make answers worse, slower and more expensive, all at once.
Why conversations get expensive#
Here is the answer to the question above. A model holds no memory, so on turn 10 you send turns 1 to 9 again, plus the new one. On turn 20 you send nineteen old turns. The total grows with the square of the conversation length, not in a straight line.
Double the turns and the history line roughly quadruples. This is why long chats get summarised: after 10 or so turns, a product replaces the oldest turns with a short summary, and the bill stops climbing on a curve. Prompt caching, covered in the chapter on latency and cost, attacks the system prompt line instead.
The two numbers for speed#
People say "the model is slow" and mean one of two very different things.
Time to first token is how long the user waits before anything appears. It grows with the length of the prompt, because the model reads the whole prompt before it can write. A short prompt might take a few hundred milliseconds. A prompt with 100,000 tokens of documents can take many seconds.
Output speed is how fast tokens arrive after that, in tokens per second. Hosted models commonly run somewhere between 30 and 150 tokens a second for a single request, and it varies a lot by model size and provider.
The total is simple: time to first token, plus output tokens divided by output speed. A 500 token answer at 50 tokens a second is ten seconds of generation. Nobody will stare at a spinner for ten seconds, which is why you stream, and why the cheapest speed fix is usually asking for a shorter answer.
Setting max_tokens: 4000 does not make answers longer. It sets where the model gets cut
off. Set it too low and you get JSON that stops halfway through a bracket. Set it with
care, then control length with the instructions themselves ("answer in at most three
sentences"), which the model actually follows.
Where the money goes#
Checkpoint#
1. Why does a long chat cost far more than the sum of its messages?
2. Your answers start appearing after 4 seconds. Prompts include 60,000 tokens of retrieved documents. What is the most likely cause?
3. Which change most directly cuts cost on a system where every call carries the same 5,000 token system prompt?
- Everything on one call shares the window: instructions, tools, history, documents, the message and the answer.
- Chat cost grows with the square of the turns, because history is resent every time.
- Total time is time to first token plus output tokens divided by speed. 500 tokens at 50 a second is ten seconds.
- A fuller context is not a smarter one. Facts in the middle of a long prompt get used least.
Everything a model sees on one call has to fit in its context window: the system prompt, tool definitions, the whole conversation so far, any retrieved documents, the new message, and room for the answer. Because the model has no memory, history is resent on every turn, so the cost of a conversation grows with the square of its length, which is why products summarise old turns. Speed is two numbers: time to first token, which grows with prompt length, and output tokens per second. The fixes follow the bill: cache and trim the system prompt, retrieve fewer and better chunks, ask for shorter answers, and route simple work to smaller models.