KV cache vs prompt caching vs semantic caching
Read firstHow an LLM works
An agent sends the same 12,000 token system prompt and tool list on every turn, followed by a few hundred tokens that actually changed. Without caching, the model reads all 12,000 again each time, and you pay for all of them. With caching set up well, most of that work and most of that bill disappear. With caching set up slightly wrong, for example a timestamp at the top of the system prompt, it quietly disappears and nothing errors.
Three different things get called "caching" in LLM systems, at three layers:
- The KV cache lives inside one request on the GPU. It is why generating the 500th token is not 500 times slower than the first.
- Prompt caching keeps that KV cache between requests that start with the same text. It is what providers discount.
- A semantic cache sits in your application, in front of the model. It returns a previous answer for a question that means the same thing, and skips the model entirely.
Your support bot's system prompt starts with 'Today is 2026-11-03 14:32:07' and then has 8,000 tokens of policies. The provider supports prompt caching, but your cache hit rate is zero. Why, and what would you change?
The KV cache: inside one request#
A transformer generates one token at a time, and each new token attends to every token before it. To do that, each layer computes a key and a value vector for every earlier token. Those do not change once computed, so the model keeps them: the KV cache. Each new token computes its own key and value, appends them, and attends over the stored ones. Without it, generating token 500 would mean recomputing keys and values for all 499 before it.
That is also why an LLM request has two phases with different costs. Prefill reads the whole prompt at once and builds its KV cache, which is heavy on compute and runs in parallel. Decode then produces tokens one by one, each reading the whole cache, which is limited by memory bandwidth. Time to first token is mostly prefill. Tokens per second is decode.
The cost is memory, and it is larger than people expect:
More than a whole 80 GB H100, before the 70B weights themselves. Long contexts limit how many requests a GPU can run at once, which is why KV cache memory, not compute, often caps throughput.
Inference servers spend a lot of effort on this memory. vLLM's PagedAttention (2023) stores the cache in fixed-size blocks, like operating system pages, so requests of different lengths do not fragment GPU memory, and blocks can be shared between requests with the same prefix. That sharing is the bridge to the next layer.