How an LLM actually works

You paste an email and type "make this shorter". A second later words start appearing, one small piece at a time. Under all of it is one operation, repeated a few hundred times: look at the text so far, and guess what comes next. Nothing else. Once you see that loop clearly, most of the strange behaviour you have seen from these tools starts to make sense.

You do not need any maths for this page. You need one picture of the loop and a feel for how the next word gets picked.

Text goes in as tokens#

A model does not read letters or words. It reads tokens, which are chunks of text from a fixed list of about 100,000 to 200,000 pieces that the model learned before it was ever trained. Common words are one token. Rare words get split.

"Refund my order" is three tokens. "Unrefundable" might be three on its own: "Un", "refund", "able". A space usually travels with the word after it, so " order" and "order" are different tokens.

For English, a useful rule is about four characters per token, or three quarters of a word. That rule breaks for other scripts. A sentence in Hindi or Tamil often costs two to four times as many tokens as the same sentence in English, because the tokenizer saw much less of those scripts and splits them into smaller pieces. That matters the day your bill arrives, because everything about a model is measured in tokens: the price, the speed, and how much it can read at once.

One step: score every token#

Here is the whole machine, drawn once.

The model in the middle is a neural network called a transformer. Its job, learned from reading a huge amount of text, is narrow: given these tokens, how likely is each possible next token? It does not output a sentence. It outputs a score for every single token in its vocabulary, and something outside the model has to pick one.

Two rounds of training give it that skill. First it reads trillions of tokens from the web, books and code, guessing the next token over and over, until it is very good at continuing text. Then it is tuned on conversations, with people (and other models) rating answers, so that the most likely continuation of a question is a helpful answer rather than three more questions.

Picking the next token#

This is the part almost everyone has wrong. The model does not choose "the best word". It hands over a probability for every token, and a sampler rolls a weighted die. Play with the two knobs every API gives you.

Temperature reshapes the probabilities before the roll. At 0 the top token always wins, so the same prompt gives the same words every time. At 1 you get the model's own probabilities. Above 1 the unlikely tokens get a real chance, and that is where answers start to wander.

Top-p cuts the tail off. At 0.9 the sampler only considers the smallest group of top tokens that together hold 90% of the probability. " yesterday" can never be picked, no matter how high you push the temperature.

So the answer to the question at the top: usually no. At any temperature above zero you are rolling dice, and even at zero many providers do not promise identical output, because of how the arithmetic is split across a GPU. If your product needs the same answer twice, store the first one.

The loop, and why output is the slow part#

A 300 word answer is roughly 400 tokens, which means 400 trips around Figure 1. Each trip has to run the whole model, and each one waits for the trip before it, because token 201 depends on token 200.

Reading your prompt is different. All of the prompt's tokens are known up front, so the model processes them together in one big pass. That is why providers charge less for input tokens than for output tokens, often three to five times less, and why the first word of an answer can arrive quickly while the last one arrives seconds later.

It is also why every serious product streams: it sends each token to the screen as it is picked instead of waiting for the full answer. The total time is the same. The wait the user feels is not. (The machinery that makes this fast at scale, batching and the KV cache, is its own chapter: Design an LLM inference platform.)

What the model does not have#

Four limits explain most of what goes wrong when people build on these models.

No memory between calls. Every request starts from nothing. A chat app "remembers" because it sends the whole conversation again on every turn. If you did not send it, the model never saw it.

A knowledge cutoff. The model knows what was in its training text, which stopped at some date. It does not know your order status, your company's refund policy, or this morning's news, unless you put them in the prompt.

No access to anything. A model cannot open a URL, query a database or run code. It can only produce text. When a product appears to do those things, your code is doing them, which is what the chapter on tool calling is about.

No fact checking. The model produces likely text. Usually likely text is true, because true statements are common in what it read. When it does not know, the likely text is a confident sentence that sounds like the true ones. That is a hallucination, and it is not a bug that will be patched out. It falls straight out of how the loop works.

Checkpoint#