SFT vs RLHF vs DPO

Give a freshly pretrained model the prompt "How do I reset my UPI PIN?" and it may well reply with "How do I change my UPI limit? How do I add a bank account?", because on the web, a question like that is usually followed by more questions in an FAQ list. The model has read trillions of tokens and knows a great deal. What it has learned to do is continue text, not answer people.

Everything between that and a helpful assistant is called post-training, and it happens in stages. Supervised fine-tuning (SFT) shows the model examples of good answers. Preference tuning then teaches it which of two answers is better, with RLHF (reinforcement learning from human feedback) or the simpler DPO (direct preference optimisation). Reasoning models add a further stage of reinforcement learning on problems with checkable answers. Interviews for AI engineering roles ask about these more and more, usually to see whether you know which one a product team would ever touch.

Three stages, three kinds of data#

The stages differ mostly in what the training data looks like, and that is the easiest way to keep them apart.

Data:   trillions of tokens of text: web pages, books, code
Learns: predict the next token
Result: a base model. Knows a lot, continues text, does not follow instructions.
Cost:   thousands of GPUs for weeks to months. Done by model labs.

SFT teaches what a good answer looks like by example. Preference tuning teaches which of two answers is better, which turns out to be much easier for people to label than writing ideal answers from scratch, and it captures qualities that are hard to demonstrate: concise rather than padded, honest about uncertainty rather than confident, declining the right requests and only those.

OpenAI's InstructGPT paper (2022) showed how much these stages matter. After SFT and RLHF, people preferred the answers of a 1.3 billion parameter model over those of the original 175 billion parameter GPT-3. Most of what users experience as "a good model" is post-training, not size.