In plain words
To make chatbots helpful and polite, companies collect pairs of answers to the same question and ask people which one is better.
The older way to use those pairs, RLHF, has three stages: train a separate judge on the pairs (a reward model), let the chatbot practise against that judge, and adjust it step by step. It works, but it is complicated and can be unstable.
Direct preference optimisation, or DPO, skips the judge. It trains the chatbot directly on the pairs: make the preferred answer a bit more likely, make the rejected answer a bit less likely, and do not drift too far from where you started. Same data, one simpler step.
Everyday example
Learning from marked answer sheets
A student preparing for exams gets old answer sheets where a teacher has marked, for each question, which of two answers is better. The student does not need to first build a full marking scheme. She just studies the pairs: write more like the good one, less like the weak one. DPO learns the same way, straight from the marked pairs.
Where the example breaks: The student understands why one answer is better. DPO only shifts probabilities towards the chosen wording, and it can only learn from the pairs it is given. It cannot try out new answers and get feedback, the way RLHF practice can.
How it actually works
DPO was introduced in 2023 by researchers at Stanford, in a paper titled "Your Language Model is Secretly a Reward Model".
Start with preference pairs
Each training example has a prompt, a chosen answer and a rejected answer. Hugging Face's training library shows the simplest case: prompt "The sky is", chosen " blue.", rejected " green."
Keep a frozen copy
Before training, a copy of the starting model is set aside as a reference. It never changes. It anchors how far the trained model is allowed to move.
Push chosen up, rejected down
For each pair, DPO compares how much more likely the model now finds the chosen answer than the reference did, against the same for the rejected answer. Training widens that gap. It is an ordinary supervised loss, so no sampling or reward model is needed during training.
The hidden judge
The paper's key insight is mathematical: the chatbot's own probabilities, compared with the reference, already act as a reward model. So optimising them directly reaches the same goal RLHF aims for. A setting called beta controls how strongly the model may move away from the reference.
Where it is used
DPO spread quickly in open models. Hugging Face's Zephyr (2023) used it with answers ranked by another AI, not people. Meta used DPO for its Llama 3 models, reporting in 2024 that it needed less compute than PPO-based RLHF at large scale and performed better for them.
- Introduced
- May 2023, Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn (Stanford)
- Presented at
- NeurIPS 2023
- Replaces
- The reward model and the reinforcement learning loop of RLHF
- Needs
- Pairs of chosen and rejected answers, plus a frozen reference copy of the model
Questions people ask
What is the difference between DPO and RLHF?
Both learn from pairs of preferred and rejected answers. RLHF first trains a reward model, then uses reinforcement learning to improve the chatbot against it. DPO trains the chatbot on the pairs directly, in one supervised step, with no separate reward model.
Is DPO better than RLHF?
It is simpler, cheaper and more stable, and it matched or beat RLHF in the original paper's tests. But RLHF lets the model try new answers during training and get fresh scores for them, which DPO, learning only from fixed pairs, cannot do.
Does DPO still need humans?
It needs preference pairs, which can come from people or from another AI model. Zephyr, for example, used rankings from a stronger AI and no human labelling. The quality of those pairs largely decides the quality of the result.
Is DPO a kind of fine-tuning?
Yes. It is a fine-tuning method applied after instruction tuning, to shape the style and safety of a model's answers using comparisons rather than single correct answers.