In plain words
It is hard to write down rules for a good answer. Should it be short or detailed? Friendly or formal? But show people two answers side by side and most can quickly say which one is better.
RLHF, short for reinforcement learning from human feedback, turns that quick judgement into training. People compare thousands of pairs of answers. Those choices are used to build a second model, a reward model, that learns to predict which answers people will like and gives each one a score.
Then the chatbot practises. It writes answers, the reward model scores them, and the chatbot is adjusted to earn higher scores. Over many rounds it drifts towards answers that are clearer, more helpful and more polite.
RLHF is a big reason ChatGPT felt so different from earlier AI when it launched in 2022.
Everyday example
A cook learning a family's taste
A new cook makes two versions of dal each day, and the family says which they prefer. Soon the cook notices patterns: less salt, more tadka, thinner on weekdays. Now the cook can judge new dishes the family's way, and keeps adjusting recipes towards what would win. The family's picks are human feedback; the cook's sense of their taste is the reward model.
Where the example breaks: A good cook knows the family is pleased because the food is better. The model only learns what wins votes, and that can include flattery or confident-sounding answers. It does not understand why people preferred something.
How it actually works
RLHF is a stage of fine-tuning that comes after pre-training and instruction tuning. OpenAI's InstructGPT paper of 2022 laid out the recipe most labs then followed.
Start from an instruction-tuned model
First, people write example answers and the model is fine-tuned on them, so it already follows instructions reasonably well.
Collect comparisons
The model writes several answers to the same prompt. Human raters rank them. InstructGPT used about 40 contractors to produce these rankings.
Train a reward model
A separate model learns from the rankings to give any answer a single score that predicts how much people would like it.
Reinforcement learning
The chatbot generates answers, the reward model scores them, and an algorithm called PPO updates the chatbot towards higher scores. This is reinforcement learning. A penalty keeps it close to its starting point so it does not drift into odd text that tricks the scorer.
Newer variations
Direct Preference Optimization (DPO, 2023) skips the separate reward model and learns straight from the preference pairs. Anthropic's Constitutional AI (2022) uses an AI judge guided by written principles in place of many human labels. Labs now mix human, AI and automatic feedback.
- Founding paper
- June 2017, Paul Christiano and colleagues at OpenAI and DeepMind, teaching simulated robots and Atari game agents from human comparisons
- Applied to chatbots
- InstructGPT, March 2022; ChatGPT, November 2022, was trained with "the same methods"
- Striking result
- People preferred a 1.3 billion parameter InstructGPT model to the 175 billion parameter GPT-3
- Known side effect
- Can encourage sycophancy: Anthropic found people and reward models sometimes prefer agreeable answers over correct ones
Questions people ask
What does RLHF stand for?
Reinforcement learning from human feedback. Reinforcement learning means learning by trial and reward. Human feedback means the reward comes from people's judgements, usually which of two answers is better, rather than from a fixed score like points in a game.
Is RLHF the same as fine-tuning?
It is one kind of fine-tuning. Plain supervised fine-tuning copies example answers. RLHF learns from preferences between answers, which is useful when good behaviour is easier to recognise than to write down.
Who does the human feedback for RLHF?
Paid raters, often contractors hired through data-labelling companies, working from written guidelines. Experts are used for specialist topics like medicine or code. Some apps also collect thumbs-up and thumbs-down from users.
Does RLHF make AI safe?
It helps a lot with tone, refusing harmful requests and following instructions, but it is not a guarantee. Models trained this way can still be talked into bad behaviour (jailbreaks) and can learn to please rather than to be right. It is one tool in alignment work.
What is the difference between RLHF and DPO?
Both learn from pairs of better and worse answers. RLHF first trains a reward model and then uses reinforcement learning against it. DPO, introduced by Stanford researchers in 2023, skips both and updates the model directly from the pairs, which is simpler and often more stable.
Further reading
- Training language models to follow instructions with human feedback (Ouyang et al.)arXiv, 2022
- Deep reinforcement learning from human preferences (Christiano et al.)arXiv, 2017
- Introducing ChatGPTOpenAI, 2022
- Direct Preference Optimization (Rafailov et al.)arXiv, 2023
- Towards Understanding Sycophancy in Language Models (Sharma et al.)arXiv, 2023