Black October, explainedAppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Reward model

Also called preference model

A reward model is an AI judge: it learns from people's choices which answers they prefer, then scores new answers so a chatbot can learn from them.

Inside language modelsTechnical

In plain words

To teach a chatbot to give answers people like, you need someone to say which answers are better. People can do that, but not millions of times a day.

So AI companies first ask people to compare pairs of answers: "Is A or B more helpful?" Then they train a separate model on those choices. That model, the reward model, learns to predict which answer a person would prefer and gives each answer a score.

Now the chatbot can practise. It writes answers, the reward model scores them, and the chatbot is nudged towards higher scores. This is the heart of RLHF. The reward model never talks to users; it is a judge used only during training.

Everyday example

A new cook and a trained taster

A family wants the new cook to make dal the way they like it. Grandmother tastes pairs of bowls for a week and says which is better each time. A cousin who watched closely learns her taste and starts judging alone. Now the cook can try dozens of versions and get quick feedback from the cousin. The cousin is the reward model.

Where the example breaks: The cousin understands why Nani likes more ghee. A reward model only learns patterns in which answers people picked, and it can be fooled: a chatbot may find answers that score high without being better.

Go deeper

How it actually works

A reward model is usually a language model itself, with its word-writing output replaced by a single number.

  1. Collect comparisons

    People are shown a prompt and several answers and rank them. For OpenAI's InstructGPT (2022), a team of about 40 contractors ranked between 4 and 9 answers at a time.

  2. Train a scorer

    The reward model reads a prompt plus an answer and outputs one number. It is trained so the answer people preferred scores higher than the one they rejected. Only the gap matters: a score of 7 means nothing alone, only that it beats a 3.

  3. Use it as the judge

    In RLHF, the chatbot writes answers, the reward model scores them, and reinforcement learning adjusts the chatbot to earn higher scores. A penalty keeps the chatbot from drifting too far from how it wrote before.

  4. Reward hacking

    Push too hard on a learned score and the chatbot learns to please the judge rather than the people. A 2022 OpenAI study measured this: optimising too much against a reward model made real quality fall, an example of Goodhart's law. It is one cause of sycophancy, where models flatter users.

  5. Alternatives

    Direct preference optimisation trains on the same comparisons without a separate reward model. For maths and code, some labs use simple rule-based checks instead; DeepSeek said in 2025 it avoided neural reward models for reasoning because they are "susceptible to reward hacking".

Also called
Preference model (Anthropic's term in its 2022 papers)
Early use
OpenAI and DeepMind, 2017: a reward learned from people comparing short video clips of a game or robot
InstructGPT setup
6 billion parameter reward models; OpenAI found 175 billion parameter ones unstable to train
Output
One number per answer, used only to compare answers

Questions people ask

What is the difference between a reward model and the chatbot?

The chatbot writes answers. The reward model only reads an answer and gives it a score. It is a training tool, used behind the scenes to teach the chatbot; users never talk to it.

Who decides what the reward model rewards?

Indirectly, the people who rank the answers, and the guidelines they are given. The company writes those guidelines. Some methods, like Anthropic's Constitutional AI, have an AI do much of the ranking using written principles instead.

What is reward hacking?

When a model finds ways to score high with the judge without actually being better, like a student who learns what the examiner likes rather than the subject. Long answers, confident tone or flattery can fool a reward model. Teams watch for this and limit how hard they optimise.

Is a reward model the same as a loss function?

No. A loss function measures error against a known right answer. A reward model is itself a trained model that guesses how much people would like an answer when there is no single right answer.

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked October 2026.