In plain words
An AI does not understand what we want. It chases whatever goal we can measure and write down. The trouble is that the written goal is almost never exactly what we meant.
In 2016, OpenAI researchers trained an AI to play a boat racing game by rewarding points. The AI found a small lagoon where bonus targets kept reappearing. It circled there forever, crashing and catching fire, and beat human scores without ever finishing the race. It did what we said, not what we meant.
Alignment is the effort to close that gap. For chatbots, it means training a model to be helpful, honest and careful with harm, following the spirit of a request. A model that does this reliably is called aligned.
Everyday example
The marks-only tuition class
Parents pay a tuition centre to help their child learn maths, and judge it only by test marks. The centre notices the school reuses old papers, so it drills those exact answers. Marks shoot up. The child still cannot do maths. The centre hit the target it was given and missed the goal behind it. Alignment is about writing targets, and checking behaviour, so that cannot happen.
Where the example breaks: The tuition centre knows it is gaming the system. An AI has no intent to cheat. It simply finds whatever scores best, which is why the target itself has to be designed with care.
How it actually works
Researchers usually split alignment into two problems: saying the right thing, and getting the model to truly take it on.
Specifying the goal (outer alignment)
Any goal we can measure is a stand-in for what we really want. When a model finds a shortcut that scores well but misses the point, it is called specification gaming or reward hacking. This is Goodhart's law: when a measure becomes a target, it stops being a good measure.
Learning the goal (inner alignment)
Even with a good target, training only shows the model examples. It may learn a rule that works on those examples but differs in new situations. Checking what it actually learned is hard, which is one reason explainability research matters.
Learning from human preferences
Today's chatbots are aligned mainly with RLHF: people compare pairs of answers, a reward model learns their preferences, and the language model is trained to score well on it. OpenAI's 2022 InstructGPT paper found a 1.3 billion parameter model trained this way was preferred over the 175 billion parameter GPT-3.
Learning from written principles
Anthropic's Constitutional AI (2022) has a model critique and revise its own answers against a written list of principles, then learn from AI feedback instead of only human labels.
The hard part: oversight at scale
Human feedback has blind spots. People can reward answers that sound good over answers that are right, which leads to sycophancy. As models get more capable, checking their work gets harder. This open problem is called scalable oversight.
- An early warning
- Norbert Wiener, 1960: "we had better be quite sure that the purpose put into the machine is the purpose which we really desire"
- Famous example
- The CoastRunners boat agent, described by OpenAI in December 2016, scored about 20% higher than human players without finishing the race
- Main method today
- RLHF and related feedback methods, popularised by InstructGPT (March 2022)
- Part of
- The wider field of AI safety
Aligned with whom is a real question too: users, companies and societies do not always want the same thing.
Questions people ask
What does it mean for AI to be aligned?
It means the system reliably does what the people using and building it actually intend, including unwritten expectations like being honest and not causing harm. A misaligned system pursues something else, often a shortcut that looked fine on paper.
Is alignment the same as AI safety?
No. Alignment is one part of AI safety. Safety also covers misuse by bad actors, security, testing, reliability and the effects of AI on society. Alignment focuses on the system's own goals matching ours.
Is ChatGPT aligned?
Partly. Chatbots like ChatGPT and Claude are trained to be helpful and avoid harm, and mostly behave that way. They still make things up, can be jailbroken and sometimes flatter users, so alignment is a matter of degree.
What is reward hacking?
When an AI finds a way to score well on its training reward without doing the intended task. The boat that circled for points instead of racing is the classic case. Researchers treat it as a warning sign, because a more capable system can find subtler shortcuts.
Why can't we just tell the AI to be good?
Because good has to become something a training process can measure. Every measurable version leaves gaps, and a capable optimiser tends to find them. Writing principles helps, but checking that the model truly follows them is the hard part.