In plain words
In reinforcement learning, an AI learns by trying things and getting rewards or penalties. The hard part is that rewards often come late. A chess move that wins the game might have been made twenty moves earlier.
Q-learning solves this by keeping a score for every move in every situation. The score, called a Q-value, answers one question: if the AI makes this move here, and plays well afterwards, how much reward should it expect in total?
At first all the scores are guesses. Each time the AI makes a move, it looks at the reward it got plus the best score available from where it landed, and nudges its old score towards that. Over many tries, the value of reaching the goal flows backwards, step by step, to the moves that led there.
Once the scores are learned, playing is easy: in each situation, pick the move with the highest score.
Everyday example
Finding the best chai stall in a new neighbourhood
You move to a new area and try different lanes on your morning walk. One lane ends at a wonderful chai stall. Soon you value not just that lane, but the turn that leads to it, and then the street before that turn. You still try a new lane now and then, in case there is something better. That is Q-learning: value spreading back from rewards, plus a little exploring.
Where the example breaks: You remember a neighbourhood as a picture in your head. Basic Q-learning has no map at all: it only stores one number per situation and move, which becomes impossible when there are billions of situations.
How it actually works
Q-learning was introduced by Chris Watkins in his 1989 PhD thesis, "Learning from Delayed Rewards". The Q stands for the quality of an action in a given state.
The update rule
After taking action a in state s, getting reward r and landing in state s′, update:
Q(s,a) ← Q(s,a) + α × (r + γ × max Q(s′,·) − Q(s,a)). The bracket is the surprise: how much better or worse things went than expected.Two dials
The learning rate α sets how much each new experience overrides the old score. The discount γ, between 0 and 1, sets how much future rewards count. A discount near 1 makes the agent far-sighted; near 0, it only cares about the next reward. Our picture uses 0.9, so each step back is worth 90% of the next.
Explore versus exploit
If the agent always picks its current best move, it may never discover better ones. A common fix is epsilon-greedy: usually pick the best move, occasionally a random one. DeepMind's Atari agent started fully random and lowered the random share to 10% over its first million frames.
It provably works, in small worlds
Watkins and Peter Dayan proved in 1992 that Q-learning converges to the best scores as long as every action keeps being tried in every state and the scores are stored in a table. It needs no model of how the world works, only experience.
Deep Q-networks
A table cannot hold every possible video game screen. DeepMind's deep Q-network (DQN), published in Nature in 2015, replaced the table with a convolutional neural network that estimates Q-values from raw pixels. Two tricks kept it stable: replaying stored past experiences in random order, and a separate copy of the network for the targets, refreshed only now and then.
- Introduced
- 1989, Chris Watkins's PhD thesis "Learning from Delayed Rewards"
- Convergence proof
- Watkins and Dayan, Machine Learning journal, 1992
- Big moment
- DQN, published in Nature in February 2015, learned 49 Atari 2600 games from pixels and score alone
- DQN result
- Beat earlier methods on 43 of 49 games and reached a level comparable to a professional human games tester across the set
- Type
- Model-free and value-based: it learns scores for actions, not a model of the world
Questions people ask
What does the Q in Q-learning stand for?
Quality. Q(s, a) is the quality of taking action a in state s: the total future reward the agent expects if it takes that action and then keeps acting well. Learning these values is the whole method.
What is a Q-table?
A table with one row per situation and one column per possible action, each cell holding a Q-value. It works for small problems like grid mazes, but grows impossibly large for games or the real world, which is why deep Q-networks replace it with a neural network.
What is the difference between Q-learning and deep Q-learning?
Plain Q-learning stores every value in a table. Deep Q-learning uses a neural network to estimate the values, so it can handle huge numbers of situations, like screens of a video game, and generalise to ones it has not seen.
Is Q-learning used in ChatGPT?
Not in its classic form. The published recipe for ChatGPT-style chatbots, RLHF, scores answers with a reward model and updates the chatbot with a different algorithm, PPO. Q-learning is a classic value-learning method, best known from games and control problems.
How did DeepMind's Atari AI work?
It used deep Q-learning. Given only the screen pixels and the score, one network with the same settings for every game learned 49 Atari games. In Breakout it discovered the trick of digging a tunnel so the ball bounced behind the wall.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- Q-learning (Watkins and Dayan), Machine Learning 8, 1992Springer, abstract via University of South Carolina, 1992
- Human-level control through deep reinforcement learning (Mnih et al.), NatureNature, PDF via Dalhousie University, 2015
- From Pixels to Actions: Human-level control through Deep Reinforcement LearningGoogle Research, 2015
- Q-learningWikipedia, 2026