In plain words
Imagine standing on a hillside in thick fog, trying to reach the bottom of the valley. You cannot see it. But you can feel the slope under your feet. So you step in the direction that goes down most steeply, then feel again, and step again.
That is gradient descent. The hill is the model's error, measured by a loss function. Where you stand is the current setting of the model's parameters. The slope, called the gradient, says which way to nudge each setting to make the error smaller.
Each step is small, and training takes millions of them. Almost every neural network, from photo apps to large language models, learns this way.
Everyday example
Finding the right amount of sugar in chai
You make chai and it is too bitter. Next cup, a little more sugar: better. Next, a little more: now too sweet, so back off slightly. Each taste tells you which direction to adjust and roughly how much. You never needed a formula for the perfect chai, just repeated small corrections guided by how far off the last cup was.
Where the example breaks: Chai has one knob. A large model has billions, and gradient descent adjusts them all at once, guided by maths rather than taste. It can also settle in a dip that is good but not the best possible.
How it actually works
Gradient descent is an optimisation method: a recipe for finding settings that make a number, the loss, as small as possible.
Compute the gradient
For the current parameters, work out how the loss changes if each parameter moves a tiny bit. Collected together, those rates of change form the gradient, which points uphill in the steepest direction. In a neural network, backpropagation computes it efficiently.
Step the other way
Move every parameter a little against the gradient:
new weight = weight − learning rate × gradient. The learning rate sets the size of the step.Repeat until it flattens
Recompute, step, repeat. Steps shrink naturally as the slope flattens near the bottom. When further steps barely change the loss, training has converged.
Use small batches
Computing the gradient over all the data for every step is slow. Stochastic gradient descent estimates it from one example; mini-batch descent, the usual choice, uses a small batch, often tens to hundreds of examples. The estimate is noisy, but steps are cheap.
Smarter variants
Optimisers such as Adam (2014) keep a running memory of past gradients and give each parameter its own step size. They are still gradient descent at heart.
- First suggested
- By the mathematician Augustin-Louis Cauchy in 1847
- Older than AI
- Google's glossary notes it is "much, much older" than machine learning
- Common forms
- Stochastic (SGD), mini-batch, and adaptive optimisers such as Adam
- Partner
- Backpropagation finds the slope; gradient descent takes the step
Questions people ask
What is gradient descent in simple terms?
A way to find the lowest point of a surface you cannot see, by repeatedly feeling the slope and stepping downhill. In AI, the surface is the model's error, so going downhill means making fewer mistakes.
What is the difference between gradient descent and backpropagation?
Backpropagation works out the slope: how much each weight in a neural network contributed to the error. Gradient descent uses that slope to update the weights. One measures, the other moves.
What is stochastic gradient descent?
A faster version that estimates the slope from one example, or a small batch, instead of the whole dataset. Each step is a little noisy, but it can take many more steps in the same time, and the noise can even help.
Can gradient descent get stuck?
It can settle in a dip that is not the lowest point, or crawl across flat stretches. In very large neural networks this matters less than once feared, and tricks like momentum and adaptive step sizes help it keep moving.