In plain words
During training, a model keeps adjusting itself to make fewer mistakes. It knows which direction to adjust from gradient descent. The learning rate decides how far to move in that direction each time.
Too small, and training crawls: every step is so cautious that reaching a good model takes forever. Too big, and the model overshoots: it jumps past the best setting, then back past it the other way, and can bounce further and further off.
The learning rate is chosen by people before training starts, which makes it a hyperparameter rather than something the model learns. It is often the first setting engineers tune when training goes wrong, and big models usually change it during training, ending with smaller steps.
Everyday example
Adjusting a hotel shower
The water is cold. Turn the knob a hair at a time and you stand shivering for ages. Swing it hard and you get scalded, swing back and freeze, and keep swinging. A sensible turn, then smaller turns as it gets close, gets you there fast. The size of each turn is your learning rate.
Where the example breaks: In a shower you feel the temperature instantly and have one knob. A model adjusts billions of settings with one shared rate, which is why modern optimisers adapt the step for each setting.
How it actually works
In the update rule of gradient descent, the learning rate is the multiplier on the gradient.
Where it sits
new weight = weight − learning rate × gradient. Google's crash course example: a gradient of 2.5 with a learning rate of 0.01 moves the weight by 0.025.Too low
Training takes many more iterations to converge, wasting time and compute, and may stall before reaching a good model. Google lists too low a learning rate as one cause of underfitting.
Too high
The loss bounces around or keeps rising instead of settling. In neural networks it can blow up the numbers inside the network, which is one reason engineers lower it when training becomes unstable.
Schedules
Rather than one fixed value, large models often use a schedule: a short warm-up from a small rate, then a gradual decay so final steps are fine adjustments.
Adaptive optimisers
Methods such as AdaGrad and Adam give each parameter its own effective step size based on its recent gradients. A base learning rate still has to be chosen.
- Type
- Hyperparameter: set by people, not learned
- Also called
- Step size, often written as LR
- Typical values
- Small fractions such as 0.01 or 0.001; the right value depends on the model and optimiser
- Works with
- Gradient descent, backpropagation, batch size and number of epochs
Questions people ask
What happens if the learning rate is too high?
The model overshoots the best settings on every step. The loss jumps around instead of falling, or climbs steadily, and training can fail completely. Lowering the learning rate is the standard first fix.
What happens if the learning rate is too low?
Training still moves in the right direction but very slowly, so it needs far more steps, time and compute. It may also stop early in a poor spot and leave the model underfit.
What is the difference between a learning rate and a parameter?
Parameters are the values the model learns, such as its weights. The learning rate is a hyperparameter: a setting people choose to control how the learning happens. It is not part of the finished model.
What is a good learning rate?
There is no universal number. People try a few values, often spaced by factors of 3 or 10, and watch the loss curve. Adaptive optimisers like Adam make training less sensitive to the exact choice.