In plain words
A neural network learns by passing a correction backwards from its answer, layer by layer, telling each part how to change. That correction is called the gradient.
At each layer, the correction gets multiplied by a number. If those numbers are mostly smaller than one, the signal shrinks at every step. After twenty or thirty layers it can be so tiny that the first layers hardly change at all.
That was a real wall for years. Deep networks looked powerful on paper, but their early layers simply would not learn. The same thing made older language models forget the start of long sentences.
The fixes came in pieces: simpler neuron rules that do not squash the signal, careful starting values, and above all shortcut connections that let the correction skip past layers. Today's very deep networks, including transformers, rely on these tricks.
Everyday example
A message passed down a long line
A teacher whispers a correction to the last student in a long line, and each student passes it forward, but only a quarter as loud as they heard it. By the time it reaches the front, nobody can hear it. The students at the back adjust; the ones at the front never learn they were wrong.
Where the example breaks: Whispers fade the same way every time. In a network the multipliers vary: they can also be bigger than one, and then the signal grows out of control instead, the exploding gradient problem.
How it actually works
During backpropagation, the chain rule makes each early layer's gradient a product of many factors, one per later layer. Many factors below one make that product shrink exponentially.
Where the small factors come from
Each factor includes the slope of the layer's activation function. The sigmoid's slope is at most 0.25, and almost zero when the neuron is saturated near 0 or 1. Ten sigmoid layers can shrink the signal by a factor of more than a million.
First identified
Sepp Hochreiter's 1991 diploma thesis pinned this down as a main reason deep networks failed. It hit recurrent networks hardest, because a long sentence acts like a very deep network unrolled over time.
Fix 1: memory cells (LSTM)
Hochreiter and Jürgen Schmidhuber's 1997 long short-term memory network added gated memory cells that let the error flow back without fading. It could learn links across more than 1,000 time steps.
Fix 2: better activations and starting weights
ReLU passes positive values with a slope of exactly 1, so it does not shrink them. Stanford's CS231n notes report it sped up training about six times compared with tanh in one well-known study. Glorot and Bengio's 2010 work showed that careful weight initialisation also keeps signals a healthy size.
Fix 3: shortcut connections
Residual networks (2015) add each layer's input straight to its output, giving the gradient an express lane backwards. ResNet trained networks 152 layers deep and won the 2015 ImageNet classification contest. Transformers use the same residual connections, plus layer normalisation.
- Identified
- Sepp Hochreiter, diploma thesis, 1991
- Opposite problem
- Exploding gradients, often handled by gradient clipping: scaling the gradient down when it gets too big
- Classic culprit
- Sigmoid and tanh activations, whose slopes are small or near zero over most inputs
- Key fixes
- LSTM (1997), ReLU, careful initialisation (2010), batch normalisation, residual connections (2015)
Questions people ask
What causes the vanishing gradient problem?
Repeated multiplication of small numbers. Backpropagation multiplies one factor per layer, and with activations like sigmoid those factors are usually well below one. Across many layers, the product shrinks towards zero.
How does ReLU help with vanishing gradients?
For positive inputs, ReLU's slope is exactly 1, so it passes the signal back unchanged instead of shrinking it. Its weakness is the opposite side: for negative inputs the slope is 0, which can leave some neurons stuck.
What is the difference between vanishing and exploding gradients?
Both come from multiplying many factors. If they are mostly below one, the gradient vanishes and early layers stop learning. If they are mostly above one, it explodes, causing wild, unstable updates. Gradient clipping is the usual fix for exploding gradients.
Do transformers have vanishing gradients?
Much less, by design. Transformers wrap every block in a residual shortcut and use normalisation layers, so the gradient has a direct path back. That helps them stack dozens of layers deep.
Why did LSTMs replace simple recurrent networks?
Simple recurrent networks could not learn links between words far apart, because the gradient faded over many time steps. LSTMs added a memory cell with gates that lets the signal survive, so they handled much longer sequences.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- Vanishing gradient problemWikipedia, 2026
- Long Short-Term Memory (Hochreiter and Schmidhuber), Neural ComputationJohannes Kepler University Linz, 1997
- Understanding the difficulty of training deep feedforward neural networks (Glorot and Bengio)PMLR, 2010
- Deep Residual Learning for Image Recognition (He et al.)arXiv, 2015
- CS231n: Neural Networks Part 1, setting up the architectureStanford University