In plain words
Every model makes mistakes on new data, and those mistakes come from two very different places.
Bias is the error of being too simple. Ask a straight ruler to trace a curved road and it will be wrong in the same way every time, no matter how much data you give it. That is underfitting.
Variance is the error of being too jumpy. A very flexible model bends to fit every quirk of the examples it saw. Train it on a slightly different batch and you get a very different model. It memorised the noise, which is overfitting.
Making a model more flexible usually lowers bias but raises variance, and the other way round. The best model on new data sits somewhere in between: flexible enough to catch the real pattern, steady enough to ignore the noise. There is also a floor of error no model can remove, because some of the data is plain noise.
Everyday example
Three ways to guess the cricket score
A friend predicts every T20 innings will end at 160, whatever the pitch or team: simple, steady and often wrong in the same direction. That is bias. Another friend rebuilds his formula after every match from the last game alone, so his guesses swing wildly: variance. The friend worth listening to uses a few things that really matter, like pitch and batting line-up, and nothing more.
Where the example breaks: Friends can be asked why they guessed. A model's bias and variance are averages over many possible training sets, which we never see directly; in practice we estimate them by watching training and test error.
How it actually works
For squared error, the expected error of a model on a new point splits exactly into three parts: bias squared, plus variance, plus irreducible noise.
Bias: wrong assumptions
Bias measures how far the model's average prediction, across many possible training sets, sits from the truth. Fitting a straight line to curved data gives high bias whatever the data. High bias shows up as high error on both training and test data.
Variance: sensitivity to the sample
Variance measures how much the prediction would change if we retrained on a different sample of the same size. A very high-degree curve through a few points has high variance. It shows up as low training error but much higher test error.
The trade-off in one dial
Many models have a dial for flexibility. In k-nearest neighbours, using more neighbours raises bias and lowers variance. Polynomial degree and the strength of regularisation work in a similar way. Test error forms a U shape as you turn the dial.
How people work with it
Hold back a test set or use validation to find the bottom of the U. Add regularisation or dropout to cut variance. Average many models, as a random forest does, to reduce variance. More training data also lowers variance.
The modern twist
Huge neural networks often fit their training data perfectly and still do well on new data, which the classic picture does not predict. In 2019 Belkin and colleagues described "double descent": test error rises near the point where the model can just fit the data, then falls again as the model grows much larger.
- The decomposition
- Expected error = bias² + variance + irreducible error (for squared error)
- Classic paper
- Geman, Bienenstock and Doursat, "Neural networks and the bias/variance dilemma", Neural Computation, 1992
- Too simple
- High bias, underfitting: poor on training and test data
- Too flexible
- High variance, overfitting: great on training data, poor on new data
Does the bias-variance trade-off still apply to giant neural networks?
One view
The classic U still rules
The bias, variance and noise split is exact maths for squared error, and the U-shaped test curve is seen across countless everyday models. Regularisation and validation, built on this view, remain standard practice.
The other view
Big networks break the U
Belkin and colleagues showed that past the point where a model can fit its training data exactly, test error can fall again as it grows. Very large networks live in that second region, so "too flexible" is not the whole story.
Most likely: Both hold in their own range. The decomposition is always true, and the U shape describes models below the point where they can fit the data exactly. Far beyond it, as with today's large networks, a second descent can appear, so the simple "keep models small" advice does not carry over.
Questions people ask
What is bias in machine learning?
Here, bias means error from too-simple assumptions: the model cannot represent the real pattern, so it is wrong in a consistent way. This is a statistical meaning, different from unfair bias against groups of people, though both words are common in AI.
What is variance in machine learning?
Variance is how much a model changes when trained on a different sample of data. High variance means the model latched onto quirks of its particular examples, so it does well on them and poorly on new data.
How do you know if a model has high bias or high variance?
Compare training and test error. Both high: likely high bias, so try a more flexible model or better features. Training low, test much higher: likely high variance, so try more data, regularisation or a simpler model.
Is underfitting the same as high bias?
Roughly, yes. Underfitting is the symptom you see, poor results even on training data, and high bias is the usual cause. Likewise, overfitting is the symptom of high variance.
Can you have low bias and low variance at the same time?
That is the goal, and more good data helps reach it. With a fixed amount of data, though, lowering one usually raises the other, and some error from noise in the data always remains.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change