In plain words
Imagine estimating a flat's rent. A rough first guess gets you part of the way. Then you look at how far off you are and make a second, small correction. Then another correction for what is still wrong. Each fix is modest, but together they get close.
Gradient boosting works like that. It starts with a simple guess, then adds a small decision tree trained only on the errors that remain. Then another tree for the errors after that, and so on, for many rounds.
Each tree is deliberately weak and its correction is scaled down, so the model creeps towards the answer instead of lurching. Popular tools such as XGBoost and LightGBM do this very efficiently, and they are a common choice for predictions from tables of data, such as prices, risk scores or demand.
Everyday example
A tailor's fittings
A tailor cuts a kurta from rough measurements. At the first fitting it is loose at the shoulders, so he takes it in a little. Second fitting: the sleeves are long, so he trims them. Each fitting fixes only what is still wrong, never starting over. After a few rounds it fits. Every new tree in gradient boosting is one more fitting, aimed at the remaining error.
Where the example breaks: A tailor stops when it fits. A boosting model will keep fitting the remaining errors, including random noise in its examples, so it must be stopped by checking on fresh data, or it ends up overfitting.
How it actually works
Boosting means combining many weak models in sequence. Gradient boosting, set out by Jerome Friedman in 2001, makes that sequence a form of gradient descent.
Start simple
Begin with a basic prediction for everyone, such as zero or the average value.
Measure what is left
For each training example, compute how the loss would change if its prediction moved, the negative gradient. For ordinary squared error, that is simply the residual: the true answer minus the current prediction.
Fit a small tree to it
Train a shallow decision tree to predict those residuals from the features. Unlike a random forest, whose trees are independent, each new tree depends on all the trees before it.
Add it, scaled down
Add the new tree's output times a small factor, the learning rate or shrinkage (often around 0.1). Small steps limit overfitting. Repeat for many rounds.
Stop at the right time
Boosted trees can overfit as rounds are added. Watch the error on a validation set and stop when it stops improving (early stopping), and limit tree depth.
- Key paper
- Jerome Friedman, "Greedy function approximation: a gradient boosting machine", Annals of Statistics, 2001
- Popular tools
- XGBoost (2016), LightGBM (2017) and scikit-learn's histogram-based version
- Competition record
- The XGBoost paper reports it was used in 17 of 29 winning Kaggle solutions published in 2015
- Best at
- Tables of data; for images and text, neural networks usually do better
Questions people ask
What is gradient boosting in simple terms?
A way to build a model from many small decision trees added one after another. Each new tree is trained on the errors still left, and its correction is scaled down. Together they make accurate predictions.
What is XGBoost?
XGBoost is a popular open-source software library for gradient boosting, described in a 2016 paper by Tianqi Chen and Carlos Guestrin. It is the same idea, engineered to be fast and to scale to very large datasets. LightGBM is a similar library.
What is the difference between gradient boosting and random forest?
A random forest grows many deep trees independently and averages them. Gradient boosting grows shallow trees in sequence, each correcting the last. Boosting can overfit if it runs too long; a forest does not overfit as trees are added.
Why is it called gradient boosting?
Boosting is combining weak models in sequence so each fixes the previous ones' mistakes. Gradient, because each new tree follows the gradient of the loss, the direction that reduces the error fastest, just like gradient descent in neural networks.