In plain words
Ask one person to guess the weight of a pumpkin and they may be way off. Ask fifty people and average their guesses, and the average is usually closer than a typical single guess. Some guess too high, some too low, and the errors partly cancel.
Ensemble learning applies the same idea to AI. Instead of trusting one model, you train several and combine their answers: by voting, for a yes or no question, or by averaging, for a number.
The trick only works if the models make different mistakes. Ten copies of the same model would just repeat one error ten times. So ensembles add variety: each model sees a different random slice of the data, uses different settings, or is a different kind of model altogether.
Two famous recipes are bagging, where many models train side by side on random samples, as in a random forest, and boosting, where models train one after another, each fixing the last one's mistakes, as in gradient boosting.
Everyday example
Asking for directions in a new city
You are lost in an unfamiliar town. The first person you ask confidently points left. You ask four more: three point right, one points left. You go right, with the majority. Any one stranger could be wrong, but it is unlikely most of them are wrong in the same way. An ensemble of models takes the same vote.
Where the example breaks: Strangers on a street often share the same wrong idea, say a road that recently closed. Models trained on the same data can too, which is why ensembles work hard to make their members differ.
How it actually works
scikit-learn's guide puts it simply: ensemble methods combine the predictions of several models to improve generalisability and robustness over a single one.
Bagging: many models in parallel
Short for bootstrap aggregating, described by Leo Breiman in 1996. Each model trains on a different random sample of the data, and their predictions are averaged or voted. It mainly reduces variance, and works best with strong, flexible models such as deep decision trees.
Random forests
Breiman's 2001 random forest is bagging with extra randomness: each tree also considers only a random subset of features at each split. More variety, less correlated mistakes.
Boosting: models in sequence
AdaBoost, introduced by Freund and Schapire in 1995, trains simple models one after another, giving more weight to the examples the previous ones got wrong. Gradient boosting, described by Friedman in 2001, fits each new tree to the remaining errors. Boosting works best with weak, simple models.
Voting and stacking
A voting ensemble combines quite different models, such as a tree, a regression model and a neural network, by majority vote or by averaging their probabilities. Stacking goes further: a final model learns how to combine the others' predictions.
The cost
An ensemble needs more computing to train and to run, since every member must make a prediction. It is also harder to explain than a single model. So where speed or clear explanations matter, teams weigh that cost against the gain.
- Main recipes
- Bagging (parallel, cuts variance), boosting (sequential, cuts bias), stacking (a model combines models)
- Key papers
- Bagging, Breiman 1996; AdaBoost, Freund and Schapire 1995; random forests, Breiman 2001
- Famous win
- The Netflix Prize (US$1 million, September 2009) went to BellKor's Pragmatic Chaos, whose solution blended many models
- Origin of the boosting idea
- Kearns and Valiant asked whether many weak learners could make one strong one; Schapire showed they could in 1990
Questions people ask
What is the difference between bagging and boosting?
Bagging trains many models independently and in parallel on random samples, then averages them; it mainly reduces variance. Boosting trains models one after another, each focusing on the previous ones' errors; it mainly reduces bias.
Is a random forest an ensemble?
Yes. A random forest is an ensemble of decision trees built with bagging plus random feature choice. Each tree votes, or their probabilities are averaged, to give the final answer.
Why do ensembles work better than single models?
Each model makes some errors. If the models are varied, their errors are not all the same, so combining them cancels part of the noise. The ensemble keeps what the models agree on and smooths over where they individually stumble.
Are ensemble models always better?
No. They usually improve accuracy, but they cost more to train and run and are harder to explain. If the members are too similar, or all weak in the same way, combining them adds little.
What is stacking in machine learning?
Stacking feeds the predictions of several different models into one final model that learns how to weigh them. The final model is usually trained with cross-validation, so it learns from honest predictions rather than memorised ones.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change