In plain words
Think of a flowchart. Is it raining? Take an umbrella. Not raining? Then is it sunny? Each question splits the possibilities until you land on an answer. A decision tree is that kind of flowchart, except the computer writes the questions itself by studying past examples.
During training it looks at many cases where the right answer is already known and picks the one question that best separates them. Then it does the same inside each branch, asking the next most useful question, until each group mostly shares one answer.
To make a prediction, a new case starts at the top and follows the yes and no branches down to the end, called a leaf, which holds the answer. You can read every step, so trees are easy to explain. Their weak spot: a tree grown too deep memorises its examples, which is overfitting.
Everyday example
Picking a ripe mango
At the fruit stall you press a mango. Soft? Then you smell it. Sweet? Buy it. Not sweet? Probably not ready. Hard? Put it back. That is a decision tree: a few questions in a fixed order, each answer sending you down one branch, ending in a decision. A machine learns the same kind of questions from thousands of mangoes someone has already tasted.
Where the example breaks: Your mango questions come from experience and common sense. A tree picks its questions only by which split best separates the examples it was shown, so with too few or odd examples it can learn questions that make no sense.
How it actually works
scikit-learn describes decision trees as a supervised learning method for both classification and regression that learns "simple decision rules inferred from the data features".
Find the best question
For every feature and every possible cut-off ("income above 50,000?"), measure how mixed the answers would be on each side, using a score such as Gini impurity or entropy. Keep the question that leaves the two groups purest.
Repeat inside each branch
Split each group again, and again. This is called recursive partitioning. Each choice is only the best at that moment, so the method is greedy: it cannot guarantee the best possible tree overall.
Answer at the leaves
A leaf holds the most common answer among its training examples (for categories) or their average (for numbers). So a tree's predictions jump in steps rather than changing smoothly.
Stop it memorising
Left alone, a tree keeps splitting until every example sits in its own leaf. Limits such as a maximum depth, a minimum number of examples per leaf, or pruning branches afterwards keep it general.
Better in groups
A small change in the data can produce a very different tree. Combining many trees fixes much of this: a random forest averages many varied trees, and gradient boosting adds trees that fix each other's mistakes.
- Used for
- Classification (which category) and regression (which number)
- Classic methods
- CART (Breiman and colleagues, 1984), ID3 (Ross Quinlan, 1986) and its successor C4.5
- In practice
- scikit-learn, a popular Python library, uses an optimised version of CART
- Strength
- A "white box": every prediction can be traced question by question
- Weakness
- Overfits easily and can change a lot when the data changes a little
Questions people ask
What is a decision tree in simple terms?
A flowchart of yes or no questions that ends in an answer. In machine learning the computer chooses the questions and their order by studying examples with known answers, then uses the flowchart to answer new cases.
Is a decision tree supervised or unsupervised learning?
Supervised. It learns from labelled examples, cases where the right answer is already known, and picks questions that separate those answers. It can predict a category, such as spam or not spam, or a number, such as a price.
What is the difference between a decision tree and a random forest?
A random forest is many decision trees, each trained on a random sample of the data and features, whose answers are combined. One tree is easier to read; a forest is usually more accurate and more stable.
What are the disadvantages of decision trees?
A deep tree overfits, memorising its training examples. Small changes in the data can produce a completely different tree. Predictions move in steps, so trees are poor at extending a trend beyond the data they saw.
Why are decision trees still used when neural networks exist?
They are fast, need little data preparation and are easy to explain. For tables of data, such as spreadsheets of customers or transactions, groups of trees are often as accurate as neural networks or better.