In plain words
Most yes or no models do not really answer yes or no. They give a score, like 0.83 for "this payment looks like fraud", and someone picks a cut-off. Set the cut-off low and you catch more fraud but annoy more honest customers. Set it high and you annoy fewer people but miss more fraud.
A ROC curve draws every possible cut-off at once. Across the bottom: the share of honest cases wrongly flagged, the false alarms. Up the side: the share of real cases caught. Each point on the curve is one cut-off.
A useless model, no better than flipping a coin, makes a straight diagonal line. A good model bends up towards the top-left corner: it catches most real cases while raising few false alarms.
AUC means area under the curve. A perfect model scores 1.0; coin flipping scores 0.5. It also has a neat meaning: pick one real case and one honest case at random, and AUC is the chance the model gave the real case the higher score.
Everyday example
A smoke alarm's sensitivity dial
Imagine a smoke alarm with a dial. Turned to maximum, it catches every fire but also screams every time someone fries pakoras. Turned to minimum, it never bothers you while cooking, but misses small fires. Now test it at every dial position, and plot fires caught against false alarms. That line is the alarm's ROC curve.
Where the example breaks: A smoke alarm's dial changes the hardware. In a model, the scores stay the same and only the cut-off moves, so the whole curve comes from one trained model.
How it actually works
ROC stands for receiver operating characteristic. The name comes from World War II radar, where it measured how well radar receiver operators told real aircraft from noise.
Two rates
True positive rate (TPR, also called recall or sensitivity) is TP ÷ (TP + FN): the share of real positives caught. False positive rate (FPR) is FP ÷ (FP + TN): the share of real negatives wrongly flagged. Both come from a confusion matrix.
Sweep the threshold
Sort the test set by model score. Start with the cut-off above every score: nothing flagged, point (0, 0). Lower it step by step; each step flags more cases. End with everything flagged, point (1, 1). Joining the points draws the curve.
Read the area
AUC is the area under that curve, from 0 to 1. The diagonal line is random guessing, AUC 0.5. As Fawcett's 2006 introduction puts it, AUC is the probability the model ranks a random positive above a random negative. The curve in our picture has an area of exactly 5/6, about 0.83.
Pick an operating point
AUC compares models; deployment still needs one cut-off. If false alarms are costly, choose a point further left, with a lower FPR. If misses are costly, as in disease screening, choose a point higher up.
When it misleads
ROC curves are not affected by how rare the positive class is, which is useful but can flatter a model when positives are very rare. For heavily imbalanced data, Google's course suggests a precision-recall curve may compare models better. See precision and recall.
- Name
- Receiver operating characteristic, from World War II radar signal analysis
- Spread to
- Psychology and signal detection, then medical diagnostic tests and radiology
- In machine learning
- K. A. Spackman was among the first to use ROC graphs to compare learning algorithms, in 1989
- Benchmarks
- AUC 1.0 is perfect, 0.5 is coin flipping; below 0.5 means the model is using its information backwards
Questions people ask
What is a good AUC score?
Higher is better, with 1.0 perfect and 0.5 no better than chance. What counts as good depends on the field and on how hard the problem is, so there is no single cut-off. Compare against existing models and simple baselines on the same data.
What does an AUC of 0.5 mean?
The model ranks cases no better than random. Pick one real positive and one real negative, and it gives the positive the higher score only half the time. Its ROC curve sits on the diagonal line.
What is the difference between a ROC curve and a precision-recall curve?
Both sweep the cut-off. ROC plots catch rate against false alarm rate. A precision-recall curve plots precision against recall. For very rare positives, Google's course suggests precision-recall curves often give a clearer comparison.
Why use AUC instead of accuracy?
Accuracy depends on one chosen cut-off and can look great when one class is rare. AUC judges the ranking across all cut-offs, so it says how well the model separates the two classes before anyone picks a threshold.
What does ROC stand for?
Receiver operating characteristic. The method was first used to analyse radar signals in World War II, measuring how well a radar receiver's operator could tell real targets from noise. The name stuck when it spread to medicine and machine learning.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change