In plain words
Your bank sends a message: did you just spend a large amount in another city? You did not. Something noticed that the payment did not look like you. That is anomaly detection.
The idea is simple. Instead of learning what fraud looks like, which is rare and keeps changing, the system learns what normal looks like from lots of everyday data: the usual amounts, times, places. Anything far from that pattern gets a high oddness score and is flagged for a closer look.
Because it does not need examples of every possible problem, it can catch new kinds of trouble nobody has seen before. It is used to spot card fraud, hacking attempts on networks, machines about to fail and errors in data. The catch: unusual is not always bad, so flags usually go to a person or a second check.
Everyday example
The watchman on your street
A society watchman knows the daily rhythm: the milk van at six, office cars leaving at nine, the vegetable seller at eleven. Nobody gave him a list of suspicious things. He just knows normal so well that a stranger trying gates at 2 a.m. stands out instantly. Anomaly detection learns that same sense of normal from data, then raises a hand when something does not fit.
Where the example breaks: The watchman understands why something is odd. A model only measures how far a point is from the usual pattern, so a harmless change, like your first trip abroad, can trigger it, while a careful fraudster who looks normal can slip past.
How it actually works
Google's glossary calls it "the process of identifying outliers": if a feature usually averages 100 with a standard deviation of 10, a value of 200 should be flagged as suspicious.
Mostly unsupervised
Labelled examples of anomalies are rare, so most methods are unsupervised: they model the bulk of the data and score how far each point sits from it. If some labelled cases exist, the problem can be treated as rare-class classification.
Statistics
The simplest approach: measure the usual average and spread of each feature, and flag values many standard deviations away.
Isolation forest
Build random trees that split the data at random points. Odd points get separated from the rest in very few splits, so a short average path means likely anomaly (Liu, Ting and Zhou, 2008).
Density and distance
Methods such as Local Outlier Factor flag points in much emptier neighbourhoods than their neighbours. Others, such as the one-class SVM, learn a boundary around normal data.
Tuning the alarm
Every method gives a score, and people choose the cut-off. Too sensitive means many false alarms; too relaxed means missed fraud. Precision and recall measure that balance.
- Also called
- Outlier detection; novelty detection when the training data is known to be clean
- Used for
- Fraud detection, network intrusion detection, equipment fault detection, system health monitoring
- Classic survey
- Chandola, Banerjee and Kumar, "Anomaly detection: a survey", ACM Computing Surveys, 2009
- Common methods
- Statistical thresholds, isolation forest, Local Outlier Factor, one-class SVM
Questions people ask
What is anomaly detection in simple terms?
Finding the odd ones out. A system learns the normal pattern in data, such as your usual spending, and flags anything that differs a lot from it, so a person or another check can look closer.
How is anomaly detection used in fraud detection?
The system learns each customer's usual amounts, places, times and devices. A payment that breaks that pattern, like a big purchase abroad at night, gets a high score and may be blocked or trigger a message asking you to confirm.
Is anomaly detection supervised or unsupervised?
Usually unsupervised, because examples of real anomalies are rare and new kinds keep appearing. Supervised methods exist too, but they are used less, because they need labelled examples of anomalies.
What is the difference between outlier detection and novelty detection?
In outlier detection, the training data already contains some odd points and the method must look past them. In novelty detection, the training data is known to be clean, and the question is whether a new point is unlike anything seen. Both are kinds of anomaly detection.