Early stopping in AI: what it is, explained simply

In plain words

A model learns by going over its training data again and again. Each pass, called an epoch, it gets a little better on those examples. But after a point it starts memorising their quirks, and it gets worse on anything new. That is overfitting.

Early stopping watches for that moment. Alongside training, it keeps checking the model on a separate pile of examples it never trains on, called the validation set. While the validation score keeps improving, training continues.

When the validation score stops improving for a few rounds in a row, training stops, and the best version seen so far is kept. It is cheap, needs almost no change to how training works, and is one of the most common ways to keep models from overfitting.

Everyday example

Taking the chai off the stove

You keep the chai boiling to bring out the flavour, and every minute or so you taste a spoonful. It gets stronger and better, then one taste is slightly bitter. You take it off at its best, not when the timer says. Training is the boiling, the spoonful is the validation check, and stopping at the peak is early stopping.

Where the example breaks: Chai cannot be un-boiled. A model can: training saves a copy whenever the validation score hits a new best, so even after a few worse rounds it can go back to the best saved copy.

How it actually works

Early stopping treats the number of training steps as a setting to tune, and tunes it during a single training run.

Hold out a validation set

Split off part of the data that the model never learns from. After every epoch, or every few hundred steps, measure the loss on it. Training loss usually keeps falling, while validation loss falls, flattens, then rises again, making a U-shaped curve.

Save the best so far

Each time validation loss reaches a new low, save a copy of the model's parameters. Saving is occasional, so it barely slows training.

Wait with patience

Validation scores are noisy, so one bad round is not a reason to stop. A patience setting says how many rounds with no improvement to allow. A minimum change can be set so tiny gains do not count.

Stop and restore

When patience runs out, stop and reload the best saved copy. In tools like Keras this restore is a separate option that is off by default, a common source of surprise.

Why it counts as regularisation

Stopping early limits how far the model's settings can travel from where they started. In a simple linear model trained by gradient descent, this can be shown to be equivalent to L2 regularisation, a standard penalty on large weights. That is why it sits under regularisation.

Described as

"Probably the most commonly used form of regularization in deep learning" (Goodfellow, Bengio and Courville, 2016)

Watches

Usually validation loss; sometimes accuracy or another score

Key settings

Patience (rounds to wait) and minimum improvement

Trade-off

It needs a validation set, so slightly less data is left for training

Some teams then retrain on all the data, including the validation set, for the number of steps early stopping found best.

Questions people ask

Why do we stop training while the training loss is still going down?

Because training loss only measures the examples the model has already seen. Falling training loss with rising validation loss means memorising, not learning. What matters is how the model does on new data, and that is what the validation set stands in for.

What is patience in early stopping?

The number of checks in a row with no improvement that training will allow before stopping. With patience 5, five rounds without a new best end the run. Higher patience waits out noise but costs more time.

Is early stopping the same as regularisation?

It is usually counted as a form of regularisation, because it limits overfitting. Strictly it changes nothing about the model or its loss, it just ends training at a good moment. Google's glossary calls it not a formal regularisation method, but effective.

Can you use early stopping with the test set?

No. If the test set decides when to stop, it has influenced training and no longer gives an honest final score. Use a separate validation set, or a fold from cross-validation, to decide when to stop.