In plain words
A flexible model can fit its training examples perfectly, quirks and all, and then stumble on anything new. That problem is overfitting. Regularisation is the name for everything we do to hold the model back from that kind of memorising.
The classic method adds a penalty. Normally training only tries to reduce the error on the examples. With regularisation, it also pays a cost for using large internal numbers, its weights. Large weights are what let a model twist itself around every odd point, so the penalty nudges it towards simpler, smoother answers.
There are many other kinds: randomly switching off parts of a network during training (dropout), stopping training early, or creating extra varied examples. All of them trade a slightly worse fit on the practice data for a better fit on real data.
Everyday example
Packing light
Packing for a trip, you could bring something for every possible situation: three umbrellas, five chargers. A strict baggage limit forces you to pack what you will really need. Regularisation is that baggage limit. The model may still use whatever it needs, but every extra bit of complexity has a cost, so it keeps only what earns its place.
Where the example breaks: A baggage limit is a fixed rule. The strength of a regularisation penalty is a setting people tune: too strict and the model becomes too simple to learn the pattern, which is underfitting.
How it actually works
Goodfellow, Bengio and Courville define regularisation as any change to a learning algorithm "intended to reduce its generalization error" but not its training error. American texts spell it regularization.
Add a penalty to the loss
Training minimises error + λ × complexity. The regularisation rate λ (lambda) is a hyperparameter: higher means simpler models and less overfitting, too high means underfitting.
L2: shrink every weight
The penalty is the sum of the squared weights. It pulls all weights towards zero without making them exactly zero. Also called weight decay, ridge regression or Tikhonov regularisation.
L1: switch some weights off
The penalty is the sum of the weights' absolute sizes. It tends to push many weights to exactly zero, which also picks out the features that matter. With a linear model this is called the lasso.
Early stopping
Watch the loss on a validation set and stop when it starts rising. Goodfellow et al. call it probably the most common form of regularisation in deep learning.
Noise and variety
Dropout randomly turns off units in a neural network during training. Data augmentation adds flipped, cropped or shifted copies of examples. Both make memorising harder.
- Purpose
- Better results on new data, at the cost of a slightly worse fit on training data
- Main kinds
- L2 (weight decay), L1 (lasso), dropout, early stopping, data augmentation
- Key setting
- The regularisation rate λ: too low overfits, too high underfits
- Spelling
- Regularisation (British), regularization (American): the same thing
Questions people ask
What is regularization in simple terms?
A way to stop a model from memorising its training data. Usually it adds a cost for complexity, such as large weights, so the model prefers simpler patterns that also work on new data.
What is the difference between L1 and L2 regularization?
L2 penalises squared weights and shrinks them all a little. L1 penalises their absolute size and tends to set many weights to exactly zero, leaving a sparser model that uses fewer features.
Is dropout a type of regularization?
Yes. Dropout randomly switches off parts of a neural network during training so no part can rely too heavily on another. It reduces overfitting without adding a penalty term to the loss.
Can regularization cause underfitting?
Yes. If the penalty is too strong, the model is forced to be too simple and misses the real pattern, doing badly even on its training data. The strength is tuned on validation data.