AppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Batch normalisation

Also called batch normalization, batchnorm

Batch normalisation re-centres and rescales the numbers flowing between layers of a neural network, batch by batch, which makes deep networks train faster and more steadily.

How machines learnTechnical

In plain words

Inside a neural network, each layer passes a stream of numbers to the next. As training changes the earlier layers, those numbers drift: sometimes huge, sometimes tiny, sometimes all bunched off to one side. The next layer keeps having to adjust to a moving target.

Batch normalisation adds a small tidy-up step. A network trains on small groups of examples at a time, called batches. For each batch, the step shifts the numbers so their average is zero and their spread is one. Then it applies a stretch and a shift that the network learns for itself, so nothing useful is lost.

The result, first shown by two Google researchers in 2015, was that networks could train with bigger steps and far fewer rounds, and were less fussy about how they were set up. Exactly why it works so well is still argued about.

Everyday example

Rescaling marks across exam shifts

An exam is held in several shifts, and one shift got a harder paper. Comparing raw marks would be unfair. So each shift's marks are adjusted to the same average and the same spread before anyone compares them. Batch normalisation does this to the numbers inside a network, one batch at a time, so the next layer always gets them on a familiar scale.

Where the example breaks: Exam rescaling follows one fixed rule. Batch normalisation also learns its own final stretch and shift, and once training ends it uses averages saved from training instead of the current batch.

Go deeper

How it actually works

A batch normalisation layer usually sits between a layer's weighted sums and its activation function.

  1. Measure the batch

    For each feature channel, compute the mean and variance of its values across the examples in the current batch. These statistics change with every batch.

  2. Normalise

    Subtract the mean and divide by the square root of the variance (plus a tiny number to avoid dividing by zero). Each channel now has mean 0 and variance 1 within the batch.

  3. Learned scale and shift

    Multiply by a learned value gamma and add a learned value beta. These are ordinary parameters, trained with everything else, so the network can undo the normalisation where that helps.

  4. Different behaviour when used

    During training the layer keeps running averages of the means and variances. When the trained model is used, it normalises with those saved averages, so a single input gets the same answer whatever else is in the batch. Forgetting to switch modes is a classic bug.

  5. Limits and alternatives

    Because it depends on batch statistics, it works less well with very small batches and is awkward for recurrent networks. Layer normalisation, which normalises across one example's features instead, avoids this, and it is what the original transformer design uses.

Introduced
February 2015, by Sergey Ioffe and Christian Szegedy at Google
Headline result
The same accuracy on an image classifier with 14 times fewer training steps
Other benefits reported
Higher learning rates, less sensitivity to starting weights, a mild regularising effect, sometimes replacing dropout
Spelling
Batch normalization in American English; often shortened to batch norm or BN

Why does batch normalisation work?

One view

It steadies each layer's inputs

The original 2015 paper argued that layer inputs keep shifting as earlier layers learn, which it called internal covariate shift, and that fixing their average and spread removes this drag on training.

The other view

It smooths the path for training

A 2018 MIT study found that steadier inputs have little to do with its success. It showed that batch norm makes the error surface much smoother, so gradients are more reliable and bigger steps are safe.

Most likely: It clearly works, and the original explanation is probably not the whole story. The smoother-training account comes with direct experiments behind it, but the full picture is still being studied.

Questions people ask

What does batch normalisation do in simple words?

It keeps the numbers inside a neural network on a steady scale. For each batch of training examples, it shifts and rescales each layer's outputs to an average of zero and a spread of one, then applies a learned adjustment.

What is the difference between batch norm and layer norm?

Batch norm averages across the examples in a batch, for each feature. Layer norm averages across all the features of a single example. Layer norm does not depend on batch size, and the original transformer design uses it.

Does batch normalisation prevent overfitting?

It can help a little. Because each example is normalised together with whichever others share its batch, its output in training is never quite the same twice, which has a mild regularising effect. The original paper found it sometimes removed the need for dropout, but it is not mainly a cure for overfitting.

Why does batch norm behave differently during training and use?

In training it uses the current batch's averages. When the model is used, there may be only one input, so it switches to averages saved during training. That keeps answers consistent, but forgetting the switch gives wrong results.

Was this clear?

1 is confusing, 5 is crystal clear. One tap, no account needed.

Spotted something wrong, or know a better example? Suggest a change

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked November 2026.