In plain words
A model does not learn from its data in one go. It works through the examples a small bunch at a time, adjusting itself after each bunch. When it has gone through every example once, that is one epoch.
One pass is rarely enough. So training starts again from the top and goes through the same data a second time, a third time, and so on, getting a little better each round. The number of epochs is a setting people choose before training.
Too few epochs and the model has not learned enough. Too many and it starts memorising the examples instead of learning the pattern, which is overfitting. Engineers watch the results after each epoch to decide when to stop.
Everyday example
Revising a textbook before exams
A student reads the whole physics textbook once in October: that is one epoch. It helps, but plenty does not stick. She reads it again in December and once more in February, understanding more each time. Reading it twenty times in a row, though, she would start reciting pages without thinking, and that would not help with new questions.
Where the example breaks: The student's later readings build on understanding. A model on its second epoch simply sees the same examples again, usually in a shuffled order, and adjusts its numbers a little further.
How it actually works
Epoch, batch and iteration are three related counts in training.
Batch
Training data is split into small groups called batches. The model processes one batch, measures the loss and updates its weights.
Iteration
One update from one batch is an iteration. Google's glossary example: 1,000 examples in batches of 50 means 20 iterations per epoch.
Epoch
When every example has been used once, one epoch is complete. Data is usually shuffled between epochs so batches differ each time.
Choosing how many
The number of epochs is a hyperparameter. A common approach is to check a validation set after each epoch and stop when it stops improving, which is called early stopping.
Large language models
Language models train on such huge datasets that each piece of text may be seen only once or a few times. A 2023 study found that repeating data for up to about 4 epochs worked almost as well as fresh data, with diminishing returns beyond that.
- Definition
- One full pass over the entire training set, each example processed once
- Iterations per epoch
- Number of examples divided by batch size
- Type
- Hyperparameter, chosen by people
- Too many
- Risks overfitting; too few risks underfitting
Questions people ask
What is an epoch in simple terms?
One complete run through all the training data. If a model is trained for 10 epochs, it has seen every training example 10 times, each time adjusting itself to do a little better.
What is the difference between an epoch, a batch and an iteration?
A batch is a small group of examples. An iteration is one update of the model using one batch. An epoch is enough iterations to cover every example once. With 1,000 examples and batches of 50, one epoch is 20 iterations.
How many epochs should a model train for?
It depends on the model and data. Small models may train for tens or hundreds of epochs; huge language models may see their data only once or a few times. People watch validation results and stop when they stop improving.
Is an epoch in AI the same as Unix epoch time?
No. In computing, the Unix epoch is the starting point for clock time, 1 January 1970. In machine learning, an epoch is one pass through the training data. Same word, unrelated meanings.