In plain words
A model in training needs to know how badly it is doing. Not a vague feeling: a single number. The loss function is the rule that turns its mistakes into that number, called the loss.
Compare each guess with the right answer and measure the gap. A tiny gap adds a little to the loss, a big gap adds a lot. Add it all up and you have the score. Zero would mean every answer was perfect.
Everything else in training exists to make that number smaller. Gradient descent adjusts the model a little in whichever direction lowers the loss, then checks again. So choosing the loss function quietly decides what the model treats as a serious mistake.
Everyday example
Guessing the weight of a pumpkin at a fair
At a village fair, people guess a pumpkin's weight. The organiser charges a forfeit for each miss: off by 1 kilo costs 1 rupee, off by 2 costs 4, off by 3 costs 9. Small slips are cheap, wild guesses are costly. The rule for the forfeit is the loss function, and the total collected is the loss.
Where the example breaks: At the fair, everyone guesses once. A model gets its loss after every batch of examples, millions of times, and changes itself after each one to bring the number down.
How it actually works
Formally, a loss function takes the model's predictions and the true labels and returns a number that training tries to minimise.
Mean squared error
For numbers, as in regression, the classic choice squares each gap and averages them:
MSE = average of (answer − guess)². Google's glossary example: gaps of 1, 1, 3, 2 and 1 give squares adding to 16; averaged over the 5 examples, that is 16 ÷ 5 = 3.2.Absolute error
Mean absolute error skips the squaring. Because squares grow fast, MSE punishes big misses and outliers far more heavily, which may or may not be what you want.
Cross-entropy for categories
For classification, the model outputs a probability for each class. Cross-entropy loss is small when the model gave high probability to the right class and very large when it was confidently wrong. Language models train on this, scoring the probability they gave to the real next token.
The loss curve
Plotting loss over training shows progress. It should fall and then flatten, which is called convergence. If loss on held-back data starts rising while training loss keeps falling, the model is overfitting.
Loss is not the goal itself
Loss is a stand-in for what we really want, such as useful answers. A model can lower its loss in ways people did not intend, which is why it is checked with other measures on a test set.
- Also called
- Cost function, objective, error function
- For numbers
- Mean squared error, mean absolute error
- For categories and next words
- Cross-entropy (log loss)
- Who uses it
- Training: gradient descent and backpropagation follow it downhill
Questions people ask
What is a loss function in simple words?
A rule that scores how wrong a model's predictions are, as one number. Training repeatedly adjusts the model to make that number smaller, so the loss function defines what getting better means.
What is the difference between loss and accuracy?
Accuracy counts how many answers were right. Loss also measures how wrong or how confident each answer was, and it changes smoothly, which is what training needs. A model can keep the same accuracy while its loss improves.
Why square the errors?
Squaring makes every error positive and makes big errors count far more than small ones: off by 3 costs 9, not 3. It also makes the maths of finding the minimum convenient.
What loss function does ChatGPT use?
During pre-training, large language models use cross-entropy loss on next-token prediction: the loss is low when the model gave high probability to the word that actually came next. Later tuning stages add other objectives, such as reward scores in RLHF.