In plain words
Many AI models end by giving a score to every possible answer. A photo app might score "cat" 2.0, "dog" 1.0, "bird" 0.1 and "car" minus 1. Those scores are useful for ranking, but they are not chances: they can be negative and they do not add up to anything.
Softmax is the small piece of maths that fixes this. It makes every score positive, then shares out 100% in proportion. Higher scores get a much bigger share, because softmax stretches the gaps: a score just one point higher ends up almost three times as likely.
The result reads like a forecast: 64% cat, 23% dog, 10% bird, 3% car. Nothing gets exactly zero, so the model always keeps a little doubt.
You meet softmax all over modern AI. It turns a chatbot's logits into odds for the next word, and inside a transformer it decides how much attention each word pays to every other word.
Everyday example
Sharing a prize pool
A quiz pays out ₹1,000 among four teams, but the organisers want the leader rewarded strongly. So each team's points are first blown up, the bigger the more so, and then the pool is split in proportion. The top team takes far more than its raw lead suggests, the last team still gets a few rupees, and the shares always total ₹1,000.
Where the example breaks: A real prize is money someone keeps. Softmax shares out belief, not rewards: the percentages are the model's confidence, which can be wrong, and a model can be 90% sure of a mistake.
How it actually works
For scores z1 to zK, softmax gives answer i the share e^zi ÷ (e^z1 + … + e^zK), where e is about 2.718. The outputs are positive and sum to exactly 1.
Exponentiate, then normalise
Raising e to each score makes everything positive and turns differences into ratios: two scores one apart end up about 2.7 times apart. Dividing by the total then scales them to add to 1. In the picture, e^2.0 is 7.39 and the four values total 11.58, so cat gets 64%.
A soft version of picking the maximum
Picking only the top score, called argmax, is all or nothing. Softmax is a smooth version of that choice, which is why some call it softargmax. Being smooth matters: training with gradient descent needs every small change in a score to change the output a little.
Temperature changes how sharp it is
Dividing all scores by a number before softmax changes the spread. A big divisor flattens the chances; a small one makes the top answer dominate. Chatbots expose this as the temperature setting.
Where it shows up
It is the usual last step of a classification network that picks one answer from many. In the 2017 transformer paper it appears twice: it turns attention scores into weights, and it turns the final output into next-token probabilities.
When not to use it
Softmax assumes exactly one answer is right. If a photo can contain both a dog and a cat, models use a separate yes or no score for each label instead. In practice, code also subtracts the largest score first so the numbers never overflow; the result is unchanged.
- Name coined
- By John S. Bridle, in two 1989 conference papers published in 1990
- Older roots
- The same formula is the Boltzmann distribution from physics, used by Ludwig Boltzmann in 1868
- Output
- One probability per option, each between 0 and 1, all adding to 1
- Used in
- Image classifiers, next-token prediction in chatbots, attention inside transformers
Questions people ask
What does softmax do in a neural network?
It converts the network's final raw scores into probabilities that add up to 1, one for each possible answer. The biggest score gets the biggest probability, and every option gets at least a sliver.
What is the difference between softmax and sigmoid?
Sigmoid squashes one score into a number between 0 and 1, good for a single yes or no question. Softmax handles a whole list of scores together, so the options compete and their chances add to 1. With two options, softmax gives the same answer as a sigmoid of the gap between the two scores.
What is the difference between logits and softmax?
Logits are the raw scores a model produces. Softmax is the function applied to them. Logits go in, probabilities come out. Adding the same number to every logit changes nothing after softmax, because only the gaps between scores matter.
Why is it called softmax?
Because it is a soft version of picking the maximum. A hard max gives the top score 100% and everything else zero. Softmax leans heavily towards the top score but leaves something for the others.
Does a softmax probability mean the model is right that often?
Not necessarily. A softmax output looks like a confidence level, but a 2017 study found modern neural networks are often poorly calibrated. Whether 80% really means right eight times in ten has to be measured separately, on test data.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change