In plain words
When a chatbot is about to write its next word, it does not pick from a short list. It gives a score to every word piece it knows, often more than 100,000 of them.
Those first scores are called logits. They are just numbers on a ruler: some big, some small, some below zero. A higher score means "more likely to fit here", but on their own they are not percentages and do not add up to anything in particular.
A final step called softmax turns the logits into proper chances that add up to 100%. The model then picks a word using those chances. Settings like temperature work by adjusting the logits just before that step.
Everyday example
Judges at a cooking contest
Three judges give raw points to four dishes: one dish gets 31, another 24, one gets a minus for being burnt. Raw points are hard to compare across contests. So the organisers convert them into shares of a prize pool that add up to 100%. Logits are the raw points; softmax is the conversion into shares.
Where the example breaks: Contest points are given by people with reasons. Logits come out of the model's maths, and a single logit means little alone: only the gaps between them matter.
How it actually works
"Logit" is the name for the last numbers a neural network produces before they are turned into probabilities.
One score per token in the vocabulary
At the end of a transformer, the final layer turns the model's internal vector into one number for every token in its vocabulary. A model with a 200,000 token vocabulary produces 200,000 logits for every single word it writes.
Softmax turns scores into chances
Softmax raises the number e (about 2.718) to the power of each logit, then divides each result by the total. Bigger logits grow much bigger, every result is positive, and they add to exactly 1. The Hugging Face course puts it plainly: logits are "the raw, unnormalized scores outputted by the last layer of the model".
Only the gaps matter
Adding the same amount to every logit changes nothing after softmax. What counts is the difference: a token whose logit is 1 higher than another's is about 2.7 times as likely.
Where logits get adjusted
Temperature divides all logits by a number before softmax, which sharpens or flattens the chances. Some APIs let developers add a bias to chosen tokens' logits to ban or encourage words; OpenAI's "logit_bias" accepts values from -100 to 100.
Also used in training
During training, the model's logits are compared with the true next token through a loss function called cross-entropy. In distillation, a small model learns from a big model's softened probabilities, made from its logits.
- Name comes from
- Statistics: Joseph Berkson coined "logit", short for logistic unit, in 1944
- In AI today
- Any raw score fed into softmax (or a similar function) is commonly called a logit
- Softmax formula
p = exp(z) / sum of exp(all z)- Seeing them
- Some APIs return log probabilities (logprobs) so developers can see how confident the model was
Questions people ask
What is the difference between logits and probabilities?
Logits are raw scores that can be any size, positive or negative. Probabilities are between 0 and 1 and add up to 1. Softmax converts logits into probabilities. The order stays the same: the highest logit always becomes the highest probability.
Why are they called logits?
The word comes from statistics, where a logit is the logarithm of the odds of something. Statistician Joseph Berkson named it in 1944. Machine learning borrowed the word for the raw scores before softmax, which work in a similar way.
Can logits be negative?
Yes. A negative logit just means a low score. After softmax it becomes a small but still positive probability. Only the differences between logits matter, not whether they are above or below zero.
How do logits relate to temperature?
Temperature divides every logit by the same number before softmax. Below 1, the gaps grow and the top word dominates. Above 1, the gaps shrink and less likely words get a real chance. The logits themselves come from the model; temperature only reshapes them.