Perplexity score in AI: what it means, explained

In plain words

A language model's whole job is guessing the next word. Perplexity checks how good those guesses are on text the model has never seen.

Picture the model reading a real sentence one word at a time. Each time, before it sees the next word, it has spread its bets over possible words. If it put a big bet on the word that actually came, it was not surprised. If it put a tiny bet, it was very surprised. Perplexity averages that surprise over the whole text.

The number has a friendly meaning: it is how many choices the model was effectively torn between, on average. A perplexity of 1 means it was always sure and right. A perplexity of 50 means it was as unsure as if picking from 50 equally likely words. Lower is better.

Everyday example

Finishing your mother's sentences

When your mother says "Did you eat your...", you know the next word is almost surely "lunch" or "dinner". You are choosing between about two words. A new neighbour hearing her might be torn between ten. You know her better, so your perplexity is lower. Language models are scored the same way, on real text.

Where the example breaks: Knowing someone's words is not the same as being helpful. A model can have a low perplexity and still give wrong or unhelpful answers, because perplexity only scores word guessing.

How it actually works

Perplexity comes from information theory. It was proposed for speech recognition at IBM in 1977 and became a standard way to test language models.

Score every real next word

Run the model over a test text. At each position, record the probability it gave the word that really came next. These come from its logits after the softmax step.

Average the surprise

Take the negative log of each probability, which is the model's surprise, and average it. This is the same quantity the model minimises as its loss in training.

Turn it back into a count

Raise e (or 2) to that average. The result is the effective number of choices. Older word-count models on Wall Street Journal text scored about 962 using single words, 170 using word pairs and 109 using triples.

Compare with care

Perplexity depends on the test text and on how text is split into tokens, so only compare models measured the same way. It suits left-to-right models like GPT, not fill-in-the-blank models like BERT.

First proposed

1977, by Jelinek, Mercer, Bahl and Baker at IBM, to measure how hard a speech recognition task is

Best possible

1: the model gave every real next word 100% chance

Random guessing

Equal to the number of options. Guessing evenly among 3 colours gives perplexity 3

A known limit

A better perplexity does not guarantee better results on real tasks like translation, says the standard textbook

Questions people ask

Is perplexity the score the same as Perplexity the app?

No. This page is about a number that scores language models. Perplexity is also the name of an AI search app. Same word, two different things: one is a measurement, the other is a product you can use.

Is a lower perplexity better?

Yes. Lower perplexity means the model gave higher probability to the words that really came, so it was less surprised by real text. Compare only models tested on the same text with the same kind of tokenization.

What is a good perplexity score?

There is no universal good number. It depends on the test text and the tokenizer. Simple, repetitive text gives low scores and varied text gives higher ones. The useful question is whether a model beats another on the same test.

Why don't chatbot makers report perplexity much?

Because users care about helpful, correct answers, and perplexity only measures word prediction. It stays useful while training a model, since it is just its loss in another form, but finished chatbots are judged with task benchmarks and human ratings.

Perplexity (the score)