AppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Top-k and top-p sampling

Also called top p, nucleus sampling, top k

Top-k and top-p are settings that trim an AI's list of possible next words to the sensible ones, before it picks one at random.

Inside language modelsTechnical

In plain words

When a chatbot writes, it has a ranked list of possible next words, each with a chance. It usually picks one with a little randomness, so replies don't sound robotic. But the bottom of that list is full of odd words, and picking one by luck can derail a sentence.

Top-k and top-p cut off the unlikely tail before the pick.

Top-k keeps a fixed number of words: the top 40, say, every time.

Top-p keeps a share of the chance instead. With top-p 0.9, it keeps the top words until together they cover 90% of the chance. If the model is sure, that might be one word. If many words fit, it might be dozens. That flexibility is why top-p, also called nucleus sampling, is popular.

Everyday example

Picking a place for dinner

Friends vote on where to eat. A top-k rule says: always shortlist the top three, then draw one. A top-p rule says: shortlist places until they hold 90% of the votes. If 95% want the biryani place, the shortlist is just that one. If votes are split five ways, the shortlist grows to fit. Either way, the place with one stray vote never wins by luck.

Where the example breaks: Votes come from people with reasons. A model's chances come from patterns in its training text, so a trimmed list can still contain wrong words that merely sound likely.

Go deeper

How it actually works

Both are sampling filters. They act on the model's next-token probabilities, alongside temperature, and do not change the model itself.

  1. Start from the probabilities

    The model turns its logits into a probability for every token in its vocabulary, often tens of thousands. Temperature first sharpens or flattens that spread.

  2. Top-k: keep the k best

    Sort tokens by probability and keep only the top k. Their chances are rescaled to add up to 100%, then one is drawn. A k of 1 means always taking the top token, which is greedy decoding.

  3. Top-p: keep the smallest set that covers p

    Add up probabilities from the top until the total reaches p, for example 0.9, and drop the rest. The cut-off moves with the model's confidence, which a fixed k cannot do.

  4. Using both together

    Google's Gemini documentation describes applying top-k first, then top-p, then picking with temperature. OpenAI advises changing temperature or top-p, not both. As of November 2026, some of Google's newest Gemini models ignore custom values for all three.

Top-k sampling
Described in a 2018 Facebook AI Research story-writing paper that sampled from the 10 most likely words
Top-p (nucleus) sampling
Proposed in 2019 by researchers at the University of Washington and the Allen Institute for AI
Typical values
Top-p of 1 switches the filter off. OpenAI's API defaults top_p to 1
Why not just take the top word
Always picking the most likely word tends to produce dull, repetitive text

Questions people ask

What is the difference between top-k and top-p?

Top-k keeps a fixed number of candidate words, whatever the situation. Top-p keeps a fixed share of the probability, so it keeps fewer words when the model is confident and more when many words fit.

What is nucleus sampling?

Another name for top-p sampling. The "nucleus" is the small core of likely words that together cover the chosen share of probability. The rest, the unlikely tail, is dropped before the model picks.

Should I change temperature or top-p?

Usually just one. OpenAI's documentation recommends altering temperature or top-p, but not both. For most uses the defaults are fine. Lower values give steadier, more predictable text.

What does top-p of 1 mean?

It keeps every word in the list, so the filter does nothing. Values like 0.9 or 0.95 drop the least likely words. Very low values, like 0.1, leave only the top few and make replies very predictable.

Do top-k and top-p stop hallucinations?

No. They only stop the model picking very unlikely words. A wrong fact can be the most likely word if the model learned it badly or is guessing, so trimming the list does not make answers true. See hallucination.

Was this clear?

1 is confusing, 5 is crystal clear. One tap, no account needed.

Spotted something wrong, or know a better example? Suggest a change

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked November 2026.