In plain words
A big chatbot writes one word at a time, and each word needs a full, slow pass through the whole model. That is a large part of why answers take time to appear.
Speculative decoding adds a small helper model that is much faster but less clever. The helper races ahead and guesses the next few words. The big model then checks all those guesses in a single pass, which takes about as long as writing one word itself.
Every guess the big model agrees with is kept. At the first wrong guess, the big model puts in its own word and the helper starts guessing again. Done carefully, the final text is the same kind the big model would have written alone, just faster.
Everyday example
A junior drafting, a senior signing off
In an office, a senior officer must approve every line of a routine letter. Writing it herself takes all day. Instead, a junior drafts five lines quickly. She reads them together: the first three are fine, the fourth is wrong, so she rewrites the fourth and hands it back. Most routine lines pass, so letters go out far faster, and every line still has her approval.
Where the example breaks: The senior officer reads lines one after another. The big model really does check all the guesses at the same moment, in parallel, which is where the saving comes from.
How it actually works
The idea was published by Google Research in November 2022, and independently by DeepMind in early 2023.
Why one word is slow
For each new token, the big model must read all of its billions of parameters from memory. Google's researchers note that big models are often held back by memory speed rather than arithmetic, leaving spare computing power. Checking several tokens at once reuses the same read, so it costs little more than producing one.
Draft
A much smaller model (the draft model) proposes the next few tokens, one after another. Because it is small, this is quick.
Verify in one pass
The big model processes the prompt plus all the drafted tokens together, and works out at each position what it would have picked. Thanks to the KV cache, the earlier text is not recomputed.
Accept or correct
Drafted tokens are kept up to the first one the big model rejects; that one is replaced by a token the big model picks. A special accept rule (keep each guess with a chance based on how the big model rates it, and draw any replacement from adjusted probabilities) makes the result match the big model's own probabilities exactly, so quality does not drop.
How much faster
Google's paper reported 2 to 3 times faster output on a large T5 model, and DeepMind 2 to 2.5 times on its 70 billion parameter Chinchilla. Gains are biggest on predictable text, where the draft guesses right more often. Google says it uses the technique to make AI Overviews in Search respond faster.
- First published
- November 2022, Yaniv Leviathan, Matan Kalman and Yossi Matias at Google Research
- Independent version
- "Speculative sampling", Charlie Chen and colleagues at DeepMind, February 2023
- Inspired by
- Speculative execution in computer chips, such as branch prediction
- Changes the answer?
- No, when done exactly: the output follows the big model's own probabilities
Questions people ask
Does speculative decoding make answers worse?
Not when it is done exactly as described in the original papers. The big model checks every word, and the acceptance rule keeps its output the same as it would have been. It only changes how fast the answer appears, not what the model would say.
What is a draft model?
A smaller, faster model, often from the same family as the big one, used only to guess upcoming words. Its guesses are suggestions; the big model has the final say. Some systems instead add small extra prediction parts to the big model itself.
Why does it help more on some text than others?
Speed depends on how often the draft guesses right. Predictable text, like code boilerplate or a common phrase, gives long runs of accepted guesses. Creative or unusual text gets more corrections, so the gain is smaller.
Is speculative decoding the same as a reasoning model thinking ahead?
No. A reasoning model writes extra hidden steps to think better. Speculative decoding is a speed trick inside inference: it changes nothing about how the model thinks, only how quickly its words are produced.