In plain words
Big AI models are clever but slow and expensive to run. Small models are quick and cheap, but trained on their own they are usually weaker. Distillation gets the best of both.
First a large "teacher" model is trained. Then a small "student" model is trained to copy the teacher's answers instead of learning only from the original data.
The trick is that the teacher gives richer answers than a plain label. Shown a photo of a car, a label just says "car". The teacher says "car, very likely; a bit like a truck; nothing like a carrot". Those odds tell the student how things relate, so it learns more from each example.
Many of the small, fast models in phones and apps were trained partly this way.
Everyday example
A senior cook teaching an apprentice
An apprentice could learn biryani only from recipe cards, which say what to do but not why. Working beside a senior cook is better. The cook says "this smell means the onions are nearly ready", "this rice is close, but a minute short". Those near-misses and judgements teach far more than the recipe alone. The apprentice ends up cooking much like the master, with far less experience.
Where the example breaks: The apprentice can eventually surpass the master. A distilled student usually stays a little below its teacher, and it also copies the teacher's mistakes and blind spots.
How it actually works
Geoffrey Hinton, Oriol Vinyals and Jeff Dean at Google named and popularised the method in March 2015, building on earlier model compression work.
Train or pick a teacher
The teacher is a large, accurate model, or even a group of models whose answers are averaged.
Collect soft targets
For each training example, the teacher's full set of probabilities is recorded, not just its top answer. A raised temperature spreads these probabilities out so that the small differences between wrong answers are visible. Hinton noted that the relative odds of the wrong answers tell the student a lot.
Train the student to match
The student is trained to produce the same probability spread as the teacher, often mixed with the true labels. Because each example carries more information, the student learns well from less data.
Distilling language models
With large language models, a common shortcut is to have the teacher write many answers, then fine-tune the student on that text. DeepSeek fine-tuned smaller Qwen and Llama models on 800,000 samples from its R1 reasoning model in January 2025.
- Key paper
- "Distilling the Knowledge in a Neural Network", Hinton, Vinyals and Dean, Google, March 2015
- Early result
- On a speech task, one distilled model kept more than 80% of the gain from an ensemble of 10 models (Hinton et al.)
- DistilBERT, 2019
- 40% smaller and 60% faster than BERT, keeping 97% of its language understanding
- In use
- Google trained its Gemma 2 2B and 9B models with distillation (2024); DeepSeek released distilled models from 1.5B to 70B parameters (2025)
Questions people ask
What is a distilled model?
A smaller model trained to imitate a larger one. It is cheaper and faster to run, often small enough for a laptop or phone, while keeping much of the larger model's ability. Names like "R1-Distill" signal this.
What is the difference between distillation and quantization?
Distillation trains a new, smaller model with fewer parameters. Quantization keeps the same model but stores each number with fewer bits. Both make models cheaper to run, and they are often combined.
What is the difference between distillation and fine-tuning?
Fine-tuning adapts a model to a task using some set of examples. Distillation is a specific goal: making a student copy a teacher. In language models, distillation is often done by fine-tuning the student on the teacher's outputs.
Is distilling someone else's model allowed?
It depends on the licence. Open models often allow it. Many commercial AI services forbid using their outputs to train competing models in their terms of use. Whether that is enforceable is a live legal question.