What is model distillation, and why does it make small models smarter?

Large models are expensive to run, and many tasks do not need all their power. Distillation is the standard way to pass what a big model knows into a small one. The big model is the teacher; the small one is the student. Instead of learning only from the original data, the student learns to reproduce what the teacher does.

The idea is older than chatbots. Geoffrey Hinton, Oriol Vinyals and Jeff Dean gave it its name in a 2015 paper, Distilling the Knowledge in a Neural Network, and used it to shrink speech-recognition models used at very large scale. In October 2026 it sits behind many of the small and mid-size models people run on their own computers.

Why a teacher's answers are better than raw data

Hinton's key observation is about what a trained model "knows" beyond the right answer. Show an image classifier a photo of a BMW, and it gives a high probability to "car", a tiny probability to "garbage truck", and an even tinier one to "carrot". The correct label alone says only "car". The teacher's full set of probabilities, which the paper calls soft targets, also says that a car looks more like a truck than like a vegetable. That extra information is how the teacher generalizes, and a student trained on it learns faster and from fewer examples.

To make those tiny probabilities visible, the paper raises a setting called temperature while producing the teacher's outputs, which spreads probability more evenly across answers. The student trains at the same high temperature, then runs normally once training is done. The authors found it worked best to combine the soft targets with the true labels, giving the labels a smaller weight.

How it works for language models today

With chatbots, the most common form is simpler: the teacher writes complete answers, and the student is fine-tuned on them. DeepSeek's R1 release in January 2025 is a clear, documented case. The company took about 800,000 examples curated with its large reasoning model and used them to fine-tune six smaller open models, from 1.5 to 70 billion parameters, based on Qwen and Llama. It reported that passing the large model's reasoning patterns down in this way gave better results than training the small models with reinforcement learning on their own.

This explains why distilled models punch above their weight. The teacher's answers are consistent, well formatted and show their reasoning step by step, while ordinary web text is messy and rarely explains itself. The student copies a polished way of working rather than discovering it from scratch.

A worked example: distilling for one task

Say you need a small local model that classifies support emails into six categories with a one-line reason.

1. Collect 1,000 to 2,000 real emails, with personal details removed.
2. Ask a strong teacher model to label each one in a fixed format, and spot-check a sample by hand.
3. Drop or fix the answers you disagree with; the student will copy mistakes faithfully.
4. Fine-tune a 3 to 8 billion parameter student on the cleaned pairs, as described in the guide to fine-tuning a small model.
5. Test teacher and student on 100 emails neither has seen, and compare accuracy, speed and cost.

When can you distill a model for your own use, and what do the terms of service allow?

Before step 2, check whether the teacher's owner allows it. There are three common situations.

Open models with permissive terms. Some open-weight models say yes explicitly. DeepSeek states that its R1 series allows commercial use, any modification and derivative works, including distillation to train other language models. Its own distilled versions show another rule worth knowing: each student also keeps the licence of the model it was built on, Apache 2.0 for the Qwen-based ones and Meta's Llama licences for the others.

Open models with custom licences. Some community licences limit using a model or its outputs to improve other large language models. Read the licence and any linked use policy before you start; the companion article on open-weight licences explains what to look for.

Closed models through an API or app. Here the terms of service decide. Anthropic's commercial terms, for example, say customers own their outputs, but they also forbid using the service to build a competing product, including training competing AI models, without Anthropic's express approval. Distilling a closed model into a small private classifier that does not compete with the provider is a different situation from building a rival general assistant, but the wording varies by company and plan, so read the current terms for the exact service you use.

When in doubt, the safest teachers are open models whose licence explicitly permits distillation, or your own labelled data checked by a person. For anything you will sell, get legal advice on the specific terms.