A large model that has seen millions of examples knows more than the label of each training point. It knows which wrong answers are almost right, which classes are easily confused, and how confident it should be in each case. Knowledge distillation is the technique that transfers that extra knowledge into a smaller model, so the student can run where the teacher cannot: on a phone, inside a latency budget, or at a fraction of the serving cost.
The idea, popularised by Hinton and co-authors in 2015, is simple to state and surprisingly deep in its consequences. Train the small model not only on the hard labels, but also on the teacher's full probability distribution over classes. Those soft targets carry information that a single correct label cannot.
What a soft label actually contains
Consider an image classifier trained on handwritten digits. For a particular image of a 7, the hard label says "7" and nothing else. The teacher's output might say: 0.92 for 7, 0.06 for 1, 0.01 for 9, and near zero for the rest. That 0.06 for 1 is not noise. It says that this 7 looks somewhat like a 1, and that the boundary between the two classes is close here. Across a whole dataset, these relative probabilities encode the similarity structure of the classes, sometimes called the dark knowledge of the teacher.
A student trained only on hard labels sees a one-hot vector at every example. It must infer the class geometry from the inputs alone. A student trained on soft labels is handed that geometry directly, at every step, for free. This is why a distilled student can often match or even exceed the accuracy of a model of the same size trained from scratch on the same data, and why it can sometimes approach the teacher's accuracy with an order of magnitude fewer parameters.
The distillation loss
The standard formulation uses a temperature parameter T to soften the teacher's logits before the softmax. For a logit vector z, the softened probability of class i is
At T = 1 this is the ordinary softmax. As T grows, the distribution flattens and the differences between the small probabilities become visible. At very high T, the softened probabilities approach a uniform distribution, and the relative ordering of the logits is all that remains.
The student is trained on a weighted combination of two losses. The first is the usual cross-entropy with the hard label y. The second is the Kullback-Leibler divergence between the teacher's softened distribution and the student's softened distribution, both computed at the same temperature T:
Here z_t and z_s are the teacher and student logits, and sigma denotes the softmax. The T squared factor is not cosmetic. The gradient of the KL term with respect to the student logits scales roughly as 1 / T squared, so multiplying by T squared keeps the two terms on a comparable scale as the temperature changes. Without it, raising the temperature would silently shrink the distillation signal.
In practice, alpha is often set around 0.1 to 0.3, so most of the weight sits on the teacher, and T is typically between 2 and 5. Both are worth tuning: a temperature that is too low hides the dark knowledge, and one that is too high washes out the signal entirely.
Why it works: three views
It is a better regulariser than label smoothing. Label smoothing replaces the one-hot target with a mixture of the true label and a uniform distribution. Distillation replaces it with the teacher's distribution, which is a data-dependent smoothing. The student is told not merely to be less confident, but to be uncertain in exactly the directions the teacher finds confusing. That is a strictly more informative target.
It is function matching, not just label matching. The KL term asks the student to reproduce the teacher's function across the training distribution, not just its argmax. Matching a function is a stronger constraint than matching its predictions, and it transfers the teacher's inductive bias: the smoothness of its decision boundaries, the shape of its confidence, and the invariances it has learned. This is also why distillation can work when the student never sees a hard label at all, as long as it sees enough teacher outputs.
It reduces the variance of the training signal. Hard labels are noisy estimates of the true conditional distribution. The teacher's soft outputs are a smoothed, lower-variance estimate of the same quantity, averaged over the teacher's own training. Fitting a smoother target is an easier optimisation problem, and the student converges faster and generalises better, especially on small datasets.
A minimal implementation
The loss is only a few lines once the teacher is frozen. The key detail is to compute the KL term on log-softmax outputs at temperature T, and to multiply by T squared:
import torch import torch.nn.functional as F def distillation_loss(student_logits, teacher_logits, targets, T=4.0, alpha=0.2): # Hard-label loss at temperature 1 ce = F.cross_entropy(student_logits, targets) # Soft-label loss at temperature T log_p_s = F.log_softmax(student_logits / T, dim=1) p_t = F.softmax(teacher_logits / T, dim=1) kl = F.kl_div(log_p_s, p_t, reduction="batchmean") return alpha * ce + (1 - alpha) * (T ** 2) * kl # Teacher is frozen and run in evaluation mode teacher.eval() with torch.no_grad(): teacher_logits = teacher(inputs) student_logits = student(inputs) loss = distillation_loss(student_logits, teacher_logits, targets) loss.backward() A common mistake is to forget the no-grad context around the teacher, which wastes memory and lets gradients flow into a model you never intend to update. Another is to apply the temperature to the cross-entropy term as well; the hard-label loss should always use T = 1.
Beyond logits: feature and self-distillation
Logit matching is only the simplest form. Feature distillation matches intermediate representations, for example by asking the student's hidden layer to predict the teacher's hidden layer through a small projection. This transfers more of the teacher's internal structure, and it matters most when the student architecture differs from the teacher, so the logits alone underdetermine what the student should learn.
Self-distillation drops the separate teacher altogether: a model is trained, then used as its own teacher in a second round, or a deeper network teaches its own shallower layers. The gains are smaller than with a strong external teacher, but the technique still helps, largely because the soft targets act as a regulariser and because the second round sees a smoother, less noisy version of the training signal.
Born-again networks take this to an extreme: a sequence of models of the same size, each distilled from the previous one, can outperform the original. The improvement comes from the same mechanism, the soft targets carry information that the hard labels alone do not, and each generation refines the estimate.
Limitations and common pitfalls
Distillation is not a free lunch, and its failures are instructive.
The teacher's errors are inherited. If the teacher is systematically wrong on some slice of the data, the student learns those errors with high confidence, because the soft targets present them as ground truth. Distillation amplifies teacher bias. Check the teacher's per-class accuracy before distilling, and consider filtering or down-weighting examples where the teacher disagrees strongly with the label.
The capacity gap can bite. A student that is far smaller than the teacher may simply lack the capacity to represent the teacher's function, and forcing it to match soft targets it cannot express can hurt more than it helps. Teacher assistants, intermediate-size models distilled first and then used as teachers, are the standard workaround.
Temperature and alpha interact. A high temperature with a low alpha can leave the hard labels dominating, while a low temperature with a high alpha makes the KL term nearly vacuous. Tune them together, not one at a time, and validate on held-out data rather than on the teacher's training accuracy.
Calibration is not guaranteed. A distilled student inherits the teacher's confidence profile, including its overconfidence. If calibrated probabilities matter for your application, measure the student's calibration separately and apply temperature scaling or another recalibration step after distillation.
Distillation needs data the teacher is good on. The technique transfers what the teacher knows about the training distribution. If you distil on data far from the teacher's training distribution, the soft targets are unreliable and the student learns noise. Unlabelled data from the same distribution is ideal; out-of-distribution data is a trap.
When to use it
Distillation is the right tool when a large model exists that performs well but cannot be deployed: too slow, too large, too expensive to serve. It is also useful when labelled data is scarce but a strong teacher and plentiful unlabelled data are available, and when you want a small model that inherits a large model's robustness rather than its raw accuracy alone.
It is the wrong tool when no good teacher exists, when the student must learn something the teacher does not know, or when the deployment constraint is memory rather than compute and a simpler compression method such as quantisation or pruning would do. Distillation, pruning, and quantisation compose well, and the best small models usually combine all three.
Takeaway
Hard labels tell a model what is correct. Soft labels tell it how the world is structured: which classes are neighbours, where the boundaries are fuzzy, and how confident to be. Knowledge distillation works because it transfers that structure, not just the answers, and it does so through a loss that is simple enough to write in a few lines and rich enough to close much of the gap between a large teacher and a small student. The student does not need to be as big as the teacher. It needs to be taught what the teacher actually knows.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
