Inmachine learning,knowledge distillation ormodel distillation is the process of transferring knowledge from a largemodel to a smaller one. While large models (such as verydeep neural networks orensembles of many models) have more knowledge capacity than small models, this capacity might not be fully utilized. It can be just as computationally expensive to evaluate a model even if it utilizes little of its knowledge capacity. Knowledge distillation transfers knowledge from a large model to a smaller one without loss ofvalidity. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware (such as amobile device).^[1]

There is also a less common technique calledReverse Knowledge Distillation, where knowledge is transferred from a smaller model to a larger one.^[2]

Model distillation is not to be confused withmodel compression, which describes methods to decrease the size of a large model itself, without training a new model. Model compression generally preserves the architecture and the nominal parameter count of the model, while decreasing the bits-per-parameter.

Knowledge distillation has been successfully used in several applications of machine learning such asobject detection,^[3]acoustic models,^[4] andnatural language processing.^[5]Recently^[when?], it has also been introduced tograph neural networks applicable to non-grid data.^[6]

Knowledge transfer from a large model to a small one somehow needs to teach the latter without loss of validity. If both models are trained on the same data, the smaller model may have insufficient capacity to learn aconcise knowledge representation compared to the large model. However, some information about a concise knowledge representation is encoded in thepseudolikelihoods assigned to its output: when a model correctly predicts a class, it assigns a large value to the output variable corresponding to such class, and smaller values to the other output variables. The distribution of values among the outputs for a record provides information on how the large model represents knowledge. Therefore, the goal of economical deployment of a valid model can be achieved by training only the large model on the data, exploiting its better ability to learn concise knowledge representations, and then distilling such knowledge into the smaller model, by training it to learn thesoft output of the large model.^[1]

Given a large model as a function of the vector variable${\displaystyle \mathbf{x}}$, trained for a specificclassification task, typically the final layer of classification networks is asoftmax in the form

${\displaystyle y_i(\mathbf{x}|t)=\frac{e^{\frac{z_i(\mathbf{x})}{t}}}{\sum _j{e}^{\frac{z_j(\mathbf{x})}{t}}}}$

where${\displaystyle t}$ is thetemperature, a parameter which is set to 1 for a standard softmax. The softmax operator converts thelogit values${\displaystyle z_i(\mathbf{x})}$ to pseudo-probabilities: higher temperature values generate softer distributions of pseudo-probabilities among the output classes. Knowledge distillation consists of training a smaller network, called thedistilled model, on adata set called thetransfer set which could correspond to the original training set or consist of new, possibly unlabeled data. Across-entropyloss function is typically used, computed between the output of the distilled model${\displaystyle \mathbf{y}(\mathbf{x}|t)}$ and the output of the large model${\displaystyle \hat{\mathbf{y}}(\mathbf{x}|t)}$ on the same record (or the average of the individual outputs, if the large model is an ensemble), using a high value of softmax temperature${\displaystyle t}$ for both models:^[1]

${\displaystyle E(\mathbf{x}|t)=-\sum _i{\hat{y}}_i(\mathbf{x}|t)\log y_i(\mathbf{x}|t).}$

In this context, a high temperature increases theentropy of the output, therefore providing more information to learn for the distilled model compared to hard targets, and at the same time reducing thevariance of thegradient between different records, thus allowing a higherlearning rate.^[1]

If ground truth is available for the transfer set, the process can be strengthened by adding to the loss the cross-entropy between the output${\displaystyle y_i(\mathbf{x}|1)}$ of the distilled model computed with${\displaystyle t=1}$, and the known label${\displaystyle {\overline{y}}_i}$

${\displaystyle E(\mathbf{x}|t)=-t^2\sum _i{\hat{y}}_i(\mathbf{x}|t)\log y_i(\mathbf{x}|t)-\sum _i{\overline{y}}_i\log y_i(\mathbf{x}|1)}$

where the component of the loss with respect to the large model is weighted by a factor of${\displaystyle t^2}$ since, as the temperature increases, the gradient of the loss with respect to the model weights scales by a factor of${\displaystyle \frac{1}{t^2}}$.^[1]

Under the assumption that the logits have zeromean, it is possible to show that model compression is a special case of knowledge distillation. The gradient of the knowledge distillation loss${\displaystyle E}$ with respect to the logit of the distilled model${\displaystyle z_i}$ is given by

where${\displaystyle {\hat{z}}_i}$ are the logits of the large model. For large values of${\displaystyle t}$ this can be approximated as

${\displaystyle \frac{1}{t}\left(\frac{1+\frac{z_i}{t}}{N+\sum _j\frac{z_j}{t}}-\frac{1+\frac{{\hat{z}}_i}{t}}{N+\sum _j\frac{{\hat{z}}_j}{t}}\right)}$

and under the zero-mean hypothesis${\displaystyle \sum _j{z}_j=\sum _j{\hat{z}}_j=0}$ it becomes${\displaystyle \frac{z_i-{\hat{z}}_i}{N{T}^2}}$, which is the derivative of${\displaystyle \frac{1}{2}{\left(z_i-{\hat{z}}_i\right)}^2}$, i.e. the loss is equivalent to matching the logits of the two models, as done in model compression.^[1]

The Optimal Brain Damage (OBD) algorithm is as follows:^[7]

Do until a desired level of sparsity or performance is reached:

Train the network (by methods such as backpropagation) until a reasonable solution is obtained

Compute the saliencies for each parameter

Delete some lowest-saliency parameters

Deleting a parameter means fixing the parameter to zero. The "saliency" of a parameter${\displaystyle \theta }$ is defined as${\displaystyle \frac{1}{2}({\mathrm{\partial }}_{\theta }^{2}L){\theta }^2}$, where${\displaystyle L}$ is the loss function. The second-derivative${\displaystyle {\mathrm{\partial }}_{\theta }^{2}L}$ can be computed bysecond-order backpropagation.

The idea for optimal brain damage is to approximate the loss function in a neighborhood of optimal parameter${\displaystyle {\theta }^{\ast }}$ byTaylor expansion:$${\displaystyle L(\theta )\approx L({\theta }^{\ast })+\frac{1}{2}\sum _i({\mathrm{\partial }}_{{\theta }_i}^{2}L({\theta }^{\ast }))({\theta }_i-{\theta }_i^{\ast }{)}^2}$$where${\displaystyle \mathrm{\nabla }L({\theta }^{\ast })\approx 0}$, since${\displaystyle {\theta }^{\ast }}$ is optimal, and the cross-derivatives${\displaystyle {\mathrm{\partial }}_{{\theta }_i}{\mathrm{\partial }}_{{\theta }_j}L}$ are neglected to save compute. Thus, the saliency of a parameter approximates the increase in loss if that parameter is deleted.

A related methodology wasmodel compression orpruning, where a trained network is reduced in size. This was first done in 1965 byAlexey Ivakhnenko and Valentin Lapa inUSSR (1965).^[8][9][10] Their deep networks were trained layer by layer throughregression analysis. Superfluous hidden units were pruned using a separate validation set.^[11] Other neural network compression methods include Biased Weight Decay^[12] and Optimal Brain Damage.^[7]

An early example of neural network distillation was published byJürgen Schmidhuber in 1991, in the field ofrecurrent neural networks (RNNs). The problem was sequence prediction for long sequences, i.e.,deep learning. Their approach was to use two RNNs. One of them (theautomatizer) predicted the sequence, and another (thechunker) predicted the errors of the automatizer. Simultaneously, the automatizer predicted the internal states of the chunker. After the automatizer manages to predict the chunker's internal states well, it would start fixing the errors, and soon the chunker is obsoleted, leaving just one RNN in the end.^[13][14]

The idea of using the output of one neural network to train another neural network was also studied as the teacher-student network configuration.^[15] In 1992, several papers studied thestatistical mechanics of teacher-student configurations with committee machines^[16][17] or parity machines.^[18]

Compressing the knowledge of multiple models into a singleneural network was calledmodel compression in 2006: compression was achieved by training a smaller model on large amounts of pseudo-data labelled by a higher-performing ensemble, optimizing to match thelogit of the compressed model to the logit of the ensemble.^[19] The knowledge distillationpreprint ofGeoffrey Hinton et al. (2015)^[1] formulated the concept and showed some results achieved in the task ofimage classification.

Knowledge distillation is also related to the concept ofbehavioral cloning discussed by Faraz Torabi et. al.^[20]

• Distilling the knowledge in a neural network – Google AI

• Deep learning

• CS1: long volume value
• Articles with short description
• Short description matches Wikidata
• All articles with vague or ambiguous time
• Vague or ambiguous time from June 2025