Full search page →

Knowledge distillation: learning from a teacher’s soft answers

Optional · supplementary note 2 of 2 for DLCV 01 · Deep Learning for Image Processing and Computer Vision. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Batch normalization: standardize inside the network
  2. Knowledge distillation: learning from a teacher’s soft answers

Paper guides · 1

  1. Guide to the Modern view surveys

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

A student who only sees the answer key learns “this photo is a tabby cat”. A student who also hears the teacher say “mostly tabby cat, a little tiger cat, almost certainly not a car” learns which classes resemble each other. Knowledge distillation trains a (usually smaller) student network to imitate the full output distribution of a (usually larger) teacher [39].

Let the teacher produce logits v\mathbf{v} and the student logits z\mathbf{z} for one input. A softmax with temperature TT turns logits into probabilities,

qi=exp⁡(zi/T)∑jexp⁡(zj/T),pi=exp⁡(vi/T)∑jexp⁡(vj/T),q_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}, \qquad p_i = \frac{\exp(v_i / T)}{\sum_j \exp(v_j / T)},

where qiq_i and pip_i are the student’s and teacher’s probability for class ii and T=1T = 1 is the ordinary softmax; a larger TT flattens the distribution, so the small probabilities of wrong classes become visible. Hinton et al. train the student with a weighted sum of two cross-entropies [39]:

L=α T2(−∑ipilog⁡qi)+(1−α)(−log⁡qy(1)),\mathcal{L} = \alpha\, T^2 \left( -\sum_i p_i \log q_i \right) + (1 - \alpha) \left( -\log q^{(1)}_y \right),

where yy is the true class, qy(1)q^{(1)}_y is the student’s probability of yy at T=1T = 1, and α∈[0,1]\alpha \in [0, 1] weights the soft term against the hard-label term (the paper found a considerably lower weight on the hard term to work best). The gradient of the soft term with respect to the student logit ziz_i is (qi−pi)/T(q_i - p_i)/T. Its size shrinks as TT grows, which is why the paper multiplies it by T2T^2 [39]. After training, the student is run at T=1T = 1.

Code · Python · 31 lines
import numpy as np

def softmax(z, T=1.0):
    z = np.asarray(z, dtype=float) / T
    e = np.exp(z - z.max())
    return e / e.sum()

teacher = np.array([9.0, 6.0, 2.0, -3.0])      # logits for: tabby cat, tiger cat, dog, car
for T in (1, 4):
    print(f"T={T}: teacher probabilities", softmax(teacher, T).round(3))

def soft_ce(zs, zt, T):
    return -(softmax(zt, T) * np.log(softmax(zs, T))).sum()   # cross-entropy with soft targets

def kd_loss(zs, zt, y, T, alpha):
    hard = -np.log(softmax(zs, 1.0)[y])                      # ordinary cross-entropy at T = 1
    return alpha * T**2 * soft_ce(zs, zt, T) + (1 - alpha) * hard

student = np.array([5.0, 4.0, 3.0, 0.0])
print("loss T=4, alpha=0.9, true class 0:", round(kd_loss(student, teacher, 0, 4, 0.9), 3))

# the gradient of the soft term with respect to the student logits is (q - p) / T
T = 4.0
analytic = (softmax(student, T) - softmax(teacher, T)) / T
numeric = np.array([(soft_ce(student + 1e-6 * np.eye(4)[i], teacher, T)
                     - soft_ce(student - 1e-6 * np.eye(4)[i], teacher, T)) / 2e-6 for i in range(4)])
print("max |analytic - numeric| gradient:", np.abs(analytic - numeric).max())

for T in (1, 4, 16, 64):                        # why the T^2 factor: gradient size shrinks like 1/T^2
    g = (softmax(student, T) - softmax(teacher, T)) / T
    print(f"T={T}: gradient norm {np.linalg.norm(g):.4f}, times T^2: {T**2 * np.linalg.norm(g):.4f}")

Output:

Code · 8 lines
T=1: teacher probabilities [0.952 0.047 0.001 0.   ]
T=4: teacher probabilities [0.59  0.279 0.102 0.029]
loss T=4, alpha=0.9, true class 0: 16.462
max |analytic - numeric| gradient: 1.0457410631881103e-10
T=1: gradient norm 0.3608, times T^2: 0.3608
T=4: gradient norm 0.0653, times T^2: 1.0441
T=16: gradient norm 0.0051, times T^2: 1.2975
T=64: gradient norm 0.0003, times T^2: 1.3377

At T=1T = 1 the teacher is almost certain (0.952 for the first class) and the second class is barely visible; at T=4T = 4 the cat-versus-dog-versus-car ordering shows up. The analytic gradient matches a finite-difference check. The last four lines show that the soft-term gradient norm falls steeply with TT, while the product with T2T^2 levels off (about 1.34 at T=64T = 64): the 1/T21/T^2 scaling is an approximation that holds at high temperature.

Link to this tutorial. DINO’s loss H(a,b)=−a⊤log⁡bH(a, b) = -a^\top \log b is the same soft-target cross-entropy, with two differences: the teacher is not a bigger pretrained network but a moving average of the student, and no labels are used. Its “sharpening” of the teacher is a softmax with a low temperature. Compressing a large model into smaller ones, as the text says is done for DINOv2, is the classic teacher-to-student setting. In the paper’s preliminary MNIST experiment, a large regularized network made 67 test errors, a smaller unregularized one made 146, and the smaller one trained to match the large one’s soft targets at T=20T = 20 made 74 [39]. This is one dataset and one setting, so treat it as an illustration, not a general rate. Common mistakes: using different temperatures for teacher and student in the soft term, forgetting the T2T^2 factor when mixing with the hard loss, and evaluating the student at T>1T > 1.

References

  1. Shyandram, “深度學習的數位影像處理介紹 (An introduction to digital image processing with deep learning),” Medium post, 2024; updated 2026. link
  2. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
  3. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
  4. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
  5. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
  6. O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
  7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
  8. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
  9. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
  10. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
  11. A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
  12. W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
  13. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
  14. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
  16. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
  17. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
  18. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
  19. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
  20. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
  21. M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
  22. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
  23. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
  24. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
  25. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
  26. H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
  27. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
  28. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
  29. T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
  30. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
  31. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
  32. L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
  33. R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
  34. M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
  35. J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
  36. scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
  37. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
  38. S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
  39. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531