Knowledge distillation: learning from a teacher’s soft answers
Optional · supplementary note 2 of 2 for DLCV 01 · Deep Learning for Image Processing and Computer Vision. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
A student who only sees the answer key learns “this photo is a tabby cat”. A student who also hears the teacher say “mostly tabby cat, a little tiger cat, almost certainly not a car” learns which classes resemble each other. Knowledge distillation trains a (usually smaller) student network to imitate the full output distribution of a (usually larger) teacher [39].
Let the teacher produce logits and the student logits for one input. A softmax with temperature turns logits into probabilities,
where and are the student’s and teacher’s probability for class and is the ordinary softmax; a larger flattens the distribution, so the small probabilities of wrong classes become visible. Hinton et al. train the student with a weighted sum of two cross-entropies [39]:
where is the true class, is the student’s probability of at , and weights the soft term against the hard-label term (the paper found a considerably lower weight on the hard term to work best). The gradient of the soft term with respect to the student logit is . Its size shrinks as grows, which is why the paper multiplies it by [39]. After training, the student is run at .
Code · Python · 31 lines
import numpy as np
def softmax(z, T=1.0):
z = np.asarray(z, dtype=float) / T
e = np.exp(z - z.max())
return e / e.sum()
teacher = np.array([9.0, 6.0, 2.0, -3.0]) # logits for: tabby cat, tiger cat, dog, car
for T in (1, 4):
print(f"T={T}: teacher probabilities", softmax(teacher, T).round(3))
def soft_ce(zs, zt, T):
return -(softmax(zt, T) * np.log(softmax(zs, T))).sum() # cross-entropy with soft targets
def kd_loss(zs, zt, y, T, alpha):
hard = -np.log(softmax(zs, 1.0)[y]) # ordinary cross-entropy at T = 1
return alpha * T**2 * soft_ce(zs, zt, T) + (1 - alpha) * hard
student = np.array([5.0, 4.0, 3.0, 0.0])
print("loss T=4, alpha=0.9, true class 0:", round(kd_loss(student, teacher, 0, 4, 0.9), 3))
# the gradient of the soft term with respect to the student logits is (q - p) / T
T = 4.0
analytic = (softmax(student, T) - softmax(teacher, T)) / T
numeric = np.array([(soft_ce(student + 1e-6 * np.eye(4)[i], teacher, T)
- soft_ce(student - 1e-6 * np.eye(4)[i], teacher, T)) / 2e-6 for i in range(4)])
print("max |analytic - numeric| gradient:", np.abs(analytic - numeric).max())
for T in (1, 4, 16, 64): # why the T^2 factor: gradient size shrinks like 1/T^2
g = (softmax(student, T) - softmax(teacher, T)) / T
print(f"T={T}: gradient norm {np.linalg.norm(g):.4f}, times T^2: {T**2 * np.linalg.norm(g):.4f}")Output:
Code · 8 lines
T=1: teacher probabilities [0.952 0.047 0.001 0. ]
T=4: teacher probabilities [0.59 0.279 0.102 0.029]
loss T=4, alpha=0.9, true class 0: 16.462
max |analytic - numeric| gradient: 1.0457410631881103e-10
T=1: gradient norm 0.3608, times T^2: 0.3608
T=4: gradient norm 0.0653, times T^2: 1.0441
T=16: gradient norm 0.0051, times T^2: 1.2975
T=64: gradient norm 0.0003, times T^2: 1.3377At the teacher is almost certain (0.952 for the first class) and the second class is barely visible; at the cat-versus-dog-versus-car ordering shows up. The analytic gradient matches a finite-difference check. The last four lines show that the soft-term gradient norm falls steeply with , while the product with levels off (about 1.34 at ): the scaling is an approximation that holds at high temperature.
Link to this tutorial. DINO’s loss is the same soft-target cross-entropy, with two differences: the teacher is not a bigger pretrained network but a moving average of the student, and no labels are used. Its “sharpening” of the teacher is a softmax with a low temperature. Compressing a large model into smaller ones, as the text says is done for DINOv2, is the classic teacher-to-student setting. In the paper’s preliminary MNIST experiment, a large regularized network made 67 test errors, a smaller unregularized one made 146, and the smaller one trained to match the large one’s soft targets at made 74 [39]. This is one dataset and one setting, so treat it as an illustration, not a general rate. Common mistakes: using different temperatures for teacher and student in the soft term, forgetting the factor when mixing with the hard loss, and evaluating the student at .
References
- Shyandram, “深度學習的數位影像處理介紹 (An introduction to digital image processing with deep learning),” Medium post, 2024; updated 2026. link
- R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
- S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
- Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
- O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
- A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
- W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
- J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
- O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
- Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
- M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
- M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
- L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
- S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
- A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
- H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
- J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
- T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
- Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
- S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
- L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
- R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
- M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
- J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
- scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
- S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
- G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531