知識蒸餾:向老師模型的軟答案學習
選讀・DLCV 01・深度學習的影像處理與電腦視覺 的補充資料(第 2/2 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
只看到標準答案的學生,學到的是「這張照片是虎斑貓」。如果學生還聽到老師說「多半是虎斑貓,有一點像另一種貓,幾乎肯定不是汽車」,就會知道哪些類別彼此相像。知識蒸餾(knowledge distillation)訓練一個(通常較小的)學生網路,去模仿一個(通常較大的)教師網路的完整輸出分布 [39]。
設教師對某個輸入輸出邏輯值(logits),學生輸出 。帶溫度(temperature) 的 softmax 把邏輯值轉成機率:
其中 、 分別是學生與教師對類別 的機率, 就是一般的 softmax; 越大,分布越平坦,錯誤類別的小機率就看得出來。Hinton 等人用兩個交叉熵的加權和來訓練學生 [39]:
其中 是真實類別, 是學生在 時對 的機率, 是軟項與硬標籤項的權重(論文發現給硬標籤項相當低的權重效果最好)。軟項對學生邏輯值 的梯度是 ,它的大小會隨 變大而縮小,所以論文把它乘上 [39]。訓練完成後,學生以 執行。
程式碼 · Python · 31 行
import numpy as np
def softmax(z, T=1.0):
z = np.asarray(z, dtype=float) / T
e = np.exp(z - z.max())
return e / e.sum()
teacher = np.array([9.0, 6.0, 2.0, -3.0]) # logits for: tabby cat, tiger cat, dog, car
for T in (1, 4):
print(f"T={T}: teacher probabilities", softmax(teacher, T).round(3))
def soft_ce(zs, zt, T):
return -(softmax(zt, T) * np.log(softmax(zs, T))).sum() # cross-entropy with soft targets
def kd_loss(zs, zt, y, T, alpha):
hard = -np.log(softmax(zs, 1.0)[y]) # ordinary cross-entropy at T = 1
return alpha * T**2 * soft_ce(zs, zt, T) + (1 - alpha) * hard
student = np.array([5.0, 4.0, 3.0, 0.0])
print("loss T=4, alpha=0.9, true class 0:", round(kd_loss(student, teacher, 0, 4, 0.9), 3))
# the gradient of the soft term with respect to the student logits is (q - p) / T
T = 4.0
analytic = (softmax(student, T) - softmax(teacher, T)) / T
numeric = np.array([(soft_ce(student + 1e-6 * np.eye(4)[i], teacher, T)
- soft_ce(student - 1e-6 * np.eye(4)[i], teacher, T)) / 2e-6 for i in range(4)])
print("max |analytic - numeric| gradient:", np.abs(analytic - numeric).max())
for T in (1, 4, 16, 64): # why the T^2 factor: gradient size shrinks like 1/T^2
g = (softmax(student, T) - softmax(teacher, T)) / T
print(f"T={T}: gradient norm {np.linalg.norm(g):.4f}, times T^2: {T**2 * np.linalg.norm(g):.4f}")輸出:
程式碼 · 8 行
T=1: teacher probabilities [0.952 0.047 0.001 0. ]
T=4: teacher probabilities [0.59 0.279 0.102 0.029]
loss T=4, alpha=0.9, true class 0: 16.462
max |analytic - numeric| gradient: 1.0457410631881103e-10
T=1: gradient norm 0.3608, times T^2: 0.3608
T=4: gradient norm 0.0653, times T^2: 1.0441
T=16: gradient norm 0.0051, times T^2: 1.2975
T=64: gradient norm 0.0003, times T^2: 1.3377在 時,教師幾乎完全確定(第一類 0.952),第二類幾乎看不到; 時,貓、狗、汽車之間的相對順序就浮現出來。解析梯度與有限差分檢查相符。最後四行顯示,軟項梯度的範數隨 急遽下降,乘上 之後則趨於平穩( 時約 1.34): 的縮放是高溫下才成立的近似。
與本教學的關聯。 DINO 的損失 就是同樣的軟目標交叉熵,差別有兩點:教師不是更大的預訓練網路,而是學生的移動平均;而且完全不使用標註。對教師輸出的「銳化」就是用低溫度的 softmax。把大模型壓縮成較小的模型(正文提到 DINOv2 這樣做)則是典型的教師到學生設定。在論文的初步 MNIST 實驗中,大型正則化網路的測試錯誤數為 67,較小而未正則化的網路為 146,而同一個小網路改為在 時去符合大網路的軟目標,錯誤數為 74 [39]。這只是單一資料集、單一設定,應視為示範,而非通用的比率。常見錯誤:軟項中教師與學生使用不同的溫度;與硬標籤損失混合時忘了 因子;以及在 下評估學生。
參考文獻
- Shyandram, “深度學習的數位影像處理介紹,” Medium post, 2024; updated 2026. link
- R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
- S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
- Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
- O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
- A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
- W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
- J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
- O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
- Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
- M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
- M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
- L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
- S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
- A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
- H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
- J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
- T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
- Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
- S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
- L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
- R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
- M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
- J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
- scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
- S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
- G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531