完整搜尋頁面 →

知識蒸餾:向老師模型的軟答案學習

選讀・DLCV 01・深度學習的影像處理與電腦視覺 的補充資料(第 2/2 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 2

  1. 批次正規化:在網路內部做標準化
  2. 知識蒸餾:向老師模型的軟答案學習

論文導讀 · 1

  1. 現代觀點綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

只看到標準答案的學生,學到的是「這張照片是虎斑貓」。如果學生還聽到老師說「多半是虎斑貓,有一點像另一種貓,幾乎肯定不是汽車」,就會知道哪些類別彼此相像。知識蒸餾(knowledge distillation)訓練一個(通常較小的)學生網路,去模仿一個(通常較大的)教師網路的完整輸出分布 [39]。

設教師對某個輸入輸出邏輯值(logits)v\mathbf{v},學生輸出 z\mathbf{z}。帶溫度(temperature)TT 的 softmax 把邏輯值轉成機率:

qi=exp⁡(zi/T)∑jexp⁡(zj/T),pi=exp⁡(vi/T)∑jexp⁡(vj/T),q_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}, \qquad p_i = \frac{\exp(v_i / T)}{\sum_j \exp(v_j / T)},

其中 qiq_i、pip_i 分別是學生與教師對類別 ii 的機率,T=1T = 1 就是一般的 softmax;TT 越大,分布越平坦,錯誤類別的小機率就看得出來。Hinton 等人用兩個交叉熵的加權和來訓練學生 [39]:

L=α T2(−∑ipilog⁡qi)+(1−α)(−log⁡qy(1)),\mathcal{L} = \alpha\, T^2 \left( -\sum_i p_i \log q_i \right) + (1 - \alpha) \left( -\log q^{(1)}_y \right),

其中 yy 是真實類別,qy(1)q^{(1)}_y 是學生在 T=1T = 1 時對 yy 的機率,α∈[0,1]\alpha \in [0, 1] 是軟項與硬標籤項的權重(論文發現給硬標籤項相當低的權重效果最好)。軟項對學生邏輯值 ziz_i 的梯度是 (qi−pi)/T(q_i - p_i)/T,它的大小會隨 TT 變大而縮小,所以論文把它乘上 T2T^2 [39]。訓練完成後,學生以 T=1T = 1 執行。

程式碼 · Python · 31 行
import numpy as np

def softmax(z, T=1.0):
    z = np.asarray(z, dtype=float) / T
    e = np.exp(z - z.max())
    return e / e.sum()

teacher = np.array([9.0, 6.0, 2.0, -3.0])      # logits for: tabby cat, tiger cat, dog, car
for T in (1, 4):
    print(f"T={T}: teacher probabilities", softmax(teacher, T).round(3))

def soft_ce(zs, zt, T):
    return -(softmax(zt, T) * np.log(softmax(zs, T))).sum()   # cross-entropy with soft targets

def kd_loss(zs, zt, y, T, alpha):
    hard = -np.log(softmax(zs, 1.0)[y])                      # ordinary cross-entropy at T = 1
    return alpha * T**2 * soft_ce(zs, zt, T) + (1 - alpha) * hard

student = np.array([5.0, 4.0, 3.0, 0.0])
print("loss T=4, alpha=0.9, true class 0:", round(kd_loss(student, teacher, 0, 4, 0.9), 3))

# the gradient of the soft term with respect to the student logits is (q - p) / T
T = 4.0
analytic = (softmax(student, T) - softmax(teacher, T)) / T
numeric = np.array([(soft_ce(student + 1e-6 * np.eye(4)[i], teacher, T)
                     - soft_ce(student - 1e-6 * np.eye(4)[i], teacher, T)) / 2e-6 for i in range(4)])
print("max |analytic - numeric| gradient:", np.abs(analytic - numeric).max())

for T in (1, 4, 16, 64):                        # why the T^2 factor: gradient size shrinks like 1/T^2
    g = (softmax(student, T) - softmax(teacher, T)) / T
    print(f"T={T}: gradient norm {np.linalg.norm(g):.4f}, times T^2: {T**2 * np.linalg.norm(g):.4f}")

輸出:

程式碼 · 8 行
T=1: teacher probabilities [0.952 0.047 0.001 0.   ]
T=4: teacher probabilities [0.59  0.279 0.102 0.029]
loss T=4, alpha=0.9, true class 0: 16.462
max |analytic - numeric| gradient: 1.0457410631881103e-10
T=1: gradient norm 0.3608, times T^2: 0.3608
T=4: gradient norm 0.0653, times T^2: 1.0441
T=16: gradient norm 0.0051, times T^2: 1.2975
T=64: gradient norm 0.0003, times T^2: 1.3377

在 T=1T = 1 時,教師幾乎完全確定(第一類 0.952),第二類幾乎看不到;T=4T = 4 時,貓、狗、汽車之間的相對順序就浮現出來。解析梯度與有限差分檢查相符。最後四行顯示,軟項梯度的範數隨 TT 急遽下降,乘上 T2T^2 之後則趨於平穩(T=64T = 64 時約 1.34):1/T21/T^2 的縮放是高溫下才成立的近似。

與本教學的關聯。 DINO 的損失 H(a,b)=−a⊤log⁡bH(a, b) = -a^\top \log b 就是同樣的軟目標交叉熵,差別有兩點:教師不是更大的預訓練網路,而是學生的移動平均;而且完全不使用標註。對教師輸出的「銳化」就是用低溫度的 softmax。把大模型壓縮成較小的模型(正文提到 DINOv2 這樣做)則是典型的教師到學生設定。在論文的初步 MNIST 實驗中,大型正則化網路的測試錯誤數為 67,較小而未正則化的網路為 146,而同一個小網路改為在 T=20T = 20 時去符合大網路的軟目標,錯誤數為 74 [39]。這只是單一資料集、單一設定,應視為示範,而非通用的比率。常見錯誤:軟項中教師與學生使用不同的溫度;與硬標籤損失混合時忘了 T2T^2 因子;以及在 T>1T > 1 下評估學生。

參考文獻

  1. Shyandram, “深度學習的數位影像處理介紹,” Medium post, 2024; updated 2026. link
  2. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th ed., Pearson, 2018. publisher page
  3. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed., Academic Press, 2008. publisher page
  4. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989. doi:10.1162/neco.1989.1.4.541
  5. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
  6. O. Russakovsky, J. Deng, H. Su, et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y · arXiv:1409.0575
  7. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” NeurIPS, 2012. proceedings
  8. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ICLR, 2015. arXiv:1409.1556
  9. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594 · arXiv:1409.4842
  10. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90 · arXiv:1512.03385
  11. A. Krizhevsky, “Learning multiple layers of features from tiny images,” technical report, University of Toronto, 2009 (the CIFAR-10/100 datasets). dataset page
  12. W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” NeurIPS, 2016. arXiv:1701.04128
  13. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” CVPR, 2015. arXiv:1411.4038
  14. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” MICCAI, 2015. arXiv:1505.04597
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017. ML anthology
  16. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021. arXiv:2010.11929
  17. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision Transformer using shifted windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986 · arXiv:2103.14030
  18. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022, arXiv:2201.03545; and S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders,” CVPR, 2023, arXiv:2301.00808.
  19. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951 · arXiv:2104.14294
  20. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553 · arXiv:2111.06377
  21. M. Oquab, T. Darcet, T. Moutakanni, et al., “DINOv2: Learning robust visual features without supervision,” Transactions on Machine Learning Research, 2024, arXiv:2304.07193; and O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, et al., “DINOv3,” arXiv preprint, 2025, arXiv:2508.10104.
  22. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision Mamba: Efficient visual representation learning with bidirectional state space model,” ICML, 2024, arXiv:2401.09417; and Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, J. Jiao, and Y. Liu, “VMamba: Visual state space model,” NeurIPS, 2024, arXiv:2401.10166.
  23. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NeurIPS, 2015, arXiv:1506.01497; and N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with Transformers,” ECCV, 2020, arXiv:2005.12872.
  24. A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” ICCV, 2023, arXiv:2304.02643; N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, et al., “SAM 2: Segment anything in images and videos,” ICLR, 2025, arXiv:2408.00714; and N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, et al., “SAM 3: Segment anything with concepts,” ICLR, 2026, arXiv:2511.16719.
  25. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, et al., “Learning transferable visual models from natural language supervision,” ICML, 2021. arXiv:2103.00020
  26. H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” NeurIPS, 2023. arXiv:2304.08485
  27. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, 2020. arXiv:2006.11239
  28. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR, 2022, arXiv:2112.10752; and W. Peebles and S. Xie, “Scalable diffusion models with Transformers,” ICCV, 2023, arXiv:2212.09748.
  29. T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” CVPR, 2023. arXiv:2211.09800
  30. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, 2022. doi:10.1109/TNNLS.2021.3084827 · arXiv:2004.02806
  31. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys, vol. 54, no. 10s, 2022. doi:10.1145/3505244 · arXiv:2101.01169
  32. L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, 2021. doi:10.1109/TPAMI.2020.2992393 · arXiv:1902.06162
  33. R. Balestriero, M. Ibrahim, V. Sobal, et al., “A cookbook of self-supervised learning,” arXiv preprint, 2023. arXiv:2304.12210
  34. M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: A survey and outlook,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, 2025. doi:10.1109/TPAMI.2024.3506283 · arXiv:2307.13721
  35. J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, 2024. doi:10.1109/TPAMI.2024.3369699 · arXiv:2304.00685
  36. scikit-learn developers, “sklearn.datasets.load_digits” (a copy of the test set of the UCI optical recognition of handwritten digits dataset). docs
  37. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML, PMLR 37, pp. 448–456, 2015. PMLR · arXiv:1502.03167
  38. S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?” NeurIPS, vol. 31, 2018. proceedings · arXiv:1805.11604
  39. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” NIPS 2014 Deep Learning Workshop, arXiv preprint, 2015. arXiv:1503.02531