完整搜尋頁面 →

評估用基準資料集細說

選讀・DLCV 02・深度影像分類 的補充資料(第 4/9 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 9

  1. 巨觀與微觀平均的計算實例
  2. Cohen’s kappa:扣除巧合的一致性
  3. 多標籤指標細說
  4. 評估用基準資料集細說
  5. ImageNet 的數字有多可靠?
  6. 殘差網路為什麼好訓練:展開後的梯度
  7. 另外兩種 CNN 設計:DenseNet 與 RegNet
  8. Squeeze-and-excitation:CNN 裡的注意力
  9. 對比式學習細說:SimCLR 與 MoCo

論文導讀 · 1

  1. 現代觀點綜述導讀

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

本則補充是正文評估基準簡表背後的完整表格。

下表列出最常見的評估集。大小以所引用的論文或官方頁面為準。

基準衡量什麼大小/類別數常回報的指標
ImageNet-1k 驗證集 [2]分佈內準確率;因為 ILSVRC 測試集的標籤保留在評估伺服器上,它成了實質上的測試集50,000 張,1,000 類top-1(top-5)
ImageNet-ReaL [8]同一批驗證影像,重新標註為多標籤的「重新評估」標籤46,837 張影像、57,553 個標籤(3,163 張沒有任何標籤的影像被捨棄)ReaL 準確率:top-1 預測只要落在標籤集合內就算對
ImageNetV2 [7]依原始收集流程重做一次所得到的新樣本上的泛化能力每個版本 10,000 張(每類 10 張);三種抽樣版本,通常使用 MatchedFrequencytop-1
ImageNet-C [42]對合成損壞的穩健性驗證影像 × 15 種損壞 × 5 個嚴重程度mean corruption error
ImageNet-R [43]各種表現形式:藝術、卡通、雕塑、玩具等30,000 張,200 類200 類上的 top-1
ImageNet-A [44]經篩選、專門騙倒 ResNet-50 的自然照片7,500 張,200 類200 類上的 top-1
ImageNet-Sketch [63]轉換到黑白素描領域50,000 張,1,000 類(每類 50 張)top-1
ObjectNet [64]控制偏差的照片,刻意變化旋轉、背景與視角;只有測試集50,000 張,313 類(其中 113 類與 ImageNet 重疊)重疊類別上的 top-1
ImageNet-LT [73]長尾分佈的訓練、平衡的測試11.58 萬張訓練影像,1,000 類,每類 5 到 1,280 張;平衡的測試集top-1,常依類別頻率分組回報
iNaturalist [6]天然長尾、細粒度的物種見正文的訓練用資料集表top-1

細粒度與遷移基準。 骨幹網路也要看它的特徵能多好地遷移到小型、專門的資料集上,方式是在凍結特徵上做線性探測,或是微調(見正文的遷移學習一節)。

資料集領域大小/類別數常回報的指標
CUB-200-2011 [65]鳥類物種,附部位與屬性標註11,788 張,200 類top-1
Stanford Cars [66]汽車品牌、型號與年份196 類;訓練 8,144/測試 8,041top-1
FGVC-Aircraft [67]三層階層的飛機型號10,000 張,100 種型號各類別平均準確率
Oxford-IIIT Pets [68]貓狗品種37 類,每類約 200 張各類別平均準確率
Oxford Flowers-102 [69]花卉物種102 類,每類 40 到 258 張;訓練 2,040/測試 6,149各類別平均準確率
Food-101 [70]菜餚;訓練標籤刻意保留雜訊101,000 張,101 類(每類訓練 750/測試 250)top-1
Caltech-101 [71]一般物體101 類加背景類,每類約 40 到 800 張各類別平均準確率

Cars 與 Flowers 的訓練/測試大小依 Kornblith 等人 [33] 所列的切分。注意其中好幾個基準回報的是各類別平均準確率(mean per-class accuracy,即巨觀平均的召回率),而不是一般的準確率,因為它們的測試集並不平衡;在同一個比較裡混用兩者是常見的錯誤。

有兩個測試組把這類資料集綁在一起,用一個數字總結遷移能力:

  • Kornblith 等人 [33] 使用了 12 個資料集:Food-101、CIFAR-10、CIFAR-100、Birdsnap、SUN397、Stanford Cars、FGVC-Aircraft、PASCAL VOC 2007 分類、Describable Textures、Oxford-IIIT Pets、Caltech-101 與 Oxford Flowers-102。依資料集不同,回報 top-1 準確率、各類別平均準確率或 11 點 mAP(VOC 2007),並比較線性探測、微調與從頭訓練三種設定。
  • VTAB [72] 包含 19 個任務,分成三組:7 個自然(一般相機影像)、4 個專門(衛星與醫學影像)、8 個結構(場景幾何,例如計數或距離,大多來自模擬環境)。在 VTAB-1k 中,每個任務只提供 1,000 個標註訓練樣本(800 個訓練、200 個驗證),因此它測試的是:一個表徵能否從很少的標籤,適應到與預訓練資料很不一樣的任務。

零樣本評估。 CLIP [40] 這類影像–文字模型在評估時,完全不在目標資料集上訓練:把類別名稱寫成文字提示,再把影像歸到最相似的提示(見正文的 CLIP 一節)。CLIP 論文用這種方式在 30 多個既有資料集上做了評估,回報零樣本 CLIP 在 27 個資料集中的 16 個上勝過以 ResNet-50 特徵做的線性探測,並發現對 ImageNet 集成 80 個提示模板,比單一預設提示多了 3.5 個百分點。這帶來兩點提醒。第一,提示與類別名稱的寫法都是超參數;在測試集上調整它們,和在測試集上調學習率一樣,會洩漏測試資訊。第二,預訓練資料是網路規模、而且大多未公開,所以「零樣本」並不保證測試影像(或近乎重複的影像)從未被看過。

參考文獻

  1. Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 12, pp. 6999–7019, 2022. doi:10.1109/TNNLS.2021.3084827
  2. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y
  3. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
  4. A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” technical report, University of Toronto, 2009. dataset page
  5. T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, “ImageNet-21K Pretraining for the Masses,” NeurIPS, 2021. arXiv
  6. G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, et al., “The iNaturalist Species Classification and Detection Dataset,” CVPR, 2018. arXiv
  7. B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do ImageNet Classifiers Generalize to ImageNet?,” ICML, PMLR vol. 97, 2019. paper
  8. L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord, “Are We Done with ImageNet?,” arXiv:2006.07159, 2020. arXiv
  9. C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks,” NeurIPS, 2021. arXiv
  10. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25 (NIPS), 2012. paper
  11. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ICLR, 2015. arXiv
  12. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594
  13. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016. doi:10.1109/CVPR.2016.308
  14. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90
  15. K. He, X. Zhang, S. Ren, and J. Sun, “Identity Mappings in Deep Residual Networks,” ECCV, 2016. arXiv
  16. G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely Connected Convolutional Networks,” CVPR, 2017. arXiv
  17. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, 2017. arXiv
  18. M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ICML, 2019. arXiv
  19. I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár, “Designing Network Design Spaces,” CVPR, 2020. arXiv
  20. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022. arXiv
  21. J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation Networks,” arXiv:1709.01507, 2017 (journal version in IEEE TPAMI). arXiv
  22. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
  23. H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training Data-Efficient Image Transformers & Distillation through Attention,” ICML, PMLR vol. 139, 2021. paper
  24. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986
  25. S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, 2022. arXiv
  26. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” ICLR, 2018. arXiv
  27. S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features,” ICCV, 2019. arXiv
  28. E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “RandAugment: Practical Automated Data Augmentation with a Reduced Search Space,” CVPR Workshops, 2020. doi:10.1109/CVPRW50498.2020.00359
  29. I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. arXiv
  30. I. Loshchilov and F. Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts,” ICLR, 2017. arXiv
  31. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep Networks with Stochastic Depth,” ECCV, 2016. doi:10.1007/978-3-319-46493-0_39
  32. R. Wightman, H. Touvron, and H. Jégou, “ResNet Strikes Back: An Improved Training Procedure in timm,” arXiv:2110.00476, 2021. arXiv
  33. S. Kornblith, J. Shlens, and Q. V. Le, “Do Better ImageNet Models Transfer Better?,” CVPR, 2019. arXiv
  34. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. arXiv
  35. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. arXiv
  36. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951
  37. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553
  38. M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
  39. O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
  40. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, PMLR vol. 139, 2021. paper
  41. J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao, “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence (arXiv:2301.05712, 2023). arXiv
  42. D. Hendrycks and T. Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations,” ICLR, 2019. arXiv
  43. D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, et al., “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization,” ICCV, 2021. arXiv
  44. D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural Adversarial Examples,” CVPR, 2021. arXiv
  45. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
  46. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut Learning in Deep Neural Networks,” Nature Machine Intelligence, 2020. arXiv
  47. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. arXiv
  48. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning Deep Features for Discriminative Localization,” CVPR, 2016. doi:10.1109/CVPR.2016.319
  49. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization,” ICCV, 2017. doi:10.1109/ICCV.2017.74
  50. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” CVPR, 2018. doi:10.1109/CVPR.2018.00286
  51. G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” arXiv:1503.02531, 2015. arXiv
  52. X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
  53. M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
  54. T. Fawcett, “An Introduction to ROC Analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006. doi:10.1016/j.patrec.2005.10.010
  55. J. Davis and M. Goadrich, “The Relationship Between Precision-Recall and ROC Curves,” Proc. 23rd International Conference on Machine Learning (ICML), pp. 233–240, 2006. doi:10.1145/1143844.1143874
  56. T. Saito and M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets,” PLOS ONE, vol. 10, no. 3, e0118432, 2015. doi:10.1371/journal.pone.0118432
  57. D. Chicco and G. Jurman, “The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation,” BMC Genomics, vol. 21, article 6, 2020. doi:10.1186/s12864-019-6413-7
  58. M. Sokolova and G. Lapalme, “A Systematic Analysis of Performance Measures for Classification Tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427–437, 2009. doi:10.1016/j.ipm.2009.03.002
  59. J. Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960. doi:10.1177/001316446002000104
  60. M.-L. Zhang and Z.-H. Zhou, “A Review on Multi-Label Learning Algorithms,” IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 8, pp. 1819–1837, 2014. doi:10.1109/TKDE.2013.39
  61. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes (VOC) Challenge,” International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010. doi:10.1007/s11263-009-0275-4
  62. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. paper
  63. H. Wang, S. Ge, E. P. Xing, and Z. C. Lipton, “Learning Robust Global Representations by Penalizing Local Predictive Power,” NeurIPS, 2019 (introduces ImageNet-Sketch). arXiv
  64. A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz, “ObjectNet: A Large-Scale Bias-Controlled Dataset for Pushing the Limits of Object Recognition Models,” NeurIPS, 2019. dataset page
  65. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The Caltech-UCSD Birds-200-2011 Dataset,” Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. dataset page
  66. J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3D Object Representations for Fine-Grained Categorization,” IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 554–561, 2013 (introduces Stanford Cars). doi:10.1109/ICCVW.2013.77
  67. S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-Grained Visual Classification of Aircraft,” arXiv:1306.5151, 2013. arXiv
  68. O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar, “Cats and Dogs,” CVPR, 2012 (introduces Oxford-IIIT Pets). dataset page
  69. M.-E. Nilsback and A. Zisserman, “Automated Flower Classification over a Large Number of Classes,” Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), 2008 (introduces Oxford Flowers-102). dataset page
  70. L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101 – Mining Discriminative Components with Random Forests,” ECCV, 2014. dataset page
  71. L. Fei-Fei, R. Fergus, and P. Perona, “One-Shot Learning of Object Categories,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006 (the reference given for Caltech-101). dataset page
  72. X. Zhai, J. Puigcerver, A. Kolesnikov, et al., “A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark,” arXiv:1910.04867, 2019. arXiv
  73. Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, “Large-Scale Long-Tailed Recognition in an Open World,” CVPR, 2019 (introduces ImageNet-LT). arXiv