評估用基準資料集細說
選讀・DLCV 02・深度影像分類 的補充資料(第 4/9 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
本則補充是正文評估基準簡表背後的完整表格。
下表列出最常見的評估集。大小以所引用的論文或官方頁面為準。
| 基準 | 衡量什麼 | 大小/類別數 | 常回報的指標 |
|---|---|---|---|
| ImageNet-1k 驗證集 [2] | 分佈內準確率;因為 ILSVRC 測試集的標籤保留在評估伺服器上,它成了實質上的測試集 | 50,000 張,1,000 類 | top-1(top-5) |
| ImageNet-ReaL [8] | 同一批驗證影像,重新標註為多標籤的「重新評估」標籤 | 46,837 張影像、57,553 個標籤(3,163 張沒有任何標籤的影像被捨棄) | ReaL 準確率:top-1 預測只要落在標籤集合內就算對 |
| ImageNetV2 [7] | 依原始收集流程重做一次所得到的新樣本上的泛化能力 | 每個版本 10,000 張(每類 10 張);三種抽樣版本,通常使用 MatchedFrequency | top-1 |
| ImageNet-C [42] | 對合成損壞的穩健性 | 驗證影像 × 15 種損壞 × 5 個嚴重程度 | mean corruption error |
| ImageNet-R [43] | 各種表現形式:藝術、卡通、雕塑、玩具等 | 30,000 張,200 類 | 200 類上的 top-1 |
| ImageNet-A [44] | 經篩選、專門騙倒 ResNet-50 的自然照片 | 7,500 張,200 類 | 200 類上的 top-1 |
| ImageNet-Sketch [63] | 轉換到黑白素描領域 | 50,000 張,1,000 類(每類 50 張) | top-1 |
| ObjectNet [64] | 控制偏差的照片,刻意變化旋轉、背景與視角;只有測試集 | 50,000 張,313 類(其中 113 類與 ImageNet 重疊) | 重疊類別上的 top-1 |
| ImageNet-LT [73] | 長尾分佈的訓練、平衡的測試 | 11.58 萬張訓練影像,1,000 類,每類 5 到 1,280 張;平衡的測試集 | top-1,常依類別頻率分組回報 |
| iNaturalist [6] | 天然長尾、細粒度的物種 | 見正文的訓練用資料集表 | top-1 |
細粒度與遷移基準。 骨幹網路也要看它的特徵能多好地遷移到小型、專門的資料集上,方式是在凍結特徵上做線性探測,或是微調(見正文的遷移學習一節)。
| 資料集 | 領域 | 大小/類別數 | 常回報的指標 |
|---|---|---|---|
| CUB-200-2011 [65] | 鳥類物種,附部位與屬性標註 | 11,788 張,200 類 | top-1 |
| Stanford Cars [66] | 汽車品牌、型號與年份 | 196 類;訓練 8,144/測試 8,041 | top-1 |
| FGVC-Aircraft [67] | 三層階層的飛機型號 | 10,000 張,100 種型號 | 各類別平均準確率 |
| Oxford-IIIT Pets [68] | 貓狗品種 | 37 類,每類約 200 張 | 各類別平均準確率 |
| Oxford Flowers-102 [69] | 花卉物種 | 102 類,每類 40 到 258 張;訓練 2,040/測試 6,149 | 各類別平均準確率 |
| Food-101 [70] | 菜餚;訓練標籤刻意保留雜訊 | 101,000 張,101 類(每類訓練 750/測試 250) | top-1 |
| Caltech-101 [71] | 一般物體 | 101 類加背景類,每類約 40 到 800 張 | 各類別平均準確率 |
Cars 與 Flowers 的訓練/測試大小依 Kornblith 等人 [33] 所列的切分。注意其中好幾個基準回報的是各類別平均準確率(mean per-class accuracy,即巨觀平均的召回率),而不是一般的準確率,因為它們的測試集並不平衡;在同一個比較裡混用兩者是常見的錯誤。
有兩個測試組把這類資料集綁在一起,用一個數字總結遷移能力:
- Kornblith 等人 [33] 使用了 12 個資料集:Food-101、CIFAR-10、CIFAR-100、Birdsnap、SUN397、Stanford Cars、FGVC-Aircraft、PASCAL VOC 2007 分類、Describable Textures、Oxford-IIIT Pets、Caltech-101 與 Oxford Flowers-102。依資料集不同,回報 top-1 準確率、各類別平均準確率或 11 點 mAP(VOC 2007),並比較線性探測、微調與從頭訓練三種設定。
- VTAB [72] 包含 19 個任務,分成三組:7 個自然(一般相機影像)、4 個專門(衛星與醫學影像)、8 個結構(場景幾何,例如計數或距離,大多來自模擬環境)。在 VTAB-1k 中,每個任務只提供 1,000 個標註訓練樣本(800 個訓練、200 個驗證),因此它測試的是:一個表徵能否從很少的標籤,適應到與預訓練資料很不一樣的任務。
零樣本評估。 CLIP [40] 這類影像–文字模型在評估時,完全不在目標資料集上訓練:把類別名稱寫成文字提示,再把影像歸到最相似的提示(見正文的 CLIP 一節)。CLIP 論文用這種方式在 30 多個既有資料集上做了評估,回報零樣本 CLIP 在 27 個資料集中的 16 個上勝過以 ResNet-50 特徵做的線性探測,並發現對 ImageNet 集成 80 個提示模板,比單一預設提示多了 3.5 個百分點。這帶來兩點提醒。第一,提示與類別名稱的寫法都是超參數;在測試集上調整它們,和在測試集上調學習率一樣,會洩漏測試資訊。第二,預訓練資料是網路規模、而且大多未公開,所以「零樣本」並不保證測試影像(或近乎重複的影像)從未被看過。
參考文獻
- Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 12, pp. 6999–7019, 2022. doi:10.1109/TNNLS.2021.3084827
- O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, pp. 211–252, 2015. doi:10.1007/s11263-015-0816-y
- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. doi:10.1109/5.726791
- A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” technical report, University of Toronto, 2009. dataset page
- T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, “ImageNet-21K Pretraining for the Masses,” NeurIPS, 2021. arXiv
- G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, et al., “The iNaturalist Species Classification and Detection Dataset,” CVPR, 2018. arXiv
- B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do ImageNet Classifiers Generalize to ImageNet?,” ICML, PMLR vol. 97, 2019. paper
- L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord, “Are We Done with ImageNet?,” arXiv:2006.07159, 2020. arXiv
- C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks,” NeurIPS, 2021. arXiv
- A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25 (NIPS), 2012. paper
- K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ICLR, 2015. arXiv
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” CVPR, 2015. doi:10.1109/CVPR.2015.7298594
- C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” CVPR, 2016. doi:10.1109/CVPR.2016.308
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. doi:10.1109/CVPR.2016.90
- K. He, X. Zhang, S. Ren, and J. Sun, “Identity Mappings in Deep Residual Networks,” ECCV, 2016. arXiv
- G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely Connected Convolutional Networks,” CVPR, 2017. arXiv
- A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, 2017. arXiv
- M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ICML, 2019. arXiv
- I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár, “Designing Network Design Spaces,” CVPR, 2020. arXiv
- Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” CVPR, 2022. arXiv
- J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation Networks,” arXiv:1709.01507, 2017 (journal version in IEEE TPAMI). arXiv
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ICLR, 2021. arXiv
- H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training Data-Efficient Image Transformers & Distillation through Attention,” ICML, PMLR vol. 139, 2021. paper
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00986
- S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in Vision: A Survey,” ACM Computing Surveys, 2022. arXiv
- H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” ICLR, 2018. arXiv
- S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features,” ICCV, 2019. arXiv
- E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “RandAugment: Practical Automated Data Augmentation with a Reduced Search Space,” CVPR Workshops, 2020. doi:10.1109/CVPRW50498.2020.00359
- I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. arXiv
- I. Loshchilov and F. Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts,” ICLR, 2017. arXiv
- G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep Networks with Stochastic Depth,” ECCV, 2016. doi:10.1007/978-3-319-46493-0_39
- R. Wightman, H. Touvron, and H. Jégou, “ResNet Strikes Back: An Improved Training Procedure in timm,” arXiv:2110.00476, 2021. arXiv
- S. Kornblith, J. Shlens, and Q. V. Le, “Do Better ImageNet Models Transfer Better?,” CVPR, 2019. arXiv
- T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” ICML, 2020. arXiv
- K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020. arXiv
- M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging Properties in Self-Supervised Vision Transformers,” ICCV, 2021. doi:10.1109/ICCV48922.2021.00951
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022. doi:10.1109/CVPR52688.2022.01553
- M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, et al., “DINOv2: Learning Robust Visual Features without Supervision,” Transactions on Machine Learning Research, 2024. arXiv
- O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, et al., “DINOv3,” arXiv:2508.10104, 2025. arXiv
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, PMLR vol. 139, 2021. paper
- J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao, “A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence (arXiv:2301.05712, 2023). arXiv
- D. Hendrycks and T. Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations,” ICLR, 2019. arXiv
- D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, et al., “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization,” ICCV, 2021. arXiv
- D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural Adversarial Examples,” CVPR, 2021. arXiv
- I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” in Proc. International Conference on Learning Representations (ICLR), 2015. arXiv
- R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut Learning in Deep Neural Networks,” Nature Machine Intelligence, 2020. arXiv
- C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” ICML, 2017. arXiv
- B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning Deep Features for Discriminative Localization,” CVPR, 2016. doi:10.1109/CVPR.2016.319
- R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization,” ICCV, 2017. doi:10.1109/ICCV.2017.74
- B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” CVPR, 2018. doi:10.1109/CVPR.2018.00286
- G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” arXiv:1503.02531, 2015. arXiv
- X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, “Scaling Vision Transformers,” CVPR, 2022. arXiv
- M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” CVPR, 2023. arXiv
- T. Fawcett, “An Introduction to ROC Analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006. doi:10.1016/j.patrec.2005.10.010
- J. Davis and M. Goadrich, “The Relationship Between Precision-Recall and ROC Curves,” Proc. 23rd International Conference on Machine Learning (ICML), pp. 233–240, 2006. doi:10.1145/1143844.1143874
- T. Saito and M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets,” PLOS ONE, vol. 10, no. 3, e0118432, 2015. doi:10.1371/journal.pone.0118432
- D. Chicco and G. Jurman, “The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation,” BMC Genomics, vol. 21, article 6, 2020. doi:10.1186/s12864-019-6413-7
- M. Sokolova and G. Lapalme, “A Systematic Analysis of Performance Measures for Classification Tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427–437, 2009. doi:10.1016/j.ipm.2009.03.002
- J. Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960. doi:10.1177/001316446002000104
- M.-L. Zhang and Z.-H. Zhou, “A Review on Multi-Label Learning Algorithms,” IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 8, pp. 1819–1837, 2014. doi:10.1109/TKDE.2013.39
- M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes (VOC) Challenge,” International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010. doi:10.1007/s11263-009-0275-4
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. paper
- H. Wang, S. Ge, E. P. Xing, and Z. C. Lipton, “Learning Robust Global Representations by Penalizing Local Predictive Power,” NeurIPS, 2019 (introduces ImageNet-Sketch). arXiv
- A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz, “ObjectNet: A Large-Scale Bias-Controlled Dataset for Pushing the Limits of Object Recognition Models,” NeurIPS, 2019. dataset page
- C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The Caltech-UCSD Birds-200-2011 Dataset,” Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. dataset page
- J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3D Object Representations for Fine-Grained Categorization,” IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 554–561, 2013 (introduces Stanford Cars). doi:10.1109/ICCVW.2013.77
- S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-Grained Visual Classification of Aircraft,” arXiv:1306.5151, 2013. arXiv
- O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar, “Cats and Dogs,” CVPR, 2012 (introduces Oxford-IIIT Pets). dataset page
- M.-E. Nilsback and A. Zisserman, “Automated Flower Classification over a Large Number of Classes,” Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), 2008 (introduces Oxford Flowers-102). dataset page
- L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101 – Mining Discriminative Components with Random Forests,” ECCV, 2014. dataset page
- L. Fei-Fei, R. Fergus, and P. Perona, “One-Shot Learning of Object Categories,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006 (the reference given for Caltech-101). dataset page
- X. Zhai, J. Puigcerver, A. Kolesnikov, et al., “A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark,” arXiv:1910.04867, 2019. arXiv
- Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, “Large-Scale Long-Tailed Recognition in an Open World,” CVPR, 2019 (introduces ImageNet-LT). arXiv