完整搜尋頁面 →

Gulrajani 與 Lopez-Paz:In Search of Lost Domain Generalization(ICLR 2021)

選讀・DLCV 14・穩健性:分布偏移、對抗樣本與異常 的論文導讀(第 3/3 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. 有效穩健性與線上的準確率
  2. 測試時的熵最小化
  3. 領域對抗訓練與梯度反轉層
  4. 決定評估結果的攻擊設定
  5. 隨機平滑的實作
  6. PatchCore 逐步解析

論文導讀 · 3

  1. Madry 等人:Towards Deep Learning Models Resistant to Adversarial Attacks(ICLR 2018)
  2. Athalye 等人:Obfuscated Gradients Give a False Sense of Security(ICML 2018)
  3. Gulrajani 與 Lopez-Paz:In Search of Lost Domain Generalization(ICLR 2021)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

許多領域泛化演算法報告了勝過單純訓練的進步,但它們用的資料集、架構不同,尤其是選擇超參數的規則(常常沒有說明)也不同,有時還用了測試領域 [19]。

教材流程圖 — 來源訓練、來源驗證先選出固定模型,再測目標領域;使用目標驗證的 oracle 是另外揭露的設定。
教材流程圖 — 來源訓練、來源驗證先選出固定模型,再測目標領域;使用目標驗證的 oracle 是另外揭露的設定。

方法

把模型選擇視為問題的一部分。DG 演算法必須附帶一套只使用訓練領域來選擇超參數與檢查點的規則。論文定義了三種規則:訓練領域驗證集(從每個訓練領域留出一部分)、留一領域交叉驗證(輪流留出一個訓練領域,其餘訓練,再取平均),以及測試領域驗證,這是一種偷看有標註測試領域資料的神諭,必須如實揭露。然後它實作了 DomainBed,一個包含七個多領域資料集(包括 Colored MNIST、Rotated MNIST、VLCS、PACS、Office-Home、TerraIncognita 與 DomainNet)、九個演算法與三種選擇規則的測試平台,對每個演算法與測試領域做 20 次試驗的隨機超參數搜尋,並將整個研究重複三次。

結果

在 ResNet-50、強資料擴增與仔細調參下,ERM 在訓練領域驗證下與所有先前發表的結果相當或更好,而且在任何一種選擇規則下,沒有演算法平均勝過 ERM 超過約一個百分點。訓練領域驗證比留一領域交叉驗證更有效。神諭規則給出較高的數字,這既顯示了可能的進步空間,也顯示神諭選出的結果有多樂觀。

判讀證據

主要的表格針對每種選擇規則,報告每個資料集在各測試領域上的平均準確率,以及三次重複的標準誤。沿著一列讀,就能在相同的基礎上比較一個演算法與 ERM;在不同表格之間比較同一個演算法,就能看到選擇規則的影響。要檢查:(a) 在相信任何差異之前,先看標準誤;(b) 附錄,其中彙整了文獻中三十多個演算法所報告的結果,可以看出已發表的數字與受控的結果差多遠。

限制與問題

結論是關於這些資料集、這些演算法與這個調參預算的;某個方法在別處仍可能有幫助。這些資料集是中小規模的影像基準;後來改變這個領域的基礎模型骨幹,並不在這項研究的範圍內。

結論

這是本篇反覆出現的教訓最清楚的例子:當基準線調得好、並且測試資料不進入模型選擇時,穩健性的進步就消失了。

參考文獻

  1. D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt and D. Song, “Natural Adversarial Examples,” CVPR, 2021. CVF
  2. P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts,” ICML, PMLR 139, 2021. PMLR
  3. J. Yang, K. Zhou, Y. Li and Z. Liu, “Generalized Out-of-Distribution Detection: A Survey,” International Journal of Computer Vision, vol. 132, no. 12, pp. 5635–5662, 2024. doi · arXiv
  4. D. Hendrycks and T. Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations,” ICLR, 2019. arXiv
  5. D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo et al., “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization,” ICCV, 2021. CVF
  6. A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum and B. Katz, “ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models,” NeurIPS, 2019. proceedings
  7. B. Recht, R. Roelofs, L. Schmidt and V. Shankar, “Do ImageNet Classifiers Generalize to ImageNet?,” ICML, PMLR 97, 2019. PMLR
  8. R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht and L. Schmidt, “Measuring Robustness to Natural Distribution Shifts in Image Classification,” NeurIPS, 2020. proceedings
  9. J. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon and L. Schmidt, “Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization,” ICML, PMLR 139, 2021. PMLR
  10. D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer and B. Lakshminarayanan, “AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty,” ICLR, 2020. arXiv
  11. R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann and W. Brendel, “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness,” ICLR, 2019. arXiv
  12. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, PMLR 139, 2021. PMLR
  13. A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave and L. Schmidt, “Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP),” ICML, PMLR 162, 2022. PMLR
  14. M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs et al., “Robust Fine-Tuning of Zero-Shot Models,” CVPR, 2022. CVF
  15. S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel and M. Bethge, “Improving robustness against common corruptions by covariate shift adaptation,” NeurIPS, 2020. proceedings
  16. D. Wang, E. Shelhamer, S. Liu, B. Olshausen and T. Darrell, “Tent: Fully Test-Time Adaptation by Entropy Minimization,” ICLR, 2021. arXiv
  17. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand and V. Lempitsky, “Domain-Adversarial Training of Neural Networks,” Journal of Machine Learning Research, vol. 17, no. 59, pp. 1–35, 2016. JMLR
  18. K. Zhou, Z. Liu, Y. Qiao, T. Xiang and C. C. Loy, “Domain Generalization: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. doi · arXiv
  19. I. Gulrajani and D. Lopez-Paz, “In Search of Lost Domain Generalization,” ICLR, 2021. arXiv
  20. Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan and J. Snoek, “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” NeurIPS, 2019. proceedings
  21. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow and R. Fergus, “Intriguing properties of neural networks,” ICLR, 2014. arXiv
  22. I. J. Goodfellow, J. Shlens and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” ICLR, 2015. arXiv
  23. A. Madry, A. Makelov, L. Schmidt, D. Tsipras and A. Vladu, “Towards Deep Learning Models Resistant to Adversarial Attacks,” ICLR, 2018. arXiv
  24. F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal and M. Hein, “RobustBench: a standardized adversarial robustness benchmark,” NeurIPS Datasets and Benchmarks Track, 2021. proceedings · leaderboard
  25. N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik and A. Swami, “Practical Black-Box Attacks against Machine Learning,” ACM Asia Conference on Computer and Communications Security (AsiaCCS), pp. 506–519, 2017. doi
  26. N. Carlini and D. Wagner, “Towards Evaluating the Robustness of Neural Networks,” IEEE Symposium on Security and Privacy (SP), pp. 39–57, 2017. doi
  27. D. Tsipras, S. Santurkar, L. Engstrom, A. Turner and A. Madry, “Robustness May Be at Odds with Accuracy,” ICLR, 2019. arXiv
  28. H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. El Ghaoui and M. I. Jordan, “Theoretically Principled Trade-off between Robustness and Accuracy,” ICML, PMLR 97, 2019. PMLR
  29. A. Athalye, N. Carlini and D. Wagner, “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples,” ICML, PMLR 80, 2018. PMLR
  30. F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” ICML, PMLR 119, 2020. PMLR
  31. J. Cohen, E. Rosenfeld and J. Z. Kolter, “Certified Adversarial Robustness via Randomized Smoothing,” ICML, PMLR 97, 2019. PMLR
  32. K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno and D. Song, “Robust Physical-World Attacks on Deep Learning Visual Classification,” CVPR, 2018. CVF
  33. T. B. Brown, D. Mané, A. Roy, M. Abadi and J. Gilmer, “Adversarial Patch,” arXiv:1712.09665, 2017. arXiv
  34. D. Hendrycks and K. Gimpel, “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks,” ICLR, 2017. arXiv
  35. W. Liu, X. Wang, J. D. Owens and Y. Li, “Energy-based Out-of-distribution Detection,” NeurIPS, 2020. proceedings
  36. K. Lee, K. Lee, H. Lee and J. Shin, “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks,” NeurIPS, 2018. proceedings
  37. Y. Sun, Y. Ming, X. Zhu and Y. Li, “Out-of-Distribution Detection with Deep Nearest Neighbors,” ICML, PMLR 162, 2022. PMLR
  38. D. Hendrycks, M. Mazeika and T. Dietterich, “Deep Anomaly Detection with Outlier Exposure,” ICLR, 2019. arXiv
  39. J. Zhang, J. Yang, P. Wang, H. Wang, Y. Lin, H. Zhang et al., “OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection,” arXiv:2306.09301, 2023. arXiv
  40. P. Bergmann, M. Fauser, D. Sattlegger and C. Steger, “MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection,” CVPR, 2019. CVF
  41. L. Heckler-Kram, J.-H. Neudeck, U. Scheler, R. König and C. Steger, “The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection,” International Journal of Computer Vision, vol. 134, no. 4, 2026. doi · arXiv
  42. K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox and P. Gehler, “Towards Total Recall in Industrial Anomaly Detection,” CVPR, 2022. CVF
  43. J. Bitterwolf, M. Müller and M. Hein, “In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation,” ICML, PMLR 202, 2023. PMLR
  44. J. Liang, R. He and T. Tan, “A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts,” International Journal of Computer Vision, vol. 133, no. 1, pp. 31–64, 2025. doi · arXiv
  45. N. Akhtar, A. Mian, N. Kardan and M. Shah, “Advances in Adversarial Attacks and Defenses in Computer Vision: A Survey,” IEEE Access, vol. 9, pp. 155161–155196, 2021. doi · arXiv
  46. G. Pang, C. Shen, L. Cao and A. van den Hengel, “Deep Learning for Anomaly Detection: A Review,” ACM Computing Surveys, vol. 54, no. 2, pp. 1–38, 2021. doi · arXiv
  47. J. Liu, G. Xie, J. Wang, S. Li, C. Wang, F. Zheng and Y. Jin, “Deep Industrial Image Anomaly Detection: A Survey,” Machine Intelligence Research, vol. 21, no. 1, pp. 104–135, 2024. doi · arXiv
  48. Z. Wang, T. Pang, C. Du, M. Lin, W. Liu and S. Yan, “Better Diffusion Models Further Improve Adversarial Training,” ICML, PMLR 202, 2023. PMLR
  49. J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. Peng et al., “OpenOOD: Benchmarking Generalized Out-of-Distribution Detection,” NeurIPS Datasets and Benchmarks Track, 2022. proceedings
  50. J. Jeong, Y. Zou, T. Kim, D. Zhang, A. Ravichandran and O. Dabeer, “WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation,” CVPR, 2023. CVF