NIST FRVT Part 3:族群效應(Grother、Ngan 與 Hanaoka,2019)
選讀・DLCV 02A・臉部辨識與度量學習 的論文導讀(第 2/3 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
挑戰
關於臉部辨識偏差的公開說法,常常根據小型研究、不同的任務(例如性別分類),或單一種錯誤。NIST 著手在大型營運資料集上量化當代演算法的族群差異 [2]。

方法
把許多送交的一對一與一對多演算法,在四個附有性別、年齡與種族或出生國家詮釋資料的大型美國政府影像集合上執行,並回報固定門檻下各群體的假陽性率與假陰性率 [2]。
結果
執行摘要的主要結果是:假陽性差異遠大於假陰性差異,各族群之間常常相差 10 倍到超過 100 倍,且效應的大小甚至方向都取決於演算法與資料集;影像品質解釋了幾項假陰性的差異;有些準確的一對多演算法的假陽性差異測不出來 [2]。
判讀證據
對任何你要引用的數字,一定要檢查三件事:是哪一種錯誤(FMR 或 FNMR、FPIR 或 FNIR)、在什麼門檻下、在哪個資料集上。報告在冗長的附錄中給出每種演算法的結果,所以「臉部辨識對 X 有偏差」不是報告所支持的陳述;「演算法 A 在資料集 D 上、門檻 t 下,群體之間的誤匹配率相差 X」才是。資料集不是公開的,所以結果無法在 NIST 以外重現,而且這些演算法是當時送交的原型,不一定是已部署的產品 [2]。
限制與問題
四個資料集中有三個以出生國家作為種族的代理 [2];報告測量差異,但不決定哪些差異可以接受;而且演算法自 2019 年以來已經改變,所以目前每種演算法的結果應取自 NIST 持續進行的評估 [3]。
結論
它是公平性一節主要論點的依據,也是建議在工作點門檻下針對每個群體回報兩種錯誤率的依據。
明確保留操作點
對一對一驗證,假設相似度 愈大愈像同身分, 就接受。依報告定義的比較分層 :
是閾值, 必須說明影像對/人口如何分組,跨群體、同群體影像對不一定屬同一分層。兩種錯誤都依閾值與影像人口改變。各群體分別設定閾值,與在共同閾值比較群體,是不同操作政策問題;任一錯誤率本身都不是完整公平定義 [2]。
一對多搜尋還依賴圖庫與搜尋決策,FPIR/FNIR 不能直接代換這兩種驗證錯誤。觀測差異時要連同不確定性、小樣本比較數一起讀;沒有偵測到差異,不代表所有人口、拍攝條件完全相同 [2]、[3]。
參考文獻
- M. Wang and W. Deng, “Deep face recognition: A survey,” Neurocomputing, vol. 429, pp. 215–244, 2021. doi
- P. Grother, M. Ngan and K. Hanaoka, “Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects,” NIST Interagency Report 8280, National Institute of Standards and Technology, 2019. doi
- National Institute of Standards and Technology, “Face Recognition Technology Evaluation (FRTE) 1:1 Verification,” evaluation web page, accessed 2026. NIST
- K. Zhang, Z. Zhang, Z. Li and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016. doi
- J. Deng, J. Guo, E. Ververas, I. Kotsia and S. Zafeiriou, “RetinaFace: Single-shot multi-level face localisation in the wild,” in Proc. IEEE/CVF CVPR, 2020. CVF
- Y. Taigman, M. Yang, M. Ranzato and L. Wolf, “DeepFace: Closing the gap to human-level performance in face verification,” in Proc. IEEE CVPR, 2014. CVF
- F. Schroff, D. Kalenichenko and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in Proc. IEEE CVPR, 2015. CVF
- S. Chopra, R. Hadsell and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in Proc. IEEE CVPR, vol. 1, pp. 539–546, 2005. doi
- R. Hadsell, S. Chopra and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in Proc. IEEE CVPR, vol. 2, pp. 1735–1742, 2006. doi
- A. Hermans, L. Beyer and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv:1703.07737, 2017. arXiv
- F. Wang, X. Xiang, J. Cheng and A. L. Yuille, “NormFace: L2 hypersphere embedding for face verification,” in Proc. ACM International Conference on Multimedia, pp. 1041–1049, 2017. doi
- W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj and L. Song, “SphereFace: Deep hypersphere embedding for face recognition,” in Proc. IEEE CVPR, 2017. CVF
- H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li and W. Liu, “CosFace: Large margin cosine loss for deep face recognition,” in Proc. IEEE/CVF CVPR, 2018. CVF
- F. Wang, J. Cheng, W. Liu and H. Liu, “Additive margin softmax for face verification,” IEEE Signal Processing Letters, vol. 25, no. 7, pp. 926–930, 2018. doi
- J. Deng, J. Guo, N. Xue and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” in Proc. IEEE/CVF CVPR, 2019. CVF
- J. Deng, J. Guo, J. Yang, N. Xue, I. Kotsia and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 5962–5979, 2022. doi
- Y. Wen, K. Zhang, Z. Li and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in Proc. ECCV, pp. 499–515, 2016. doi
- J. Deng, J. Guo, T. Liu, M. Gong and S. Zafeiriou, “Sub-center ArcFace: Boosting face recognition by large-scale noisy web faces,” in Proc. ECCV, pp. 741–757, 2020. doi
- Q. Meng, S. Zhao, Z. Huang and F. Zhou, “MagFace: A universal representation for face recognition and quality assessment,” in Proc. IEEE/CVF CVPR, 2021. CVF
- M. Kim, A. K. Jain and X. Liu, “AdaFace: Quality adaptive margin for face recognition,” in Proc. IEEE/CVF CVPR, 2022. CVF
- X. An, J. Deng, J. Guo, Z. Feng, X. Zhu, J. Yang and T. Liu, “Killing two birds with one stone: Efficient and robust training of face recognition CNNs by Partial FC,” in Proc. IEEE/CVF CVPR, 2022. CVF
- A. Martin, G. Doddington, T. Kamm, M. Ordowski and M. Przybocki, “The DET curve in assessment of detection task performance,” in Proc. Eurospeech, pp. 1895–1898, 1997. ISCA
- G. B. Huang, M. Mattar, T. Berg and E. Learned-Miller, “Labeled Faces in the Wild: A database for studying face recognition in unconstrained environments,” in Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, Marseille, 2008. HAL
- I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller and E. Brossard, “The MegaFace benchmark: 1 million faces for recognition at scale,” in Proc. IEEE CVPR, 2016. CVF
- University of Washington, “MegaFace and MF2: Million-scale face recognition,” project web page, accessed 2026. project page
- Y. Guo, L. Zhang, Y. Hu, X. He and J. Gao, “MS-Celeb-1M: A dataset and benchmark for large-scale face recognition,” in Proc. ECCV, pp. 87–102, 2016. doi
- C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, J. Cheney and P. Grother, “IARPA Janus Benchmark-B face dataset,” in Proc. IEEE CVPR Workshops, 2017. CVF
- B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney and P. Grother, “IARPA Janus Benchmark – C: Face dataset and protocol,” in Proc. International Conference on Biometrics (ICB), pp. 158–165, 2018. doi
- Z. Zhu, G. Huang, J. Deng, Y. Ye, J. Huang, X. Chen, J. Zhu, T. Yang, J. Lu, D. Du and J. Zhou, “WebFace260M: A benchmark unveiling the power of million-scale deep face recognition,” in Proc. IEEE/CVF CVPR, 2021. CVF
- K. Peng, A. Mathur and A. Narayanan, “Mitigating dataset harms requires stewardship: Lessons from 1000 papers,” in Proc. NeurIPS Datasets and Benchmarks Track, 2021. proceedings
- G. Bae, M. de La Gorce, T. Baltrušaitis, C. Hewitt, D. Chen, J. Valentin, R. Cipolla and J. Shen, “DigiFace-1M: 1 million digital face images for face recognition,” in Proc. IEEE/CVF WACV, 2023. CVF
- M. Kim, F. Liu, A. Jain and X. Liu, “DCFace: Synthetic face generation with dual condition diffusion model,” in Proc. IEEE/CVF CVPR, 2023. CVF
- F. Boutros, V. Struc, J. Fierrez and N. Damer, “Synthetic data for face recognition: Current state and future prospects,” Image and Vision Computing, vol. 135, art. 104688, 2023. doi
- M. Wang, W. Deng, J. Hu, X. Tao and Y. Huang, “Racial Faces in the Wild: Reducing racial bias by information maximization adaptation network,” in Proc. IEEE/CVF ICCV, pp. 692–702, 2019. doi
- J. Buolamwini and T. Gebru, “Gender Shades: Intersectional accuracy disparities in commercial gender classification,” in Proc. Conference on Fairness, Accountability and Transparency, PMLR 81, pp. 77–91, 2018. PMLR
- P. Drozdowski, C. Rathgeb, A. Dantcheva, N. Damer and C. Busch, “Demographic bias in biometrics: A survey on an emerging challenge,” IEEE Transactions on Technology and Society, vol. 1, no. 2, pp. 89–103, 2020. doi
- Z. Yu, Y. Qin, X. Li, C. Zhao, Z. Lei and G. Zhao, “Deep learning for face anti-spoofing: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, pp. 5609–5631, 2023. doi
- European Commission, “What personal data is considered sensitive?,” data protection guidance referring to Article 9 of the GDPR, accessed 2026. European Commission
- European Commission, “AI Act” (Regulation (EU) 2024/1689), policy page, accessed 2026. European Commission
- M. Turk and A. Pentland, “Eigenfaces for recognition,” Journal of Cognitive Neuroscience, vol. 3, no. 1, pp. 71–86, 1991. doi
- P. N. Belhumeur, J. P. Hespanha and D. J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711–720, 1997. doi