Full search page →

Margin losses side by side: one formula and its edge cases

Optional · supplementary note 3 of 6 for DLCV 02A · Face Recognition and Metric Learning. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 6

  1. Aligning a face: the least-squares similarity transform
  2. How large must the scale s be?
  3. Margin losses side by side: one formula and its edge cases
  4. Beyond one center per class: sub-centers, quality-aware margins and Partial FC
  5. Measuring very small error rates
  6. Before deep learning: eigenfaces and Fisherfaces

Paper guides · 3

  1. ArcFace (Deng et al., CVPR 2019; TPAMI 2022)
  2. NIST FRVT Part 3: Demographic Effects (Grother, Ngan and Hanaoka, 2019)
  3. Guide to the Modern view surveys

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

This note writes the three margins as one formula, derives their two-class boundaries, and gives the gradient used in the toy code.

One formula. The ArcFace journal paper writes SphereFace, ArcFace and CosFace as special cases of a combined margin target logit [16]

ψ(θ)=cos⁡(m1θ+m2)−m3,\psi(\theta) = \cos(m_1 \theta + m_2) - m_3,

where m1m_1 is SphereFace’s multiplicative angular margin, m2m_2 ArcFace’s additive angular margin and m3m_3 CosFace’s additive cosine margin. (m1,m2,m3)=(1,0,0)(m_1, m_2, m_3) = (1, 0, 0) is normalized softmax; (1,0.5,0)(1, 0.5, 0) is ArcFace; (1,0,0.35)(1, 0, 0.35) is CosFace; and the paper also tries combinations of the three [16].

Two-class boundaries. Let the centers W1W_1 and W2W_2 be an angle φ\varphi apart and consider a feature on the arc between them, at angle θ1\theta_1 from W1W_1 and θ2=φ−θ1\theta_2 = \varphi - \theta_1 from W2W_2. Class 1 is satisfied when ψ(θ1)>cos⁡θ2\psi(\theta_1) \gt \cos\theta_2.

  • ArcFace: cos⁡(θ1+m)>cos⁡(φ−θ1)\cos(\theta_1 + m) \gt \cos(\varphi - \theta_1). Both arguments lie in [0,π][0, \pi] (when θ1+m≤π\theta_1 + m \le \pi), where cosine is decreasing, so the condition is θ1+m<φ−θ1\theta_1 + m \lt \varphi - \theta_1, that is θ1<(φ−m)/2\theta_1 \lt (\varphi - m)/2. By symmetry class 2 is satisfied when θ1>(φ+m)/2\theta_1 \gt (\varphi + m)/2, so the zone where neither is satisfied has width exactly mm, independent of φ\varphi.
  • SphereFace (on the first branch of ψ\psi): mθ1<φ−θ1m\theta_1 \lt \varphi - \theta_1, so θ1<φ/(m+1)\theta_1 \lt \varphi/(m+1); the gap is φ−2φ/(m+1)=φ(m−1)/(m+1)\varphi - 2\varphi/(m+1) = \varphi(m-1)/(m+1). Close centers get a small margin and distant centers a large one.
  • CosFace: cos⁡θ1−cos⁡(φ−θ1)>m\cos\theta_1 - \cos(\varphi - \theta_1) \gt m. Using cos⁡A−cos⁡B=−2sin⁡A+B2sin⁡A−B2\cos A - \cos B = -2\sin\frac{A+B}{2}\sin\frac{A-B}{2}, this is 2sin⁡φ2sin⁡(φ2−θ1)>m2\sin\frac{\varphi}{2}\sin\big(\frac{\varphi}{2} - \theta_1\big) \gt m, so θ1<φ2−arcsin⁡ ⁣(m2sin⁡(φ/2))\theta_1 \lt \frac{\varphi}{2} - \arcsin\!\big(\frac{m}{2\sin(\varphi/2)}\big). The angular gap is 2arcsin⁡ ⁣(m2sin⁡(φ/2))2\arcsin\!\big(\frac{m}{2\sin(\varphi/2)}\big): larger for close centers, and undefined (no satisfiable region on the arc) if 2sin⁡(φ/2)<m2\sin(\varphi/2) \lt m. For φ=80∘\varphi = 80^\circ and m=0.35m = 0.35 it is about 31∘31^\circ, as in Figure 2.

The gradient of the ArcFace logit. Training uses the derivative of the target logit with respect to the cosine c=cos⁡θc = \cos\theta, because the cosine is what the network computes. With θ=arccos⁡c\theta = \arccos c,

∂∂c scos⁡(θ+m)=s sin⁡(θ+m) 1sin⁡θ,\frac{\partial}{\partial c}\, s\cos(\theta + m) = s\,\sin(\theta + m)\,\frac{1}{\sin\theta},

since ddθcos⁡(θ+m)=−sin⁡(θ+m)\frac{d}{d\theta}\cos(\theta + m) = -\sin(\theta + m) and dθdc=−1/sin⁡θ\frac{d\theta}{dc} = -1/\sin\theta. This is the factor np.sin(th + margin) / np.sin(th) in the toy code. It is larger than 1 whenever sin⁡(θ+m)>sin⁡θ\sin(\theta + m) \gt \sin\theta, that is for θ<(π−m)/2\theta \lt (\pi - m)/2, so the margin amplifies the gradient on samples that are already close to their center, until the softmax probability saturates. Two edge cases follow. Near θ=0\theta = 0 the factor divides by sin⁡θ≈0\sin\theta \approx 0, so implementations clip the cosine away from ±1\pm 1 (the toy code clips at 1−10−71 - 10^{-7}). For θ+m>π\theta + m \gt \pi the factor becomes negative and the logit is no longer monotone, as noted in the main text; such samples need a fallback penalty.

Why easy samples stop contributing. The gradient with respect to the logits is pj−[j=yi]p_j - [j = y_i], where pjp_j is the softmax probability. When s ψ(θyi)s\,\psi(\theta_{y_i}) is far above every scos⁡θjs\cos\theta_j, pyi→1p_{y_i} \to 1 and all these differences go to zero. The margin moves the point where this happens: a sample must be inside its class’s satisfied region by a clear amount before its gradient vanishes, which is exactly how margins create compact clusters.

References

  1. M. Wang and W. Deng, “Deep face recognition: A survey,” Neurocomputing, vol. 429, pp. 215–244, 2021. doi
  2. P. Grother, M. Ngan and K. Hanaoka, “Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects,” NIST Interagency Report 8280, National Institute of Standards and Technology, 2019. doi
  3. National Institute of Standards and Technology, “Face Recognition Technology Evaluation (FRTE) 1:1 Verification,” evaluation web page, accessed 2026. NIST
  4. K. Zhang, Z. Zhang, Z. Li and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016. doi
  5. J. Deng, J. Guo, E. Ververas, I. Kotsia and S. Zafeiriou, “RetinaFace: Single-shot multi-level face localisation in the wild,” in Proc. IEEE/CVF CVPR, 2020. CVF
  6. Y. Taigman, M. Yang, M. Ranzato and L. Wolf, “DeepFace: Closing the gap to human-level performance in face verification,” in Proc. IEEE CVPR, 2014. CVF
  7. F. Schroff, D. Kalenichenko and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in Proc. IEEE CVPR, 2015. CVF
  8. S. Chopra, R. Hadsell and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in Proc. IEEE CVPR, vol. 1, pp. 539–546, 2005. doi
  9. R. Hadsell, S. Chopra and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in Proc. IEEE CVPR, vol. 2, pp. 1735–1742, 2006. doi
  10. A. Hermans, L. Beyer and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv:1703.07737, 2017. arXiv
  11. F. Wang, X. Xiang, J. Cheng and A. L. Yuille, “NormFace: L2 hypersphere embedding for face verification,” in Proc. ACM International Conference on Multimedia, pp. 1041–1049, 2017. doi
  12. W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj and L. Song, “SphereFace: Deep hypersphere embedding for face recognition,” in Proc. IEEE CVPR, 2017. CVF
  13. H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li and W. Liu, “CosFace: Large margin cosine loss for deep face recognition,” in Proc. IEEE/CVF CVPR, 2018. CVF
  14. F. Wang, J. Cheng, W. Liu and H. Liu, “Additive margin softmax for face verification,” IEEE Signal Processing Letters, vol. 25, no. 7, pp. 926–930, 2018. doi
  15. J. Deng, J. Guo, N. Xue and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” in Proc. IEEE/CVF CVPR, 2019. CVF
  16. J. Deng, J. Guo, J. Yang, N. Xue, I. Kotsia and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 5962–5979, 2022. doi
  17. Y. Wen, K. Zhang, Z. Li and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in Proc. ECCV, pp. 499–515, 2016. doi
  18. J. Deng, J. Guo, T. Liu, M. Gong and S. Zafeiriou, “Sub-center ArcFace: Boosting face recognition by large-scale noisy web faces,” in Proc. ECCV, pp. 741–757, 2020. doi
  19. Q. Meng, S. Zhao, Z. Huang and F. Zhou, “MagFace: A universal representation for face recognition and quality assessment,” in Proc. IEEE/CVF CVPR, 2021. CVF
  20. M. Kim, A. K. Jain and X. Liu, “AdaFace: Quality adaptive margin for face recognition,” in Proc. IEEE/CVF CVPR, 2022. CVF
  21. X. An, J. Deng, J. Guo, Z. Feng, X. Zhu, J. Yang and T. Liu, “Killing two birds with one stone: Efficient and robust training of face recognition CNNs by Partial FC,” in Proc. IEEE/CVF CVPR, 2022. CVF
  22. A. Martin, G. Doddington, T. Kamm, M. Ordowski and M. Przybocki, “The DET curve in assessment of detection task performance,” in Proc. Eurospeech, pp. 1895–1898, 1997. ISCA
  23. G. B. Huang, M. Mattar, T. Berg and E. Learned-Miller, “Labeled Faces in the Wild: A database for studying face recognition in unconstrained environments,” in Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, Marseille, 2008. HAL
  24. I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller and E. Brossard, “The MegaFace benchmark: 1 million faces for recognition at scale,” in Proc. IEEE CVPR, 2016. CVF
  25. University of Washington, “MegaFace and MF2: Million-scale face recognition,” project web page, accessed 2026. project page
  26. Y. Guo, L. Zhang, Y. Hu, X. He and J. Gao, “MS-Celeb-1M: A dataset and benchmark for large-scale face recognition,” in Proc. ECCV, pp. 87–102, 2016. doi
  27. C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, J. Cheney and P. Grother, “IARPA Janus Benchmark-B face dataset,” in Proc. IEEE CVPR Workshops, 2017. CVF
  28. B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney and P. Grother, “IARPA Janus Benchmark – C: Face dataset and protocol,” in Proc. International Conference on Biometrics (ICB), pp. 158–165, 2018. doi
  29. Z. Zhu, G. Huang, J. Deng, Y. Ye, J. Huang, X. Chen, J. Zhu, T. Yang, J. Lu, D. Du and J. Zhou, “WebFace260M: A benchmark unveiling the power of million-scale deep face recognition,” in Proc. IEEE/CVF CVPR, 2021. CVF
  30. K. Peng, A. Mathur and A. Narayanan, “Mitigating dataset harms requires stewardship: Lessons from 1000 papers,” in Proc. NeurIPS Datasets and Benchmarks Track, 2021. proceedings
  31. G. Bae, M. de La Gorce, T. Baltrušaitis, C. Hewitt, D. Chen, J. Valentin, R. Cipolla and J. Shen, “DigiFace-1M: 1 million digital face images for face recognition,” in Proc. IEEE/CVF WACV, 2023. CVF
  32. M. Kim, F. Liu, A. Jain and X. Liu, “DCFace: Synthetic face generation with dual condition diffusion model,” in Proc. IEEE/CVF CVPR, 2023. CVF
  33. F. Boutros, V. Struc, J. Fierrez and N. Damer, “Synthetic data for face recognition: Current state and future prospects,” Image and Vision Computing, vol. 135, art. 104688, 2023. doi
  34. M. Wang, W. Deng, J. Hu, X. Tao and Y. Huang, “Racial Faces in the Wild: Reducing racial bias by information maximization adaptation network,” in Proc. IEEE/CVF ICCV, pp. 692–702, 2019. doi
  35. J. Buolamwini and T. Gebru, “Gender Shades: Intersectional accuracy disparities in commercial gender classification,” in Proc. Conference on Fairness, Accountability and Transparency, PMLR 81, pp. 77–91, 2018. PMLR
  36. P. Drozdowski, C. Rathgeb, A. Dantcheva, N. Damer and C. Busch, “Demographic bias in biometrics: A survey on an emerging challenge,” IEEE Transactions on Technology and Society, vol. 1, no. 2, pp. 89–103, 2020. doi
  37. Z. Yu, Y. Qin, X. Li, C. Zhao, Z. Lei and G. Zhao, “Deep learning for face anti-spoofing: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, pp. 5609–5631, 2023. doi
  38. European Commission, “What personal data is considered sensitive?,” data protection guidance referring to Article 9 of the GDPR, accessed 2026. European Commission
  39. European Commission, “AI Act” (Regulation (EU) 2024/1689), policy page, accessed 2026. European Commission
  40. M. Turk and A. Pentland, “Eigenfaces for recognition,” Journal of Cognitive Neuroscience, vol. 3, no. 1, pp. 71–86, 1991. doi
  41. P. N. Belhumeur, J. P. Hespanha and D. J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711–720, 1997. doi