Full search page →

The gradient of the soft Dice loss

Optional · supplementary note 3 of 4 for DLCV 04 · Image Segmentation: Semantic, Instance and Panoptic. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 4

  1. Gridding in stacked dilated convolutions
  2. Dense CRF post-processing
  3. The gradient of the soft Dice loss
  4. Boundary IoU in detail

Paper guides · 2

  1. Panoptic Segmentation: a reading guide
  2. Mask2Former: a reading guide

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

This note derives the derivative of the soft Dice coefficient and explains its unusual behaviour.

The derivative. Use the sum form D=2I/SD = 2I / S with I=∑ipigiI = \sum_i p_i g_i and S=∑ipi+∑igiS = \sum_i p_i + \sum_i g_i, where pi∈[0,1]p_i \in [0, 1] are predicted probabilities and gi∈{0,1}g_i \in \{0, 1\} the labels. By the quotient rule, for pixel jj,

∂D∂pj=2gjS−2IS2=2 (gjS−I)S2,\frac{\partial D}{\partial p_j} = \frac{2 g_j S - 2 I}{S^2} = \frac{2\,(g_j S - I)}{S^2},

because ∂I/∂pj=gj\partial I / \partial p_j = g_j and ∂S/∂pj=1\partial S / \partial p_j = 1. (Milletari et al. [15] give the analogous expression for their squared denominator.)

What it says.

  • Every background pixel (gj=0g_j = 0) has the same probability derivative ∂D/∂pj=−2I/S2\partial D/\partial p_j = -2I/S^2, whatever its current probability. Minimizing 1−D1-D therefore lowers those probabilities, but equal derivatives with respect to probabilities do not imply equal derivatives with respect to network logits.
  • Every foreground pixel gets the same positive gradient 2(S−I)/S22(S - I)/S^2.
  • The size of the push depends on global sums. When the predicted and true masks are both small, SS is small and the gradients are large; this is how Dice “listens” to small objects, and also why it can be unstable early in training.

Compare derivatives with respect to the same variable. For an independent binary sigmoid, pj=σ(zj)p_j=\sigma(z_j) with logit zjz_j, the Dice loss has ∂(1−D)/∂zj=−(∂D/∂pj)pj(1−pj)\partial(1-D)/\partial z_j=-(\partial D/\partial p_j)p_j(1-p_j). Binary cross-entropy has logit derivative pj−gjp_j-g_j, whereas its probability derivative is (pj−gj)/(pj(1−pj))(p_j-g_j)/(p_j(1-p_j)). A multiclass softmax requires its full Jacobian, which couples class channels; it is not just an independent sigmoid factor.

The code checks the probability derivative against finite differences.

Code · Python · 17 lines
import numpy as np

rng = np.random.default_rng(1)
g = (rng.random(200) < 0.05).astype(float)       # 200 pixels, about 5% foreground
p = rng.uniform(0.05, 0.6, 200)                  # some soft prediction

def dice(p, g):
    return 2 * (p * g).sum() / (p.sum() + g.sum())

I, S = (p * g).sum(), p.sum() + g.sum()
grad = 2 * (g * S - I) / S ** 2                  # analytic dD/dp_j
eps = 1e-6
num = np.array([(dice(p + eps * np.eye(200)[j], g) - dice(p - eps * np.eye(200)[j], g)) / (2 * eps)
                for j in range(200)])
print("max |analytic - numeric|:", f"{np.abs(grad - num).max():.1e}")
print("distinct gradient values on background pixels:", np.unique(grad[g == 0].round(12)).size)
print(f"foreground gradient {grad[g == 1][0]:+.5f}, background gradient {grad[g == 0][0]:+.5f}")

Output

max |analytic - numeric|: 9.0e-12
distinct gradient values on background pixels: 1
foreground gradient +0.02793, background gradient -0.00099

Edge cases. If the ground truth is empty (g=0g = 0), then I=0I = 0 and D=0D = 0 for any nonzero prediction, so the loss is 1 even for a nearly perfect prediction; if both are zero the ratio is undefined. Implementations add a smoothing constant to the numerator and denominator, compute the sums over the whole batch rather than per image, or add cross-entropy so that the gradient never depends on the Dice ratio alone. For the independent-sigmoid case above, the additional factor pj(1−pj)p_j(1-p_j) can make saturated pixels learn slowly despite a nonzero probability derivative.

References

  1. S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 7, 2022. doi · arXiv
  2. X. Li, H. Ding, H. Yuan, W. Zhang, J. Pang, G. Cheng, K. Chen, Z. Liu and C. C. Loy, “Transformer-based visual segmentation: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, 2024. doi · arXiv
  3. T. Zhou, W. Xia, F. Zhang, B. Chang, W. Wang, Y. Yuan, E. Konukoglu and D. Cremers, “Image segmentation in foundation model era: A survey,” arXiv:2408.12957, 2024. arXiv
  4. A. Kirillov, K. He, R. Girshick, C. Rother and P. Dollár, “Panoptic segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  5. J. Long, E. Shelhamer and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. doi · arXiv
  6. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS vol. 9351, 2015. doi
  7. L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in Proc. European Conference on Computer Vision (ECCV), 2018. doi · arXiv
  8. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, 2018. doi · arXiv
  9. F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in Proc. International Conference on Learning Representations (ICLR), 2016. arXiv
  10. L.-C. Chen, G. Papandreou, F. Schroff and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv:1706.05587, 2017. arXiv
  11. P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou and G. Cottrell, “Understanding convolution for semantic segmentation,” in Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), 2018. doi · arXiv
  12. H. Zhao, J. Shi, X. Qi, X. Wang and J. Jia, “Pyramid scene parsing network,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · arXiv
  13. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. International Conference on Learning Representations (ICLR), 2021. arXiv
  14. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez and P. Luo, “SegFormer: Simple and efficient design for semantic segmentation with transformers,” in Advances in Neural Information Processing Systems (NeurIPS), 2021. NeurIPS · arXiv
  15. F. Milletari, N. Navab and S.-A. Ahmadi, “V-Net: Fully convolutional neural networks for volumetric medical image segmentation,” in Proc. Fourth International Conference on 3D Vision (3DV), 2016. doi · arXiv
  16. T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi
  17. B. Cheng, A. G. Schwing and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” in Advances in Neural Information Processing Systems (NeurIPS), 2021. NeurIPS · arXiv
  18. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth and B. Schiele, “The Cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi · arXiv
  19. B. Cheng, R. Girshick, P. Dollár, A. C. Berg and A. Kirillov, “Boundary IoU: Improving object-centric image segmentation evaluation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi · arXiv
  20. K. He, G. Gkioxari, P. Dollár and R. Girshick, “Mask R-CNN,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi · arXiv
  21. S. Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, 2017. doi
  22. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, “Feature pyramid networks for object detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi
  23. A. Kirillov, Y. Wu, K. He and R. Girshick, “PointRend: Image segmentation as rendering,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. doi · arXiv
  24. A. Kirillov, R. Girshick, K. He and P. Dollár, “Panoptic feature pyramid networks,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi · arXiv
  25. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov and S. Zagoruyko, “End-to-end object detection with transformers,” in Proc. European Conference on Computer Vision (ECCV), 2020. doi · arXiv
  26. B. Cheng, I. Misra, A. G. Schwing, A. Kirillov and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. doi · arXiv
  27. J. Jain, J. Li, M. Chiu, A. Hassani, N. Orlov and H. Shi, “OneFormer: One transformer to rule universal image segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. doi · arXiv
  28. T. Kerssies, N. Cavagnero, A. Hermans, N. Norouzi, G. Averta, B. Leibe, G. Dubbelman and D. de Geus, “Your ViT is secretly an image segmentation model,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. CVF · arXiv
  29. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson et al., “Segment anything,” in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi · arXiv
  30. N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma et al., “SAM 2: Segment anything in images and videos,” in Proc. International Conference on Learning Representations (ICLR), 2025. OpenReview · arXiv
  31. N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris et al., “SAM 3: Segment anything with concepts,” in Proc. International Conference on Learning Representations (ICLR), 2026. OpenReview · arXiv
  32. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal et al., “Learning transferable visual models from natural language supervision,” in Proc. International Conference on Machine Learning (ICML), PMLR 139, 2021. PMLR
  33. B. Li, K. Q. Weinberger, S. Belongie, V. Koltun and R. Ranftl, “Language-driven semantic segmentation,” in Proc. International Conference on Learning Representations (ICLR), 2022. OpenReview · arXiv
  34. S. Cho, H. Shin, S. Hong, A. Arnab, P. H. Seo and S. Kim, “CAT-Seg: Cost aggregation for open-vocabulary semantic segmentation,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi · arXiv
  35. C. Zhu and L. Chen, “A survey on open-vocabulary detection and segmentation: Past, present, and future,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, 2024. doi
  36. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, “The PASCAL Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, no. 2, 2010. doi
  37. B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso and A. Torralba, “Scene parsing through ADE20K dataset,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi · CVF
  38. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan et al., “Microsoft COCO: Common objects in context,” in Proc. European Conference on Computer Vision (ECCV), 2014. doi · arXiv
  39. H. Caesar, J. Uijlings and V. Ferrari, “COCO-Stuff: Thing and stuff classes in context,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi · arXiv
  40. G. Neuhold, T. Ollmann, S. Rota Bulò and P. Kontschieder, “The Mapillary Vistas dataset for semantic understanding of street scenes,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017. doi
  41. P. Krähenbühl and V. Koltun, “Efficient inference in fully connected CRFs with Gaussian edge potentials,” in Advances in Neural Information Processing Systems (NeurIPS), 2011. NeurIPS · arXiv