Full search page →

DLCV 04 · Does high pixel accuracy mean a good segmentation?

AdvancedDLCV 04 · Image Segmentation: Semantic, Instance and PanopticImage lab

Description

Simulate a segmentation network that predicts at a coarse output stride and compare pixel accuracy, mean IoU and the IoU of the smallest class.

Try this: At the default output stride 8 the prediction gets about 95% of the pixels right, yet the small squares have an IoU of 0. Lower the stride to 4, then 2: at which stride does the thin pole come back, and why does it need a finer stride than the squares? Then raise the stride to 32: how many of the five classes still have a non-zero IoU? Finally switch the enlarging method to bilinear at stride 8: which numbers change, and which one stays at 0?

Predict first

A network predicts at output stride 8, so one score cell covers 8 × 8 pixels. Its cell scores are the best possible (the exact class fractions). The thin pole is 3 pixels wide and the small squares are 6 × 6. What do the scores look like?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Pixel accuracy stays high while the small classes disappear. In an 8 × 8 cell the pole fills at most 3 × 8 = 24 of the 64 pixels, and none of the five squares fills more than 24 of 64 either, because they straddle cell borders. The background holds the largest share of every such cell, so the whole cell is labelled background. The pole and the squares together hold only about 2.7% of the pixels, so losing them costs little pixel accuracy, but every class counts one fifth of mIoU. Pixel accuracy mostly measures the large background.

Lab · DLCV 04 · Does high pixel accuracy mean a good segmentation?

Predict first

A network predicts at output stride 8, so one score cell covers 8 × 8 pixels. Its cell scores are the best possible (the exact class fractions). The thin pole is 3 pixels wide and the small squares are 6 × 6. What do the scores look like?

Loading image…

Ground truth (5 classes)
Prediction of the simulated network
Wrong pixels in red
Pixel accuracy, mIoU and the IoU of every class

Try this: At the default output stride 8 the prediction gets about 95% of the pixels right, yet the small squares have an IoU of 0. Lower the stride to 4, then 2: at which stride does the thin pole come back, and why does it need a finer stride than the squares? Then raise the stride to 32: how many of the five classes still have a non-zero IoU? Finally switch the enlarging method to bilinear at stride 8: which numbers change, and which one stays at 0?

The label map is synthetic (128 × 128, drawn by code, no random numbers): a background of 85% of the pixels, a disk, a block, a 3-pixel pole and five 6 × 6 squares, all in separate classes. The network is idealized: at output stride s its score for each class in an s × s cell is the fraction of the cell’s pixels that belong to the class, which is the best a stride-s head can report. With nearest enlargement the class with the largest fraction labels the whole cell (ties go to the lowest class number); with bilinear enlargement the fractions are interpolated per class and the largest score labels each pixel. It is the construction of the chapter’s toy experiment on a smaller scene, so its numbers are this lab’s own and not the chapter’s. The scores follow the chapter: pixel accuracy is the share of correct pixels, the IoU of a class is true positives divided by true positives plus misses plus false alarms, and mIoU is the mean of the IoU over all five classes, the background included, computed from one confusion matrix.

Method and image sources

Open in the Playground →