Full search page →

DLCV 14 · Why can an invisible change flip a prediction?

AdvancedDLCV 14 · Robustness: Distribution Shift, Adversarial Examples and AnomaliesImage lab

Description

Train a linear classifier on noisy stripe images of 64, 256 or 1,024 pixels, then change every pixel by the same few levels, once along the loss gradient (FGSM) and once with random signs.

Try this: At the default setting (32 × 32 pixels, ε = 8, FGSM) accuracy falls from 99.3% to 2.7%, while random signs of the same size leave it at 99.3%. The mean logit shift is +5.8, against a median clean margin of 3.5. Now switch the size to 16 × 16 and then to 8 × 8 at the same ε: FGSM leaves 76.3% and 94.7%. Next find the ε at which FGSM first brings accuracy below 50%: ε = 5 for 32 × 32, ε = 11 for 16 × 16, and none up to ε = 16 for 8 × 8 (70.7% at ε = 16). The classes differ by the same total amount at every size, so why does the same ε matter more when the image has more pixels?

Predict first

Every pixel of a 32 × 32 test image is changed by at most 8 of 255 levels, far below the pixel noise of 51 levels. Which change brings the accuracy of the trained classifier close to zero: signs that follow the loss gradient (FGSM), random signs, both, or neither?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Only FGSM: accuracy 2.7% against 99.3% for random signs. The logit is a sum over 1,024 pixels. If all 1,024 changes of 8 levels have the sign that raises the loss, the logit moves by ε‖w‖₁ = 5.8, more than the median margin of 3.5. Random signs push half the pixels each way and cancel: the typical shift is about ε‖w‖₂ = 0.2. The change per pixel is the same; what differs is whether the 1,024 small changes add up.

Lab · DLCV 14 · Why can an invisible change flip a prediction?

Predict first

Every pixel of a 32 × 32 test image is changed by at most 8 of 255 levels, far below the pixel noise of 51 levels. Which change brings the accuracy of the trained classifier close to zero: signs that follow the loss gradient (FGSM), random signs, both, or neither?

Loading image…

Left: a clean test image (noisy stripes). Middle: the change, stretched to the full gray range for display (white = +ε, black = −ε). Right: the perturbed image. Under each, the logit and the class it gives.
Accuracy on 300 test images as ε grows

Try this: At the default setting (32 × 32 pixels, ε = 8, FGSM) accuracy falls from 99.3% to 2.7%, while random signs of the same size leave it at 99.3%. The mean logit shift is +5.8, against a median clean margin of 3.5. Now switch the size to 16 × 16 and then to 8 × 8 at the same ε: FGSM leaves 76.3% and 94.7%. Next find the ε at which FGSM first brings accuracy below 50%: ε = 5 for 32 × 32, ε = 11 for 16 × 16, and none up to ε = 16 for 8 × 8 (70.7% at ε = 16). The classes differ by the same total amount at every size, so why does the same ε matter more when the image has more pixels?

Synthetic and tiny on purpose: no photographs. Each image is 0.5 plus or minus an oblique cosine stripe pattern (two periods across, one down) plus Gaussian noise with standard deviation 0.2 (51 of 255 levels), clipped to [0, 1]; class A has the stripes in one phase and class B in the opposite phase. The stripe amplitude per pixel is 3 × 0.2 / √d for d pixels, so the two classes differ by the same total amount at every size (before clipping, the best possible classifier is equally accurate at every size); only the number of pixels that carry the evidence changes. The classifier is a logistic regression with one weight per pixel, trained in your browser by 50 steps of full-batch gradient descent (step 8, weight decay 0.001) on 300 images and tested on 300 other images (seeds 11 and 12). Its logit is z = w·(x − 0.5) + b, so the gradient of the cross-entropy loss with respect to the pixels is (p − y)·w and FGSM, x′ = clip(x + ε·sign(∇ₓL), 0, 1), moves every pixel by ε toward the wrong class. With no clipping the logit moves by exactly ε‖w‖₁; random signs move it by about ε‖w‖₂ and cancel on average. Here ‖w‖₂ is 7.5, 7.5 and 6.9 for 64, 256 and 1,024 pixels, while ‖w‖₁ is 54.4, 102.8 and 186.2. ε is counted in levels of 8-bit brightness (1 level = 1/255). This is a linear toy that isolates the argument of Goodfellow et al.; it is not the digits experiment of the chapter, a deep network is not linear, and one draw of data and training is shown, so read the trend, not the exact percentages.

Method and image sources

Open in the Playground →