DLCV 14 · Why can an invisible change flip a prediction?
AdvancedDLCV 14 · Robustness: Distribution Shift, Adversarial Examples and AnomaliesImage lab
Description
Train a linear classifier on noisy stripe images of 64, 256 or 1,024 pixels, then change every pixel by the same few levels, once along the loss gradient (FGSM) and once with random signs.
Try this: At the default setting (32 × 32 pixels, ε = 8, FGSM) accuracy falls from 99.3% to 2.7%, while random signs of the same size leave it at 99.3%. The mean logit shift is +5.8, against a median clean margin of 3.5. Now switch the size to 16 × 16 and then to 8 × 8 at the same ε: FGSM leaves 76.3% and 94.7%. Next find the ε at which FGSM first brings accuracy below 50%: ε = 5 for 32 × 32, ε = 11 for 16 × 16, and none up to ε = 16 for 8 × 8 (70.7% at ε = 16). The classes differ by the same total amount at every size, so why does the same ε matter more when the image has more pixels?
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Only FGSM: accuracy 2.7% against 99.3% for random signs. The logit is a sum over 1,024 pixels. If all 1,024 changes of 8 levels have the sign that raises the loss, the logit moves by ε‖w‖₁ = 5.8, more than the median margin of 3.5. Random signs push half the pixels each way and cancel: the typical shift is about ε‖w‖₂ = 0.2. The change per pixel is the same; what differs is whether the 1,024 small changes add up.
Lab · DLCV 14 · Why can an invisible change flip a prediction?
Loading image…
Try this: At the default setting (32 × 32 pixels, ε = 8, FGSM) accuracy falls from 99.3% to 2.7%, while random signs of the same size leave it at 99.3%. The mean logit shift is +5.8, against a median clean margin of 3.5. Now switch the size to 16 × 16 and then to 8 × 8 at the same ε: FGSM leaves 76.3% and 94.7%. Next find the ε at which FGSM first brings accuracy below 50%: ε = 5 for 32 × 32, ε = 11 for 16 × 16, and none up to ε = 16 for 8 × 8 (70.7% at ε = 16). The classes differ by the same total amount at every size, so why does the same ε matter more when the image has more pixels?
Synthetic and tiny on purpose: no photographs. Each image is 0.5 plus or minus an oblique cosine stripe pattern (two periods across, one down) plus Gaussian noise with standard deviation 0.2 (51 of 255 levels), clipped to [0, 1]; class A has the stripes in one phase and class B in the opposite phase. The stripe amplitude per pixel is 3 × 0.2 / √d for d pixels, so the two classes differ by the same total amount at every size (before clipping, the best possible classifier is equally accurate at every size); only the number of pixels that carry the evidence changes. The classifier is a logistic regression with one weight per pixel, trained in your browser by 50 steps of full-batch gradient descent (step 8, weight decay 0.001) on 300 images and tested on 300 other images (seeds 11 and 12). Its logit is z = w·(x − 0.5) + b, so the gradient of the cross-entropy loss with respect to the pixels is (p − y)·w and FGSM, x′ = clip(x + ε·sign(∇ₓL), 0, 1), moves every pixel by ε toward the wrong class. With no clipping the logit moves by exactly ε‖w‖₁; random signs move it by about ε‖w‖₂ and cancel on average. Here ‖w‖₂ is 7.5, 7.5 and 6.9 for 64, 256 and 1,024 pixels, while ‖w‖₁ is 54.4, 102.8 and 186.2. ε is counted in levels of 8-bit brightness (1 level = 1/255). This is a linear toy that isolates the argument of Goodfellow et al.; it is not the digits experiment of the chapter, a deep network is not linear, and one draw of data and training is shown, so read the trend, not the exact percentages.