DLCV 01 · How much of the image can one output unit see?
IntermediateML 03 · Convolution, Image Filtering and CNNsImage lab
Description
Pick a stack of convolution layers and a spot in a real photograph: the exact box of input pixels that one output unit depends on is drawn on the image, with its size, jump and the share of the image it covers.
Try this: Choose “Six 3×3 layers, stride 2 in every second” and click the image: the unit sees 29 × 29 pixels, 1.3% of the picture, and its neighbors are 8 pixels apart. Now choose “Four 3×3 layers, dilations 1, 2, 4, 8”: a similar field, 31 × 31 pixels (1.5%), but the jump stays 1, so every pixel keeps its own output unit. Finally click the top-left corner: why is the covered share smaller there than in the middle of the photograph?
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Stride 2 in the first layer wins: the field grows from 7 to 11, against 9 for a fourth layer and 9 for dilation 2 in the first layer. A stride doubles the jump, the pixel spacing between neighboring units, for every later layer, so each of the two remaining 3×3 layers now adds 4 pixels instead of 2. Dilation 2 on a layer fed by the input (jump 1) adds only 2 pixels, the same as one more layer. Where the stride sits matters: in the last layer it changes nothing, and the field stays 7.
Lab · DLCV 01 · How much of the image can one output unit see?
Loading image…
Try this: Choose “Six 3×3 layers, stride 2 in every second” and click the image: the unit sees 29 × 29 pixels, 1.3% of the picture, and its neighbors are 8 pixels apart. Now choose “Four 3×3 layers, dilations 1, 2, 4, 8”: a similar field, 31 × 31 pixels (1.5%), but the jump stays 1, so every pixel keeps its own output unit. Finally click the top-left corner: why is the covered share smaller there than in the middle of the photograph?
Real photograph: skimage.data.camera at 256 × 256 (credit below). Each layer is a square convolution with kernel k, stride s and dilation d, padded with zeros by (k_eff − 1)/2 on each side, where the effective kernel is k_eff = (k − 1)·d + 1. The size and jump come from the recurrence r ← r + (k_eff − 1)·j and j ← j·s (r = j = 1 at the input), and the box on the image from tracing the index interval of one output unit back through every layer: both agree for every stack here, and an independent SciPy computation of which input pixels influence one unit gives the same boxes. The unit snaps to the nearest position on the output grid, whose spacing is the jump. Where the box leaves the image, the part outside is zero padding and the covered share counts only the part inside. This is the theoretical receptive field: the set of pixels that can change the unit. Measured influence is not uniform; it is concentrated near the center of the box, so the effective receptive field is smaller. That is not computed here, and no weights are trained.
Method and image sources
- André Araujo, Wade Norris and Jack Sim, Computing Receptive Fields of Convolutional Neural Networks, Distill, 2019
- Wenjie Luo, Yujia Li, Raquel Urtasun and Richard Zemel, Understanding the Effective Receptive Field in Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems 29 (NIPS 2016)
- scikit-image test images (skimage.data): camera is CC0 by Lav Varshney; coins is from the Brooklyn Museum collection, no known copyright restrictions; astronaut is a NASA photograph in the public domain