Full search page →

ML 03 · Convolution, Image Filtering and CNNs

Machine LearningIntermediate27 minUpdated Nov 6, 2026

Needs: ML 02 · Neural Networks from Perceptron to Deep Networks, Images as Arrays

Navigate: this page and this track: Navigate: this page and this track
Explore this chapter 2 interactive experiments

Predict the response of flat regions and edges as you change the kernel weights.

What you’ll learn

  • Explain filtering kernels and distinguish correlation from convolution.
  • Implement filtering with explicit data types and border rules.
  • Track channels, padding, stride, dilation and output shapes.
  • Relate receptive fields, translation behavior and backpropagation.
  • Compare CNN designs through their structural and computational choices.

The big picture

An image is a grid of numbers (see Images as arrays). Many of the most useful things we can do to it are local: whether a pixel sits on an edge, inside a smooth region or in a noisy patch depends mostly on its neighbors. Convolution is the operation that formalizes “look at a small neighborhood, combine it with fixed weights, and repeat everywhere.” It is the workhorse of classical image processing, and it is also the building block of convolutional neural networks (CNNs), which learn their weights from data instead of having them designed by hand.

This tutorial has two halves. The first half (filtering) is the classical toolbox and needs only NumPy. The second half (CNNs) builds on ML 02, where you met layers, losses and backpropagation. If you want more depth on fixed filters afterward, IP 03 covers spatial filtering and IP 04 explains the frequency-domain view through the convolution theorem. For the deep-learning side, DLCV 02 studies modern classification backbones in detail, ML 04 turns to sequences, and ML 05 introduces attention, the main alternative to convolution. Standard textbook treatments are chapter 9 of Goodfellow et al. [1] and chapter 10 of Prince [10].

Part I · Filtering with a fixed kernel

The sliding weighted sum

Plain version. Every output pixel is a weighted sum of the input pixels around the same position. The small grid of weights is called a kernel (also filter or mask).

Precise version. Let II be a grayscale image, with I(x,y)I(x,y) the intensity at column xx and row yy. Let KK be a kernel of size (2a+1)×(2b+1)(2a+1)\times(2b+1), indexed from −a-a to aa vertically and −b-b to bb horizontally, so that K(0,0)K(0,0) is its center. The filtered image GG is

G(x,y)=∑i=−aa∑j=−bbK(i,j) I(x+j,  y+i).G(x,y)=\sum_{i=-a}^{a}\sum_{j=-b}^{b}K(i,j)\,I(x+j,\;y+i).

Here ii indexes kernel rows and jj kernel columns, G(x,y)G(x,y) is the output at the same position, and the double sum visits every tap of the kernel once. For a 3×33\times3 kernel, a=b=1a=b=1 and each output pixel costs nine multiplications and eight additions.

Figure 1 works through one output value by hand. The orange window covers the top-left 3×33\times3 neighborhood of the padded input. Multiply element-wise by the kernel and add: 1⋅1+0⋅2+(−1)⋅0+2⋅0+0⋅1+(−2)⋅3+1⋅2+0⋅0+(−1)⋅1=−41\cdot1+0\cdot2+(-1)\cdot0+2\cdot0+0\cdot1+(-2)\cdot3+1\cdot2+0\cdot0+(-1)\cdot1=-4. That number becomes GG at the window’s center. Sliding the window one step to the right and repeating fills in the next output.

A five by five integer image with a ring of zero padding, a three by three kernel with columns 1,2,1 then zeros then -1,-2,-1, and the resulting five by five output; the orange window and the corresponding output entry minus four are highlighted.
Figure 1 — One step of the sliding weighted sum. The orange 3 × 3 window, multiplied element-wise by K and summed, gives −4 at the matching output position. Zero padding (gray) lets the output keep the input size.

Two observations already matter for everything that follows. First, the same nine weights are used at every position: the operation is shift-invariant (also called space-invariant). Second, the operation is linear: filtering αI1+βI2\alpha I_1+\beta I_2 gives αG1+βG2\alpha G_1+\beta G_2. Linear shift-invariant (LSI) operations are exactly the ones a convolution can express, which is why convolution is so central in signal processing.

Correlation versus convolution

Plain version. The formula above is technically cross-correlation. True convolution does the same thing with the kernel rotated by 180 degrees. For symmetric kernels the two agree; for asymmetric ones they differ.

Precise version. Mathematical convolution, written K∗IK*I, is

(K∗I)(x,y)=∑i=−aa∑j=−bbK(i,j) I(x−j,  y−i)=∑i,jK(−i,−j) I(x+j,  y+i).(K*I)(x,y)=\sum_{i=-a}^{a}\sum_{j=-b}^{b}K(i,j)\,I(x-j,\;y-i) =\sum_{i,j}K(-i,-j)\,I(x+j,\;y+i).

The second form shows that convolution equals correlation with the flipped kernel K(−i,−j)K(-i,-j). A clean way to see the difference is to filter a single bright pixel (an impulse). Correlation leaves a copy of the kernel rotated by 180 degrees; convolution leaves an exact copy of the kernel. That copy is the impulse response, and it is why signal processing prefers convolution: the output for any image is the sum of shifted, scaled impulse responses.

A kernel with entries 1 to 9, an input with a single bright pixel, the correlation output showing the kernel rotated by 180 degrees (9 at top left), and the convolution output showing the kernel unchanged (1 at top left).
Figure 2 — Filtering an impulse. Correlation (what cv2.filter2D computes) stamps a 180°-rotated kernel; convolution stamps the kernel itself.

Convolution has algebraic properties that correlation lacks. It is commutative (K∗I=I∗KK*I=I*K) and associative (K2∗(K1∗I)=(K2∗K1)∗IK_2*(K_1*I)=(K_2*K_1)*I), so two filters applied in a row equal one bigger filter. Applying a 3×33\times3 box blur twice, for instance, is a single 5×55\times5 kernel. Associativity also explains separable kernels: a k×kk\times k Gaussian equals a k×1k\times1 column filter followed by a 1×k1\times k row filter, cutting the cost per pixel from k2k^2 to 2k2k operations. The convolution theorem says convolution in space is multiplication in the Fourier domain; IP 04 develops that view.

Which libraries compute which?

Library callComputesDefault border
cv2.filter2Dcorrelation [11]reflect-101 (BORDER_DEFAULT)
scipy.ndimage.correlate / convolvecorrelation / convolutionreflect (edge sample repeated)
scipy.signal.convolve2dconvolutionzero fill
torch.nn.functional.conv2d, nn.Conv2dcorrelation [31]zero padding

For hand-designed asymmetric kernels (Sobel, emboss, motion blur) the flip changes the sign or direction of the result, so flip deliberately when you need true convolution, for example with cv2.flip(K, -1). For learned kernels the distinction disappears: a network that “should” use the flipped kernel simply learns it. That is why deep-learning frameworks call their correlation “convolution” without harm.

Blur, edges and sharpening

Plain version. Positive weights that sum to one average neighbors and blur. Weights that sum to zero cancel flat regions and keep only changes, so they find edges. Adding a bit of “the image minus its blur” back to the image sharpens it.

A box blur gives every neighbor equal weight; the 3×33\times3 version has nine entries of 1/91/9. A Gaussian blur samples

K(i,j)=1Zexp⁡ ⁣(−i2+j22σ2),K(i,j)=\frac{1}{Z}\exp\!\left(-\frac{i^2+j^2}{2\sigma^2}\right),

where σ\sigma (in pixels) sets the blur width and ZZ normalizes the weights to sum to one. Nearby pixels count more than distant ones, which avoids the blocky artifacts of a box. A practical rule is to use a kernel radius of about 3σ3\sigma.

The sum of the weights tells you what happens in a flat region where every pixel equals cc: the output is c∑Kc\sum K. A blur with sum one preserves brightness; a kernel with sum two doubles it (and saturates an 8-bit image); a kernel with sum zero outputs zero.

The Sobel kernels estimate horizontal and vertical derivatives while smoothing in the orthogonal direction:

Sx=[−101−202−101]=[121][−101],Sy=Sx⊤.S_x=\begin{bmatrix}-1&0&1\\-2&0&2\\-1&0&1\end{bmatrix} =\begin{bmatrix}1\\2\\1\end{bmatrix}\begin{bmatrix}-1&0&1\end{bmatrix}, \qquad S_y=S_x^\top.

SxS_x responds to intensity changes along xx, so it highlights vertical boundaries; SyS_y highlights horizontal ones. The outputs GxG_x, GyG_y are signed: dark-to-light and light-to-dark edges have opposite signs. The gradient magnitude Gx2+Gy2\sqrt{G_x^2+G_y^2} is a nonnegative edge strength, and atan2⁡(Gy,Gx)\operatorname{atan2}(G_y,G_x) is the edge orientation. The factorization shows that Sobel is separable: a [1,2,1]⊤[1,2,1]^\top smoothing column times a [−1,0,1][-1,0,1] central difference.

The Laplacian kernel [0101−41010]\begin{bmatrix}0&1&0\\1&-4&1\\0&1&0\end{bmatrix} approximates ∂2I/∂x2+∂2I/∂y2\partial^2I/\partial x^2+\partial^2I/\partial y^2; it responds to thin lines and to both sides of an edge. Unsharp masking sharpens with

Isharp=I+λ (I−Gσ∗I),I_{\text{sharp}}=I+\lambda\,(I-G_\sigma*I),

where Gσ∗IG_\sigma*I is a Gaussian-blurred copy, I−Gσ∗II-G_\sigma*I is the high-frequency detail, and λ>0\lambda\gt0 controls strength. Large λ\lambda amplifies noise and creates bright and dark halos along edges.

Six versions of the camera-man test image: original, box blurred, Gaussian blurred, signed Sobel x response in red and blue, gradient magnitude with bright outlines, and an unsharp-masked version with crisper edges.
Figure 3 — The same image through six kernels. Box and Gaussian blur suppress fine detail; Sobel x is signed (red = dark-to-light, blue = light-to-dark); the gradient magnitude outlines objects; unsharp masking adds back amplified detail.

Try it: the kernel playground

Edit the kernel cells below and watch the output update. Start from Sobel x, then transpose it to detect horizontal edges instead of vertical ones. Make every weight of the blur preset larger and notice that flat regions brighten, exactly as the sum-of-weights rule predicts. Try the Laplacian and emboss presets and predict their kernel sums before you look.

Kernel playgroundEdit any cell or pick a preset. The output updates as you type.
Input I
Output G = K ⋆ I

The slider below compares a synthetic test image (a disc, a square, a checkerboard and stripes) with a 5×55\times5 Gaussian blur, σ=1.4\sigma=1.4. The fine stripes and the checkerboard disappear first because a blur is a low-pass filter: it removes fast intensity variations while leaving slow ones.

InputGaussian σ=1.4

Code: from scratch, then OpenCV, SciPy and PyTorch

The loop below is the whole algorithm. Instead of looping over every pixel (slow in Python), it loops over the kernel taps and adds one shifted copy of the padded image per tap, which NumPy vectorizes. The script then checks that four libraries agree once you match the definition (correlation versus convolution) and the border rule. It needs numpy, scipy, opencv-python and torch.

Code · Python · 45 lines
import numpy as np
import cv2
import torch
import torch.nn.functional as F
from scipy import ndimage, signal

def correlate2d(img, K, mode="edge"):
    """Cross-correlation of a 2-D float image with an odd-sized kernel.
    mode is any np.pad mode: "constant" (zeros), "edge", "reflect", "wrap"."""
    kh, kw = K.shape
    ph, pw = kh // 2, kw // 2
    padded = np.pad(img, ((ph, ph), (pw, pw)), mode=mode)
    H, W = img.shape
    out = np.zeros((H, W), dtype=np.float64)
    for i in range(kh):              # loop over kernel taps, not pixels:
        for j in range(kw):          # each step adds one shifted copy of the image
            out += K[i, j] * padded[i:i + H, j:j + W]
    return out

def convolve2d(img, K, mode="edge"):
    return correlate2d(img, K[::-1, ::-1], mode)     # convolution = correlation with flipped K

rng = np.random.default_rng(0)
img = rng.random((32, 40))
K = rng.standard_normal((3, 5))                      # deliberately asymmetric

ours_corr = correlate2d(img, K, "reflect")
ours_conv = convolve2d(img, K, "reflect")
# OpenCV: filter2D computes correlation; BORDER_REFLECT_101 is np.pad's "reflect"
cv_corr = cv2.filter2D(img, cv2.CV_64F, K, borderType=cv2.BORDER_REFLECT_101)
# SciPy: ndimage.correlate / ndimage.convolve with mode="mirror" (= reflect-101)
sp_corr = ndimage.correlate(img, K, mode="mirror")
sp_conv = ndimage.convolve(img, K, mode="mirror")
# PyTorch: conv2d is correlation; pad first with the same rule
t = torch.tensor(img)[None, None]
t = F.pad(t, (2, 2, 1, 1), mode="reflect")
pt_corr = F.conv2d(t, torch.tensor(K)[None, None])[0, 0].numpy()

for name, a in [("OpenCV", cv_corr), ("SciPy", sp_corr), ("PyTorch", pt_corr)]:
    print(f"{name:8s} correlation matches:", np.allclose(ours_corr, a))
print("SciPy    convolution matches:", np.allclose(ours_conv, sp_conv))
print("conv == corr for this kernel?", np.allclose(ours_conv, ours_corr))
# zero padding: our convolution equals signal.convolve2d with mode="same"
same = signal.convolve2d(img, K, mode="same", boundary="fill")
print("zero-padded 'same' convolution matches:", np.allclose(convolve2d(img, K, "constant"), same))

All checks print True except conv == corr, which is False because the random kernel is asymmetric. Note the naming trap: NumPy’s "reflect" padding (dcb|abcd|cba, edge sample not repeated) is OpenCV’s BORDER_REFLECT_101 and SciPy’s "mirror", while SciPy’s "reflect" repeats the edge sample (dcba|abcd|dcba).

The second snippet shows the two bugs that cause most beginner frustration with real images: integer saturation and the kernel-sum rule.

Code · Python · 21 lines
import numpy as np
import cv2
from skimage import data

img_u8 = data.camera()                               # 512 x 512 uint8 test image
img = img_u8.astype(np.float32) / 255.0
sobel_x = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], np.float32)

gx = cv2.filter2D(img, -1, sobel_x)                  # float in, float out: negatives kept
gy = cv2.filter2D(img, -1, sobel_x.T)
mag = np.hypot(gx, gy)
print("float Sobel x range:", gx.min().round(3), gx.max().round(3))

gx_u8 = cv2.filter2D(img_u8, -1, sobel_x)            # uint8 in, uint8 out: saturated
print("uint8 Sobel x range:", gx_u8.min(), gx_u8.max(),
      "| pixels clipped to 0:", f"{(gx_u8 == 0).mean():.0%}")

blur = cv2.GaussianBlur(img, (5, 5), 1.4)            # 5 x 5 Gaussian, sigma 1.4
print("mean brightness kept by blur:", np.isclose(img.mean(), blur.mean(), atol=1e-3))
K2 = 2 * np.ones((3, 3), np.float32) / 9             # weights sum to 2
print("mean after sum-2 kernel:", round(float(cv2.filter2D(img, -1, K2).mean() / img.mean()), 3))

In floating point the Sobel response ranges from about −3.4-3.4 to 3.33.3. With uint8 input and output, every negative response is clipped to zero (more than half the pixels here), so all light-to-dark edges vanish. Always filter in float32 and convert back only for display.

Border handling

Plain version. Near the edge of the image, part of the kernel hangs outside. You must decide what values to pretend are there.

The common choices are: zero (constant) padding, which pretends the world outside is black; replicate, which repeats the edge pixel; reflect, which mirrors the image at its border; and wrap, which treats the image as periodic, as the discrete Fourier transform implicitly does. OpenCV’s default, BORDER_REFLECT_101, mirrors without repeating the edge pixel [11]. PyTorch’s nn.Conv2d uses zero padding unless you set padding_mode to 'reflect', 'replicate' or 'circular' [31].

Top row: the camera-man image with a 24-pixel border filled four ways, black, smeared edge pixels, mirrored image and wrapped image. Bottom row: a 21 by 21 box blur of each, where zero padding darkens the borders and wrap pulls the bottom of the image into the top.
Figure 4 — Four border rules and their effect on a large 21 × 21 box blur. Zero padding darkens every border; wrap blends opposite edges; replicate and reflect keep flat borders flat.

Zero padding darkens the borders of a blurred image because black pixels are averaged in. Wrap mixes the top of the image into the bottom. Replicate and reflect are usually the least visible choices for photographs. With small kernels the difference touches only a thin frame, but in a deep CNN every layer adds its own frame: after many zero-padded layers, a large fraction of the feature map has “seen” the artificial border, which lets networks infer absolute position from padding.

Part II · Convolutional neural networks

From a fixed filter to a learned layer

Plain version. A standard dense convolution learns filters spanning all input channels, each producing one output feature map. Grouped convolutions restrict that channel access.

Precise version. Let the input be X∈RCin×H×WX\in\mathbb{R}^{C_{\mathrm{in}}\times H\times W} (channels, height, width) and the weights K∈RCout×Cin×k×kK\in\mathbb{R}^{C_{\mathrm{out}}\times C_{\mathrm{in}}\times k\times k} with bias b∈RCoutb\in\mathbb{R}^{C_{\mathrm{out}}}. A stride-one layer with zero padding pp computes

Yo,u,v=bo+∑c=1Cin∑i=0k−1∑j=0k−1Ko,c,i,j Xc, u+i−p, v+j−p.Y_{o,u,v}=b_o+\sum_{c=1}^{C_{\mathrm{in}}}\sum_{i=0}^{k-1}\sum_{j=0}^{k-1} K_{o,c,i,j}\,X_{c,\,u+i-p,\,v+j-p}.

oo indexes output channels (filters), cc input channels, (u,v)(u,v) output positions and (i,j)(i,j) kernel taps; positions outside the image read the padding value. A nonlinearity such as ReLU is then applied element-wise. An RGB filter has three k×kk\times k slices, one per color channel, and their responses are summed into a single output number per position: a filter does not produce three separate images.

Two structural assumptions are built into this formula:

  • Local connectivity: each output depends only on a k×kk\times k neighborhood, not on the whole image.
  • Weight sharing (parameter sharing): the same KK is used at every position (u,v)(u,v).

Together they are the CNN’s inductive bias, an assumption about which functions are likely. Its payoff is efficiency. A layer with Cin=3C_{\mathrm{in}}=3, Cout=32C_{\mathrm{out}}=32 and k=3k=3 has 32×(3⋅9+1)=89632\times(3\cdot9+1)=896 parameters, whatever the image size. A fully connected layer from a flattened 224×224×3224\times224\times3 image to just 32 numbers needs 150,528×32≈4.8150{,}528\times32\approx4.8 million weights, and it would have to learn the same edge detector separately at every position.

Goodfellow et al. [1] summarize the motivation as sparse interactions, parameter sharing and equivariance. Figure 5 shows what a tiny CNN learns in its first layer when trained on the 8×88\times8 digits dataset used in the classifier code later on. The learned 3×33\times3 filters look like oriented derivative and blob detectors: the network rediscovered Sobel-like kernels because they are useful, not because anyone told it to. Zeiler and Fergus [30] showed the same phenomenon at scale by visualizing the filters and feature maps of large ImageNet networks.

A digit image followed by eight learned three by three filters shown in red and blue, and below each filter its ReLU feature map, highlighting different stroke orientations of the digit.
Figure 5 — First-layer filters learned on 8 × 8 digits (top) and their ReLU responses to one test digit (bottom). Many look like oriented edge or stroke detectors.

Padding, stride and dilation

Plain version. Padding controls what happens at the border and whether the output keeps its size. Stride skips positions to shrink the output. Dilation spreads the kernel taps apart to see farther without adding weights.

For input size nn (height or width), kernel size kk, padding pp on each side, stride ss and dilation dd, the output size is [12]

nout=⌊n+2p−d(k−1)−1s⌋+1.n_{\mathrm{out}}=\left\lfloor\frac{n+2p-d(k-1)-1}{s}\right\rfloor+1.

d(k−1)+1d(k-1)+1 is the effective kernel size: dilation dd inserts d−1d-1 gaps between taps. Three padding conventions have names: valid (p=0p=0, output shrinks by k−1k-1), same (p=(k−1)/2p=(k-1)/2 for odd kk and s=1s=1, output keeps its size) and full (p=k−1p=k-1, every partial overlap counted, output grows by k−1k-1). Stride two roughly halves each spatial dimension and is a common alternative to pooling for downsampling.

Four seven by seven grids showing the taps of two neighboring outputs: valid convolution shrinking to five by five, same convolution with padding keeping seven by seven, stride two producing four by four with taps two columns apart, and dilation two with spaced-out taps keeping seven by seven.
Figure 6 — Taps of two neighboring outputs (blue and orange; overlap appears brown) under four settings. Stride moves the window by s; dilation spaces the taps d apart.

Dilated (atrous) convolutions were popularized for dense prediction by Yu and Koltun [18]: stacking layers with dilations 1, 2, 4, … grows the field of view exponentially while keeping full resolution. A single dilated layer sees only a sparse lattice of inputs, so dilations should be mixed with care to avoid “gridding” artifacts. A transposed convolution runs the forward map of a strided convolution backward to upsample; it reappears in the backpropagation section below. The convolution arithmetic guide by Dumoulin and Visin [12] has diagrams for every combination of these settings.

The snippet checks the formula against PyTorch, counts parameters, shows that a 1×11\times1 convolution is a per-pixel linear layer, and measures translation equivariance:

Code · Python · 35 lines
import torch
from torch import nn

def out_size(n, k, p=0, s=1, d=1):
    return (n + 2 * p - d * (k - 1) - 1) // s + 1

x = torch.randn(1, 3, 64, 64)                        # one RGB 64 x 64 image
for k, p, s, d in [(3, 1, 1, 1), (5, 2, 2, 1), (3, 2, 1, 2), (1, 0, 1, 1)]:
    conv = nn.Conv2d(3, 16, k, stride=s, padding=p, dilation=d)
    y = conv(x)
    n_params = sum(q.numel() for q in conv.parameters())
    assert y.shape[-1] == out_size(64, k, p, s, d)
    assert n_params == 16 * (3 * k * k + 1)
    print(f"k={k} p={p} s={s} d={d}: output {tuple(y.shape[1:])}, parameters {n_params}")

# A 1x1 convolution is the same linear layer applied at every pixel
conv1 = nn.Conv2d(16, 4, kernel_size=1)
lin = nn.Linear(16, 4)
lin.weight.data = conv1.weight.data[:, :, 0, 0].clone()
lin.bias.data = conv1.bias.data.clone()
z = torch.randn(2, 16, 8, 8)
same = torch.allclose(conv1(z), lin(z.permute(0, 2, 3, 1)).permute(0, 3, 1, 2), atol=1e-6)
print("1x1 conv == per-pixel Linear:", same)

# Translation equivariance away from borders (stride 1, no padding effects in the centre)
conv = nn.Conv2d(1, 4, 3, padding=1, bias=False)
img = torch.zeros(1, 1, 32, 32); img[..., 10:14, 8:12] = 1
shift = lambda t: torch.roll(t, shifts=(3, 5), dims=(-2, -1))
with torch.no_grad():
    a, b = conv(shift(img)), shift(conv(img))
print("shift then conv == conv then shift:", torch.allclose(a, b, atol=1e-6))
conv2 = nn.Conv2d(1, 4, 3, padding=1, stride=2, bias=False)
with torch.no_grad():
    y0, y1 = conv2(img), conv2(torch.roll(img, 1, dims=-1))   # shift by one pixel
print("stride 2, 1-pixel shift: max output change =", round((y0 - y1).abs().max().item(), 3))

It prints output shapes 16×64×6416\times64\times64, 16×32×3216\times32\times32, 16×64×6416\times64\times64 and 16×64×6416\times64\times64 with 448, 1216, 448 and 64 parameters, confirms the 1×11\times1 and equivariance checks, and shows that a one-pixel shift changes a stride-two output by a clearly nonzero amount; the exact value depends on the randomly initialized weights.

Channels and 1×1 convolutions

Plain version. Channels are the “what” dimension of a feature map; height and width are the “where”. A 1×11\times1 convolution mixes channels at each pixel without looking at neighbors.

A 1×11\times1 convolution with CinC_{\mathrm{in}} inputs and CoutC_{\mathrm{out}} outputs multiplies the channel vector at every pixel by the same Cout×CinC_{\mathrm{out}}\times C_{\mathrm{in}} matrix, as the snippet verified. Lin et al.’s Network in Network [17] used this idea as a tiny per-pixel MLP and introduced global average pooling as a classifier head. GoogLeNet [16] used 1×11\times1 layers to reduce channels before expensive 3×33\times3 and 5×55\times5 branches, and ResNet’s bottleneck block [5] wraps a 3×33\times3 layer between a 1×11\times1 reduction and a 1×11\times1 expansion. In every case the motivation is the same: spatial filtering costs k2k^2 times more than channel mixing, so do the cheap part with 1×11\times1 layers and keep the expensive part narrow.

Pooling and downsampling

Plain version. Pooling summarizes each small block of a feature map by a single number, usually the maximum or the average. It shrinks the map and makes the representation less sensitive to small shifts.

Max pooling with a 2×22\times2 window and stride two keeps the largest value in each block: “was this feature present anywhere here?”. Average pooling keeps the mean. Global average pooling averages each channel over all positions, producing one number per channel regardless of input size, which is why modern classifiers can accept different image sizes. Pooling has no learnable parameters. Its gradient is simple: max pooling routes the whole upstream gradient to the winning input of each window; average pooling spreads it equally over the window.

A four by four grid of integers divided into four colored two by two blocks, the two by two max-pooled result keeping each block's largest value, and the two by two average-pooled result keeping each block's mean.
Figure 7 — Max and average pooling with a 2 × 2 window and stride 2. Each colored block becomes one number.

Downsampling, by pooling or by strided convolution, saves computation and enlarges later receptive fields, but it is also subsampling without a proper low-pass filter. High frequencies alias, and the output can change noticeably when the input moves by a single pixel, as the snippet above measured. Zhang [20] showed that inserting a small blur before subsampling (“anti-aliased” or “blur pooling”) makes standard CNNs measurably more shift-invariant: a neat case of a classical signal-processing rule fixing a deep network.

Equivariance is not invariance

Plain version. A stride-one convolution shifts with its input away from boundary effects. This equivariance does not make the prediction invariant to translation.

Let TΔT_\Delta shift an image by Δ\Delta pixels. Away from the borders, a stride-one convolution is translation-equivariant: f(TΔX)=TΔf(X)f(T_\Delta X)=T_\Delta f(X). A classifier wants invariance: g(TΔX)=g(X)g(T_\Delta X)=g(X). CNNs get approximate invariance by stacking equivariant layers and then pooling, especially global average pooling, which discards position. Padding, strides, cropping and the finite image break exact equivariance. Rotation and scale equivariance are not built in; data augmentation or specially designed group-equivariant layers must supply them.

The distinction matters in practice. A detector must be equivariant: moving a person should move the predicted box. A classifier should be invariant: moving the person should not change the label. The same backbone serves both, with different heads.

Receptive field

Plain version. Early layers see small pieces of the image; deeper layers combine those pieces and see larger regions.

The receptive field of a unit is the set of input pixels that can influence it. Let rℓr_\ell be its width after layer ℓ\ell, and jℓj_\ell the jump, the distance in input pixels between adjacent units of layer ℓ\ell. Starting from r0=j0=1r_0=j_0=1, a layer with kernel size kℓk_\ell, dilation dℓd_\ell and stride sℓs_\ell gives

rℓ=rℓ−1+(dℓ(kℓ−1)) jℓ−1,jℓ=sℓ jℓ−1.r_\ell=r_{\ell-1}+\big(d_\ell(k_\ell-1)\big)\,j_{\ell-1}, \qquad j_\ell=s_\ell\,j_{\ell-1}.

Three stacked stride-one 3×33\times3 layers see 7×77\times7 pixels, the same area as one 7×77\times7 layer but with 3⋅9C2=27C23\cdot9C^2=27C^2 instead of 49C249C^2 weights and three nonlinearities instead of one; this was the central design argument of VGG [15]. A stride-two layer doubles the jump, so every later layer grows the field twice as fast.

Three heatmaps of the gradient of a single central output with respect to the input, for three, six, and six strided averaging layers, each with a dashed box marking the theoretical receptive field of 7, 13 and 29 pixels; the bright region is concentrated in the middle of each box.
Figure 8 — Theoretical receptive field (dashed) versus actual influence, measured as the input gradient of one central output through stacks of 3 × 3 averaging layers. Influence is Gaussian-like and concentrated in the center; strides add a checkerboard pattern.

The theoretical field says which pixels could matter. Luo et al. [3] measured the effective receptive field via input gradients, as in Figure 8, and found that it is roughly Gaussian and occupies only a fraction of the theoretical field, growing like the square root of depth for uniform layers. Two practical lessons follow: a network may need more context than its theoretical field suggests, and downsampling, dilation and large kernels are ways to enlarge the effective field without unbounded depth.

Backpropagation through a convolution

Plain version. Kernel gradients combine inputs with upstream gradients. Input gradients use the transpose of the forward linear operator, with matching stride and padding conventions.

Precise version. Take one channel, no padding, stride one: Yu,v=∑i,jKi,jXu+i, v+jY_{u,v}=\sum_{i,j}K_{i,j}X_{u+i,\,v+j}. Let LL be the loss and Gu,v=∂L/∂Yu,vG_{u,v}=\partial L/\partial Y_{u,v} the upstream gradient delivered by backpropagation, as in ML 02. Each weight Ki,jK_{i,j} is used at every output position, so by the chain rule its gradient sums over them:

∂L∂Ki,j=∑u,vGu,v Xu+i, v+j.\frac{\partial L}{\partial K_{i,j}}=\sum_{u,v}G_{u,v}\,X_{u+i,\,v+j}.

That is a valid correlation of the input XX with GG, which is why weight sharing does not complicate training: shared weights simply accumulate gradients from every location. Each input pixel Xm,nX_{m,n} contributes to every output whose window covers it, at tap (i,j)=(m−u,n−v)(i,j)=(m-u,n-v):

∂L∂Xm,n=∑i,jGm−i, n−j Ki,j,\frac{\partial L}{\partial X_{m,n}}=\sum_{i,j}G_{m-i,\,n-j}\,K_{i,j},

with GG taken as zero outside its range. This is a full convolution of GG with KK, equivalently a correlation of the zero-padded GG with the flipped kernel. It is exactly what a transposed convolution computes, which is how transposed convolutions got their name. Bias gradients are just ∑u,vGu,v\sum_{u,v}G_{u,v} per output channel. With multiple channels, the input gradient also sums over output channels, and the kernel gradient also sums over the batch.

Code · Python · 20 lines
import torch
import torch.nn.functional as F

torch.manual_seed(0)
X = torch.randn(1, 1, 6, 7, requires_grad=True)     # input
K = torch.randn(1, 1, 3, 3, requires_grad=True)     # kernel
Y = F.conv2d(X, K)                                   # 'valid' correlation: 4 x 5
G = torch.randn_like(Y)                              # pretend upstream gradient dL/dY
(Y * G).sum().backward()                             # so that dL/dY = G exactly

# 1) dL/dK: correlate the input with the upstream gradient
dK = F.conv2d(X.detach(), G)
# 2) dL/dX: 'full' convolution of G with K = correlation of zero-padded G with flipped K
dX = F.conv2d(F.pad(G, (2, 2, 2, 2)), K.detach().flip(-1, -2))
# 3) the same thing is what a transposed convolution computes
dX_t = F.conv_transpose2d(G, K.detach())

print("dL/dK matches autograd:", torch.allclose(dK, K.grad, atol=1e-5))
print("dL/dX matches autograd:", torch.allclose(dX, X.grad, atol=1e-5))
print("conv_transpose2d gives the same:", torch.allclose(dX_t, X.grad, atol=1e-5))

All three lines print True. The trick (Y * G).sum().backward() makes autograd use a chosen upstream gradient GG, a handy way to test any hand-derived backward pass.

Classic architectures in brief

Plain version. Every famous CNN is a stack of convolutions, nonlinearities and downsampling. What changed over the years was depth, how to keep deep stacks trainable, and how to spend computation wisely.

Model (year)Key ideaWhat it taught
LeNet-5 (1998) [13]conv → subsample → conv → subsample → fully connected, trained end to endLearned convolutional features beat hand-designed ones for digits
AlexNet (2012) [14]Deeper and wider, ReLU, dropout, data augmentation, GPU trainingLarge labeled data plus GPUs made CNNs win ImageNet
VGG (2014) [15]Only 3×33\times3 convolutions, 16–19 layersDepth with small kernels is simple and effective
GoogLeNet / Inception (2014) [16]Parallel 1×11\times1, 3×33\times3, 5×55\times5 branches with 1×11\times1 reductionsMulti-scale features at controlled cost
ResNet (2015) [5]Residual connections y=x+F(x)y=x+F(x), BatchNormVery deep networks (100+ layers) become trainable
U-Net (2015) [4]Encoder–decoder with skip connections at each resolutionDense, pixel-aligned outputs from few images
MobileNetV2 (2018) [7]Depthwise-separable convolutions, inverted residuals, linear bottlenecksAccurate models for phones
EfficientNet (2019) [8]Compound scaling of depth, width and resolutionScale dimensions together, not one at a time
ConvNeXt (2022) [9]7×77\times7 depthwise kernels, LayerNorm, inverted bottlenecks, transformer-era trainingA modernized pure CNN competes with vision transformers

The pattern from LeNet to ResNet is a steady increase in depth. AlexNet [14] showed that a deep CNN trained on ImageNet with GPUs, ReLU and dropout could beat traditional pipelines by a wide margin. VGG [15] argued for uniform stacks of 3×33\times3 layers, as computed in the receptive-field section. GoogLeNet [16] made width cheap with 1×11\times1 reductions. Then depth hit an optimization wall: plain networks with more layers had higher training error, which is not overfitting but a failure to optimize. ResNet [5] fixed it with identity shortcuts. DLCV 02 follows this line further, through training recipes, transfer learning and evaluation.

Residual blocks and normalization

Plain version. Instead of asking a block to compute the whole output, ask it to compute a correction to its input.

A residual block computes y=x+Fθ(x)y=x+F_\theta(x), where FθF_\theta is typically two or three convolutions with normalization and ReLU. Its Jacobian is ∂y/∂x=I+∂Fθ/∂x\partial y/\partial x=I+\partial F_\theta/\partial x: even if FθF_\theta contributes little, the identity path carries the signal forward and the gradient backward. If Fθ≈0F_\theta\approx0 is the best a block can do, it can learn that easily, so adding blocks no longer hurts optimization. When the shape changes (more channels, stride two), the shortcut becomes a projection P(x)P(x), usually a strided 1×11\times1 convolution.

Batch normalization [19] standardizes each channel using the batch mean and variance during training, then applies a learned scale and shift; at evaluation it uses running averages. It makes training faster and less sensitive to initialization, but it misbehaves with very small or correlated batches, where GroupNorm [6] (statistics over groups of channels within each sample) is more reliable. Always call model.eval() before validation so BatchNorm uses its running statistics.

Code · Python · 30 lines
import torch
from torch import nn

count = lambda m: sum(p.numel() for p in m.parameters())

C = 64
dense = nn.Conv2d(C, C, 3, padding=1, bias=False)
separable = nn.Sequential(
    nn.Conv2d(C, C, 3, padding=1, groups=C, bias=False),   # depthwise: one 3x3 per channel
    nn.Conv2d(C, C, 1, bias=False))                        # pointwise: mix channels
print("dense:", count(dense), "| depthwise+pointwise:", count(separable),
      "| ratio:", round(count(separable) / count(dense), 4), "vs 1/C + 1/9 =",
      round(1 / C + 1 / 9, 4))

class BasicBlock(nn.Module):
    """ResNet-style block: y = relu(x + F(x)), with a 1x1 projection when shapes change."""
    def __init__(self, c_in, c_out, stride=1):
        super().__init__()
        self.f = nn.Sequential(
            nn.Conv2d(c_in, c_out, 3, stride, 1, bias=False), nn.BatchNorm2d(c_out), nn.ReLU(),
            nn.Conv2d(c_out, c_out, 3, 1, 1, bias=False), nn.BatchNorm2d(c_out))
        self.proj = (nn.Identity() if stride == 1 and c_in == c_out else
                     nn.Sequential(nn.Conv2d(c_in, c_out, 1, stride, bias=False),
                                   nn.BatchNorm2d(c_out)))
    def forward(self, x):
        return torch.relu(self.proj(x) + self.f(x))

x = torch.randn(2, 64, 32, 32)
print("same shape :", tuple(BasicBlock(64, 64)(x).shape))
print("downsample :", tuple(BasicBlock(64, 128, stride=2)(x).shape))

The output reports 36,864 weights for the dense layer against 4,672 for the separable pair, a ratio of 0.1267 that matches the formula below, and block outputs of shape 64×32×3264\times32\times32 and 128×16×16128\times16\times16.

Efficient CNNs separate spatial filtering from channel mixing

Plain version. Looking around (spatial filtering) and combining features (channel mixing) need not happen in one expensive operation.

A dense k×kk\times k convolution has k2CinCoutk^2C_{\mathrm{in}}C_{\mathrm{out}} weights. A depthwise convolution filters each channel separately (k2Cink^2C_{\mathrm{in}} weights, groups=C in PyTorch); a following pointwise 1×11\times1 convolution mixes channels (CinCoutC_{\mathrm{in}}C_{\mathrm{out}} weights). With Cin=Cout=CC_{\mathrm{in}}=C_{\mathrm{out}}=C,

depthwise + pointwisedense=k2C+C2k2C2=1C+1k2.\frac{\text{depthwise + pointwise}}{\text{dense}}=\frac{k^2C+C^2}{k^2C^2}=\frac{1}{C}+\frac{1}{k^2}.

For k=3k=3 and large CC this approaches 1/91/9.

Curves comparing nine C squared weights for dense convolution with nine C plus C squared for depthwise plus pointwise convolution, for C from 16 to 256.
Figure 9 — Analytical weight counts for equal input and output channels. Fewer weights do not automatically mean lower latency: depthwise layers have low arithmetic intensity and may run slowly on some hardware.

MobileNetV2 [7] builds blocks that expand channels with a 1×11\times1 layer, filter with a depthwise 3×33\times3, then project back with a linear (no ReLU) 1×11\times1 bottleneck, with a residual connection between the narrow ends. EfficientNet [8] scales depth, width and input resolution together with a single compound coefficient. ConvNeXt [9] started from a ResNet and applied, one at a time, design choices associated with vision transformers: patchify stem, 7×77\times7 depthwise kernels, inverted bottlenecks, fewer activations, LayerNorm, and longer training with heavy augmentation. The result matched Swin Transformers of similar size, evidence that much of the transformer advantage came from training recipes and macro-design rather than from attention alone.

Optional: ResNet (in the Paper guides). Read the original figure, evidence, design choices and limitations.

Code: a complete small-image classifier

This example needs torch, numpy and scikit-learn. It uses the bundled 8×88\times8 digits dataset, so no download or GPU is needed. A stratified split gives separate training, validation and test sets: validation picks the best epoch, and the test set is evaluated once at the end. It is a teaching baseline, not an ImageNet recipe.

Code · Python · 45 lines
import copy
import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.manual_seed(0)
torch.set_num_threads(1)
d = load_digits()
X = torch.tensor(d.images[:, None] / 16.0, dtype=torch.float32)
y = torch.tensor(d.target, dtype=torch.long)
train, test = train_test_split(np.arange(len(y)), test_size=.2,
                              stratify=d.target, random_state=0)
train, val = train_test_split(train, test_size=.2,
                             stratify=d.target[train], random_state=1)
model = nn.Sequential(
    nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(),
    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(),
    nn.MaxPool2d(2),                     # B × 32 × 4 × 4
    nn.Flatten(), nn.Linear(32 * 4 * 4, 10),
)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
best_loss, best_state = float("inf"), None
for epoch in range(12):
    model.train()
    order = torch.tensor(train)[torch.randperm(len(train))]
    for idx in order.split(64):
        loss = nn.functional.cross_entropy(model(X[idx]), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    with torch.no_grad():
        val_loss = nn.functional.cross_entropy(model(X[val]), y[val]).item()
    if val_loss < best_loss:
        best_loss = val_loss
        best_state = copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
model.eval()
with torch.no_grad():
    accuracy = (model(X[test]).argmax(1) == y[test]).float().mean().item()
print("parameters:", sum(p.numel() for p in model.parameters()))
print("best validation loss:", round(best_loss, 4))
print("held-out accuracy:", round(accuracy, 4))

On our CPU run it reports 9,930 parameters and about 96% held-out accuracy (exact numbers vary slightly across PyTorch versions). Cross-entropy receives logits, with no softmax in front. The head flattens the 32×4×432\times4\times4 map rather than global-average-pooling it, because on tiny digits the position of strokes carries information; for variable-size images, adaptive or global pooling gives a fixed-size head.

Inspect the errors, not just the score. Which digit pairs get confused? Does shifting every test digit by one pixel hurt? Compare against a parameter-matched MLP and a classical SVM, as in IP 13, before crediting every gain to convolution.

Modern view

Survey literature. Several reviews map the CNN landscape, each with a different emphasis. Gu et al. [23] (Pattern Recognition, 2018) organize CNN advances by component: convolution variants, pooling, activations, loss functions, regularization and optimization, followed by applications. Khan et al. [22] (Artificial Intelligence Review, 2020) propose a taxonomy of architectures by their main innovation: spatial exploitation (kernel sizes), depth, multi-path designs (shortcuts, Inception branches), width, feature-map exploitation (channel attention), and attention-based designs. Li et al. [21] (IEEE TNNLS, 2022) cover the history from LeNet onward, compare one-, two- and multi-dimensional convolutions, and survey lightweight networks and applications with rules of thumb for choosing hyperparameters. Alzubaidi et al. [2] (Journal of Big Data, 2021) give a broad, application-oriented review of deep learning centered on CNN architectures, with discussion of data scarcity, transfer learning and hardware. These surveys predate ConvNeXt, so read them as maps of the 2012–2020 era.

CNNs after transformers (2022–2026). Vision transformers (ML 05) prompted a re-examination of convolution, and the main finding is that the gap was smaller than early comparisons suggested:

  • Modernized CNNs. ConvNeXt [9] showed that a carefully modernized ResNet matches Swin. ConvNeXt V2 [24] (CVPR 2023) added masked-autoencoder pretraining with sparse convolutions and a global response normalization layer, bringing self-supervised pretraining (see ML 06) to pure CNNs.
  • Large kernels. RepLKNet [26] (CVPR 2022) used depthwise kernels as large as 31×3131\times31, with re-parameterized small kernels to ease optimization, to enlarge the effective receptive field. UniRepLKNet [27] (CVPR 2024) extended large-kernel design to audio, video, point clouds and time series.
  • Dynamic and deformable convolution. InternImage [25] (CVPR 2023) scaled deformable convolution, whose sampling locations are predicted from the input, into a large foundation backbone for detection and segmentation.
  • Scale. Smith et al. [29] (2023) pretrained NFNet CNNs on the large JFT-4B image dataset and found that their performance followed similar compute scaling trends to vision transformers under comparable budgets, arguing that compute and data, more than architecture, dominate at scale.
  • Edge devices. MobileNetV4 [28] (ECCV 2024) designed blocks and a neural architecture search for good latency across CPUs, GPUs, DSPs and accelerators, an acknowledgment that parameter counts alone do not predict speed.

Where CNNs fit in 2026. Convolution remains a default for latency-constrained and on-device inference, for dense prediction at high resolution (restoration, segmentation, medical imaging, see DLCV 09), for small datasets where the locality bias helps, and as the stem or early stages of hybrid CNN–transformer models. Large vision foundation models are mostly transformer-based, but they frequently contain convolutions in their patch embeddings, upsampling decoders and diffusion U-Nets.

Open problems. Exact shift equivariance under downsampling (aliasing [20]); building rotation and scale symmetry in without heavy cost; understanding what the effective receptive field really needs to be for a task [3]; and fair comparison across architectures, which requires matching data, augmentation, resolution, schedule and compute before attributing a difference to convolution versus attention.

Key takeaways

  • Convolution flips the kernel; correlation does not. Check the library convention when reproducing a filter.
  • Kernel sums determine the response to constant input, but a sum of one alone does not imply smoothing.
  • Learned filters share weights spatially; channel grouping determines which inputs each filter reads.
  • Padding, stride and dilation determine shapes and sampling; downsampling may introduce aliasing.
  • Translation equivariance differs from invariance; boundaries, stride and pooling affect the actual behavior.
  • Separate spatial filtering, channel mixing and skip connections when comparing architectures; measure efficiency at matched budgets.

Exercises

  1. Design and verify. Write a 3×33\times3 kernel that detects only horizontal edges, and another that detects only diagonal edges running from top-left to bottom-right. Verify both in the kernel playground or with cv2.filter2D on a synthetic image containing a horizontal, a vertical and a diagonal line.
Hint

A horizontal edge is a change along yy, so use Sy=Sx⊤S_y=S_x^\top (rows −1,−2,−1-1,-2,-1 / 0,0,00,0,0 / 1,2,11,2,1). For a diagonal edge, put the derivative across the diagonal, for example [012−101−2−10]\begin{bmatrix}0&1&2\\-1&0&1\\-2&-1&0\end{bmatrix}. Its weights sum to zero, so flat regions give zero.

  1. Kernel sums. Why should a blur kernel’s weights sum to one? Predict, then check, what happens to a mid-gray uint8 image filtered with a 3×33\times3 kernel whose weights all equal 2/92/9, and what changes if you filter in float32 instead.
Hint

A flat region of value cc maps to c∑Kc\sum K. With sum two, a gray of 128 becomes 256, which saturates at 255 in uint8; every brighter-than-mid-gray region turns white. In float32 the values double without clipping, and you can rescale afterward.

  1. Composition. Show that applying a 3×33\times3 box blur twice equals one 5×55\times5 kernel, and write that kernel. Verify numerically away from the borders, and explain why the equality can fail at the borders.
Hint

By associativity, the combined kernel is the box convolved with itself: the outer product of [1,2,3,2,1][1,2,3,2,1] with itself, divided by 81. At the border, the first pass already invented padding values, and the second pass pads that result; a single 5×55\times5 pass pads the original image instead, so the two differ in a frame of width two.

  1. Layer arithmetic. For a 64×6464\times64 RGB input and a layer with 16 filters, k=5k=5, p=2p=2, s=2s=2: give the output shape and the number of parameters. Then compute (r,j)(r,j) after three 3×33\times3 layers with strides 1, 2, 1, and compare with all strides equal to one.
Hint

⌊(64+4−4−1)/2⌋+1=32\lfloor(64+4-4-1)/2\rfloor+1=32, so the output is 16×32×3216\times32\times32 with 16(3⋅25+1)=121616(3\cdot25+1)=1216 parameters. Receptive field: (3,1)→(5,2)→(9,2)(3,1)\to(5,2)\to(9,2). All strides one gives (7,1)(7,1).

  1. Gradient by hand. For a 1×41\times4 input X=[x1,x2,x3,x4]X=[x_1,x_2,x_3,x_4] and a kernel K=[k1,k2]K=[k_1,k_2] with valid correlation, write YY, then derive ∂L/∂K\partial L/\partial K and ∂L/∂X\partial L/\partial X in terms of G=∂L/∂YG=\partial L/\partial Y. Check your formulas with the autograd trick from the backpropagation snippet.
Hint

Y=[k1x1+k2x2,  k1x2+k2x3,  k1x3+k2x4]Y=[k_1x_1+k_2x_2,\;k_1x_2+k_2x_3,\;k_1x_3+k_2x_4]. Then ∂L/∂k1=G1x1+G2x2+G3x3\partial L/\partial k_1=G_1x_1+G_2x_2+G_3x_3 and ∂L/∂k2=G1x2+G2x3+G3x4\partial L/\partial k_2=G_1x_2+G_2x_3+G_3x_4 (a correlation of XX with GG). ∂L/∂X=[G1k1,  G1k2+G2k1,  G2k2+G3k1,  G3k2]\partial L/\partial X=[G_1k_1,\;G_1k_2+G_2k_1,\;G_2k_2+G_3k_1,\;G_3k_2], the full convolution of GG with KK.

  1. Ablation on digits. In the classifier, replace Flatten + Linear(512, 10) by global average pooling + Linear(32, 10). Compare parameter counts, validation loss and test accuracy, then evaluate both models on test digits shifted by one pixel. Explain the trade-off you observe.
Hint

nn.AdaptiveAvgPool2d(1) then nn.Flatten(). Pooling removes most head parameters and position information, which can help shift robustness but hurt on 8×88\times8 digits where stroke position matters. Keep everything else fixed, select on validation only, and report both accuracy and parameter count.

Paper guides

1 paper guide, each on a page of its own.

Open the paper guides →

References

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 9. Book
  2. L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of Big Data, 2021. Review
  3. W. Luo, Y. Li, R. Urtasun and R. Zemel, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks,” NeurIPS, 2016. Paper
  4. O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, 2015. Paper
  5. K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. Paper
  6. Y. Wu and K. He, “Group Normalization,” ECCV, 2018. Paper
  7. M. Sandler et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” CVPR, 2018. Paper
  8. M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” ICML, 2019. Paper
  9. Z. Liu et al., “A ConvNet for the 2020s,” CVPR, 2022. Paper
  10. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023, ch. 10. Book and notebooks
  11. OpenCV, “Image Filtering” (filter2D, border types), documentation, version 4.x. Documentation
  12. V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv:1603.07285, 2016. arXiv
  13. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, 1998. DOI
  14. A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS, 2012. Paper
  15. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” ICLR, 2015. arXiv
  16. C. Szegedy et al., “Going Deeper with Convolutions,” CVPR, 2015. arXiv
  17. M. Lin, Q. Chen and S. Yan, “Network In Network,” ICLR, 2014. arXiv
  18. F. Yu and V. Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” ICLR, 2016. arXiv
  19. S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” ICML, 2015. arXiv
  20. R. Zhang, “Making Convolutional Networks Shift-Invariant Again,” ICML, 2019. arXiv
  21. Z. Li, F. Liu, W. Yang, S. Peng and J. Zhou, “A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects,” IEEE Transactions on Neural Networks and Learning Systems, 2022. DOI
  22. A. Khan, A. Sohail, U. Zahoora and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artificial Intelligence Review, 2020. arXiv
  23. J. Gu et al., “Recent Advances in Convolutional Neural Networks,” Pattern Recognition, 2018. arXiv
  24. S. Woo et al., “ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders,” CVPR, 2023. arXiv
  25. W. Wang et al., “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions,” CVPR, 2023. arXiv
  26. X. Ding et al., “Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs” (RepLKNet), CVPR, 2022. arXiv
  27. X. Ding et al., “UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition,” CVPR, 2024. arXiv
  28. D. Qin et al., “MobileNetV4: Universal Models for the Mobile Ecosystem,” ECCV, 2024. arXiv
  29. S. L. Smith, A. Brock, L. Berrada and S. De, “ConvNets Match Vision Transformers at Scale,” arXiv:2310.16764, 2023. arXiv
  30. M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” ECCV, 2014. arXiv
  31. PyTorch, torch.nn.Conv2d documentation. Documentation

Check your understanding

This tutorial has 6 exercises. Can you solve them without opening the hints?

Go to the exercises ↑

Next ML 04 · Recurrent and Recursive Networks · 27 min