Start with your chapter and one important question. Predict, change a parameter, and inspect the result. Find the experiments that matter for each chapter, with a reading-only choice where appropriate.
73 experiments97 chapter guides8 tracksRuns in your browser
Choose experiments chapter by chapter 97
Each chapter has a suggested starting point. A shared experiment can support the same concept in several chapters.
This chapter is a reading and setup guide. Choose a learning route and run its first example; a separate numerical playground would not test that objective.
Keep every s-th pixel and fewer gray levels of a real photo and of a synthetic zone plate, and see false patterns (aliasing) appear unless you filter first.
Blur a real photo with a Gaussian PSF, add a little noise, then divide by the blur in the frequency domain. See the plain inverse filter fail, and find the Wiener constant K that rescues it.
Transform a real photo with an 8 × 8 block DCT or a Haar wavelet, round every coefficient to a coarse step, and see where the lost information shows up.
Clean a thresholded photo of coins that carries specks, scratches and holes. Change the operation, the shape and the size of the structuring element, and see what each choice removes, fills or keeps.
Run a real snake, the Kass–Witkin–Terzopoulos contour of IP 11, on a photograph of coins: start a circle around one coin and see how far from its rim the circle may begin before the snake settles in the wrong place.
Slide a small window over a real photograph and watch the window’s gradient vectors, its structure-tensor ellipse and the Harris response decide whether the spot is flat, an edge or a corner.
Run leave-one-out k-nearest-neighbour classification on the real iris measurements, then change the unit of one feature: no flower moves, yet the neighbours and the accuracy do.
Run leave-one-out k-nearest-neighbour classification on the real iris measurements, then change the unit of one feature: no flower moves, yet the neighbours and the accuracy do.
Grow real decision trees on 300 noisy two-moons points, bootstrap them, let them vote, and watch the test error fall and then flatten as trees are added.
Move the images of the two basis vectors and watch a real photograph follow. The determinant is the signed area scale, the singular values are the stretch along two perpendicular directions, and rank 1 flattens the picture onto a line.
Train a network with 2 inputs, m tanh hidden units and 1 sigmoid output on the two moons and the two circles, one gradient-descent step at a time. Watch the decision map and the training and test loss: too few hidden units, too small a learning rate and too large a learning rate each fail in their own way.
Transform a real photo with an 8 × 8 block DCT or a Haar wavelet, round every coefficient to a coarse step, and see where the lost information shows up.
Mix the astronaut photograph with fixed noise, look at the noise, clean-image and velocity targets side by side, then give the network a small error and see which conversion back to the clean image survives.
Pick a stack of convolution layers and a spot in a real photograph: the exact box of input pixels that one output unit depends on is drawn on the image, with its size, jump and the share of the image it covers.
Move the decision threshold on fixed classifier scores. The confusion matrix changes, and the operating point slides along the ROC and precision-recall curves built from the same scores.
Turn a predicted box a few degrees away from the ground truth in the bird’s-eye view and watch the exact rotated IoU fall while the centre distance stays at zero.
Render a Gaussian heatmap at output stride 1, 2, 4 or 8 for 500 random sub-pixel keypoints and compare three ways to read the position back: the brightest cell, a quarter-cell offset and a Taylor step.
Edit the blank probability in the frames between two equal letters of a toy recognizer output, and watch greedy best-path decoding and prefix beam search read it.
Track three coloured blocks through 30 synthetic frames with a Kalman filter, a cost matrix and the Hungarian algorithm, and watch two identities swap when only box overlap is used.
A blurry observation fits two sharp clean signals. The blurry average has the lowest expected squared error; a sharp guess looks better but is usually farther from the truth.
A blurry observation fits two sharp clean signals. The blurry average has the lowest expected squared error; a sharp guess looks better but is usually farther from the truth.
Keep every s-th pixel and fewer gray levels of a real photo and of a synthetic zone plate, and see false patterns (aliasing) appear unless you filter first.
A blurry observation fits two sharp clean signals. The blurry average has the lowest expected squared error; a sharp guess looks better but is usually farther from the truth.
Two shots of one scene are each sharp in a different region. Average them, or pick every pixel from the shot with more local detail, and score both against the all-in-focus truth.
Train a linear classifier on noisy stripe images of 64, 256 or 1,024 pixels, then change every pixel by the same few levels, once along the loss gradient (FGSM) and once with random signs.
Add the same number to every pixel of a real photo and compare 8-bit wraparound with saturating and floating-point arithmetic.
Open experimentClose experiment
Loading image…
Original photo (8-bit)Your arithmetic (see Show)Correct result: clip(pixel + amount, 0, 255)Output value against input value
Try this: With the amount at 60 and the view on overflow, see where the red pixels sit. Now switch the arithmetic to saturate: the same pixels turn white instead of black, and the PSNR against the correct result becomes infinite. Last, set the amount to −100 with wraparound: which pixels flip, and in which direction?
Camera test image reduced to 256 × 256 by 2 × 2 averaging, so the fractions differ slightly from the chapter, which counts on the 512 × 512 original. The pixels are real bytes: wraparound is done in a Uint8Array, so the sum is reduced modulo 256 exactly as NumPy does for gray + np.uint8(60). Saturate clamps the integer sum to 0..255 (what cv2.add does); float adds in floating point, then clips and rounds, which gives the same bytes for whole-number amounts. A negative amount corresponds to subtracting. PSNR compares the result with the correct clipped result, with peak 255. Brightness here is the mean pixel value, 0 to 255; this shows arithmetic on stored values, not a physical exposure model.
Move the images of the two basis vectors and watch a real photograph follow. The determinant is the signed area scale, the singular values are the stretch along two perpendicular directions, and rank 1 flattens the picture onto a line.
Open experimentClose experiment
Loading image…
Original photograph with the unit square, e1 (red), e2 (yellow) and the unit circle (dashed)The photograph after A: the unit square, Ae1 (red), Ae2 (yellow) and the ellipse. Drag the round tips, or click, or focus and use keys 1, 2 and the arrow keys
Try this: The default is a shear: det A = 1, so areas are kept, yet σ1 = 1.62 and σ2 = 0.62 (κ = 2.62), and the unit circle becomes an ellipse. Keeping the area does not keep the shape. Now choose Stretch and slide k from 1 down through 0 to −1: at k = 0 the picture collapses onto a line (rank 1, κ = ∞), and for negative k it returns as a mirror image (det A < 0). Last, choose Projection and decide which points of the picture land on the same spot, and whether any matrix could undo that.
Real photograph: skimage.data.camera at 256 × 256 (credit below). The plane origin is the centre of the photo and 64 pixels are one unit, so the photo spans −2 to 2 in x and y, with y pointing up as in a maths figure. The right picture is made by inverse mapping: each output pixel looks up where it came from (the inverse of A applied to its position) with bilinear interpolation, and squeezed regions average several samples to avoid jagged edges; checkerboard marks places the picture does not reach. A rank-1 matrix has no inverse, so it is drawn as a line whose value at each spot is the mean of the photo along the whole line of points that collapsed onto it. Dashed cyan is the image of the unit circle, an ellipse whose semi-axes (solid cyan) have lengths σ1 and σ2; they are left out when σ1 = σ2. Controls: with u = (cos θ, sin θ), Shear is I + k·u·vT (v is u turned by 90°), Stretch is I + (k − 1)·u·uT (k = 1 is the identity), Rotation is k times a rotation by θ, Reflection is k times a mirror across the line along u, Projection is k·u·uT. Because they act along the direction θ, the two sliders move all four entries a, b, c, d. "Your own" has no restriction: drag the red and yellow tips on the right picture, or click it, or focus it and press 1 or 2 to choose an arrow and the arrow keys to move it (Shift for bigger steps). Any matrix with entries between −1.95 and 1.95 in steps of 0.01 is reachable; the sliders are switched off in that mode. Singular values use the closed-form 2 × 2 formula (Blinn), checked against numpy.linalg.svd; a matrix counts as singular when σ2 is below one billionth of σ1. Numbers are rounded to two decimals.
Keep every s-th pixel and fewer gray levels of a real photo and of a synthetic zone plate, and see false patterns (aliasing) appear unless you filter first.
Try this: Set the stride to 1 and the bits to 3 (8 gray levels): the photo shows bands, yet its PSNR is higher than with stride 4 at 8 bits, and even higher than with stride 2. Which loss does PSNR punish more? Then set the bits back to 8 and compare the zone plate at strides 2, 4 and 8: where do the false rings begin, and how does that radius shrink with the stride? Finally turn the Gaussian prefilter on.
Photo: the camera test image, 256 × 256. Zone plate: synthetic, 256 × 256, I(r) = 0.5 + 0.5 cos(k r²) with r the distance from the centre and k = π / (2 r_max), so its local frequency rises linearly to 0.5 cycles per pixel at the corner. Sampling keeps pixel (0, 0) of every stride × stride block (a[::s, ::s]); gray levels use the chapter’s rule of rounding to the centre of one of 2^bits bins; the result is enlarged by pixel replication (nearest neighbour) so single samples stay visible. The optional prefilter is a Gaussian with σ = stride / 2 applied before sampling. PSNR compares the enlarged result with the original 8-bit image (peak 255). False-pattern contrast is the standard deviation of the kept zone-plate samples that lie beyond radius π / (2 · stride · k), where the true pattern is finer than the coarse grid can show; a faithful result is flat gray there, so a large value means false rings. PSNR cannot tell a false pattern from honest blur, which is why both numbers are shown.
Filter a real photo in the frequency domain and compare an ideal, a Butterworth and a Gaussian low-pass at the same cutoff.
Open experimentClose experiment
Loading image…
Original cropSpectrum kept by the filter (log scale)Filtered imagePixels outside the original range: brighter (red), darker (blue)One image row through a strong edge
Try this: Choose the ideal filter with cutoff 12 and look at the red and blue pixels around the edges. Switch to Gaussian at the same cutoff. Which filter rings, and which has the higher PSNR?
A real 128 × 128 crop of the camera test image. The transfer function multiplies its 2-D FFT (no padding, so the image is treated as periodic) and the inverse transform is not clipped before ripples are counted. Cutoff D0 is in cycles per image; Gaussian H = exp(−D²/2D0²); Butterworth H = 1/(1 + (D/D0)^2n). Overshoot counts pixels more than 0.02 outside the original image’s own gray range. PSNR compares the filtered image, clipped to [0, 1], with the original.
Blur a real photo with a Gaussian PSF, add a little noise, then divide by the blur in the frequency domain. See the plain inverse filter fail, and find the Wiener constant K that rescues it.
Open experimentClose experiment
Loading image…
Original (known)Blurred, plus noiseRestored with H*/(|H|² + K)PSNR of the restored image against K (the same blur and noise)
Try this: Leave the noise at 0.005 and raise log10 K from −7 in small steps. The restored PSNR climbs, peaks, then falls: where is the peak, and what does the image look like on each side of it? Then set the noise to 0: is K = 0 still a disaster? Finally raise the noise to 0.02 and see where the best K moves.
Real 256 × 256 camera image. The degradation follows the chapter, G = H·F + N, in the frequency domain. H is the transform of a Gaussian PSF of standard deviation σ pixels, H = exp(−2π²σ²(u² + v²)) with u and v in cycles per pixel; the blur is periodic (circular), as in the chapter’s code, so the filter matches the blur exactly and no padding is used. N is a fixed seeded white Gaussian field scaled by the noise level (a standard deviation as a fraction of the 0 to 1 range, so 0.005 is about 1.3 gray levels of 255); it is added to the blurred image and not rounded to 8 bits. The restoration is the constant-K Wiener filter of the chapter, F̂ = H*/(|H|² + K) · G. Here K is a tuning constant that you choose, not an estimate of the noise-to-signal power ratio. K = 0 is the plain inverse 1/H, where frequencies with |H| below 10^−6 are set to 0 (the convention of the chapter’s code); that is why its peak gain stops at about a million and why no-noise restoration is not perfect for large σ. PSNR compares each image, clipped to 0 to 1, with the original. The gain is the largest value of |H|/(|H|² + K) over all frequencies; for K greater than 0 it never exceeds 1/(2√K).
Multiply a real color photo by a gain and compare three ways of handling the channels that overflow: clip each one, rescale the pixel, or tone-map it.
Open experimentClose experiment
Loading image…
Original photoResult, or hue map (red: hue moved, blue: color lost, gray: unchanged)Hue of the overflowing colored pixels, before and after (10° bins)
Try this: Set the gain to 2 with per-channel clipping and look at the hue map: the wooden table and the coffee are red (hue moved) and the bright cup walls are blue (colour lost). Raise the gain to 4: why do the orange bars of the hue histogram pile up in one bin at 60° (yellow), and where do the other pixels go? Now switch to rescaling: what does not change, and what does the picture lose instead?
Photo: the coffee test image reduced to 256 × 171 (CC0, Rachel Michetti). The gain multiplies the stored 8-bit sRGB values directly, like np.clip(g * x, 0, 1) in the Images as Arrays chapter; a real exposure change acts on linear light, so the numbers would differ but the story would not. Overflow means some gained channel exceeds 255. Clip: each channel is cut at 255. Rescale: an overflowing pixel is divided by its largest channel, so the channel ratios, hence the hue, survive, but its largest channel always ends at 255. Tone map: a Reinhard-style operator L(1 + L/g²)/(1 + L) on the luma L of the gained pixel (white point g, the gain), applied as one common factor to R, G and B, followed by clipping of any channel still above 255. Hue is the HSV hue angle in degrees (red 0°, green 120°, blue 240°), close to but not identical with the chapter’s HSI hue. Pixels with HSV saturation or value below 0.1 have no usable hue and are left out of the hue statistics; a pixel that was coloured but ends below 0.1 saturation counts as colour lost. Saturation is HSV saturation, averaged over coloured pixels that overflow.
Transform a real photo with an 8 × 8 block DCT or a Haar wavelet, round every coefficient to a coarse step, and see where the lost information shows up.
Open experimentClose experiment
Loading image…
Original (yellow box = zoom)Reconstruction from the quantized coefficientsSurviving coefficients, log scale, zeros black (DCT: one 8 × 8 tile per block, block average top-left; Haar: coarsest average top-left, finer details outward)Error: reconstruction − original, × 10, gray = 0Zoom (32 × 32 pixels, 6 ×): originalZoom: reconstruction (DCT: the ruler marks the 8-pixel blocks)
Try this: Set the DCT step to 0.6 and look at the sky in the zoom box. Then switch to Haar at the same step: does the staircase go away, and which transform keeps fewer nonzero coefficients at about the same PSNR? Finally move the zoom box to the grass: how much of the texture is left?
A real 256 × 256 version of the camera test image (a 2 × 2 average of skimage.data.camera), gray values in [0, 1]. Both transforms are orthonormal: an 8 × 8 block DCT (each block coded on its own) and a 5-level 2-D Haar wavelet transform. The quantizer is uniform: every coefficient c becomes round(c / step) × step, with the same step for all coefficients including the block averages, so a coefficient survives only if |c| is at least step / 2. This is simpler than the frequency-dependent table of JPEG in IP 08, so the pictures show what the transform and the rounding do on their own. "Nonzero" counts surviving coefficients. It is not a bit rate: no entropy coder is implemented, and the real cost also depends on which coefficients survive and how they are coded. PSNR has peak 1 and uses the decoder output before clipping. Because both transforms are orthonormal, the squared error in pixels equals the squared error in the coefficients: what the quantizer removes is exactly the error, and the decoder cannot bring it back, because many different images give the same quantized coefficients. A bottleneck autoencoder (ML 06A) loses detail in the same way, except that its basis is learned from data instead of fixed. The error image shows (reconstruction − original) × 10 around mid-gray. "Smoothest" and "busiest" halves are the pixels with the smaller and larger Sobel gradient of the original. Haar is the simplest wavelet and is piecewise constant, so it also produces blocks, of several sizes; smoother wavelets such as the one in IP 08’s wavelet comparison blur instead.
Clean a thresholded photo of coins that carries specks, scratches and holes. Change the operation, the shape and the size of the structuring element, and see what each choice removes, fills or keeps.
Open experimentClose experiment
Loading image…
Input mask: coins with specks, scratches and holesOutput (black stays, red removed, blue added)The structuring element (each square is one pixel; the orange dot is the origin)
Try this: With opening and a disk of size 4, all clutter goes. Now switch the element to a horizontal line of size 4: which pieces survive, and why? Then raise the disk size with opening until a coin disappears. Finally switch to closing: what does it fix that opening cannot?
The photograph is skimage’s coins image (303 × 384). The clean mask smooths it (Gaussian, σ = 2), subtracts a wide Gaussian background (σ = 50) so the bright corner does not count, and thresholds with Otsu’s method; that gives exactly 24 solid coins. Clutter is then added with a fixed seed so the answer is known: 40 specks (blobs of radius 0 to 3 pixels), 10 thin scratches (5 horizontal, 5 vertical) and 18 round holes of radius 1 to 3 pixels inside the coins (scripts/figures/playground_restore_assets.py). Notation follows the chapter: A is the foreground, B the structuring element, opening = erosion then dilation, closing = dilation then erosion, gradient = dilation minus erosion. Size r means a disk of radius r (all offsets with dx² + dy² ≤ r²), a square of (2r + 1) × (2r + 1) pixels, or a line of 2r + 1 pixels. Pixels outside the image count as background. Components are 8-connected; a hole is a background region that does not touch the border; clutter means a connected piece under 200 pixels (the smallest coin has more than a thousand). In the output picture, black pixels stay foreground, red were foreground and are gone, blue are new.
Dim one side of a page and watch Otsu’s single threshold lose the text, while a threshold that follows the local light keeps it.
Open experimentClose experiment
Loading image…
The page under uneven lightTrue text mask (known exactly)One global Otsu thresholdLocal thresholdHistogram of the lit page and the Otsu threshold
Try this: Start with the lighting ramp at 0, then raise it slowly. At what ramp does the global mask start to lose text on the dim side? Then shrink the local window to 11: why does the local mask get worse?
The page is synthetic (scripts/figures/playground_images.py), so the true text mask is known exactly and IoU is a real score, not a proxy. Lighting multiplies the page by 1 − ramp · x/(w − 1). Global: Otsu’s threshold on the 256-bin histogram. Local: a pixel is ink if it is darker than (1 − c) times the Gaussian-weighted average of its window (σ = (window − 1)/6). IoU is the overlap of the ink mask with the true mask divided by their union.
Run a real snake, the Kass–Witkin–Terzopoulos contour of IP 11, on a photograph of coins: start a circle around one coin and see how far from its rim the circle may begin before the snake settles in the wrong place.
Open experimentClose experiment
Loading image…
Coins with the start circle (white dashed), snapshots at 10, 30, 100 and 300 iterations (orange), the final snake (yellow) and the true coin boundary (cyan dashed). Click or drag to set the start radiusEdge map e = |∇(G_σ ∗ image)| (dark blue: strong edge) with the start circle (dashed) and the final snake (orange)Snake energy against iterationMean distance to the true coin boundary against iteration
Try this: The default circle has radius 40 and the coin about 23: the snake ends 17.7 px from the coin boundary, caught by the neighbouring coins. (1) Lower the start radius to 28: it finds the rim (0.5 px). (2) Back at radius 40, raise the smoothness to 1.0: now it finds the coin too (0.4 px). Why does a stiffer, more contracting snake reach the coin from farther away? (3) Keep smoothness 2.0 and start inside the coin at radius 12: the snake shrinks to a small curve (17.4 px). (4) With smoothness 0.05, a start at radius 12 reaches the rim at σ = 3 (1.1 px) but stays inside at σ = 2 (10.2 px). What does the blur remove?
Real image: a 144 × 144 crop of skimage.data.coins, centred on one coin and showing parts of the eight coins around it (credit below). The snake follows IP 11 and its figure code: a closed contour of 100 points, the semi-implicit step x(t) = (A + γI)⁻¹(γ x(t−1) + f) with A built from [β, −α−4β, 2α+6β, −α−4β, β], γ = 1, α = β = the smoothness slider, and the force f = κ∇e with κ = 4, where the edge map e = |∇(G_σ ∗ image)| is scaled to a maximum of 1 and sampled by bilinear interpolation. There is no resampling, no balloon force and no step cap; points stay inside the image. It stops when no point moves more than 0.05 px in one step, or after 1500 iterations. This is not scikit-image’s active_contour: that function builds the same matrix A, but takes a Sobel gradient of the image you pass in, a spline for the force and caps every step at one pixel. The true boundary is the circle fitted by least squares to the boundary pixels of the coin’s Otsu mask (holes filled); the coin is not a perfect circle, so a snake sitting on the rim is still about half a pixel from it. Distance is the mean over the 100 points of |distance from the circle’s centre − its radius|; “found” means at most 2 px. Energy is E = ½ Σ(α|v(i+1) − v(i)|² + β|v(i+1) − 2v(i) + v(i−1)|²) − κ Σ e(v(i)), in arbitrary units; the orange dashed line is E for the fitted circle itself. A click or drag on an image sets the start radius so that the circle passes through the pointer; the circle always stays centred on the coin. Click positions are rounded to whole pixels.
Slide a small window over a real photograph and watch the window’s gradient vectors, its structure-tensor ellipse and the Harris response decide whether the spot is flat, an edge or a corner.
Open experimentClose experiment
Loading image…
Photograph with the probe window (yellow) and marks A, B, C. Click or drag to move the probeThe window magnified, with a gradient arrow in each pixel (orange)Harris response R over the whole image (red: corner, blue: edge, white: flat)One dot per window pixel: its gradient (Ix, Iy). The orange ellipse is the structure tensorThe two eigenvalues of this window (dot) among all windows of the image (grey)
Try this: Place the probe on the three marked points with the sliders (A: x 200, y 20; B: x 80, y 52; C: x 30, y 80) and compare the dots: a tiny huddle at the origin, a cloud in two directions and a thin line. Then try the left tripod leg (x 122, y 232) and the right leg (x 183, y 220): the line turns, yet λ2/λ1 stays below 0.0025 for both, because both are edges. Finally raise the window from 9 to 25 pixels: the share of the image labelled corner grows from 1.4% to 8.9%. Why does a bigger window find more corners?
Real photograph: skimage.data.camera at 256 × 256 (see the credit below). Gradients are Gaussian derivatives with σ = 1 pixel, as in the IP 12 code. The window is W × W pixels with Gaussian weights of σ = W/6, so W = 9 gives the chapter’s integration scale σ = 1.5 (the chapter code truncates the Gaussian at 4σ instead of at the box). M is the weighted sum of the outer products of the gradient vectors, R = det M − 0.05 · tr(M)². Gradients are in gray levels per pixel (0 to 255) and eigenvalues in their squares. The flat, edge and corner labels use a cutoff chosen for this lab, not part of Harris and Stephens: an eigenvalue counts as large when it is at least 5% of the largest λ1 anywhere in the image at that window size. The Harris detector proper keeps local maxima of R above a threshold. The heat map shows R divided by its image maximum, compressed by a square root: red positive, blue negative, white near zero.
Edit 3 × 3 weights and inspect smoothing, sharpening and edge responses.
Open experimentClose experiment
Kernel playgroundEdit any cell or pick a preset. The output updates as you type.
Input IOutput G = K ⋆ I
Try this: start with the identity kernel, then make the weights sum to zero. What happens in a flat region?
This computes cross-correlation with an unflipped kernel and replicated boundaries. Signed zero-sum responses use mid-gray as zero; values outside the display range are clipped.
Try this: Enter an unseen context and set α to zero. Why is the estimate undefined? Increase α and inspect where the probability goes.
Each nonempty line is one sentence. Tokens are lowercased and split on whitespace; punctuation stays attached. The first 8,000 characters are used. Vocabulary comes only from the corpus, plus [EOS] and [UNK]. [BOS] pads the start but is never predicted. The last n − 1 query tokens form the context; an empty query means sentence start. This is a counting demonstration, not a Chinese word segmenter.
Train byte-pair encoding on a tiny weighted vocabulary and apply the learned merge order to a new word.
Open experimentClose experiment
Learned merges
4
Training pieces, weighted
31
Input pieces
5
Merge counts come only from the fixed training vocabulary. Encoding applies the learned rules in order; it does not refit them to your input.
Try this: Compare “lower”, “slowest” and “glow”. Which pieces transfer to unseen words, and which merges only compress the training vocabulary?
Character-level BPE with an explicit end marker ▁; lexicographic tie breaking. This is not a byte tokenizer or a pretrained vocabulary. New characters remain individual pieces; spaces inside the input are treated as characters.
Pad a three-word sentence with zero vectors and compare a correct masked mean with a mean that counts the padding in its denominator.
Open experimentClose experiment
Pooled h
0.233
Masked-mean h (reference)
0.7
P(class 1)
0.763
Reference P(class 1)
0.971
Real / counted length
3 / 9
Zero padding must not change a masked mean. Counting the padding in the denominator shrinks h by 3 divided by the counted length and pulls P(class 1) toward 0.5, although the sentence is the same. An unweighted mean also ignores word order; only the position-weighted diagnostic tells the two sentences apart.
Try this: Keep the padding bug selected and drag the padding count from 0 to 9: predict h and P(class 1) before each step, and explain why the classifier drifts toward 0.5 although the sentence never changed. Then switch to the masked mean and confirm that padding cannot matter. Last, try position weights on the reversed sentence: that is the only mode where word order changes the answer.
Fixed scalar embeddings: dogs=1, chase=0.4, cats=0.7; classifier p=σ(5h). The masked mean divides by the 3 real tokens, so zero padding adds nothing. The bug divides by 3 + padding. The reference bar always shows the masked mean of the same sentence. A plain mean ignores word order by construction, so swapping the sentence changes nothing there; position weights 1,2,3 are a diagnostic, not a trained syntax model.
Combine emission scores, transition scores and BIO span constraints to choose a complete tag path.
Open experimentClose experiment
Best total score
7.3
Local path score
6.7
Local path legal?
Yes
The highlighted cells maximize the total path score, not each token separately. An I tag can only continue a B or I when constraints are enforced.
Try this: Raise the score for I at “York”. Can a transition preference change the best path even when the local winner stays fixed?
O=outside, B=begin, I=inside one entity type. Scores are synthetic, not probabilities. All 81 paths are enumerated exactly; dynamic programming would find the same maximum more efficiently on long sequences.
Follow fixed logits through temperature scaling, top-k truncation, nucleus truncation and renormalization.
Open experimentClose experiment
Candidates retained
3
Entropy, bits
1.201
Probability sum
1
Temperature rescales relative logits. Truncation sets excluded candidates to zero and renormalizes the survivors; it cannot improve the information available to the model.
Try this: Keep temperature fixed and lower top-p. Why can the output entropy change abruptly even though the slider moves continuously?
Five synthetic candidate tokens with logits [3,2,1,0,−1]. This applies top-k first, then top-p to its renormalized distribution. No model is trained and no quality score is inferred from entropy.
Inspect BM25 term saturation and document-length normalization on a fully visible local corpus.
Open experimentClose experiment
Top document
D2
Top score
1.86
Matching documents
4
IDF rewards terms found in fewer documents. k₁ controls how repeated matches saturate; b controls the penalty for length relative to this corpus mean.
Try this: Search “red”, then “robots objects”. Change length normalization and saturation; explain each rank swap from document terms.
Six synthetic documents, lowercase English word tokenization, distinct query terms, positive smoothed IDF. A lexical relevance score is not evidence that a document answers a question.
Change a right-context clue, hide it causally, then attach different output heads to the same small contextual vector.
Open experimentClose experiment
Context positions used
3
First output probability
0.931
Output probabilities sum
1
The target content is always absent. A bidirectional objective can use right context; a causal next-token objective cannot. An output head maps representations to the labels required by a task.
Try this: Change the right-hand clue under each mask. When must the masked-position prediction remain identical, and why?
A hand-built uniform context encoder and two linear heads illustrate BERT-style visibility and task interfaces. These are not BERT embeddings or pretrained predictions. The sequence head reads this toy pooled context, not an actual [CLS] representation.
Run leave-one-out k-nearest-neighbour classification on the real iris measurements, then change the unit of one feature: no flower moves, yet the neighbours and the accuracy do.
Open experimentClose experiment
Loading image…
k-NN decision map on the iris flowers (axes in the original centimetres; ring = misclassified in leave-one-out)Leave-one-out accuracy as the unit of the second feature changes (the data are the same in every column)
Try this: Start with sepal width and petal width, 5 neighbours, petal width in decimetres (×0.1). Step the unit to centimetres (×1) and millimetres (×10): how do the accuracy and the number of changed nearest neighbours move? Then switch to petal length and petal width. Is the effect as large? Finally compare with the standardized setting.
Data: the 150 iris flowers bundled with scikit-learn (three species, 50 each), two of the four measurements at a time, in centimetres. The classifier is the plain k-NN rule with Euclidean distance. Only the second feature is rescaled (the first stays in centimetres); "standardized" divides both features by their standard deviation. Accuracy is leave-one-out: each flower is classified by the other 149, and the map is trained on all 150. The measurements are rounded to 0.1 cm, so many distances tie exactly, and the fifth and sixth neighbour of a flower are often equally far away. Ties are settled in a fixed way: equal distances are ordered by a fixed random ranking of the flowers (the file lists the species one after another, so ordering by position would favour setosa), and a tie between classes goes to the tied class with the nearest member. The last metric counts the answers that rest on such a tie. scikit-learn orders equal distances in its own way: on the 108 settings checked in the tests (3 feature pairs, 6 units, 6 values of k) it agrees with this lab on every answer that does not rest on a tie, and its leave-one-out count differs from this one by at most 7 flowers. Rescaling is reversible, so it adds no information; the accuracy is a single 150-flower sample, not a general result.
Move a decision boundary by changing class frequency and the cost of a false alarm.
Open experimentClose experiment
Decision boundary x
0
Posterior P(1 | x)
0.731
Required posterior for class 1
0.5
At x = 0.5, choose class 1. Compare conditional risks: choosing 0 costs 0.731; choosing 1 costs 0.269. The boundary obeys 2x + log(prior odds) = log(false-positive cost).
Try this: Keep the observation fixed. Can increasing the false-positive cost switch the decision without changing the posterior?
A synthetic population: X|class 0 ~ N(−1,1), X|class 1 ~ N(1,1). The slider sets the true class prior. A false negative costs 1; correct decisions cost 0. Curves show class-conditional densities, not posterior probabilities.
Change the bandwidth and inspect individual kernels, the estimated density and leave-one-out prediction.
Open experimentClose experiment
Mean leave-one-out log density
-1.827
Peak density
0.283
Small bandwidth preserves local bumps; large bandwidth merges them. Leave-one-out evaluation prevents each sample from predicting itself.
Try this: Find a bandwidth that improves the leave-one-out log score. Why can a spiky fit score worse on a held-out point?
Ten fixed synthetic observations; Gaussian kernels integrate to one. The score averages log density at each point after excluding that point. It is a density score in nats, not classification accuracy. The plot shows individual contributions divided by ten.
Fit polynomial basis functions with ridge regression and separate training error from recovery of the underlying function.
Open experimentClose experiment
Training mean squared error
0.0538
True-function grid error
0.029
Squared coefficient norm
5.403
Changing λ alters the fitted coefficients, not the observations. Large penalties can remove noise-fitting but also suppress the curvature needed to represent the signal.
Try this: At degree seven, increase regularization. Does the lowest training error also give the lowest underlying-function error?
The blue fit solves sum of squared residuals + λ times the squared non-intercept coefficients. Twelve fixed synthetic points have deterministic perturbations. The dashed curve is sin(πx); its grid error is available only because this is a known synthetic problem. Polynomial features are linear in their coefficients.
Rotate a one-dimensional projection and compare reconstruction error with class separation.
Open experimentClose experiment
Retained variance fraction
0.662
Mean squared reconstruction distance
1.917
Projected class-centroid gap
1
At 0°, both classes project to the same six positions. At 90°, within-class horizontal variation disappears and class centers remain distinct.
Try this: Find the projection that minimizes reconstruction error, then the one that separates the colored classes. Why are they different?
The centered synthetic observations have horizontal variation within each class and a vertical class difference. Principal component analysis maximizes retained variance without using labels. The centroid gap shown here is descriptive, not a complete classification metric.
Warp a signal in time and inspect a dynamic time warping cost matrix and its optimal alignment path.
Open experimentClose experiment
Aligned total cost
0.231
Same-index total cost
3.591
Path cells
14
Dark cells have larger amplitude mismatch; gray cells are forbidden by the band. The path minimizes accumulated allowed cost. A wider band cannot raise the optimum but can permit implausible repetitions.
Try this: Increase the time warp with a zero-width band, then widen the band. What does the lower cost allow the alignment to ignore?
Two twelve-sample sine signals: a(t)=sin(2πt), b(t)=sin(2πt^w). Local cost is squared amplitude difference. The path must start and finish at both endpoints; only diagonal, vertical and horizontal forward steps are allowed. Cost is a sum, not normalized by path length.
Step through Gaussian-mixture EM and observe responsibilities, likelihood and the symmetric-initialization trap.
Open experimentClose experiment
Component means
-1.857 / 1.537
Log likelihood (nats)
-15.863
Gain since initialization
10.555
The E step recomputes soft memberships; the M step takes responsibility-weighted means. With identical initial means every responsibility is 1/2, so symmetry cannot break on its own.
Try this: Set initial separation to zero and add iterations. Why do the two means remain identical? Restart with a small separation.
A constrained two-component mixture: both weights stay 1/2 and both standard deviations stay at the selected σ. EM updates only the means. Each point is colored by its current responsibility for component 1; these are component memberships, not supervised labels.
Move and widen a Gaussian approximation to a two-mode target; compare penalties for missing mass and filling the valley.
Open experimentClose experiment
KL(p ‖ q)
8.777
KL(q ‖ p)
0.69
KL(p‖q) weights mismatch where the target has mass; KL(q‖p) weights mismatch where the approximation puts mass. A single Gaussian may cover both modes or settle on one, depending on the objective.
Try this: With separated modes, compare a narrow approximation on one mode against a wide one covering both. Which KL prefers each?
Target p is an equal mixture N(−a,0.65²) and N(a,0.65²). Approximation q is N(μ,σ²). Both are numerically normalized on [−12,12] with spacing 0.02; KL values are grid approximations in nats. This is an adjustable distribution family, not an optimizer.
Grow real decision trees on 300 noisy two-moons points, bootstrap them, let them vote, and watch the test error fall and then flatten as trees are added.
Open experimentClose experiment
Loading image…
Decision map of the chosen view over the 300 training points (circle = class 0, square = class 1; paler colour = the trees disagree; hollow points = missing from that member’s bootstrap sample)Test error of the bagged vote for every M from 1 to 100 (1000 held-out points); the ring marks the current M
Try this: Start with 50 trees at the fully grown depth 14 and drag M from 1 to 100: the vote falls from 14.60% to 11.10% over the first ten trees, then barely moves (10.25% at both 50 and 100). Switch the view to One member and drag M to see how ragged single bootstrap trees are, then back to the vote. Lower the depth to 2: the members now disagree on only 4.88% of the test points per pair, and the vote (10.70%) is no better than one tree (10.50%). Finally try depth 5: one tree already scores 9.30% against 10.15% for 50 bagged trees, a gap of less than one percentage point, about the noise of a 1000-point test set, so bagging does not replace choosing a sensible depth. Why does a vote of similar trees gain nothing?
Data: the two-moons set from scikit-learn make_moons with Gaussian noise 0.3, generated once and stored in the page: 300 training points (150 per class, data seed 2) and a separate held-out test set of 1000 points (seed 1002). The data seed is the one of 30 random draws whose bagging gain is closest to the average gain, so this is a typical case, not a favourable one. Trees: CART with Gini impurity and axis-aligned splits at the midpoint between neighbouring values; the depth limit comes from the slider (14 grows every tree here until its leaves are pure). Bagging: member b is trained on 300 rows drawn with replacement from one fixed seeded stream (seed 20260618), so moving M only adds or removes trees and the curve does not change. The vote is a plain majority; for an even M an exact tie counts as half an error. Test error is measured on the 1000 held-out points; its standard error is about 1 percentage point at an error of 10%, so differences smaller than that between two different settings are weak evidence on their own. Disagreement: for every pair of the M members, the share of the 1000 test points on which the two predict different classes, averaged over all pairs (0 means identical members; the closer to 0, the more alike the members, which is a proxy for how correlated their mistakes are). Ties between equally good splits are broken by a fixed rule (feature 1 before feature 2, lower threshold first), while scikit-learn visits the features in random order: in the tests, trees without such a tie give predictions identical to DecisionTreeClassifier, and the vote error of the full 100-tree ensembles differs from the same ensembles built with scikit-learn trees by at most 0.7 percentage points. Best possible 8.5% is the error of the ideal classifier on the distribution that make_moons samples, computed by numerical integration of the two class densities; the lab does not use it. Real bagging also gives each tree out-of-bag points for free error estimates; this lab uses the separate test set instead.
Set a rejection threshold, shift both class distributions and compare accepted coverage with selective error.
Open experimentClose experiment
Accepted coverage
0.666
Error among accepted predictions
0.068
Deployment-optimal boundary
0
The classifier still places its decision boundary at zero, while the deployment-optimal boundary moves with δ. Rejecting uncertain predictions does not repair confidence under shift.
Try this: Hold the rejection threshold fixed while shifting the population. Can high reported confidence coexist with more accepted mistakes?
The fixed classifier uses training posterior sigmoid(2x), learned for equal-prior N(−1,1) and N(1,1). Deployment shifts both means by δ without updating the classifier. Coverage and accepted error are numerical integrals over the synthetic deployment population, not estimates from a benchmark.
Compare a fixed two-unit network with and without nonlinear activations on the XOR corners.
Open experimentClose experiment
Correct corners
3 / 4
P(1 | 0,1)
0.881
P(1 | 1,1)
0.119
A linear hidden layer collapses to one affine decision boundary. Nonlinear hidden units create pieces that can assign the two opposite XOR corners to the same class.
Try this: Start with the linear activation and try every bias: can all four corners be right at once? Then switch to ReLU and tune the gain and bias. Why does the nonlinearity change the answer?
Weights are hand constructed, not trained: h₁=a(g(x+y)), h₂=a(g(x+y−1)), p=σ(4(h₁−2h₂)/g−2+b). Color is P(class 1); circles show known corner labels.
Move paired and unpaired unit vectors, then inspect the normalized matching loss used by contrastive learning.
Open experimentClose experiment
Positive probability
0.896
Matching loss, nats
0.109
Positive cosine
0.94
Loss = −log(exp(sim⁺/τ) / Σ exp(sim/τ)). Temperature sharpens whichever candidate already has the highest similarity. Collapsed unit vectors give equal probabilities and loss log(4).
Try this: Bring a negative closer to the anchor than the positive. Does a lower temperature help the correct pair? Then collapse all vectors.
One anchor, one positive and three negative candidates; vectors are controlled directly rather than trained. The same matching geometry explains image–text retrieval, but these are not CLIP features or semantic predictions.
Compare a fixed recurrent decay with input-dependent write decisions while distractors pass through a sequence.
Open experimentClose experiment
State h₁ → h₈ (before the second write)
0.75 → 0.75
Final state
-0.563
∂h₁₂ / ∂h₁
0.25
Largest |state|
0.75
Write opportunities
2
Fixed decay trades persistence against interference at every step. Selection can preserve state through irrelevant inputs, but the gate must identify relevance correctly; the oracle here removes that learning problem. Retention 1 preserves perfectly and writes nothing.
Try this: Compare the two policies: with selective updates the state written at position 1 survives the distractors, with the same update at every step the distractors overwrite it. Then raise the distractor amplitude and see which state changes. Finally set retention to one: a state that never changes keeps its old value perfectly, so nothing new can be written either. Why is that not memory?
Scalar recurrence hₜ=aₜhₜ₋₁+(1−aₜ)xₜ with h₀=0. Retention aₜ=1 keeps the old state, aₜ=0 replaces it with the input. Relevant positions 1 and 9 are provided by an oracle; in the selective policy every other position has aₜ=1. With retention 1 at the relevant positions the state stays at its starting value 0: perfect preservation, nothing written. This illustrates gating and Mamba’s input-selectivity motivation; it is not an implementation of Mamba, LSTM or GRU.
Compare bidirectional and causal attention using the same query, keys and values.
Open experimentClose experiment
Visible tokens
3
Weight sum
1
Attention output
[0.7517, 0.7517]
The score column includes temperature scaling. A masked score is −∞ and its weight is exactly zero. Changing the mask changes normalization over the remaining keys.
Try this: Choose the first query in causal mode: its weight must be 1 on itself. Switch to bidirectional mode, then lower the temperature.
All Q, K and V vectors are fixed, synthetic two-dimensional numbers, not learned language representations. Scores are Q · K / (√2 τ); τ is an extra temperature control and τ = 1 gives the usual scaling. Causal attention masks positions after the query before softmax. The output is the weighted sum of V. These weights illustrate routing, not an explanation of linguistic meaning.
Apply one positive temperature to fixed validation logits and compare accuracy, log loss and squared probability error.
Open experimentClose experiment
Accuracy
75%
Mean log loss, nats
0.498
Brier score
0.177
Positive temperature preserves logit ordering and binary decisions. It can adjust overconfidence or underconfidence, but cannot correct which examples the classifier gets wrong.
Try this: Find a temperature with lower log loss than T=1. Why does accuracy stay fixed even while confidence changes?
Eight synthetic validation observations, not a benchmark. Binary probabilities use σ(logit/T), with a fixed 0.5 decision threshold. Choosing T here illustrates validation tuning; it does not establish performance on an unseen test set.
Check the derivative of exp(x), separating truncation error from floating-point cancellation.
Open experimentClose experiment
Analytic derivative
2.71828183
Numerical derivative
2.71828183
Relative error
2.155e-11
Large h introduces approximation error. Very small h subtracts nearly identical floating-point values, losing useful digits. Central differences cancel the leading truncation term but still suffer roundoff.
Try this: Sweep h from 10⁻¹ to 10⁻¹⁶. Find the useful range for forward and central differences rather than choosing the smallest step.
Computed with JavaScript binary64 arithmetic; exact means the analytic derivative exp(x) evaluated in the same arithmetic. A zero measured error is rounded agreement, not proof of exact real arithmetic.
Train a network with 2 inputs, m tanh hidden units and 1 sigmoid output on the two moons and the two circles, one gradient-descent step at a time. Watch the decision map and the training and test loss: too few hidden units, too small a learning rate and too large a learning rate each fail in their own way.
Open experimentClose experiment
Loading image…
Two moons: decision map at the chosen step (blue circles = class 0, orange triangles = class 1; filled = training, hollow = test; ringed = wrong; black line = 0.5 boundary)Two moons: cross-entropy loss against the step (solid = training, dashed = test)Two circles: decision map at the chosen step (same symbols)Two circles: cross-entropy loss against the step (solid = training, dashed = test)
Try this: Start from the default (m = 3, η = 1, 1000 steps), where both curved boundaries are learned. (1) Drag the steps slider up from 0: after 100 steps the moons are only 85.0% right (close to one tilted straight line, the loss on a plateau near 0.30), after 300 steps 94.5%, after 500 steps 98.0%. (2) Set η to 0.3 and lower m to 2, then 1: at 1000 steps the moons stay at 86.0% and 85.0%, the circles at 77.5% and 61.5%. (3) Go back to m = 3 and try η = 0.03, 0.1 and 0.3: at 1000 steps the moons are 84.5%, 85.0% and 94.5% right. A small η is not wrong, only slow: run offline for 5000 steps, beyond the 2000-step limit of this lab, η = 0.1 reaches 98.0% on the moons. (4) Try η = 3, 10 and 30: the loss now goes up on many steps, and with the largest η the training does not settle.
Data: scikit-learn make_moons (noise 0.2, random_state 0) and make_circles (noise 0.1, factor 0.5, random_state 0), 200 training points each, plus a held-out test set of 200 points each (random_state 1) that is never used for training. Network and loss as in ML 02: z₁ = W₁x + b₁, a₁ = tanh(z₁), z₂ = W₂a₁ + b₂, p = σ(z₂), binary cross-entropy averaged over the training points; the gradient comes from backpropagation (checked against finite differences and against an independent NumPy run). Training is plain full-batch gradient descent, θ ← θ − η∇L, with a fixed η: no momentum, mini-batches or regularization. Initial weights: Xavier normal (variance 2 / (fan-in + fan-out)) with zero biases, from one fixed seed (5), so a slider never re-rolls them; unit j always gets the same random numbers. Another seed changes the details: over seeds 1 to 20 (η = 1, 2000 steps), 3 hidden units reached at least 95% training accuracy on both datasets in all 20 runs, 1 unit never did, and 2 units reached it on the moons in at most 1 of the 20 runs and on the circles never. A step is one parameter update; step 0 is the untrained network. When the loss goes up and down the iteration is chaotic, so the curves at large η can differ in their details between browsers, though not in their story. The test-loss curve is sampled every 4 steps.
Move and widen a Gaussian posterior, then compute expected decoder loss and the exact KL penalty.
Open experimentClose experiment
Expected decoder NLL
0.806
KL(q || prior), nats
0.638
NLL + β KL
1.444
Reconstruction favors latent values near x=1 and a narrow posterior. KL penalizes both moving away from zero and changing variance away from one. Prior agreement alone does not guarantee useful reconstruction.
Try this: Set q(z|x) equal to the prior. Why is KL zero but reconstruction still poor for x=1? Can shrinking σ improve one term while worsening the other?
One observation x=1, q=N(μ,σ²), prior=N(0,1), fixed decoder p(x|z)=N(z,0.5²). Expected negative log likelihood is analytic; no samples or network training. β=1 gives the negative evidence lower bound for this model; other β values are weighted objectives.
Push a Gaussian through an invertible nonlinear map and preserve probability mass with its derivative.
Open experimentClose experiment
Derivative at zero
2
Output density at zero
0.199
Numerical area (±4 base σ)
0.99988
pₓ(f(z)) = p_z(z) / |f′(z)|. A local stretch spreads the same probability over more output space. Omitting the Jacobian would change the total probability mass.
Try this: Increase the nonlinear stretch near zero. Why does the output density at zero fall even though zero still maps to zero?
One-dimensional normalizing flow x=az+b tanh(z), with a>0 and b≥0. The derivative is always positive, so the map is invertible. Curves are computed from the exact change-of-variables formula, without fitting a model.
Transport particles through a known velocity field and compare integration error at equal step counts.
Open experimentClose experiment
Mean endpoint error
0.189926
Field evaluations / particle
4
Exact endpoint from x₀=0
1.532
Step count and function evaluations are different costs. Reducing integration error cannot fix an incorrect learned field; this oracle example keeps field error at zero so the two issues stay separate.
Try this: Set the rate to zero: why is one Euler step exact? Then use a nonzero rate and compare steps with actual field evaluations.
Known field v(x,t)=λx+1, integrated from t=0 to 1. This isolates solver error relevant to flow-matching sampling; no velocity network is trained and no generation quality is measured. The dashed paths use the analytic solution.
Mix the astronaut photograph with fixed noise, look at the noise, clean-image and velocity targets side by side, then give the network a small error and see which conversion back to the clean image survives.
Open experimentClose experiment
Loading image…
x_t, the network input: the only picture it gets to seeε, the noise that was addedx₀, the clean imagev = √ᾱ·ε − √(1−ᾱ)·x₀, the velocityx̂₀ recovered from the network’s imperfect output of the chosen targetError x̂₀ − x₀ (gain ×3, mid-gray = 0)
Try this: Start at −20 dB with an error of 0.05 in the noise output. Slide the signal level up to 0 dB: where do the three recoveries become equally good? Then go to +20 dB: which target is now the most error-sensitive, and why does the answer flip?
Image: the astronaut photograph (256 × 256, RGB), with pixel values mapped to [-1, 1] as in the chapter. The mix is x_t = √ᾱ x₀ + √(1−ᾱ) ε with a fixed seeded standard-normal ε, and v = √ᾱ ε − √(1−ᾱ) x₀ is the velocity of the ML 08 chapter (the diffusion velocity, not the flow-matching transport velocity). The slider sets the signal-to-noise ratio ᾱ/(1−ᾱ) of x_t in dB. The network is not trained: its output is the exact target plus a fixed seeded error, and the same error size (in the units of the [-1, 1] pixel scale) is applied whichever target is predicted. A real network does not make equal errors for every target, so this isolates the conversion, not the learning. x̂₀ is clipped to [-1, 1] as an image decoder would, and PSNR is measured against x₀ with both mapped to [0, 1] (peak 1). All panels share one gray mapping (gray = 0.5 + u/2.5).
Enumerate all sixteen binary 2×2 images under a first-order pixel model, then change the generation order.
Open experimentClose experiment
Image probability
0.064
Bits per pixel
0.991
All-image probability sum
1
Joint probability multiplies normalized conditional factors. All sixteen image probabilities sum to one. A high-copy model prefers sequences with fewer transitions, which need not correspond to the same spatial pattern in each order.
Try this: Compare row stripes and a checkerboard when copying is likely. Switch raster to column order: which model assumptions changed?
First pixel has probability 1/2; each later pixel copies the previous one with probability c. The chain rule is exact, but this model deliberately retains only one previous pixel. Order dependence here comes from that restricted conditional model.
Pick a stack of convolution layers and a spot in a real photograph: the exact box of input pixels that one output unit depends on is drawn on the image, with its size, jump and the share of the image it covers.
Open experimentClose experiment
Loading image…
Photograph with the receptive field of one output unit (yellow box). Click or drag to move the unitEverything this unit can see, enlargedThe stack, layer by layer: r is the receptive field and j the jump after that layer; bars are traced exactly
Try this: Choose “Six 3×3 layers, stride 2 in every second” and click the image: the unit sees 29 × 29 pixels, 1.3% of the picture, and its neighbors are 8 pixels apart. Now choose “Four 3×3 layers, dilations 1, 2, 4, 8”: a similar field, 31 × 31 pixels (1.5%), but the jump stays 1, so every pixel keeps its own output unit. Finally click the top-left corner: why is the covered share smaller there than in the middle of the photograph?
Real photograph: skimage.data.camera at 256 × 256 (credit below). Each layer is a square convolution with kernel k, stride s and dilation d, padded with zeros by (k_eff − 1)/2 on each side, where the effective kernel is k_eff = (k − 1)·d + 1. The size and jump come from the recurrence r ← r + (k_eff − 1)·j and j ← j·s (r = j = 1 at the input), and the box on the image from tracing the index interval of one output unit back through every layer: both agree for every stack here, and an independent SciPy computation of which input pixels influence one unit gives the same boxes. The unit snaps to the nearest position on the output grid, whose spacing is the jump. Where the box leaves the image, the part outside is zero padding and the covered share counts only the part inside. This is the theoretical receptive field: the set of pixels that can change the unit. Measured influence is not uniform; it is concentrated near the center of the box, so the effective receptive field is smaller. That is not computed here, and no weights are trained.
Move the decision threshold on fixed classifier scores. The confusion matrix changes, and the operating point slides along the ROC and precision-recall curves built from the same scores.
Open experimentClose experiment
Precision
0.66667
Recall
0.8
ROC AUC (all thresholds)
0.84
F1
0.72727
Accuracy
0.7
Mean error cost
0.3
True positives (TP): 4; false positives (FP): 2; false negatives (FN): 1; true negatives (TN): 3. Lowering the threshold can recover missed positives while admitting false positives. Precision and recall use different denominators; accuracy alone cannot express unequal error costs.
Try this: Find a threshold with no false positives and note where the dot sits on the ROC curve. How many positives are missed? Then raise the false-positive cost c: the curves do not move, but which point has the lowest mean error cost does.
Ten fixed synthetic scores with five positive labels. Predict class 1 when score ≥ threshold; no model is trained. Each vertex of the curves is one threshold (one of the ten scores); lines between vertices are only drawn to guide the eye. ROC: x is FP/(FP+TN), y is recall TP/(TP+FN); AUC is the trapezoid area. Precision-recall: precision is undefined if nothing is predicted positive, so no dot is drawn. Mean error cost = (c×FP+FN)/10, with false-negative cost fixed at 1. Adjusting c changes evaluation, not scores, decisions or curves.
When do overlapping detections represent one object? Inspect how greedy suppression makes that decision.
Open experimentClose experiment
IoU(A, B)
0.551
IoU(B, C)
0
Kept boxes
2
IoU = intersection area / union area. A suppressed box never suppresses another box; only previously kept boxes participate. A: kept · B: suppressed · C: kept.
Try this: Move B onto A, then raise the IoU threshold. Find a setting that keeps both. Can the rule tell duplicates from two nearby objects?
Three fixed-score boxes of the same class. First keep scores ≥ the score threshold; then visit boxes by descending score and suppress any box whose intersection-over-union with a kept box is strictly greater than the IoU threshold. Scores are toy inputs, not model accuracy.
Turn a predicted box a few degrees away from the ground truth in the bird’s-eye view and watch the exact rotated IoU fall while the centre distance stays at zero.
Open experimentClose experiment
Loading image…
Bird’s-eye view: ground-truth box, predicted box and their exact overlapRotated IoU against yaw error for the three shapes, at the chosen centre offset
Try this: At the default setting (bus or truck, 12 × 2.5 m, yaw error 10°, centres identical) the rotated IoU is 0.657 although the centre distance is 0 m; the enclosing rectangles, which ignore the yaw, give 0.539. Lower the yaw error to 5° and the IoU is still only 0.808. Switch to the car (4.5 × 1.8 m) and the same 10° costs less, 0.796, then to the pedestrian (0.7 × 0.7 m): 0.863 at 10°, and never below 0.707 at any yaw error. The last row of the table gives the yaw error at which each shape falls below IoU 0.7: 8.5° for the bus and 16.55° for the car. Which class would a matching rule that uses an IoU threshold of 0.7 punish most for a heading mistake, and what does a rule that only looks at centre distance say about all three?
The ground-truth box is axis-aligned at the origin with its length along x. The prediction has exactly the right size and is turned by the yaw error about its own centre, then moved along the ground truth’s length by the centre offset. Sizes are illustrative, the ones used in the chapter’s Figures 4 and 6 (pedestrian 0.7 × 0.7 m, car 4.5 × 1.8 m, bus or truck 12 × 2.5 m), not measurements of any dataset. Rotated IoU is exact: the overlap polygon comes from Sutherland–Hodgman clipping and its area from the shoelace formula, and IoU = overlap / (area of the ground truth + area of the prediction − overlap); the code is the chapter’s Python snippet ported to TypeScript and tested against it. The axis-aligned IoU takes the smallest axis-aligned rectangle around each box and computes the ordinary IoU of those two rectangles; it ignores the real shape, so it is lower than the rotated IoU for small yaw errors (the enclosing rectangle of a turned box is larger than the box) and higher for large ones. The centre distance is the Euclidean distance between the two centres on the ground plane. IoU cannot see a 180° flip of the heading, so the yaw error is limited to 0° to 90°. Only the bird’s-eye view is compared here (no height). The nuScenes benchmark matches a detection to an object by the 2D centre distance on the ground plane, to decouple detection from object size and orientation, and scores heading in a separate error term; KITTI’s car threshold of 0.7 is a 3D box overlap.
Simulate a segmentation network that predicts at a coarse output stride and compare pixel accuracy, mean IoU and the IoU of the smallest class.
Open experimentClose experiment
Loading image…
Ground truth (5 classes)Prediction of the simulated networkWrong pixels in redPixel accuracy, mIoU and the IoU of every class
Try this: At the default output stride 8 the prediction gets about 95% of the pixels right, yet the small squares have an IoU of 0. Lower the stride to 4, then 2: at which stride does the thin pole come back, and why does it need a finer stride than the squares? Then raise the stride to 32: how many of the five classes still have a non-zero IoU? Finally switch the enlarging method to bilinear at stride 8: which numbers change, and which one stays at 0?
The label map is synthetic (128 × 128, drawn by code, no random numbers): a background of 85% of the pixels, a disk, a block, a 3-pixel pole and five 6 × 6 squares, all in separate classes. The network is idealized: at output stride s its score for each class in an s × s cell is the fraction of the cell’s pixels that belong to the class, which is the best a stride-s head can report. With nearest enlargement the class with the largest fraction labels the whole cell (ties go to the lowest class number); with bilinear enlargement the fractions are interpolated per class and the largest score labels each pixel. It is the construction of the chapter’s toy experiment on a smaller scene, so its numbers are this lab’s own and not the chapter’s. The scores follow the chapter: pixel accuracy is the share of correct pixels, the IoU of a class is true positives divided by true positives plus misses plus false alarms, and mIoU is the mean of the IoU over all five classes, the background included, computed from one confusion matrix.
Render a Gaussian heatmap at output stride 1, 2, 4 or 8 for 500 random sub-pixel keypoints and compare three ways to read the position back: the brightest cell, a quarter-cell offset and a Taylor step.
Open experimentClose experiment
Loading image…
Heatmap cells around one keypoint (values 0 to 1): true position, decoded position and the brightest cellMean error of the three decoders at each stride, over 500 keypointsHistogram of the 500 errors for the chosen decoding and stride
Try this: At the default setting (stride 4, plain argmax) the 500 keypoints are off by 1.58 px on average, and the table of means shows the error doubling with every doubling of the stride: 0.37, 0.78, 1.58 and 3.12 px at strides 1, 2, 4 and 8, close to 0.383 times the stride. Switch the decoding to the quarter offset (about half of that) and then to the Taylor step: at stride 8 it brings 3.12 px down to 0.41 px. Now keep the Taylor step and raise σ from 2 to 4 cells: at stride 8 its error grows from 0.41 to 1.58 px. A wider blob should be easier to read, so why does the same noise hurt more?
A toy that mirrors the chapter’s code, with its own random numbers (so the means differ in the second decimal from the chapter’s run). 500 keypoints are drawn uniformly in [16, 240] × [16, 240] pixels of a 256 × 256 input. The heatmap at stride s is exp(−‖p − x/s‖² / (2σ²)) on integer cells p, centred on the exact unrounded position, plus Gaussian noise of standard deviation 0.01 per cell, floored at 10⁻⁶; a cell’s centre is the integer position, as in the chapter, so the brightest cell i maps to s · i. Argmax decodes s · (brightest cell). Quarter offset moves the brightest cell by ¼ cell on each axis toward the brighter neighbour. The Taylor step is one Newton step on the logarithm of the heatmap at the brightest cell, with central differences, kept only if the 2 × 2 Hessian is negative definite. The error is the Euclidean distance to the true position in input pixels. Only the (2R + 1)² cells around the true cell, R = ⌈2σ⌉ + 2, are generated; outside them the Gaussian is below 0.14 (against a peak of at least 0.77 and noise of 0.01), so the rest of the map cannot change the answer. The grid shows one fixed keypoint at (102.6, 78.9), the same noise for every decoder. Real networks add errors that this toy leaves out; the lab isolates the error that comes from the grid alone.
Edit the blank probability in the frames between two equal letters of a toy recognizer output, and watch greedy best-path decoding and prefix beam search read it.
Open experimentClose experiment
Loading image…
Toy network output: one row per frame, one column per symbol (orange bar: the frames between the repeated letters)The five most probable texts, with all alignments of each added up
Try this: At the default setting (blank 0.45 and O 0.55 in each of the two frames between the O’s, greedy decoding) the largest entry of every frame gives the path B O O O O K, which merges to BOK: one O is lost. Yet the correct word has the largest total probability, 0.410, against 0.310 for BOK, because every alignment with a blank in either gap frame reads BOOK. Switch to prefix beam search (beam 3): it writes BOOK. Now move the blank probability. At 0.50 greedy decoding also writes BOOK (a tie goes to the blank). With beam search, 0.40 still gives BOOK, but at 0.35 it writes BOK (0.377 against 0.343 for BOOK). At 0.05 the network almost surely saw one long O: both decoders write BOK and the loss −ln p(BOOK) is 2.819. Finally choose SEE: greedy decoding flips at 0.50 and beam search between 0.35 and 0.40, just as for BOOK. Why can the sum over alignments recover a double letter that no single frame votes for?
A hand-made toy, not the output of a trained recognizer. Each letter of the word has one sure frame (the letter 0.85, the blank 0.15); between two equal letters there are two gap frames in which the network hesitates between the blank (probability q, the slider) and the letter (1 − q). Every other entry is 0, so the matrix asks one question only: blank or letter? BOOK has 6 frames (B, O, two gap frames, O, K) and SEE has 5. The collapsing map merges repeats first and then deletes blanks, so a double letter can be written only with a blank between its copies. Greedy decoding takes the largest entry of every frame; a tie goes to the blank, as NumPy’s argmax goes to the first index. The right-hand bars add up every alignment of each text by enumeration (4⁶ = 4,096 alignments for BOOK, 3⁵ = 243 for SEE); the probability of the correct word is also computed with the forward algorithm, and the tests check the two against each other. Prefix beam search keeps, for each prefix, the probability of alignments that end in a blank and of those that end in the last letter, and only the 3 best prefixes after every frame; pruning can lose some alignments, so its own probability can be a little below the exact one (0.366 against 0.377 for BOK at q = 0.35). The same code reproduces the 5-frame ‘cat’ example of the chapter (greedy ca, beam 3 cat 0.344, exact 0.363).
Track three coloured blocks through 30 synthetic frames with a Kalman filter, a cost matrix and the Hungarian algorithm, and watch two identities swap when only box overlap is used.
Open experimentClose experiment
Loading image…
Synthetic frame: true boxes (white dashed), predicted boxes (dotted) and track boxes (solid, with trails), coloured by track idCost matrix of the inspected frame, with the Hungarian assignment outlinedIdentity ribbon over the whole clip (click to choose the frame)
Try this: At the default setting (λ = 1, box overlap only) the orange and blue walkers swap tracks at frame 15, right after they meet: 2 identity switches, and 30 of the 90 object-frames carry the wrong id. Lower λ in steps of 0.05: the swap disappears between λ = 0.9 and 0.8, and at λ = 0.5 the mean IoU of the tracks with their own objects rises from 0.582 to 0.850. Now set the jitter to 0: overlap alone still swaps, so the turn and not the noise causes it. Finally set the jitter to 5 pixels: even λ = 0.5 gets 4 identity switches. At frame 13 the colour cost 1 − cos of track #1 is 0.47 for the blue walker and 0.61 for its own orange walker. Why is colour less reliable now?
Synthetic and tiny on purpose: everything is drawn by code, no photographs. Scene: three 30 × 40 pixel person-shaped blocks in a 160 × 128 frame over 30 frames (orange, blue and green shirts, each with its own trouser colour) plus Gaussian pixel noise of 0.03. The orange and blue walkers approach each other along one line at 4 pixels per frame, stand at the same x at frame 14 (the blue one 8 pixels lower and in front) and turn back; the green one walks along the bottom. The scene uses meet-and-turn rather than a pass-through because a pass-through at constant velocity is kept apart by motion alone and swaps only by chance under noise, whereas after a turn the straight-line guess points at the wrong walker (the case the chapter calls the bounce). The tracker follows the chapter: a Kalman filter on [cx, cy, w, h, vx, vy] (q = 1, r = 4), C = λ(1 − IoU) + (1 − λ)(1 − m·e) where e is the unit-length colour histogram (4 bins per channel) of the pixels inside the detection box and m a track memory updated with α = 0.9, and the Hungarian algorithm. Simplified: every frame has exactly one detection per object (no misses, false alarms, gating, birth or death); detections arrive in a rotating order and carry their true object only for scoring; the jitter is a fixed field of random numbers (seed 16) scaled by the slider, so it never re-rolls. Metrics: an identity switch is an object matched to a different track id than in the previous frame (the chapter’s definition), so two swapped tracks count 2; the mean IoU compares each track’s filtered box with the true box of the object the track started on; MOTA = 1 − (FN + FP + IDSW) / GT with FN = FP = 0 and GT = 90 object-frames, so two switches cost only 0.022 although the ids stay wrong for 15 frames. One jitter draw is one clip: over 200 other draws at σ = 2.5, overlap alone switches in 164 and λ = 0.5 in 12; at σ = 5, in 165 and 90. A NumPy and SciPy re-implementation reproduces every number.
A blurry observation fits two sharp clean signals. The blurry average has the lowest expected squared error; a sharp guess looks better but is usually farther from the truth.
Open experimentClose experiment
Expected reconstruction MSE
0.5
Conditional-mean minimum MSE
0.25
Sharpness (adjacent total variation)
3
Averaging incompatible possibilities minimizes expected squared error, but can remove structures present in every valid clean outcome. Choosing one restores structure while risking the wrong structure.
Try this: Start from the sharp copy of A (commitment 1): its risk is higher than the blurry mean's. Slide the commitment to 0 and watch the risk fall to the minimum. Then raise the probability of A to 0.9: does the sharp copy of A catch up, and can it ever win?
Two fixed 8-sample patterns, A = 00110011 and B = 11001100, are the only clean signals that could have produced the same blurry observation. P(A|y) = p is supplied, not learned. The blurry mean is p·A + (1−p)·B. Your reconstruction blends the mean toward A: (1−m)·mean + m·A, so m = 1 is exactly A (a sharp sample) and m = 0 is the mean. Risk = the expected mean squared error, p·MSE(A, output) + (1−p)·MSE(B, output), averaged over which pattern is the truth. Sharpness is the total variation (sum of absolute differences between neighbouring samples), not a perceptual quality score.
Simulate a dim photo with photon and read noise, then compare more light, more digital gain and a denoiser.
Open experimentClose experiment
Loading image…
The scene (what we want)Sensor output after gainAfter the enhancer
Try this: At 8 photons and gain 12 the picture is as bright as the scene, yet very noisy. Lower the gain to 1: the picture goes dark, but does the signal-to-noise ratio change? Now keep gain 12 and raise the photons to 100. What changes instead, and what happens to the highlights?
Camera test image, 256 × 256. Each pixel collects Poisson(photons × scene) electrons plus Gaussian read noise (σ = 2 electrons), then the output is clip(gain × electrons / 100). Noise fields are fixed, so sliders never re-roll them. SNR is the mean signal over the standard deviation of the noise, measured before clipping, in dB. The enhancers are a Gaussian blur (σ = 1.5) and a 3 × 3 median, stand-ins for a learned denoiser; they are not one. Large photon counts use a normal approximation of the Poisson distribution.
Bake one smooth color enhancement into an M × M × M table and apply it with a nearest-node lookup or with trilinear interpolation. Compare both with the exact transform.
Open experimentClose experiment
Loading image…
Exact transform (the target)LUT, nearest nodeLUT, trilinearError map of the chosen lookup (8×, bright: wrong)
Try this: At the default (M = 9, the smooth gradient) the nearest lookup shows clear bands and the trilinear result looks like the exact one. Switch the error map to trilinear: where is the error largest, and why there? Then raise M to 33 for the nearest lookup: are the bands gone? Finally set M = 3 and compare the two interpolations.
The true enhancement is defined in code (src/lib/playground/labs/lut-trilinear.ts) and is the lab’s own toy: on each channel t, a tone curve g = t^0.9 followed by the S-curve (1 − a) g + a (3g² − 2g³) with a = 0.5; then a saturation gain of 1.15 that scales each color’s distance from its luma gray (luma 0.299 R + 0.587 G + 0.114 B); then a hue turn of 8° (a rotation of the RGB cube about its gray axis); then clipping to [0, 1]. The table has one entry per node of an M × M × M lattice over the RGB cube: node (i, j, k) stores the exact transform of the color (i/(M−1), j/(M−1), k/(M−1)), so it holds 3M³ numbers. A nearest lookup returns the stored color of the closest node. Trilinear interpolation blends the 8 corners of the cell around the color with weights that are products of the offsets, as in the chapter. Both results and the exact transform are rounded to 8-bit levels (0 to 255) before the errors are measured, and the error map is magnified 8 times. The photo is the astronaut test image reduced to 256 × 256; the gradient is a smooth blend of four corner colors computed in the lab, so it has no texture to hide banding.
Two shots of one scene are each sharp in a different region. Average them, or pick every pixel from the shot with more local detail, and score both against the all-in-focus truth.
Open experimentClose experiment
Loading image…
Shot A (sharp inside the disk)Shot B (sharp outside the disk)Average (A + B)/2Fused by local detailDecision map (white: take A; red line: true boundary)Error of the fused image (4×, bright: wrong)
Try this: Start from the defaults: the average is blurry everywhere while the focus rule is sharp. Now lower the window to 0.5 and look at the decision map: why does it fill with specks, and where do they hurt the picture? Switch the detail measure to local variance at window 0.5 (it is far less forgiving), then raise the window and smooth the map by 2. What does smoothing buy, and what does a very large window cost at the circle’s edge?
Synthetic multi-focus pair (scripts/figures/playground_image_fusion.py), the recipe of the chapter’s snippet at half the size: the camera test image (256 × 256) is the all-in-focus truth; shot A keeps it inside a disk and is blurred (Gaussian, σ = 2) outside, shot B the other way round, each with Gaussian noise of standard deviation 0.01. Because we made the blur, the true decision map (the disk) is known, which real multi-focus pairs never provide. The average is (A + B)/2. The focus rule scores each shot at each pixel with a detail measure averaged over a Gaussian window of standard deviation σ (the “window”): the Laplacian energy G_σ * (∇²I)², with ∇²I = I(x−1,y) + I(x+1,y) + I(x,y−1) + I(x,y+1) − 4 I(x,y), or the local variance G_σ * I² − (G_σ * I)². The decision map marks the pixels where shot A scores higher, and F = w A + (1 − w) B with w the decision map, optionally blurred by a Gaussian of the chosen σ (0 keeps it binary). PSNR = 10 log₁₀(1/MSE) in dB against the truth, pixel values in [0, 1]. Map agreement is the share of pixels, counted only where the picture has texture (smoothed |Laplacian| of the truth above 0.01), on which the decision equals the true disk. These numbers belong to this toy.
Run block matching on a synthetic stereo pair with known disparity, and see small windows turn noisy while large windows fatten the foreground.
Open experimentClose experiment
Loading image…
Left image (with noise)True disparity (dark: far, light: near; left strip not scored)Estimated disparity (pink: invalidated; white line: true depth edge)Error against window size at this noise level
Try this: At the default 3 × 3 window the map is noisy, most of all inside the textureless patch. Grow the window one step at a time and watch the three curves: where is the all-pixel error lowest? What happens to the white outline of the disc once the window reaches 31 × 31? Then raise the noise to 0.06 and see how much a 3 × 3 window suffers. Finally switch the left–right check on: which pixels turn pink, and what happens to bad-1?
Synthetic and tiny on purpose: the left image is 128 × 96 and is drawn by code, with fixed seeds so nothing flickers. A textured background plane has disparity 4, a brighter textured disc has disparity 12, and a flat gray rectangle lies on the background. The right image shifts every left pixel left by its disparity (the nearer surface wins) and fills the uncovered pixels with fresh texture; Gaussian sensor noise with the chosen standard deviation is added to both images. The matcher is the one of the chapter: for every disparity 0 to 15 the cost of a window is the mean absolute difference (SAD, normalized by the number of pixels that have a partner), and the winner-take-all rule takes the lowest cost. Bad-1 is the percentage of scored pixels whose disparity is off by more than 1 pixel. Scored pixels are those visible in both views and at least 16 columns from the left border (washed out in the pictures). The check matches again with the right image as reference and invalidates a pixel whose two answers differ by more than 1; bad-1 and the flat-patch score are then taken over the kept pixels. Disc fattening counts the visible background pixels labelled with the disc’s disparity (within 1), as a percentage of the disc’s true area. These are the lab’s own numbers on its own toy scene, not those of the chapter’s scene.
Compare one-step predictions, open-loop rollout and periodic observation refresh under the same actions.
Open experimentClose experiment
One-step MSE
0.00195
Open-loop MSE
1.15919
Refreshed-rollout MSE
0.02323
One-step evaluation repeatedly supplies the true state. An open-loop rollout must consume its own imperfect predictions. Periodic observations limit drift without improving the transition model itself.
Try this: Set the model coefficient to the true value 0.96. Then change it to 1.02 and increase the horizon. How much can observation refresh correct?
True scalar state: x(t+1)=0.96x(t)+u(t), x(0)=1, u(t)=a sin(0.45t). The model replaces 0.96 with a chosen coefficient. Refresh feeds the true state every k steps; k=0 disables it. Fully observed deterministic dynamics, no learned video model or replanning.
Train a linear classifier on noisy stripe images of 64, 256 or 1,024 pixels, then change every pixel by the same few levels, once along the loss gradient (FGSM) and once with random signs.
Open experimentClose experiment
Loading image…
Left: a clean test image (noisy stripes). Middle: the change, stretched to the full gray range for display (white = +ε, black = −ε). Right: the perturbed image. Under each, the logit and the class it gives.Accuracy on 300 test images as ε grows
Try this: At the default setting (32 × 32 pixels, ε = 8, FGSM) accuracy falls from 99.3% to 2.7%, while random signs of the same size leave it at 99.3%. The mean logit shift is +5.8, against a median clean margin of 3.5. Now switch the size to 16 × 16 and then to 8 × 8 at the same ε: FGSM leaves 76.3% and 94.7%. Next find the ε at which FGSM first brings accuracy below 50%: ε = 5 for 32 × 32, ε = 11 for 16 × 16, and none up to ε = 16 for 8 × 8 (70.7% at ε = 16). The classes differ by the same total amount at every size, so why does the same ε matter more when the image has more pixels?
Synthetic and tiny on purpose: no photographs. Each image is 0.5 plus or minus an oblique cosine stripe pattern (two periods across, one down) plus Gaussian noise with standard deviation 0.2 (51 of 255 levels), clipped to [0, 1]; class A has the stripes in one phase and class B in the opposite phase. The stripe amplitude per pixel is 3 × 0.2 / √d for d pixels, so the two classes differ by the same total amount at every size (before clipping, the best possible classifier is equally accurate at every size); only the number of pixels that carry the evidence changes. The classifier is a logistic regression with one weight per pixel, trained in your browser by 50 steps of full-batch gradient descent (step 8, weight decay 0.001) on 300 images and tested on 300 other images (seeds 11 and 12). Its logit is z = w·(x − 0.5) + b, so the gradient of the cross-entropy loss with respect to the pixels is (p − y)·w and FGSM, x′ = clip(x + ε·sign(∇ₓL), 0, 1), moves every pixel by ε toward the wrong class. With no clipping the logit moves by exactly ε‖w‖₁; random signs move it by about ε‖w‖₂ and cancel on average. Here ‖w‖₂ is 7.5, 7.5 and 6.9 for 64, 256 and 1,024 pixels, while ‖w‖₁ is 54.4, 102.8 and 186.2. ε is counted in levels of 8-bit brightness (1 level = 1/255). This is a linear toy that isolates the argument of Goodfellow et al.; it is not the digits experiment of the chapter, a deep network is not linear, and one draw of data and training is shown, so read the trend, not the exact percentages.
Place two feasible points on a circle, slide along the segment between them, and compare a disk with an annulus.
Open experimentClose experiment
Radius at weight λ
0.291
Smallest radius on the segment
0.291
Angle apart (degrees)
140
The chosen point lies in the hole although both endpoints are feasible: the annulus is not convex.
Try this: Choose the annulus and place both endpoints on the circle. Find two endpoints whose whole segment stays feasible, and two whose segment crosses the hole. How far apart in angle do they need to be?
Endpoints are P = 0.85(cos α, sin α) and Q = 0.85(cos β, sin β). The disk has radius 1; the annulus also requires radius ≥ 0.5. The segment is x(λ) = (1−λ)P + λQ. One counterexample disproves convexity; sampled examples alone do not prove it.
Compare coordinate clipping with the exact solution of a coupled quadratic under nonnegative constraints.
Open experimentClose experiment
Optimal objective
0.255
Clipped objective
0.5
Excess objective
0.245
Optimal (x,y)=(0, 0.3). Coupling changes the best boundary position.
Try this: Set coupling to zero, then increase it. When does clipping cease to minimize the objective?
Minimize ½[(x−a)²+2ρ(x−a)(y−1)+(y−1)²], x,y≥0. |ρ|<1 ensures a positive definite Hessian. The solver checks the interior and both boundary minima; it does not use a grid approximation.
Move one observation and compare squared, absolute and Huber losses for fitting a constant.
Open experimentClose experiment
Fitted constant
1.333
Total loss
27.917
The loss defines what an error costs. Robust residual loss and parameter regularization solve different problems.
Try this: Increase the outlier from 2 to 12. Compare the movement of the squared-loss and Huber estimates.
Five fixed observations are −1, −0.5, 0, 0.5, 1. The sixth is adjustable. Squared loss includes ½; Huber uses δ=1. For absolute loss, any point in [0,0.5] minimizes the loss; the plot chooses 0.25.
See why a step size that works on a round bowl can diverge on a narrow one.
Open experimentClose experiment
Final objective
0.034
Converges when 0 < η <
0.4
Coordinate multipliers (x, y)
0.85, 0.25
Both coordinate multipliers have magnitude below 1, so the iterates converge. A negative multiplier produces an alternating path; a multiplier close to 1 or −1 produces slow progress.
Try this: Set k = 10. Compare step sizes 0.1, 0.2 and 0.3. What happens to the y coordinate at the stability boundary?
An exact quadratic: f(x,y) = (x² + ky²)/2, starting at (3,2). Updates are x ← (1−η)x and y ← (1−ηk)y. The contour plot has a fixed viewing window; displayed objective values are never clipped.
Raise the barrier weight t on a two-variable linear program and watch the central point, its dual bound and the gap m/t.
Open experimentClose experiment
analytic center (t → 0)optimal vertexGreen: central path · orange: x*(t) · dashed: level line of c·x through x*(t)
Gap m/t
0.5
Suboptimality c·x − p*
0.1841
Distance to nearest edge
0.109
At t = 10 the central point is 0.109 from the nearest edge. Its suboptimality 0.1841 is at most the duality gap m/t = 0.5, and the gap computed from the dual point λ = 1/(t·slack) is 0.5.
Try this: Multiply t by ten. How do the distance to the boundary, the true suboptimality and the bound m/t change? Then rotate the objective: does the path still end at a vertex?
Minimize c·x over a convex pentagon with m = 5 edge constraints a_i·x ≤ b_i. The central point minimizes t·c·x − Σ log(b_i − a_i·x) (damped Newton). With λ_i = 1/(t s_i), where s_i is the slack, the dual objective is −b·λ and the duality gap equals Σ λ_i s_i = m/t exactly. The true suboptimality c·x − p* is at most m/t.
Compare soft thresholding with ordinary shrinkage and inspect its nonsmooth objective.
Open experimentClose experiment
x = −4x = 4
Proximal solution
0.5
Objective
1
Residual v − x
1
Outside the threshold, the magnitude is reduced by λ.
Try this: Keep v=1 and increase λ through 1. Why does the optimum stay at exactly zero afterward?
Solve minₓ ½(x−v)²+λ|x|. The exact proximal solution is sign(v) max(|v|−λ,0). This scalar identity extends coordinatewise to an L1 penalty; it is not a complete total-variation denoiser.
Change the true probability and the model probability; measure the extra coding cost caused by a wrong model.
Open experimentClose experiment
Expected bits per symbol (shared scale 0–7)H(P)0.722H(P,Q)1D(P∥Q)0.278
Entropy
0.722
Cross-entropy
1
KL divergence
0.278
H(P,Q) = 0.722 + 0.278 bits. Matching probabilities removes the KL penalty, but cannot remove source uncertainty.
Try this: Fix p=0.8. Find the q that minimizes cross-entropy, then compare q=0.2.
For Bernoulli sources, H(p,q)=−p log₂q−(1−p)log₂(1−q)=H(p)+D(p∥q). q stays strictly between 0 and 1. These are expected ideal bit costs, not measured file sizes.
The output has P(Y=1) = 0.5. I(X;Y) = H(Y) − h₂(p) = 1 − 0.469 = 0.531. Random output alone does not imply useful information: at p = 0.5 the output is independent of the input.
Try this: With crossover p = 0.1, move the input prior away from 0.5. Does the channel capacity change? Does the transmitted information change?
Independent bit flips define a binary symmetric channel. q = P(X=1), p = P(Y≠X), and P(Y=1) = p + q(1−2p). All values are bits per channel use. Capacity maximizes mutual information over q; here q = 0.5 achieves it.
Compare the probability mass of a typical set with its fraction of all possible binary blocks.
Open experimentClose experiment
Green: typical · Blue: other · Height: probability massk = 0k = 16
Typical probability mass
0.6575
Fraction of all strings
0.03815
H(p) (bits)
0.722
Typicality concerns information per symbol. A single likely string and a set carrying most probability are different objects.
Try this: Fix p=0.2 and increase block length. Does the typical set contain most probability while using most of all binary strings?
Independent Bernoulli(p) blocks of length n. A block is typical when |−log₂ P(block)/n−H(p)|≤ε. Each bar aggregates all C(n,k) strings with k ones. Small n can give an empty typical set; AEP is asymptotic.
No codeword is the prefix of another, so concatenated bits can be decoded without separators. Expected length is Σ p(symbol) × code length = 1.5 bits/symbol. Entropy is 1.357; their gap is 0.143. Spaces in the example only show codeword boundaries.
Try this: Compare P(A) = 0.25 with P(A) = 0.7. Why does the optimal code change in discrete jumps while entropy changes smoothly?
Huffman coding repeatedly merges the two least probable nodes. Ties use alphabetical order. B, C and D each have probability (1−P(A))/3. The expected length assumes this source distribution; the example message has its own symbol counts. Codebook transmission cost is excluded.
Compare equal allocation with water-filling across two parallel Gaussian channels.
Open experimentClose experiment
Gray: noise floor · Blue: allocated powerN1=1N2=4
Optimal bits / joint use
1.17
Equal-allocation bits
1.085
Capacity gain
0.085
P₁=3.5, P₂=0.5. Active channels share a water level; a noise floor above it receives no power.
Try this: Give the second channel high noise. Increase the total power until it first receives a positive allocation.
Independent real Gaussian channels, noise variances N₁=1 and N₂, total power P. Capacity per joint use is ½Σ log₂(1+Pᵢ/Nᵢ); water-filling uses Pᵢ=max(μ−Nᵢ,0). This assumes optimal Gaussian signaling, not a finite code.
Orange off-diagonal cells are reconstruction errors. Their total equals D; the joint distribution retains the selected source marginal.
Try this: Move distortion to its maximum. Why is sending no bits sufficient even though some symbols are wrong?
X~Bernoulli(p), p≤0.5, Hamming distortion D≤p. R(D)=h₂(p)−h₂(D). Construct X=Y XOR Z with independent Z~Bernoulli(D) and P(Y=1)=(p−D)/(1−2D). At p=D=0.5 choose constant Y=0. The curve is an asymptotic block-coding limit, not the rate of this single-symbol channel.
Slide a decision threshold across two exact binomial distributions and compute both error probabilities.
Open experimentClose experiment
Blue: H₀ · Orange: H₁ · Shade: reject H₀k=0k=10
False alarm
0.04735
Miss
0.15027
Detection probability
0.84973
Reject H₀ for k ≥ 6. Changing a threshold changes both errors; increasing evidence can improve their tradeoff.
Try this: Try thresholds that always accept or always reject H₀. Then increase n while keeping the threshold near n/2.
n independent binary observations. H₀:p=0.3, H₁:p=0.7; reject H₀ if number of ones k≥threshold. Curves show exact finite-sample probability masses. False alarm is conditional on H₀, miss on H₁; neither is a posterior probability.
Averaging either row or column gives 0.5: a single feature does not change the label distribution. Observing both can.
Try this: Set label noise to zero. Why does each feature have zero information while their pair has one bit?
A and B are independent fair bits, Y=A XOR B XOR E, E~Bernoulli(ε) independent. Thus I(A;Y)=I(B;Y)=0 and I(A,B;Y)=1−h₂(ε). This population identity does not establish a finite-sample generalization bound.