Full search page →

ML 02 · Can a tiny network learn a curved boundary?

IntermediateML 02 · Neural Networks from Perceptron to Deep NetworksImage lab

Description

Train a network with 2 inputs, m tanh hidden units and 1 sigmoid output on the two moons and the two circles, one gradient-descent step at a time. Watch the decision map and the training and test loss: too few hidden units, too small a learning rate and too large a learning rate each fail in their own way.

Try this: Start from the default (m = 3, η = 1, 1000 steps), where both curved boundaries are learned. (1) Drag the steps slider up from 0: after 100 steps the moons are only 85.0% right (close to one tilted straight line, the loss on a plateau near 0.30), after 300 steps 94.5%, after 500 steps 98.0%. (2) Set η to 0.3 and lower m to 2, then 1: at 1000 steps the moons stay at 86.0% and 85.0%, the circles at 77.5% and 61.5%. (3) Go back to m = 3 and try η = 0.03, 0.1 and 0.3: at 1000 steps the moons are 84.5%, 85.0% and 94.5% right. A small η is not wrong, only slow: run offline for 5000 steps, beyond the 2000-step limit of this lab, η = 0.1 reaches 98.0% on the moons. (4) Try η = 3, 10 and 30: the loss now goes up on many steps, and with the largest η the training does not settle.

Predict first

Set the hidden units to 1 and leave everything else as it is. Can the network separate the two moons?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

Not well. With one hidden unit the network computes a single number, tanh(w·x + b), and the output is a rising or falling function of it, so the decision boundary is a straight line however long you train. Checking every line through two of the 200 training points, the best straight line gets 175 of them right (87.5%), and a one-unit network cannot beat that. The default three hidden units bend the boundary: 97.5% of the training points and 95.5% of the held-out test points are right. On the circles the best straight line gets only 133 of 200 (66.5%).

Lab · ML 02 · Can a tiny network learn a curved boundary?

Predict first

Set the hidden units to 1 and leave everything else as it is. Can the network separate the two moons?

Loading image…

Two moons: decision map at the chosen step (blue circles = class 0, orange triangles = class 1; filled = training, hollow = test; ringed = wrong; black line = 0.5 boundary)
Two moons: cross-entropy loss against the step (solid = training, dashed = test)
Two circles: decision map at the chosen step (same symbols)
Two circles: cross-entropy loss against the step (solid = training, dashed = test)

Try this: Start from the default (m = 3, η = 1, 1000 steps), where both curved boundaries are learned. (1) Drag the steps slider up from 0: after 100 steps the moons are only 85.0% right (close to one tilted straight line, the loss on a plateau near 0.30), after 300 steps 94.5%, after 500 steps 98.0%. (2) Set η to 0.3 and lower m to 2, then 1: at 1000 steps the moons stay at 86.0% and 85.0%, the circles at 77.5% and 61.5%. (3) Go back to m = 3 and try η = 0.03, 0.1 and 0.3: at 1000 steps the moons are 84.5%, 85.0% and 94.5% right. A small η is not wrong, only slow: run offline for 5000 steps, beyond the 2000-step limit of this lab, η = 0.1 reaches 98.0% on the moons. (4) Try η = 3, 10 and 30: the loss now goes up on many steps, and with the largest η the training does not settle.

Data: scikit-learn make_moons (noise 0.2, random_state 0) and make_circles (noise 0.1, factor 0.5, random_state 0), 200 training points each, plus a held-out test set of 200 points each (random_state 1) that is never used for training. Network and loss as in ML 02: z₁ = W₁x + b₁, a₁ = tanh(z₁), z₂ = W₂a₁ + b₂, p = σ(z₂), binary cross-entropy averaged over the training points; the gradient comes from backpropagation (checked against finite differences and against an independent NumPy run). Training is plain full-batch gradient descent, θ ← θ − η∇L, with a fixed η: no momentum, mini-batches or regularization. Initial weights: Xavier normal (variance 2 / (fan-in + fan-out)) with zero biases, from one fixed seed (5), so a slider never re-rolls them; unit j always gets the same random numbers. Another seed changes the details: over seeds 1 to 20 (η = 1, 2000 steps), 3 hidden units reached at least 95% training accuracy on both datasets in all 20 runs, 1 unit never did, and 2 units reached it on the moons in at most 1 of the 20 runs and on the circles never. A step is one parameter update; step 0 is the untrained network. When the loss goes up and down the iteration is chaotic, so the curves at large η can differ in their details between browsers, though not in their story. The test-loss curve is sampled every 4 steps.

Method and image sources

Open in the Playground →