ML 02 · Can a tiny network learn a curved boundary?
IntermediateML 02 · Neural Networks from Perceptron to Deep NetworksImage lab
Description
Train a network with 2 inputs, m tanh hidden units and 1 sigmoid output on the two moons and the two circles, one gradient-descent step at a time. Watch the decision map and the training and test loss: too few hidden units, too small a learning rate and too large a learning rate each fail in their own way.
Try this: Start from the default (m = 3, η = 1, 1000 steps), where both curved boundaries are learned. (1) Drag the steps slider up from 0: after 100 steps the moons are only 85.0% right (close to one tilted straight line, the loss on a plateau near 0.30), after 300 steps 94.5%, after 500 steps 98.0%. (2) Set η to 0.3 and lower m to 2, then 1: at 1000 steps the moons stay at 86.0% and 85.0%, the circles at 77.5% and 61.5%. (3) Go back to m = 3 and try η = 0.03, 0.1 and 0.3: at 1000 steps the moons are 84.5%, 85.0% and 94.5% right. A small η is not wrong, only slow: run offline for 5000 steps, beyond the 2000-step limit of this lab, η = 0.1 reaches 98.0% on the moons. (4) Try η = 3, 10 and 30: the loss now goes up on many steps, and with the largest η the training does not settle.
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
Not well. With one hidden unit the network computes a single number, tanh(w·x + b), and the output is a rising or falling function of it, so the decision boundary is a straight line however long you train. Checking every line through two of the 200 training points, the best straight line gets 175 of them right (87.5%), and a one-unit network cannot beat that. The default three hidden units bend the boundary: 97.5% of the training points and 95.5% of the held-out test points are right. On the circles the best straight line gets only 133 of 200 (66.5%).
Lab · ML 02 · Can a tiny network learn a curved boundary?
Loading image…
Try this: Start from the default (m = 3, η = 1, 1000 steps), where both curved boundaries are learned. (1) Drag the steps slider up from 0: after 100 steps the moons are only 85.0% right (close to one tilted straight line, the loss on a plateau near 0.30), after 300 steps 94.5%, after 500 steps 98.0%. (2) Set η to 0.3 and lower m to 2, then 1: at 1000 steps the moons stay at 86.0% and 85.0%, the circles at 77.5% and 61.5%. (3) Go back to m = 3 and try η = 0.03, 0.1 and 0.3: at 1000 steps the moons are 84.5%, 85.0% and 94.5% right. A small η is not wrong, only slow: run offline for 5000 steps, beyond the 2000-step limit of this lab, η = 0.1 reaches 98.0% on the moons. (4) Try η = 3, 10 and 30: the loss now goes up on many steps, and with the largest η the training does not settle.
Data: scikit-learn make_moons (noise 0.2, random_state 0) and make_circles (noise 0.1, factor 0.5, random_state 0), 200 training points each, plus a held-out test set of 200 points each (random_state 1) that is never used for training. Network and loss as in ML 02: z₁ = W₁x + b₁, a₁ = tanh(z₁), z₂ = W₂a₁ + b₂, p = σ(z₂), binary cross-entropy averaged over the training points; the gradient comes from backpropagation (checked against finite differences and against an independent NumPy run). Training is plain full-batch gradient descent, θ ← θ − η∇L, with a fixed η: no momentum, mini-batches or regularization. Initial weights: Xavier normal (variance 2 / (fan-in + fan-out)) with zero biases, from one fixed seed (5), so a slider never re-rolls them; unit j always gets the same random numbers. Another seed changes the details: over seeds 1 to 20 (η = 1, 2000 steps), 3 hidden units reached at least 95% training accuracy on both datasets in all 20 runs, 1 unit never did, and 2 units reached it on the moons in at most 1 of the 20 runs and on the circles never. A step is one parameter update; step 0 is the untrained network. When the loss goes up and down the iteration is chaotic, so the curves at large η can differ in their details between browsers, though not in their story. The test-loss curve is sampled every 4 steps.
Method and image sources
- scikit-learn documentation, sklearn.datasets.make_moons (the two-moons data, generated here, no download)
- scikit-learn documentation, sklearn.datasets.make_circles (the two-circles data)
- F. Pedregosa et al., Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research, 12, 2825-2830, 2011
- D. E. Rumelhart, G. E. Hinton and R. J. Williams, Learning representations by back-propagating errors, Nature, 323, 533-536, 1986 (backpropagation)
- X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, Proceedings of AISTATS, PMLR 9, 249-256, 2010 (Xavier initialisation)
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016 (multilayer perceptrons, cross-entropy, gradient descent)