Two hidden units can bend a decision region
BeginnerNLP 03 · Neural Networks for NLPInteractive plot
Description
Compare a fixed two-unit network with and without nonlinear activations on the XOR corners.
Try this: Start with the linear activation and try every bias: can all four corners be right at once? Then switch to ReLU and tune the gain and bias. Why does the nonlinearity change the answer?
Hints
The chapter section that uses this lab:
Explanation
Answer the prediction first, or open the explanation when you are ready.
Show the explanation
A linear activation collapses the network to one linear score that falls steadily as x + y grows. XOR needs the middle value of x + y to be class 1 and both ends to be class 0, which a single threshold on that score cannot do. At best three of the four corners are right.
Lab · Two hidden units can bend a decision region
- Correct corners
- 3 / 4
- P(1 | 0,1)
- 0.881
- P(1 | 1,1)
- 0.119
A linear hidden layer collapses to one affine decision boundary. Nonlinear hidden units create pieces that can assign the two opposite XOR corners to the same class.
Try this: Start with the linear activation and try every bias: can all four corners be right at once? Then switch to ReLU and tune the gain and bias. Why does the nonlinearity change the answer?
Weights are hand constructed, not trained: h₁=a(g(x+y)), h₂=a(g(x+y−1)), p=σ(4(h₁−2h₂)/g−2+b). Color is P(class 1); circles show known corner labels.
Method sources
- Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning, MIT Press, 2016
- Ashish Vaswani et al., Attention Is All You Need, NeurIPS, 2017
- Jonathan Ho, Ajay Jain & Pieter Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS, 2020
- Tim Salimans & Jonathan Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR, 2022