Full search page →

Two hidden units can bend a decision region

BeginnerNLP 03 · Neural Networks for NLPInteractive plot

Description

Compare a fixed two-unit network with and without nonlinear activations on the XOR corners.

Try this: Start with the linear activation and try every bias: can all four corners be right at once? Then switch to ReLU and tune the gain and bias. Why does the nonlinearity change the answer?

Predict first

With the linear activation selected, can some output bias classify all four XOR corners correctly?

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

A linear activation collapses the network to one linear score that falls steadily as x + y grows. XOR needs the middle value of x + y to be class 1 and both ends to be class 0, which a single threshold on that score cannot do. At best three of the four corners are right.

Lab · Two hidden units can bend a decision region

Predict first

With the linear activation selected, can some output bias classify all four XOR corners correctly?

Correct corners
3 / 4
P(1 | 0,1)
0.881
P(1 | 1,1)
0.119

A linear hidden layer collapses to one affine decision boundary. Nonlinear hidden units create pieces that can assign the two opposite XOR corners to the same class.

Try this: Start with the linear activation and try every bias: can all four corners be right at once? Then switch to ReLU and tune the gain and bias. Why does the nonlinearity change the answer?

Weights are hand constructed, not trained: h₁=a(g(x+y)), h₂=a(g(x+y−1)), p=σ(4(h₁−2h₂)/g−2+b). Color is P(class 1); circles show known corner labels.

Method sources

Open in the Playground →