Why XOR needs an extra layer

Topic: Neural units · more questions on this topic
‹ Linear hidden units

The four inputs (0,0), (1,0), (0,1), (1,1) have labels 0, 1, 1, 0, respectively. A threshold unit outputs 1 when its net input is nonnegative.

  1. Explain geometrically why a single linear threshold unit cannot classify all four inputs correctly.

  2. Construct a two-hidden-unit threshold network for these labels, specifying all weights and biases.

  3. Explain what would be lost by replacing all the activations in a multilayer network by linear functions.

  1. The two positive corners and the two negative corners lie on opposing diagonals of a square. Their convex hulls meet at (1/2,1/2), so no straight line strictly separates the two classes. Equivalently, the four threshold inequalities are incompatible.

  2. Let

h_1=1\{x_1+x_2-1/2\ge 0\}

be OR, let

h_2=1\{-x_1-x_2+3/2\ge 0\}

be NAND, and let

y=1\{h_1+h_2-3/2\ge 0\}

be AND of the two hidden units. The weights and biases are the coefficients in those three net inputs. The four inputs give (h_1,h_2) equal to (0,1), (1,1), (1,1), (1,0), and the output net inputs -1/2, +1/2, +1/2, -1/2, which match the labels 0,1,1,0.

  1. Composing affine maps gives another affine map. Added depth alone would then create no nonlinear decision feature, so it would not solve this classification problem.