Backpropagation on XOR

Section of Backpropagation, part of Statistical learning

Backpropagation on XOR

Type Section
Taught in Backpropagation
Part Statistical learning

Training a 2–2–1 sigmoid network on XOR is where the learning rate and the random start stop being abstract. Some runs can remain on a plateau. Large steps can cause saturation or unstable updates. A finite trajectory is evidence about one implementation and one starting point. It is not a general convergence guarantee.

Eight random starts and five learning rates, recomputed for a seeded implementation, produce total squared-error curves on a logarithmic scale. The paths differ. Rates 0.1, 0.5, 2, 8 and 32 do not trace the same curve. The hidden units learn a nonlinear representation in which the output can separate the classes. Hidden units may swap, change sign with compensating weights, or find different valid representations. Their particular weights are not unique scientific parameters. The existence of several representations is the same symmetry as above.

The batch update below is logistic units, squared error with no factor \tfrac12, and one gradient step per epoch. The laboratory underneath is the numerical check.

# Batch backpropagation for one hidden layer, logistic units,
# squared error with no factor 1/2.
g <- function(z) 1 / (1 + exp(-z))
backprop <- function(X, Y, l = 2, eps = 0.5, epochs = 5000, seed = 1) {
  set.seed(seed)
  X1 <- cbind(1, X)                                    # bias input x0 = 1
  W1 <- matrix(runif((ncol(X) + 1) * l, -0.5, 0.5), ncol(X) + 1, l)
  W2 <- matrix(runif((l + 1) * ncol(Y), -0.5, 0.5), l + 1, ncol(Y))
  sse <- numeric(epochs)
  for (e in 1:epochs) {
    H <- g(X1 %*% W1); H1 <- cbind(1, H)               # feed forward
    O <- g(H1 %*% W2)
    sse[e] <- sum((Y - O)^2)                           # error
    dO <- -2 * (Y - O) * O * (1 - O)                   # output deltas
    dH <- (dO %*% t(W2[-1, , drop = FALSE])) * H * (1 - H)  # hidden deltas
    W2 <- W2 - eps * crossprod(H1, dO)                 # weight updates
    W1 <- W1 - eps * crossprod(X1, dH)
  }
  list(W1 = W1, W2 = W2, sse = sse)
}
X <- rbind(c(0, 0), c(0, 1), c(1, 0), c(1, 1))
Y <- matrix(c(0, 1, 1, 0), ncol = 1)
fit <- backprop(X, Y)
tail(fit$sse, 1)                                       # near 0 for most seeds

Used in

See also

Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer

Glossary and index