A numerical gradient check

Section of Backpropagation, part of Statistical learning

A numerical gradient check

Type Section
Taught in Backpropagation
Part Statistical learning

Before trusting a training run, compare the analytic gradient with central finite differences at a fixed set of weights that are not saturated. Both gradients refer to the same total squared-error objective, with no factor of \tfrac12. Agreement at one point checks the implemented derivative. It does not prove that training will converge, and it does not speak to generalisation.

The objects xor_x and xor_y are the four XOR inputs and the labels (0,1,1,0). The function backprop_r is the same batch update, returning a loss vector.

X <- cbind(1, xor_x); Y <- matrix(xor_y, ncol = 1)
w <- seq(-0.4, 0.4, length.out = 9)
loss_grad <- function(w) {
  W1 <- matrix(w[1:6], 3, 2); W2 <- matrix(w[7:9], 3, 1)
  H <- plogis(X %*% W1); Hb <- cbind(1, H)
  O <- plogis(Hb %*% W2)
  dO <- 2 * (O - Y) * O * (1 - O)
  dH <- (dO %*% t(W2[-1, , drop = FALSE])) * H * (1 - H)
  list(value = sum((O - Y)^2),
       gradient = c(crossprod(X, dH), crossprod(Hb, dO)))
}
h <- 1e-5
numeric_grad <- sapply(seq_along(w), function(j) {
  plus <- minus <- w; plus[j] <- plus[j] + h; minus[j] <- minus[j] - h
  (loss_grad(plus)$value - loss_grad(minus)$value) / (2 * h)
})
max_error <- max(abs(numeric_grad - loss_grad(w)$gradient))
max_error
#> [1] 1.885e-11
stopifnot(max_error < 1e-7)
trained <- backprop_r(seed = 1, epochs = 6000)
tail(trained$loss, 1)
#> [1] 0.0009966

The maximum absolute discrepancy is 1.885\times 10^{-11}. This checks the implemented derivative at this point. The learning curves use R’s random-number generator, so their trajectories need not match an earlier implementation in another language.

Key idea

Backpropagation evaluates the gradient by reusing local derivatives. The forward pass supplies activations. The backward pass carries sensitivities through outgoing weights. The optimiser then updates parameters. Numerical gradient checks verify that calculation. They do not certify the model, the optimiser’s global success, or its performance on new observations.

Used in

See also

Basic gradient-descent learning · Squared error is not the logistic likelihood · Backpropagation, batch form · Local minima, plateaus, and symmetry · Training error of a network · Gradient descent · A single layer of sigmoid units · Backpropagation through one hidden layer

Glossary and index