Output coding, softmax, and cross-entropy

Section of Networks in practice, part of Statistical learning

Type	Section
Taught in	Networks in practice
Part	Statistical learning
Questions using it	2

Separate sigmoid outputs, one per class, need not sum to 1. They are scores. A probability model for a single class label needs a distribution. For v classes, the simplest coding still uses v output units: for each pattern, output k is 1 if the pattern is in class k and 0 otherwise. This treats each class equally. On test data, and often on training data, every output lies in (0,1). The usual prediction is the class of the largest output.

When the outputs are meant to be class probabilities, they have to add to one. The softmax of Bridle (1990) exponentiates and then normalises:

s_k=\frac{e^{z_k}}{\sum_{l=1}^{m} e^{z_l}},\qquad k=1,\ldots,m.

Here z_k is the output of the last linear stage, often called a logit. Cross-entropy is the negative log-likelihood for the observed categorical label. Squared error is also a possible probability-scoring loss. Cross-entropy has a particularly convenient derivative with softmax: a confidently wrong output produces a substantial gradient with respect to its logits. This is a statement about the output layer, not a guarantee against vanishing gradients throughout the network.

Let the target be a one-hot vector y, with y_k=1 for the true class c, and let the loss be

E=-\sum_{k=1}^{m} y_k\ln s_k=-\ln s_c.

Differentiate -\ln s_c through s_k=e^{z_k}/\sum_l e^{z_l}, using \partial s_k/\partial z_j=s_k(\delta_{kj}-s_j). The factor 1/s_k from the logarithm cancels the s_k in the softmax Jacobian:

\begin{aligned} \frac{\partial E}{\partial z_j} &=-\sum_{k=1}^{m} y_k\frac{1}{s_k}\,s_k(\delta_{kj}-s_j) =-\sum_{k=1}^{m} y_k(\delta_{kj}-s_j)\\ &=-y_j+s_j\sum_{k=1}^{m} y_k =s_j-y_j. \end{aligned}

The result is \partial E/\partial z_j=s_j-y_j. The calculation only needs \sum_k y_k=1, so it also works for probability-valued target vectors. There is no additional sigmoid-slope factor at this layer. A confidently correct prediction can, and should, have a small gradient. A confidently wrong prediction retains an error signal. Earlier hidden layers still contribute their own derivatives during backpropagation. The saturation exercise on the backpropagation page is the comparison with squared error.

Theorem 1 (Softmax is multinomial logistic regression) With no hidden layer, z_k=\beta_k^\top x and softmax gives

\tau_k(x)=\frac{e^{\beta_k^\top x}}{\sum_l e^{\beta_l^\top x}},

which is the multinomial logistic model. The reference class is absorbed by setting \beta_1=0. Softmax is unchanged when the same vector is added to every \beta_k. Minimising the cross-entropy over the training set is maximising the multinomial likelihood. That likelihood, for one linear score per class, is logistic regression. For two classes, softmax reduces to the sigmoid:

\frac{e^{z_1}}{e^{z_1}+e^{z_2}}=g(z_1-z_2).

A network with a softmax output layer is therefore multinomial logistic regression on learned features h. The hidden layers, and the gradient through them, are the addition on this page.

Article: Softmax is multinomial logistic regression →

Example 1 (Softmax and its gradient) Question. Three output activations are z=(2,1,0.1) and the true class is 1. Compute the softmax probabilities, the cross-entropy, and the error signals s-y.

Solution. e^{z}=(7.389,2.718,1.105) with sum 11.213, so s=(0.659,0.242,0.099). These sum to 1. The cross-entropy is -\ln 0.659=0.417 and the error signals are s-y=(-0.341,0.242,0.099). The correct class’s activation is pushed up and the others down, each in proportion to its probability. Adding a constant to all of z leaves s unchanged, which is why software subtracts \max_k z_k before exponentiating, to prevent overflow.

Used in

Taught in Networks in practice
Also used in Logistic regression
Questions that use it (2) — Softmax gradient (answer) · Complexity and early stopping
All 7 questions on Networks in practice

Used in

See also