Cross-entropy
Topic of Classification trees, part of Statistical learning · Also in Networks in practice
Cross-entropy
| Type | Topic |
| Taught in | Classification trees · Networks in practice |
| Part | Statistical learning |
| Questions using it | 5 |
For class proportions \hat p_g, the impurity -\sum_g\hat p_g\log\hat p_g (the entropy of the node). As a loss for predicted probabilities s_k and a one-hot target it is -\sum_ky_k\log s_k, which with a softmax output is the multinomial negative log-likelihood.
In Classification trees
From Classification trees
\begin{align*} I_C(m)&=-\sum_{g=1}^G \hat p_{mg}\log\hat p_{mg}, &&\text{cross-entropy},\\ I_G(m)&=\sum_{g=1}^G \hat p_{mg}(1-\hat p_{mg}) =1-\sum_{g=1}^G \hat p_{mg}^2, &&\text{Gini index},\\ I_M(m)&=1-\hat p_{mg^*}, &&\text{misclassification rate}, \end{align*}
In Networks in practice
From Networks in practice
Separate sigmoid outputs, one per class, need not sum to 1. They are scores. A probability model for a single class label needs a distribution. For v classes, the simplest coding still uses v output units: for each pattern, output k is 1 if the pattern is in class k and 0 otherwise. This treats each class equally. On test data, and often on training data, every output lies in (0,1). The usual prediction is the class of the largest output.
When the outputs are meant to be class probabilities, they have to add to one. The softmax of Bridle (1990) exponentiates and then normalises:
s_k=\frac{e^{z_k}}{\sum_{l=1}^{m} e^{z_l}},\qquad k=1,\ldots,m.
Here z_k is the output of the last linear stage, often called a logit. Cross-entropy is the negative log-likelihood for the observed categorical label. Squared error is also a possible probability-scoring loss. Cross-entropy has a particularly convenient derivative with softmax: a confidently wrong output produces a substantial gradient with respect to its logits. This is a statement about the output layer, not a guarantee against vanishing gradients throughout the network.
Let the target be a one-hot vector y, with y_k=1 for the true class c, and let the loss be
E=-\sum_{k=1}^{m} y_k\ln s_k=-\ln s_c.
Differentiate -\ln s_c through s_k=e^{z_k}/\sum_l e^{z_l}, using \partial s_k/\partial z_j=s_k(\delta_{kj}-s_j). The factor 1/s_k from the logarithm cancels the s_k in the softmax Jacobian:
\begin{aligned} \frac{\partial E}{\partial z_j} &=-\sum_{k=1}^{m} y_k\frac{1}{s_k}\,s_k(\delta_{kj}-s_j) =-\sum_{k=1}^{m} y_k(\delta_{kj}-s_j)\\ &=-y_j+s_j\sum_{k=1}^{m} y_k =s_j-y_j. \end{aligned}
The result is \partial E/\partial z_j=s_j-y_j. The calculation only needs \sum_k y_k=1, so it also works for probability-valued target vectors. There is no additional sigmoid-slope factor at this layer. A confidently correct prediction can, and should, have a small gradient. A confidently wrong prediction retains an error signal. Earlier hidden layers still contribute their own derivatives during backpropagation. The saturation exercise on the backpropagation page is the comparison with squared error.
Theorem 1 (Softmax is multinomial logistic regression) With no hidden layer, z_k=\beta_k^\top x and softmax gives
\tau_k(x)=\frac{e^{\beta_k^\top x}}{\sum_l e^{\beta_l^\top x}},
which is the multinomial logistic model. The reference class is absorbed by setting \beta_1=0. Softmax is unchanged when the same vector is added to every \beta_k. Minimising the cross-entropy over the training set is maximising the multinomial likelihood. That likelihood, for one linear score per class, is logistic regression. For two classes, softmax reduces to the sigmoid:
\frac{e^{z_1}}{e^{z_1}+e^{z_2}}=g(z_1-z_2).
A network with a softmax output layer is therefore multinomial logistic regression on learned features h. The hidden layers, and the gradient through them, are the addition on this page.
Example 1 (Softmax and its gradient) Question. Three output activations are z=(2,1,0.1) and the true class is 1. Compute the softmax probabilities, the cross-entropy, and the error signals s-y.
Solution. e^{z}=(7.389,2.718,1.105) with sum 11.213, so s=(0.659,0.242,0.099). These sum to 1. The cross-entropy is -\ln 0.659=0.417 and the error signals are s-y=(-0.341,0.242,0.099). The correct class’s activation is pushed up and the others down, each in proportion to its probability. Adding a constant to all of z leaves s unchanged, which is why software subtracts \max_k z_k before exponentiating, to prevent overflow.
Used in
- Taught in Classification trees · Networks in practice
- Also used in Networks in practice
- Questions that use it (5) — Three impurities · Which split is better? · Saturation · Softmax gradient · Sum of squares versus accuracy (answer)
- All 7 questions on Classification trees
See also
Node impurity · One-hot encoding · Output coding, softmax, and cross-entropy · Likelihood function · Softmax is multinomial logistic regression · Building a classification tree · Choosing a split · Invariance to a change of units