Softmax is multinomial logistic regression
Theorem taught in Networks in practice, part of Statistical learning
Softmax is multinomial logistic regression
| Type | Theorem |
| Taught in | Networks in practice |
| Part | Statistical learning |
With no hidden layer, z_k=\beta_k^\top x and softmax gives
\tau_k(x)=\frac{e^{\beta_k^\top x}}{\sum_l e^{\beta_l^\top x}},
which is the multinomial logistic model. The reference class is absorbed by setting \beta_1=0. Softmax is unchanged when the same vector is added to every \beta_k. Minimising the cross-entropy over the training set is maximising the multinomial likelihood. That likelihood, for one linear score per class, is logistic regression. For two classes, softmax reduces to the sigmoid:
\frac{e^{z_1}}{e^{z_1}+e^{z_2}}=g(z_1-z_2).
A network with a softmax output layer is therefore multinomial logistic regression on learned features h. The hidden layers, and the gradient through them, are the addition on this page.
Used in
- Taught in Networks in practice, where the derivation and examples are.
- All 7 questions on Networks in practice
See also
Multinomial logistic regression · Baseline-category logit model · Hidden layer · Output coding, softmax, and cross-entropy · Cross-entropy · Training set · Likelihood function · Sigmoid function