Multinomial logistic regression
Section of Logistic regression, part of Statistical learning
Multinomial logistic regression
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
Two classes were enough to fix the link. With G>2 nominal classes, a single log odds no longer describes the conditional distribution. The baseline-category logit on Categorical responses is this model: one class is the reference, and each remaining class has its own linear predictor. Take the reference to be class 1. Each remaining class gets its own coefficient vector \beta_g, g\in\{2,\ldots,G\}, and the model is linear in the log odds against the reference, the same comparison linear discriminant analysis makes one pair at a time:
\log\frac{\tau_g(x)}{\tau_1(x)}=\beta_g^\top x.
Exponentiate and use \sum_{g=1}^G\tau_g(x)=1, which the binary model obtained from \tau_0=1-\tau_1. Exponentiating gives \tau_g(x)=\tau_1(x)\,e^{\beta_g^\top x} for g\ge 2. Substituting into the sum-to-one constraint,
\begin{align*} \tau_1(x) &=1-\sum_{g=2}^G\tau_g(x) =1-\tau_1(x)\sum_{g=2}^G e^{\beta_g^\top x},\\ \tau_1(x)\Biggl(1+\sum_{g=2}^G e^{\beta_g^\top x}\Biggr) &=1, \end{align*}
so
\tau_1(x)=\frac{1}{1+\sum_{g=2}^G e^{\beta_g^\top x}}, \qquad \tau_g(x)=\frac{e^{\beta_g^\top x}}{1+\sum_{g'=2}^G e^{\beta_{g'}^\top x}}, \quad g\ge 2.
The algebra uses the linear log-odds specification together with the requirement that the fitted probabilities sum to 1. The reference class is a labelling choice: a different reference reparameterises the same family of conditional distributions. The coefficient vectors are estimated jointly by maximum likelihood, or by an extension of the binary iteratively reweighted least-squares iteration. The map from linear predictors to probabilities is the softmax. An ordered response can use a multinomial model, at some loss of efficiency, because that model ignores the order. Models that use the order make further assumptions and are not developed here.
Used in
See also
Softmax is multinomial logistic regression · Baseline-category logit model · Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression