Linear discriminant analysis implies a logistic model
Section of Logistic regression, part of Statistical learning
Linear discriminant analysis implies a logistic model
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
Two Gaussian classes with a common covariance imply linear posterior log odds, so the population linear discriminant model implies a logistic model. Their estimates differ. Linear discriminant analysis uses a joint model for measurements and labels. Logistic regression uses the conditional likelihood of the labels and does not model the marginal of x. Correct extra assumptions can improve statistical efficiency. Incorrect ones can hurt. Logistic regression avoids Gaussian predictor assumptions, and it can still be misspecified through its linear log-odds form.
The algebra is Bayes’ theorem plus cancellation of the quadratic terms. Let X\mid Y=1\sim\operatorname{N}(\mu_1,\Sigma) and X\mid Y=0\sim\operatorname{N}(\mu_0,\Sigma), with priors \pi_1 and \pi_0. Then
\log\frac{\tau_1(x)}{\tau_0(x)} =\log\frac{\pi_1}{\pi_0}+\log\frac{f_1(x)}{f_0(x)}.
The normal log-density ratio, with the common quadratic form x^\top\Sigma^{-1}x cancelling, is
\begin{align*} \log\frac{f_1(x)}{f_0(x)} &=-\tfrac12(x-\mu_1)^\top\Sigma^{-1}(x-\mu_1) +\tfrac12(x-\mu_0)^\top\Sigma^{-1}(x-\mu_0)\\ &=x^\top\Sigma^{-1}(\mu_1-\mu_0) -\tfrac12(\mu_1+\mu_0)^\top\Sigma^{-1}(\mu_1-\mu_0). \end{align*}
The second line uses symmetry of \Sigma^{-1}: \mu_1^\top\Sigma^{-1}\mu_1-\mu_0^\top\Sigma^{-1}\mu_0=(\mu_1+\mu_0)^\top\Sigma^{-1}(\mu_1-\mu_0). Therefore
\log\frac{\tau_1(x)}{\tau_0(x)}=\alpha_0+\beta^\top x,
with
\beta=\Sigma^{-1}(\mu_1-\mu_0), \qquad \alpha_0=\log\frac{\pi_1}{\pi_0}-\tfrac12(\mu_1+\mu_0)^\top\Sigma^{-1}(\mu_1-\mu_0).
That is a logistic model. Linear discriminant analysis estimates these coefficients from the class means, the pooled covariance, and the priors. Logistic regression estimates them from the conditional likelihood.
Under pure prior shift the slopes can stay and the intercept can be adjusted. If f(x\mid Y) stays fixed and the source and target class-one prevalences are \pi_s and \pi_t, the log density ratio does not change, and the prior contribution changes by \mathrm{logit}\,\pi_t-\mathrm{logit}\,\pi_s:
\mathrm{logit}\,p_t(x)=\mathrm{logit}\,p_s(x)+\mathrm{logit}\,\pi_t-\mathrm{logit}\,\pi_s.
A correctly specified logistic model then needs an intercept adjustment and no refit of the slopes. This conclusion depends on stable class-conditional distributions and on valid source probabilities. An arbitrary shift of the joint distribution does not satisfy it.
Used in
See also
Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression · Iteratively reweighted least squares · Interpretation and the decision boundary