Interpretation and the decision boundary

Section of Logistic regression, part of Statistical learning

Interpretation and the decision boundary

Type Section
Taught in Logistic regression
Part Statistical learning
Questions using it 7

The fitted model is linear on the log-odds scale, which is the scale the link was chosen to linearise. From \tau_1=1/(1+e^{-\beta^\top x}) and \tau_0=1-\tau_1,

\frac{\tau_1(x)}{\tau_0(x)} =\frac{1/(1+e^{-\beta^\top x})}{e^{-\beta^\top x}/(1+e^{-\beta^\top x})} =e^{\beta^\top x},

so

\log\frac{\tau_1(x)}{\tau_0(x)}=\beta^\top x.

Hold every predictor other than x_j fixed and increase x_j by one unit. The linear predictor increases by \beta_j, so the log odds of class 1 increase by \beta_j and the odds are multiplied by e^{\beta_j}. The probability does not increase by a constant, because the logistic curve flattens near 0 and near 1. If the baseline probability is p and the odds multiplier is a, the new probability is ap/(1-p+ap).

The same one-unit reading can be derived for the linear discriminant log-odds score, since under a shared covariance that score is also linear in x. A coefficient remains a conditional association inside the fitted model. It does not, by itself, measure the effect of intervening on x_j.

The equal-cost Bayes rule predicts class 1 when \tau_1(x)\ge 1/2. Under the logistic model that event is the event that the odds are at least 1, which is the event that the log odds are at least 0:

\beta^\top x=0.

The boundary is a hyperplane, one dimension below the predictor space, and it has the same shape as the linear discriminant boundary. The agreement in shape is not an agreement in estimates. The classification rule at the probability threshold 1/2 is the sign of the linear predictor,

\eta(\beta)= \begin{cases} 1,&\beta^\top x\ge 0,\\ 0,&\beta^\top x<0. \end{cases}

Changing the decision threshold changes the trade-off between false positives and false negatives while leaving the fitted scores fixed. An ROC curve plots these two class-conditional operating characteristics over thresholds. Under pure prior shift, with the class-conditional score distributions unchanged, the ROC curve remains the same, although predictive values and the cost-optimal threshold can change.

Products and powers such as x_1, x_1^2, and x_1 x_2 can be added to the linear compound. The log odds stay linear in the coefficients, and the boundary in the original predictors becomes curved. The one-unit odds interpretation is then an interpretation in the expanded predictors.

Linear discriminant analysis implies a logistic model

Main article: Linear discriminant analysis implies a logistic model

Two Gaussian classes with a common covariance imply linear posterior log odds, so the population linear discriminant model implies a logistic model. Their estimates differ. Linear discriminant analysis uses a joint model for measurements and labels. Logistic regression uses the conditional likelihood of the labels and does not model the marginal of x. Correct extra assumptions can improve statistical efficiency. Incorrect ones can hurt. Logistic regression avoids Gaussian predictor assumptions, and it can still be misspecified through its linear log-odds form.

The algebra is Bayes’ theorem plus cancellation of the quadratic terms. Let X\mid Y=1\sim\operatorname{N}(\mu_1,\Sigma) and X\mid Y=0\sim\operatorname{N}(\mu_0,\Sigma), with priors \pi_1 and \pi_0. Then

\log\frac{\tau_1(x)}{\tau_0(x)} =\log\frac{\pi_1}{\pi_0}+\log\frac{f_1(x)}{f_0(x)}.

The normal log-density ratio, with the common quadratic form x^\top\Sigma^{-1}x cancelling, is

\begin{align*} \log\frac{f_1(x)}{f_0(x)} &=-\tfrac12(x-\mu_1)^\top\Sigma^{-1}(x-\mu_1) +\tfrac12(x-\mu_0)^\top\Sigma^{-1}(x-\mu_0)\\ &=x^\top\Sigma^{-1}(\mu_1-\mu_0) -\tfrac12(\mu_1+\mu_0)^\top\Sigma^{-1}(\mu_1-\mu_0). \end{align*}

The second line uses symmetry of \Sigma^{-1}: \mu_1^\top\Sigma^{-1}\mu_1-\mu_0^\top\Sigma^{-1}\mu_0=(\mu_1+\mu_0)^\top\Sigma^{-1}(\mu_1-\mu_0). Therefore

\log\frac{\tau_1(x)}{\tau_0(x)}=\alpha_0+\beta^\top x,

with

\beta=\Sigma^{-1}(\mu_1-\mu_0), \qquad \alpha_0=\log\frac{\pi_1}{\pi_0}-\tfrac12(\mu_1+\mu_0)^\top\Sigma^{-1}(\mu_1-\mu_0).

That is a logistic model. Linear discriminant analysis estimates these coefficients from the class means, the pooled covariance, and the priors. Logistic regression estimates them from the conditional likelihood.

Under pure prior shift the slopes can stay and the intercept can be adjusted. If f(x\mid Y) stays fixed and the source and target class-one prevalences are \pi_s and \pi_t, the log density ratio does not change, and the prior contribution changes by \mathrm{logit}\,\pi_t-\mathrm{logit}\,\pi_s:

\mathrm{logit}\,p_t(x)=\mathrm{logit}\,p_s(x)+\mathrm{logit}\,\pi_t-\mathrm{logit}\,\pi_s.

A correctly specified logistic model then needs an intercept adjustment and no refit of the slopes. This conclusion depends on stable class-conditional distributions and on valid source probabilities. An arbitrary shift of the joint distribution does not satisfy it.

Used in

See also

Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression · Iteratively reweighted least squares · Linear discriminant analysis implies a logistic model

Glossary and index