From a linear model to class probabilities
Section of Logistic regression, part of Statistical learning
From a linear model to class probabilities
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
Take two classes, coded 0 and 1. The probability to be predicted is the conditional probability of class 1, which is also the conditional mean of a binary response:
\tau_1(x)=\mathbb{P}(Y=1\mid X=x),\qquad \tau_0(x)=1-\tau_1(x).
A line fitted to y_i\in\{0,1\} can predict values below 0 or above 1, and those values are not probabilities. A generalised linear model keeps a linear predictor and passes it through an inverse link to produce a mean. For a binary response the mean is the probability, so the distribution around that mean is Bernoulli: each y_i is a single trial with success probability p_i.
The link is the function that is assumed linear. Augment the predictor vector with a leading 1, so that x has length p+1 and \beta_0 is an intercept. Then
h\bigl(\tau_1(x)\bigr)=\beta^\top x=\beta_0+\beta_1 x_1+\cdots+\beta_p x_p.
The inverse link has to return values in (0,1), and it has to be non-decreasing, so that a larger linear predictor means a larger probability of class 1. The standard normal distribution function \Phi meets both requirements and gives probit regression: the inverse link is \Phi, the distribution function of \operatorname{N}(0,1). The logistic function is the usual choice, because its inverse is the log odds and the derivatives needed for fitting stay simple. Write \sigma(\eta)=1/(1+e^{-\eta}). The model probability is
\tau_1(x)=\sigma(\beta^\top x)=\frac{1}{1+e^{-\beta^\top x}}.
Solving \tau_1=1/(1+e^{-\eta}) for the linear predictor \eta gives the link. From e^{-\eta}=(1-\tau_1)/\tau_1,
\eta=\log\frac{\tau_1}{1-\tau_1}=\log\frac{\tau_1}{\tau_0}.
So the link is the log odds of class 1,
h(\tau_1)=\log\frac{\tau_1}{1-\tau_1}=\log\frac{\tau_1}{\tau_0},
also called the logit, and the model statement is h(\tau_1)=\beta^\top x. The probability is nonlinear in x. The log odds are linear in x. That split is why a one-unit change in a predictor multiplies the odds by a constant and does not add a constant to the probability.
Logistic and probit
Main article: Logistic and probit models
Both links are smooth curves from 0 to 1, so the choice is not forced by the range of a probability. The scale of the linear predictor can be absorbed into \beta, which means the two curves should be compared after they are given the same slope at 0.
Differentiate \sigma(\eta)=(1+e^{-\eta})^{-1}. The chain rule gives
\sigma'(\eta) =(1+e^{-\eta})^{-2}\,e^{-\eta} =\frac{1}{1+e^{-\eta}}\cdot\frac{e^{-\eta}}{1+e^{-\eta}} =\sigma(\eta)\bigl(1-\sigma(\eta)\bigr),
because 1-\sigma(\eta)=e^{-\eta}/(1+e^{-\eta}). Hence \sigma'(0)=\sigma(0)(1-\sigma(0))=(1/2)(1/2)=1/4. The same identity gives the symmetry \sigma(-\eta)=1-\sigma(\eta), since
\sigma(-\eta)=\frac{1}{1+e^{\eta}}=\frac{e^{-\eta}}{e^{-\eta}+1}=1-\sigma(\eta).
The rescaled normal distribution function \Phi(c\eta) has slope c\,\varphi(0) at 0, where \varphi(0)=1/\sqrt{2\pi} is the standard normal density. Equating the slopes,
\frac{c}{\sqrt{2\pi}}=\frac14, \qquad c=\frac{\sqrt{2\pi}}{4}=\sqrt{\frac{\pi}{8}},
the last step because (\sqrt{2\pi}/4)^2=2\pi/16=\pi/8. With that scaling the curves nearly agree. The logistic function approaches 0 and 1 a little more slowly, so it is the heavier-tailed of the two links. The later algebra uses the logistic derivative \sigma(1-\sigma), which is why the logistic link is the one carried forward.
Used in
See also
Binary logistic regression · A finite stationary point is a global maximiser · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression · Iteratively reweighted least squares · Interpretation and the decision boundary · Linear discriminant analysis implies a logistic model