Logistic and probit models
Section of Logistic regression, part of Statistical learning
Logistic and probit models
| Type | Section |
| Taught in | Logistic regression |
| Part | Statistical learning |
Both links are smooth curves from 0 to 1, so the choice is not forced by the range of a probability. The scale of the linear predictor can be absorbed into \beta, which means the two curves should be compared after they are given the same slope at 0.
Differentiate \sigma(\eta)=(1+e^{-\eta})^{-1}. The chain rule gives
\sigma'(\eta) =(1+e^{-\eta})^{-2}\,e^{-\eta} =\frac{1}{1+e^{-\eta}}\cdot\frac{e^{-\eta}}{1+e^{-\eta}} =\sigma(\eta)\bigl(1-\sigma(\eta)\bigr),
because 1-\sigma(\eta)=e^{-\eta}/(1+e^{-\eta}). Hence \sigma'(0)=\sigma(0)(1-\sigma(0))=(1/2)(1/2)=1/4. The same identity gives the symmetry \sigma(-\eta)=1-\sigma(\eta), since
\sigma(-\eta)=\frac{1}{1+e^{\eta}}=\frac{e^{-\eta}}{e^{-\eta}+1}=1-\sigma(\eta).
The rescaled normal distribution function \Phi(c\eta) has slope c\,\varphi(0) at 0, where \varphi(0)=1/\sqrt{2\pi} is the standard normal density. Equating the slopes,
\frac{c}{\sqrt{2\pi}}=\frac14, \qquad c=\frac{\sqrt{2\pi}}{4}=\sqrt{\frac{\pi}{8}},
the last step because (\sqrt{2\pi}/4)^2=2\pi/16=\pi/8. With that scaling the curves nearly agree. The logistic function approaches 0 and 1 a little more slowly, so it is the heavier-tailed of the two links. The later algebra uses the logistic derivative \sigma(1-\sigma), which is why the logistic link is the one carried forward.
Used in
See also
Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Likelihood, score, and fitting · Score and Hessian of logistic regression · Iteratively reweighted least squares · Interpretation and the decision boundary · Linear discriminant analysis implies a logistic model