Complete separation
Topic of Logistic regression, part of Statistical learning
Complete separation
| Type | Topic |
| Taught in | Logistic regression |
| Part | Statistical learning |
| Questions using it | 1 |
When a hyperplane separates the two classes perfectly, the logistic log-likelihood has no maximum: it keeps rising toward 0 as the coefficients grow without bound, and fitted probabilities approach 0 and 1. Software reports huge coefficients and standard errors.
In Logistic regression
From Logistic regression
Complete separation is a concrete failure. Suppose there is a direction v such that v^\top x_i>0 whenever y_i=1 and v^\top x_i<0 whenever y_i=0. Move the coefficient vector farther along that direction: the probabilities approach the observed zeroes and ones, and the log-likelihood approaches its upper bound 0 without attaining it at a finite vector. The bound holds because each Bernoulli factor is at most 1, so L(\beta)\le 1 and \ell(\beta)\le 0, with equality only in the limit where every fitted probability is exactly the observed label. The likelihood stays bounded. The coefficients diverge. Quasi-complete separation can also prevent a finite maximum likelihood estimate. Check convergence and separation before interpreting standard errors.
Used in
- Taught in Logistic regression
- Questions that use it (1) — Complete separation (answer)
- All 7 questions on Logistic regression
See also
Likelihood function · Standard error · Binary logistic regression · A finite stationary point is a global maximiser · From a linear model to class probabilities · Logistic and probit models · Likelihood, score, and fitting · Score and Hessian of logistic regression