Complete separation

Topic: Logistic regression · more questions on this topic
‹ Multinomial probabilities Reading the deviance ›

For x=(1,2,3,4) and y=(0,0,1,1), show that \ell(\beta_0,\beta_1) increases to 0 along \beta_1=s, \beta_0=-2.5s as s\to\infty, so no maximum exists. What does this say about the claim that the log-likelihood has a single maximum?

Worked answer (click to show)

With \beta_1=s and \beta_0=-2.5s,

\eta_i=s(x_i-2.5)=(-1.5s,-0.5s,0.5s,1.5s).

For y_i=0 the summand of \ell is -\log(1+e^{\eta_i})=-\log(1+e^{-|\eta_i|}). For y_i=1 it is

\eta_i-\log(1+e^{\eta_i})=-\log(1+e^{-\eta_i})=-\log(1+e^{-|\eta_i|}).

Each term is negative and tends to 0 as s\to\infty, so \ell\uparrow 0. Since \ell<0 for every finite \beta, each probability being strictly below 1, the supremum 0 is never attained. There is no maximum, and the estimates diverge. Concavity alone does not establish a finite maximum likelihood estimate. Complete or quasi-complete separation, and rank deficiency, need separate checks.