An unbounded likelihood

Topic: Mixtures and the EM algorithm · more questions on this topic
‹ An M step Counting parameters ›

Take H=2, p=1, with \mu_1=x_1 and \sigma_1^2\to 0, and the other component fixed at some \operatorname{N}(\mu_2,\sigma_2^2) with \pi_1=\pi_2=\tfrac12. Show that \ell\to\infty.

Worked answer (click to show)

Make one component concentrate on a single observation while the other retains positive density at every remaining observation. A lower bound is enough to show divergence. The log-likelihood contains the term for observation 1,

\ln\Biggl(\frac12\cdot\frac{1}{\sqrt{2\pi\sigma_1^2}}+\frac12 f_2(x_1)\Biggr) \ge -\ln 2-\frac12\ln(2\pi\sigma_1^2),

using \varphi(x_1;x_1,\sigma_1^2)=(2\pi\sigma_1^2)^{-1/2}. For i\ge 2,

\ln\Bigl(\tfrac12 f_1(x_i)+\tfrac12 f_2(x_i)\Bigr) \ge\ln\bigl(\tfrac12 f_2(x_i)\bigr),

a finite constant that does not depend on \sigma_1. So

\ell\ge C-\tfrac12\ln(2\pi\sigma_1^2)\to\infty \qquad\text{as}\qquad\sigma_1^2\to 0.