An M step

Topic: Mixtures and the EM algorithm · more questions on this topic
‹ An E step An unbounded likelihood ›

Using the two responsibility values from the previous exercise as the memberships of the two data points x=1 and x=3, carry out the M step for \pi, \mu, and \sigma^2.

Worked answer (click to show)

Hold the E-step responsibilities fixed throughout this M step. They act like fractional counts when updating weights, means, and variances.

The memberships are \tau_{\cdot 1}=(0.6156,0.0107) and \tau_{\cdot 2}=(0.3844,0.9893). Then N_1=0.6263, N_2=1.3737 and

\pi=(N_1,N_2)/2=(0.3131,0.6869).

The means are

\mu_1=\frac{0.6156\cdot 1+0.0107\cdot 3}{0.6263}=1.034, \qquad \mu_2=\frac{0.3844\cdot 1+0.9893\cdot 3}{1.3737}=2.440.

The variances about those means are

\begin{align*} \sigma_1^2 &=\frac{0.6156(1-1.034)^2+0.0107(3-1.034)^2}{0.6263} =0.067,\\ \sigma_2^2 &=\frac{0.3844(1-2.440)^2+0.9893(3-2.440)^2}{1.3737} =0.806. \end{align*}

With only two points the first component is essentially pinned on x=1, and its variance shrinks toward zero. That is a miniature of the singularity of an unconstrained normal mixture.