What the stationary equations suggest
Section of Mixtures and the EM algorithm, part of Statistical learning
What the stationary equations suggest
| Type | Section |
| Taught in | Mixtures and the EM algorithm |
| Part | Statistical learning |
At an interior maximum the derivative of the observed log-likelihood is zero. For mixtures, the solutions of interest are off the boundary of the parameter space. Write \pi_H=1-\sum_{j=1}^{H-1}\pi_j and differentiate in a free weight \pi_j. The derivative of the sum inside the ith logarithm contributes f_j(\mathbf{x}_i)-f_H(\mathbf{x}_i), so
\frac{\partial\ell}{\partial\pi_j} =\sum_{i=1}^{n} \frac{f_j(\mathbf{x}_i)-f_H(\mathbf{x}_i)}{\sum_{m=1}^{H}\pi_m f_m(\mathbf{x}_i)}.
Definition 1 (Posterior membership) The posterior probability that observation \mathbf{x} came from component h, evaluated at \Psi, is
\tau_h(\mathbf{x};\Psi) =\frac{\pi_h f_h(\mathbf{x};\theta_h)}{\sum_{m=1}^{H}\pi_m f_m(\mathbf{x};\theta_m)},
where \theta_h holds the mean and covariance of component h. These probabilities are the responsibilities. They are nonnegative and sum to one over h.
Because \tau_h(\mathbf{x}_i;\Psi)/\pi_h=f_h(\mathbf{x}_i)/\sum_m\pi_m f_m(\mathbf{x}_i), the score in \pi_j is \sum_i\bigl(\tau_j(\mathbf{x}_i)/\pi_j-\tau_H(\mathbf{x}_i)/\pi_H\bigr). Setting it to zero forces \sum_i\tau_h(\mathbf{x}_i)=c\,\pi_h for every h, with the same c. Summing over h gives c=n, and therefore a maximum likelihood estimate \widehat\Psi satisfies the fixed-point equation
\widehat\pi_h=\frac1n\sum_{i=1}^{n}\tau_h(\mathbf{x}_i;\widehat\Psi), \qquad h=1,\ldots,H.
The display is a fixed point. The weights \widehat\pi_h appear on the right-hand side through \tau_h, which depends on the whole of \widehat\Psi. The equation does identify the missing data: the weights it asks for are the conditional probabilities of the component indicators.
Used in
See also
Finite mixture model · Posterior membership · Complete-data likelihood · EM for a normal mixture · Monotonicity of EM · Likelihood of a normal mixture · E and M steps · E step