Likelihood of a normal mixture
Section of Mixtures and the EM algorithm, part of Statistical learning
Likelihood of a normal mixture
| Type | Section |
| Taught in | Mixtures and the EM algorithm |
| Part | Statistical learning |
Fix H. The observations are continuous and p-dimensional, and each component is multivariate normal. Let \{\mathbf{x}_1,\ldots,\mathbf{x}_n\} be an iid sample from the mixture, stored as an n\times p matrix X_{\mathrm{all}}. Collect the component parameters and the weights in one vector \Psi. The constraint \sum_h\pi_h=1 leaves only \pi_1,\ldots,\pi_{H-1} free. For normal components,
\Psi=(\boldsymbol\mu_1^\top,\ldots,\boldsymbol\mu_H^\top,\boldsymbol\xi^\top,\pi_1,\ldots,\pi_{H-1})^\top,
where \boldsymbol\xi holds the distinct entries of \Sigma_1,\ldots,\Sigma_H. A symmetric p\times p covariance has p(p+1)/2 distinct entries, so the number of free parameters is
d=Hp+H\frac{p(p+1)}{2}+(H-1).
For p=2 and H=3 the count is 6+9+2=17: three means of length 2, three covariance matrices with three distinct entries each, and two free weights.
The multivariate normal density of component h is
f_h(\mathbf{x}) =(2\pi)^{-p/2}\,|\Sigma_h|^{-1/2} \exp\Bigl(-\tfrac12(\mathbf{x}-\boldsymbol\mu_h)^\top\Sigma_h^{-1}(\mathbf{x}-\boldsymbol\mu_h)\Bigr).
The observed likelihood multiplies the mixture densities. The component label has been summed out inside each factor, so every factor is a sum,
L(\Psi;X_{\mathrm{all}})=\prod_{i=1}^{n}\sum_{h=1}^{H}\pi_h f_h(\mathbf{x}_i),
and the log-likelihood is the sum of the logarithms of those sums,
\begin{align*} \ell(\Psi;X_{\mathrm{all}}) &=\sum_{i=1}^{n}\ln\Biggl(\sum_{h=1}^{H}\pi_h f_h(\mathbf{x}_i)\Biggr)\\ &=\sum_{i=1}^{n}\ln\Biggl( \sum_{h=1}^{H} \frac{\pi_h}{(2\pi)^{p/2}\,|\Sigma_h|^{1/2}} \exp\Bigl(-\tfrac12(\mathbf{x}_i-\boldsymbol\mu_h)^\top\Sigma_h^{-1}(\mathbf{x}_i-\boldsymbol\mu_h)\Bigr) \Biggr). \end{align*}
The logarithm of a sum does not become a sum of logarithms, so the observed log-likelihood does not separate across components. Newton–Raphson on \ell is a different algorithm: every derivative has to pass through the sum. EM replaces \ell by an expected complete-data log-likelihood, for which the normal formulas apply with weights.
Used in
See also
Finite mixture model · Posterior membership · Complete-data likelihood · EM for a normal mixture · Monotonicity of EM · What the stationary equations suggest · E and M steps · E step