E step
Section of Mixtures and the EM algorithm, part of Statistical learning
E step
| Type | Section |
| Taught in | Mixtures and the EM algorithm |
| Part | Statistical learning |
| Questions using it | 1 |
The complete log-likelihood is linear in the indicators, so the conditional expectation passes inside the sum. The expectation of a binary indicator is the probability that it equals one. Bayes’ rule for a discrete label and a continuous measurement identifies that probability with the posterior membership, evaluated at the current parameter:
\begin{align*} Q(\Psi;\Psi^{(k)}) &=\mathbb{E}\Biggl[ \sum_{i=1}^{n}\sum_{h=1}^{H}Z_{ih}\bigl[\ln\pi_h+\ln f_h(\mathbf{x}_i;\theta_h)\bigr] \Bigm|X_{\mathrm{all}} \Biggr]\\ &=\sum_{i=1}^{n}\sum_{h=1}^{H} \mathbb{E}(Z_{ih}\mid\mathbf{x}_i)\, \bigl[\ln\pi_h+\ln f_h(\mathbf{x}_i;\theta_h)\bigr]\\ &=\sum_{i=1}^{n}\sum_{h=1}^{H} \tau_h(\mathbf{x}_i;\Psi^{(k)})\, \bigl[\ln\pi_h+\ln f_h(\mathbf{x}_i;\theta_h)\bigr]. \end{align*}
The current parameter \Psi^{(k)} is held fixed inside \tau_h. The free parameter \Psi remains in \ln\pi_h+\ln f_h, which is what the M step maximises. Explicitly,
\tau_h\bigl(\mathbf{x}_i;\Psi^{(k)}\bigr) =\frac{\pi_h^{(k)}f_h\bigl(\mathbf{x}_i;\theta_h^{(k)}\bigr)} {\sum_{m=1}^{H}\pi_m^{(k)}f_m\bigl(\mathbf{x}_i;\theta_m^{(k)}\bigr)}.
Used in
- Taught in Mixtures and the EM algorithm
- Questions that use it (1) — An M step (answer)
- All 7 questions on Mixtures and the EM algorithm
See also
EM algorithm · EM for a normal mixture · Missing data · Finite mixture model · Posterior membership · Complete-data likelihood · Monotonicity of EM · Likelihood of a normal mixture