Gibbs’ inequality
Topic of Mixtures and the EM algorithm, part of Statistical learning
Gibbs’ inequality
| Type | Topic |
| Taught in | Mixtures and the EM algorithm |
| Part | Statistical learning |
For probability vectors p and q, \sum_ip_i\log p_i\ge\sum_ip_i\log q_i, with equality only when p=q: the Kullback–Leibler divergence is never negative. It is the step in the proof that an EM iteration never decreases the likelihood.
In Mixtures and the EM algorithm
Gibbs’ inequality is nonnegativity of Kullback–Leibler divergence, so G(\Psi;\Psi^{(k)})\le G(\Psi^{(k)};\Psi^{(k)}) for every \Psi. Split the change in the observed log-likelihood:
Used in
See also
Kullback–Leibler divergence · EM algorithm · Likelihood function · Finite mixture model · Posterior membership · Complete-data likelihood · EM for a normal mixture · Monotonicity of EM