Kullback–Leibler divergence
Section of Model selection for stochastic models, part of Statistical learning
Kullback–Leibler divergence
| Type | Section |
| Taught in | Model selection for stochastic models |
| Part | Statistical learning |
| Questions using it | 1 |
The expected log-likelihood of a new observation differs from a Kullback–Leibler divergence by a constant that does not depend on the candidate. Let f be the density of a new observation and \widehat f the fitted density. Then
D(f\Vert\widehat f) =\int\ln\Bigl(\frac{f(x)}{\widehat f(x)}\Bigr)f(x)\,dx =\int f\ln f-\mathbb{E}\bigl[\ln\widehat f(X)\bigr].
The integral \int f\ln f does not depend on which candidate produced \widehat f. Maximising \mathbb{E}[\ln\widehat f(X)] is therefore the same comparison as minimising D(f\Vert\widehat f). The divergence is not symmetric. The convention puts the density of the expectation, here the true f, in the first argument.
Used in
- Taught in Model selection for stochastic models
- Also used in Mixtures and the EM algorithm · Hierarchical clustering and kernel density estimation
- Questions that use it (1) — KL divergence (answer)
- All 7 questions on Model selection for stochastic models
See also
Gibbs’ inequality · Bayesian information criterion · Akaike information criterion · Laplace approximation · Dropping terms that do not scale with n · Reading the BIC · Cross-validation of the likelihood · BIC sign conventions