Cross-validation of the likelihood
Section of Model selection for stochastic models, part of Statistical learning
Cross-validation of the likelihood
| Type | Section |
| Taught in | Model selection for stochastic models |
| Part | Statistical learning |
BIC approximates a marginal probability of the sample already observed. Cross-validation estimates a different functional: the expected log-likelihood of a new draw, with the parameter fixed at a fit from the current sample. Smyth (2000) fits on training observations and scores the log-likelihood on test observations, and selects the model with the highest test score. The target is
k^*=\arg\max_k \mathbb{E}\bigl[\log\mathbb{P}(X\mid\Psi=\widehat\Psi_k,M_k)\bigr].
The expectation is over a new observation X. For each candidate, \widehat\Psi_k is the maximum likelihood fit on the data already in hand, so the parameter is not recomputed inside the expectation. A test set that was excluded from the fit approximates the expectation, as an average or as a sum. Multiplying the sum by a positive constant does not change the maximiser, and the two forms select the same model:
\begin{align*} k^* &\approx\arg\max_k\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}} \log\mathbb{P}(x_{\mathrm{test},i}\mid\widehat\Psi_k,M_k)\\ &\approx\arg\max_k\sum_{i=1}^{n_{\mathrm{test}}} \log\mathbb{P}(x_{\mathrm{test},i}\mid\widehat\Psi_k,M_k). \end{align*}
In v-fold cross-validation, refit every candidate on each training fold and sum its held-out log densities. Any preprocessing belongs inside those fits. The winning specification may then be refitted on all development data. An honest estimate of the selected procedure’s predictive performance uses an untouched test sample or an outer cross-validation loop. The folds used to choose the model are validation folds, even when older notation calls them test folds (Cawley and Talbot, 2010).
The Kullback–Leibler connection
Main article: Kullback–Leibler divergence
The expected log-likelihood of a new observation differs from a Kullback–Leibler divergence by a constant that does not depend on the candidate. Let f be the density of a new observation and \widehat f the fitted density. Then
D(f\Vert\widehat f) =\int\ln\Bigl(\frac{f(x)}{\widehat f(x)}\Bigr)f(x)\,dx =\int f\ln f-\mathbb{E}\bigl[\ln\widehat f(X)\bigr].
The integral \int f\ln f does not depend on which candidate produced \widehat f. Maximising \mathbb{E}[\ln\widehat f(X)] is therefore the same comparison as minimising D(f\Vert\widehat f). The divergence is not symmetric. The convention puts the density of the expectation, here the true f, in the first argument.
Used in
See also
Cross-validation · -fold cross-validation · Leave-one-out cross-validation · Nested cross-validation · Bayesian information criterion · Akaike information criterion · Laplace approximation · Dropping terms that do not scale with n