Dropping terms that do not scale with n
Section of Model selection for stochastic models, part of Statistical learning
Dropping terms that do not scale with n
| Type | Section |
| Taught in | Model selection for stochastic models |
| Part | Statistical learning |
The next cut keeps only the terms that grow with n. The maximised log-likelihood is a sum of one log-density per observation, and the sum is not divided by n, so it grows as the sample grows. The model prior was chosen before the data arrived, so \log\mathbb{P}(M_k) does not grow with n. The prior density evaluated at the maximum likelihood estimator stays O(1) under Schwarz’s conditions. The term (d_k/2)\log(2\pi) contains no n.
Drop every term that remains O(1). That removes \log\mathbb{P}(M_k), which equal prior probabilities on the models would have removed on their own, the log prior density \log\mathbb{P}(\widehat\Psi_k\mid M_k), and (d_k/2)\log(2\pi). For n observations the observed information is of order n, provided the average information per observation stays of order 1. That proviso is the regular case, and it is the step that fails near a vanishing component. Under it,
\bigl|I(\widehat\Psi_k,X_{\mathrm{all}})\bigr| =n^{d_k}\Bigl|\frac1n I(\widehat\Psi_k,X_{\mathrm{all}})\Bigr| =n^{d_k}\,O(1),
so
-\frac12\log I(\widehat\Psi_k,X_{\mathrm{all}}) =-\frac{d_k}{2}\log n+O(1).
Drop the remaining O(1) term. The approximate MAP rule is
k^*\approx\arg\max_k \ell(\widehat\Psi_k;X_{\mathrm{all}},M_k)-\frac{d_k}{2}\log n.
Definition 1 (Bayesian information criterion) Multiply the approximate MAP rule by -2, which reverses the direction of the optimisation. The same rule is
k^*\approx\arg\min_k \mathrm{BIC}_k, \qquad \mathrm{BIC}_k=-2\ell(\widehat\Psi_k;X_{\mathrm{all}},M_k)+d_k\log n.
Used in
See also
Bayesian information criterion · Akaike information criterion · Laplace approximation · Reading the BIC · Cross-validation of the likelihood · Kullback–Leibler divergence · BIC sign conventions