Reading the BIC
Section of Model selection for stochastic models, part of Statistical learning
Reading the BIC
| Type | Section |
| Taught in | Model selection for stochastic models |
| Part | Statistical learning |
BIC retains the terms that grow with sample size and discards bounded terms. It does not require exactly equal model priors or a flat parameter prior: many fixed, smooth, positive priors give the same leading expression. Their differences survive in the omitted terms, which can matter when BIC values are close. Retaining only one omitted constant would not generally recover an accurate marginal likelihood.
The penalty comes from integration, specifically the shrinking volume around a regular likelihood peak. With information of order n in each of d_k directions, that volume is of order n^{-d_k/2}. Taking logs gives -(d_k/2)\log n. Thus one extra parameter must improve the log-likelihood by more than \tfrac12\log n to improve BIC. This is a comparison rule coming from the volume of the Laplace approximation around a regular likelihood peak.
Mixtures need a separate qualification. Coincident components and zero weights make model-order comparison singular, so the ordinary Laplace argument is generally invalid there. BIC can nevertheless be consistent under particular mixture-model assumptions. Consistency and accuracy as a marginal-likelihood approximation are different properties (Drton and Plummer, 2017). The unconstrained Gaussian family with collapsing covariances does not acquire a finite global maximum likelihood estimator merely because a BIC penalty is attached. State the fitted covariance restrictions and the candidate component counts when reporting the comparison.
Used in
See also
Bayesian information criterion · Akaike information criterion · Laplace approximation · Dropping terms that do not scale with n · Cross-validation of the likelihood · Kullback–Leibler divergence · BIC sign conventions