Choosing model complexity
Topic: Model selection for stochastic models · more questions on this topic
‹ A tempting wrong turn Mixture selection and uncertainty ›
For n=100, three fitted density models have the following maximised log-likelihoods \ell and parameter counts d:
| Model | \ell | d |
|---|---|---|
| A | -180 | 5 |
| B | -170 | 9 |
| C | -166 | 14 |
Use \mathrm{BIC}=-2\ell+d\log n.
Compute BIC for each model and choose the preferred one under this convention.
Explain why choosing the largest training likelihood alone would not provide the same protection against overfitting.
Explain why a program reporting 2\ell-d\log n should select its largest value instead.
- With \log 100=4.605170,
\begin{align*} \mathrm{BIC}(A)&=360+5\log 100=383.026,\\ \mathrm{BIC}(B)&=340+9\log 100=381.447,\\ \mathrm{BIC}(C)&=332+14\log 100=396.472. \end{align*}
Model B has the smallest BIC, so B is preferred.
Extra flexibility can fit random sample variation. In nested models the global maximised likelihood cannot decrease as the model class expands, so the training likelihood contains no explicit cost for extra parameters. BIC trades fit against a d\log n penalty. A finite optimisation run, or a singular mixture fit, can complicate this ideal comparison.
The quantity 2\ell-d\log n is the negative of this BIC. Negation reverses the ordering, so the preferred model is the one with the largest reported value. The preferred statistical model is unchanged when the optimisation direction changes consistently.