A tempting wrong turn

Topic: Model selection for stochastic models · more questions on this topic
‹ KL divergence Choosing model complexity ›

A student selects the number of mixture components by maximising the log-likelihood on the training data, for H=1,\ldots,10. Why will this almost always select H=10, or the largest value tried, and which two changes to the procedure on this page fix it?

Worked answer (click to show)

If the candidate parameter spaces are nested, the supremum of the training log-likelihood cannot decrease as H increases. It need not strictly increase at every step. For unrestricted Gaussian mixtures with at least two components, the supremum is already infinite by the covariance-collapse construction: a component can be pinned on one observation while another component keeps the remaining likelihood factors positive. A numerical finite local fit does not remove that pathology. Meaningful comparisons require a stated fitting or constraint convention.

Two changes address the selection. A complexity criterion such as BIC or AIC charges a penalty that grows with the parameter count. A held-out log-likelihood scores observations that were excluded from the fit. The two targets differ: BIC approximates a marginal likelihood in regular models, and held-out log-likelihood estimates predictive performance. The ordinary Laplace justification of BIC needs qualification for singular mixtures (Drton and Plummer, 2017).