Mixture selection and uncertainty
Topic: Model selection for stochastic models · more questions on this topic
‹ Choosing model complexity
Two two-dimensional Gaussian mixtures fitted to n=80 observations have (\ell,d)=(-140,11) and (-133,17). Use \mathrm{BIC}=-2\ell+d\log n.
Explain how posterior component probabilities are calculated and why they are more informative than a hard label for an observation in an overlap region.
Calculate BIC and choose between the two fitted candidates.
Would a larger fitted training likelihood prove that a model predicts new data better? Explain.
Compute \pi_k\varphi(\mathbf{x};\boldsymbol\mu_k,\Sigma_k) for each component and divide by the sum of those weighted densities. The weight \pi_k, the component mean and covariance, and the normalising mixture density are the ingredients. A hard label retains only the maximum. The probability vector distinguishes a near tie from a confident assignment.
With \log 80=4.382027,
\begin{align*} \mathrm{BIC}&=280+11\log 80=328.202,\\ \mathrm{BIC}&=266+17\log 80=340.494. \end{align*}
The first model has the smaller BIC, so it is the one selected.
- A larger training likelihood is a statement about the sample already used to fit the model. Flexibility can fit noise. A comparison on new observations, or an explicit complexity penalty, is what speaks to prediction. A numerical likelihood value alone also does not prove that the optimisation found a global maximum.