Label switching

Topic: Mixtures and the EM algorithm · more questions on this topic
‹ Counting parameters Mixtures versus K-means ›

Show that the two-component parameter vectors (\pi,\mu_1,\sigma_1^2,\mu_2,\sigma_2^2) and (1-\pi,\mu_2,\sigma_2^2,\mu_1,\sigma_1^2) give the same density. Why does this complicate averaging the parameters across bootstrap fits?

Worked answer (click to show)

Write the density before interpreting the component names. A permutation changes the parameter labels and leaves the sum, and therefore the likelihood, unchanged. The density is

\pi\,\varphi(x;\mu_1,\sigma_1^2)+(1-\pi)\,\varphi(x;\mu_2,\sigma_2^2).

Swapping the labels and the weights, \pi\leftrightarrow 1-\pi, reorders the two summands. The likelihood is the same for both parameter vectors, and the data cannot distinguish them.

If two bootstrap fits have swapped labels, an average of their parameters, for example of the two values called \mu_1, mixes the means of different components and lands between them. The bootstrap covariance is inflated by that artificial spread. Components must be matched across fits first.