Pathologies of mixture likelihoods

Section of Mixtures and the EM algorithm, part of Statistical learning

Type	Section
Taught in	Mixtures and the EM algorithm
Part	Statistical learning

Unbounded likelihood. For H\ge 2, keep one component broad, with positive density at every observation. Place another mean exactly on one observation and let its covariance be \varepsilon^2 I. Its density at that point grows as \varepsilon^{-p}, while the broad component keeps every other likelihood factor positive. The likelihood therefore diverges as \varepsilon\downarrow 0. This construction proves unboundedness. The boundary does not contain an ordinary finite maximum that the divergence is approaching.

The same divergence occurs when a component is supported on too few points to determine a positive definite covariance. Two points can pin a component in the plane, three points in three dimensions, and so on. The remaining components are left to describe the rest of the sample, while the collapsed component is made narrower. Starting a component at one point with a very small variance is enough to begin the collapse.

In one dimension, stop and restart if a component variance falls below a threshold. In two dimensions the analogous failure is a component collapsing toward a line. A singular covariance is the algebraic form of that failure: the determinant is 0, and the precision in the normal density does not exist. The eigenvalues of a component covariance are the variances along its elliptical axes. Positive definiteness requires every one of them to be positive. If an eigenvalue becomes small, stop. A variance on its way to 0 is a failure of the unconstrained fit.

Label switching. Renaming components changes the parameter vector and leaves the density unchanged. Finite Gaussian mixtures with distinct components and positive weights are identifiable up to this permutation. Coincident components or zero weights introduce further non-identifiability (McLachlan and Peel, 2000). Before comparing component parameters across fits, decide how to match labels. Ordering means can help in one dimension. The order becomes unstable when means are close.

Nothing in the model says that the component on the left is component 1. A start with \mu_1 to the right of \mu_2 returns the swapped labelling, and the weights and covariances move with the labels. Averaging parameters across restarts, as if the subscript 1 named the same component every time, mixes distinct components.

Initialisation. The observed likelihood on one sample can have two local modes. An exact fixed point in the regular interior satisfies the stationary likelihood equations. A small numerical update is weaker evidence of stationarity, and a further argument is required before that point is called a global mode. One start can finish with components of different shapes and little overlap, and another start can finish at a different stationary point. Further iteration from a stopped update stays at that point: given the current parameter, the map is deterministic.

Different valid starts can enter different regions of the likelihood surface. Monotonicity controls the likelihood along a run. A run can still approach a singular boundary and miss every finite stationary point. Inspect the likelihood, the component weights, and the smallest covariance eigenvalues together.

One possible start assigns observations at random and fits each resulting group. Its means lie in the sample convex hull, and its covariances need not be well conditioned: a small group can already be singular. Random allocation also tends to place the means near the overall mean, giving weak initial separation. Check group sizes and covariance eigenvalues before calling EM.

A common alternative runs K-means with K=H, then uses the group proportions, means, and covariances as initial values. This often gives more separated means. The K-means criterion does not make every realised group spherical. A small or nearly collinear group can still produce a singular covariance. Initialisation supplies a feasible starting point. Whether the mixture is a good description of the data is a separate question.

Use several starts, and report how degeneracy is handled. An eigenvalue constraint or a covariance penalty changes the fitted optimisation problem and should be stated explicitly. Keeping the largest finite value encountered is a computational choice. A separate argument is required before that value is treated as an unconstrained maximum.

Spurious solutions. A component covariance on its way to zero makes the density, and therefore the log-likelihood, numerically unstable. Discard those runs. A run can also place a few points in a component whose covariance is still positive definite. The likelihood is then finite, and the component may still fail to describe a subgroup one is willing to interpret. Penalising small variances would move every component, including those that have not collapsed. A hard floor on the variances fails in both directions: a floor that is too low lets the iteration reach the undesirable mode and remain there, and a floor that is too high blocks legitimate components and blocks intermediate values the iteration may need. Requiring a minimum total weight \sum_i\tau_{ih}, with the floor set above 1, changes the algorithm, so the monotonicity argument no longer applies as stated. The eigenvalues of the component covariances, rather than a single variance, detect a collapse toward a lower-dimensional subspace. In practice the best finite solutions often have to be compared by hand. If the only aim is density estimation, a small component that improves the approximation may be kept.

Used in

Taught in Mixtures and the EM algorithm
All 7 questions on Mixtures and the EM algorithm

Used in

See also