Mixture discriminant analysis
Section of Statistical classifiers and discriminant analysis, part of Statistical learning
Mixture discriminant analysis
| Type | Section |
| Taught in | Statistical classifiers and discriminant analysis |
| Part | Statistical learning |
A single Gaussian density cannot represent a ring-shaped concentration of observations. A quadratic boundary can still be circular, so QDA can separate a ring from a central group. Density fit and classification boundary are different questions. Mixture discriminant analysis gives each class a more flexible density by combining Gaussian components (Hastie and Tibshirani, 1996; Fraley and Raftery, 2002). Its benefit should be assessed on the actual prediction task, since estimating more components also costs data.
The number of components is not fixed by the geometry. More data can support more of them, and BIC or AIC, from Model selection, is the criterion already used for that choice, typically applied separately in each class. If one class is too thin for those components, a kernel density follows the same curve, and it is the estimator that fails first as the dimension grows. When the geometry does not decide between the methods, the comparison is an error rate on observations the fit did not see.
The plug-in step is the Bayes rule with each true class density replaced by its fitted mixture. For each of the G classes, fit a normal mixture to the observations from that class. For a new \mathbf{x}, evaluate the mixture density of each class, weight by the class probability, known or estimated, and assign \mathbf{x} to the class with the largest weighted density:
g^*=\arg\max_g \pi_g\widehat f_g(\mathbf{x}), \qquad \widehat f_g(\mathbf{x})=\sum_{h=1}^{H_g}\lambda_{gh}\widehat f_{gh}(\mathbf{x}),
where \pi_g is the prior probability of class g, \lambda_{gh} is the probability of component h within the mixture for class g, \widehat f_{gh} is the fitted density of that component, and H_g is the number of components fitted to class g, chosen separately or jointly. Ordinary LDA and QDA are the special cases H_g=1, with a common covariance or a covariance of their own.
Used in
See also
Training data and classifier · Posterior class probability · Zero–one loss and risk · Bayes-optimal rule · Fisher’s direction · Classification problem · Bayes classifier · Error rates and optimal classifiers