Quadratic discriminant analysis
Section of Statistical classifiers and discriminant analysis, part of Statistical learning
Quadratic discriminant analysis
| Type | Section |
| Taught in | Statistical classifiers and discriminant analysis |
| Part | Statistical learning |
| Questions using it | 2 |
To allow different class shapes, estimate a separate covariance for each class. QDA makes that change and retains Gaussian class densities. The pairwise score difference becomes a polynomial of degree at most two. It can describe curved boundaries, though special parameter choices give degenerate or empty boundaries. The next display shows exactly which cancellation LDA loses.
The tie \widehat\pi_j\widehat f_j(\mathbf{x})=\widehat\pi_l\widehat f_l(\mathbf{x}) is now
\frac{\widehat\pi_j}{(2\pi)^{p/2}\,|\widehat\Sigma_j|^{1/2}} \exp\Bigl(-\tfrac12(\mathbf{x}-\widehat{\boldsymbol\mu}_j)^\top\widehat\Sigma_j^{-1}(\mathbf{x}-\widehat{\boldsymbol\mu}_j)\Bigr) = \frac{\widehat\pi_l}{(2\pi)^{p/2}\,|\widehat\Sigma_l|^{1/2}} \exp\Bigl(-\tfrac12(\mathbf{x}-\widehat{\boldsymbol\mu}_l)^\top\widehat\Sigma_l^{-1}(\mathbf{x}-\widehat{\boldsymbol\mu}_l)\Bigr).
The common factor (2\pi)^{p/2} still cancels. The determinants do not, because each class now has its own covariance. Taking logarithms leaves them on the boundary:
\log\frac{\widehat\pi_j\,|\widehat\Sigma_l|^{1/2}}{\widehat\pi_l\,|\widehat\Sigma_j|^{1/2}} =\frac12\Bigl[ (\mathbf{x}-\widehat{\boldsymbol\mu}_j)^\top\widehat\Sigma_j^{-1}(\mathbf{x}-\widehat{\boldsymbol\mu}_j) -(\mathbf{x}-\widehat{\boldsymbol\mu}_l)^\top\widehat\Sigma_l^{-1}(\mathbf{x}-\widehat{\boldsymbol\mu}_l) \Bigr].
The quadratic terms are \mathbf{x}^\top\widehat\Sigma_j^{-1}\mathbf{x} and \mathbf{x}^\top\widehat\Sigma_l^{-1}\mathbf{x}. They cancel only if the two inverses agree. Once the covariances differ, the difference \mathbf{x}^\top(\widehat\Sigma_j^{-1}-\widehat\Sigma_l^{-1})\mathbf{x} survives, and the boundary is quadratic in \mathbf{x}. The class regions therefore have piecewise quadratic edges. The same comparison can be written as a score for each class,
\delta_g(\mathbf{x}) =-\tfrac12\log|\widehat\Sigma_g| -\tfrac12(\mathbf{x}-\widehat{\boldsymbol\mu}_g)^\top\widehat\Sigma_g^{-1}(\mathbf{x}-\widehat{\boldsymbol\mu}_g) +\log\widehat\pi_g,
and the rule predicts the class with the largest score. What changed relative to LDA is the covariance assumption. Normality and equal costs are still in force.
QDA can represent class shapes that a single covariance cannot, and it estimates more parameters from the same sample. LDA has
Gp+\frac{p(p+1)}{2}+(G-1)
free parameters. QDA has
Gp+G\frac{p(p+1)}{2}+(G-1).
Each class covariance is estimated from that class alone, so its sampling variability is larger, especially when a class is small or p is large. If a common covariance is a reasonable approximation and the sample is limited, LDA can have the smaller error on new observations. The extra flexibility has then been spent on variation in the training sample that does not repeat. That is overfitting. How large a sample is enough depends on p: the parameter counts grow like p^2, and faster for QDA. Bivariate or trivariate marginal plots are the practical check on whether one covariance is plausible for the classes at hand. Descriptive LDA and QDA regions fitted to the two iris petal measurements illustrate the shapes. Those pictures are training-data illustrations. The laboratory below classifies flowers that were held out of the fit.
Used in
- Taught in Statistical classifiers and discriminant analysis
- Questions that use it (2) — A quadratic boundary · Parameter counts
- All 8 questions on Statistical classifiers and discriminant analysis
See also
Training data and classifier · Posterior class probability · Zero–one loss and risk · Bayes-optimal rule · Fisher’s direction · Classification problem · Bayes classifier · Error rates and optimal classifiers