LDA and QDA on held-out iris flowers

Section of Statistical classifiers and discriminant analysis, part of Statistical learning

LDA and QDA on held-out iris flowers

Type Section
Taught in Statistical classifiers and discriminant analysis
Part Statistical learning

The split holds out 15 flowers of each species and trains on the remaining 35, so both fits below see the same four predictors and the same 105 labelled flowers. The held-out set has 45 flowers. LDA forces one covariance. QDA estimates one per species. That is the only difference between the two rules being compared. The data frames train and test carry Species and the four iris measurements.

lda_fit <- MASS::lda(Species ~ ., data = train)
qda_fit <- MASS::qda(Species ~ ., data = train)
lda_pred <- predict(lda_fit, test)$class
qda_pred <- predict(qda_fit, test)$class
table(Observed = test$Species, Predicted = lda_pred)
#>             Predicted
#> Observed     setosa versicolor virginica
#>   setosa         15          0         0
#>   versicolor      0         13         2
#>   virginica       0          1        14
c(LDA_test_error = mean(lda_pred != test$Species),
  QDA_test_error = mean(qda_pred != test$Species))
#> LDA_test_error QDA_test_error
#>        0.06667        0.06667

LDA misclassifies two versicolor flowers as virginica and one virginica flower as versicolor: 3 mistakes in 45, and 3/45=0.06667. The QDA test error on the same split is also 0.06667. One test flower changes the error by 1/45, about 2.2 percentage points, so a sample of 45 held-out flowers makes a coarse comparison. Equal class counts describe one assessment population. A different prevalence, or a different cost of mistakes, changes which accuracy matters, as in the unequal-cost rule.

Key idea

A statistical classifier estimates the class probabilities \pi_g and the class densities f_g, converts them to posterior probabilities \tau_g(\mathbf{x}), and plugs those probabilities into the Bayes rule. Equal costs choose the largest posterior. Unequal costs choose the class with the smallest expected loss. The Bayes risk is a floor for a fresh observation from the true distribution. Under a shared normal covariance the log posterior odds are linear in \mathbf{x}, which is LDA. Separate covariances leave a quadratic boundary, which is QDA. Fisher’s direction \mathbf{a}\propto S^{-1}(\widehat{\boldsymbol\mu}_1-\widehat{\boldsymbol\mu}_2) is the same vector, obtained by maximising separation of the class means relative to pooled within-class scatter.

Used in

See also

Training data and classifier · Posterior class probability · Zero–one loss and risk · Bayes-optimal rule · Fisher’s direction · Classification problem · Bayes classifier · Error rates and optimal classifiers

Glossary and index