Parameter counts

Topic: Statistical classifiers and discriminant analysis · more questions on this topic
‹ A quadratic boundary Fisher’s direction ›

For p=10 and G=3, count the free parameters of LDA and of QDA. How large must the training set be, at least, for each of the three QDA class covariance matrices to be nonsingular?

Worked answer (click to show)

A covariance shared across classes and a covariance estimated separately in each class change both the parameter count and the rank requirement.

\text{LDA: } Gp+\frac{p(p+1)}{2}+(G-1)=30+55+2=87,

\text{QDA: } 30+3\times 55+2=197.

A class sample covariance matrix has rank at most n_g-1, so nonsingularity needs n_g-1\ge p, that is n_g\ge 11 in each class and at least 33 observations in total. A nearly singular matrix is already unreliable, so a practical fit uses more.