When ordinary linear discrimination is not defined

Topic: High-dimensional analysis · more questions on this topic
‹ A tempting wrong turn Principal components before linear discrimination ›

There are n=90 independent labelled observations, 30 in one class and 60 in another, with p=500 numerical predictors. You propose principal components followed by ordinary, unregularised linear discriminant analysis.

  1. Explain a fitting difficulty and an evaluation difficulty for this data set.

  2. Give upper bounds on the number of nonzero sample principal components and on the dimension in which the pooled within-class covariance can be invertible. Explain the difference.

  3. Describe how to choose and evaluate the principal-component dimension, and explain why a feasible dimension need not be a good predictive dimension.

  1. The pooled covariance is singular because its rank is at most n-g=88<500, so its ordinary inverse does not exist. With only 90 observations, tuning and test errors can be variable, and using the training or tuning data for final assessment creates optimism. Class imbalance also makes overall accuracy alone incomplete.

  2. Centred data have rank at most \min(p,n-1)=89, so there can be at most 89 nonzero sample principal components. Within-class centring loses one degree of freedom per class, giving rank at most n-g=88. Thus q\le 88 is necessary for an invertible projected pooled covariance. It is not sufficient if further dependencies remain. Using all 89 components does not solve the rank problem of ordinary linear discrimination.

  3. Select q within training-only cross-validation, refitting centring, scaling, the principal components, and the discriminant rule in every fold. Use held-out testing or an outer loop for assessment. Apply the rank restriction to the smaller training fold, not just to the full sample. High-variance directions may contain little class information, and near-singular covariance estimates can be unstable even below the rank limit.