Principal components before linear discrimination

Topic: High-dimensional analysis · more questions on this topic
‹ When ordinary linear discrimination is not defined Selection in many dimensions ›

There are 72 observations with 400 continuous predictors: 24 observations in class 1 and 48 in class 2. Ordinary linear discriminant analysis is proposed for future classification.

  1. Explain one difficulty in fitting it and one in estimating its predictive performance here.

  2. Explain how principal components could make fitting feasible. Give the maximum number of nonzero sample components and the necessary upper bound on the retained dimension for an invertible pooled within-class covariance. Explain how to choose that dimension.

  1. The pooled within-class covariance has rank at most 72-2=70, so a 400\times 400 inverse is impossible without changing the estimator. The small sample makes evaluation variable and makes resubstitution or tuning-set assessment vulnerable to optimism. Accuracy must also be interpreted against the 2{:}1 class split.

  2. Fit the principal components on the training data and project into q dimensions. Centred total data have rank at most 71, whereas pooled within-class scatter has rank at most 70. Thus q\le 70 is necessary for ordinary linear discrimination, and the projected covariance must still have full rank. Choose q by training-only validation, refitting the complete pipeline in each fold and obeying that fold’s rank limit. High explained variance is not proof of predictive value.