Principal components as a modelling step
A student wants to classify iris species from four measurements. They propose retaining the first two principal components of the standardised measurements.
Explain how the principal directions and component scores are obtained. Distinguish a direction from a score.
Explain what changes if centimetres are converted to millimetres for just one variable before covariance-based principal components, and why correlation-based principal components avoid this issue.
Explain why retaining the two components with largest variance is not guaranteed to give the best classifier. Describe where the transformation must be fitted during cross-validation.
Directions, scores and variance fractions are related objects with different dimensions and different roles. The place where the transformation was fitted is part of the answer.
Centre the variables and, here, divide by the training standard deviations. Diagonalise their covariance matrix S=U\Lambda U^\top, sorting eigenvalues from largest to smallest. A direction \mathbf{u}_j is a unit vector in predictor space. A score for observation \mathbf{x}_i is t_{ij}=\mathbf{u}_j^\top\mathbf{z}_i. The first direction maximises score variance subject to unit norm. Subsequent directions impose orthogonality. The retained variance fraction is (\lambda_1+\lambda_2)/\sum_j\lambda_j.
Multiplying one variable by 10 changes its covariance contributions, so the covariance-based directions can change. Standardisation cancels a positive change of measurement unit and makes the correlation-based analysis invariant to that change.
The decomposition ignores the labels. A low-variance direction can separate classes. Fit the centring, the scaling and the eigendecomposition on each training fold, and apply that fitted transform to its validation fold. Choose the component count and the classifier settings within the training data, leaving the final test set untouched.