A low-variance predictor

Topic: Principal component analysis · more questions on this topic
‹ How many components Principal components as a modelling step ›

Let X_1\sim\operatorname{N}(0,100) and X_2\sim\operatorname{N}(0,1) be independent, and let the response be Y=X_2 exactly. An analyst reduces the explanatory variables to their first principal component and regresses Y on it. What is the first principal direction, in the population version, and what proportion of the variance of Y can the regression explain? What does this show about using principal components before regression?

Worked answer (click to show)

Predictor variance and predictive relevance can point in different directions. One case of that disagreement is enough to reject a claim that principal components always preserve the useful information.

\operatorname{Var}(X_1)=100 and \operatorname{Var}(X_2)=1 with zero covariance, so \Sigma=\operatorname{diag}(100,1) and the first principal direction is \mathbf{a}_1=(1,0)^\top: the first component is X_1 itself. But Y=X_2 is uncorrelated with X_1, so a regression of Y on the first component explains R^2=0 of the variance of Y, even though Y is perfectly predictable from X_2. This is a statement about population linear-regression R^2. Independence of the coordinates was an extra assumption of the construction. Principal components choose directions by the variance of the explanatory variables alone, so a low-variance direction can carry all of the predictive information. Selecting components should be checked against the response, for example by cross-validation, rather than assumed to preserve predictive power.