Ridge makes the matrix invertible

Topic: High-dimensional analysis · more questions on this topic
‹ Rank A Bonferroni threshold ›

Let

X=\begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}

be n=2 observations on two perfectly collinear predictors. Show that X^\top X is singular, find its eigenvalues, and compute the ridge estimate (X^\top X+\lambda I)^{-1}X^\top y for y=(1,3)^\top and \lambda=1.

Worked answer (click to show)

X^\top X=\begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix}

has determinant 0. Its eigenvalues are 4, with eigenvector (1,1), and 0, with eigenvector (1,-1). With \lambda=1,

X^\top X+I=\begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix}

has determinant 5 and inverse \tfrac15\begin{pmatrix} 3 & -2 \\ -2 & 3 \end{pmatrix}. Also X^\top y=(4,4)^\top, so

\hat\beta_R=\tfrac15(12-8,\,-8+12)^\top=(0.8,0.8)^\top.

The solution splits the effect equally between the two identical predictors, and it is the unique ridge solution. Least squares has infinitely many solutions \beta_1+\beta_2=2.