Why a covariance is positive semi-definite

Topic: Covariance and the multivariate normal · more questions on this topic
Variances of linear combinations ›

Show that for any random vector with finite second moments, \mathbf{z}^\top\Sigma\mathbf{z}=\operatorname{Var}(\mathbf{z}^\top\mathbf{X})\ge 0. Deduce |\sigma_{jk}|\le\sqrt{\sigma_{jj}\sigma_{kk}} from the 2\times 2 principal minor. Use Sylvester’s criterion to decide whether

\begin{pmatrix} 1 & 0.9 & 0.9\\ 0.9 & 1 & -0.9\\ 0.9 & -0.9 & 1 \end{pmatrix}

could be a correlation matrix.

Worked answer (click to show)

Turn the matrix expression into the variance of a scalar. That gives a nonnegative quantity without an eigenvalue calculation. The proposed correlation matrix can then be checked separately.

For any fixed \mathbf{z},

\mathbf{z}^\top\Sigma\mathbf{z} =\mathbb{E}\bigl[\mathbf{z}^\top(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^\top\mathbf{z}\bigr] =\mathbb{E}\bigl[(\mathbf{z}^\top(\mathbf{X}-\boldsymbol{\mu}))^2\bigr] =\operatorname{Var}(\mathbf{z}^\top\mathbf{X})\ge 0.

Taking \mathbf{z} with only entries j and k nonzero shows that the 2\times 2 matrix

\begin{pmatrix}\sigma_{jj}&\sigma_{jk}\\\sigma_{jk}&\sigma_{kk}\end{pmatrix}

is positive semi-definite, so its determinant satisfies \sigma_{jj}\sigma_{kk}-\sigma_{jk}^2\ge 0, which is |\sigma_{jk}|\le\sqrt{\sigma_{jj}\sigma_{kk}}.

For the matrix given, the leading minors are 1, 1-0.81=0.19, and

\begin{aligned} \det &=1(1-0.81)-0.9\bigl(0.9-(-0.9)(0.9)\bigr)+0.9\bigl(0.9(-0.9)-0.9\bigr)\\ &=0.19-0.9(1.71)+0.9(-1.71)\\ &=0.19-1.539-1.539=-2.888<0. \end{aligned}

The determinant is negative, so the matrix is not positive semi-definite and cannot be a correlation matrix. The pattern is impossible: X_1 cannot be strongly positively correlated with both X_2 and X_3 while X_2 and X_3 are strongly negatively correlated.