Bias and the sample covariance

Section of Estimation and inference for the multivariate normal, part of Statistical learning

Type	Section
Taught in	Estimation and inference for the multivariate normal
Part	Statistical learning

Whether \widehat\Sigma is unbiased is not settled by the likelihood equation. Expand the cross-product matrix about the true mean,

A=\sum_{i=1}^n(\mathbf{X}_i-\boldsymbol{\mu})(\mathbf{X}_i-\boldsymbol{\mu})^\top -n(\overline{\mathbf{X}}-\boldsymbol{\mu})(\overline{\mathbf{X}}-\boldsymbol{\mu})^\top.

If the observations are uncorrelated, with common mean \boldsymbol{\mu} and common covariance \Sigma, each outer product (\mathbf{X}_i-\boldsymbol{\mu})(\mathbf{X}_i-\boldsymbol{\mu})^\top has expectation \Sigma, so the first sum has expectation n\Sigma. The same uncorrelatedness gives \operatorname{Var}(\overline{\mathbf{X}})=\Sigma/n, and the subtracted term then has expectation \Sigma. Therefore

\mathbb{E}(A)=(n-1)\Sigma, \qquad \mathbb{E}(\widehat\Sigma)=\frac{n-1}{n}\Sigma.

Dividing A by n estimates \Sigma by (n-1)/n times \Sigma on average, so the maximum likelihood estimator is biased low. The unbiased matrix in common use divides by the degrees of freedom that remain after estimating the mean. It is the sample covariance matrix

S=\frac{1}{n-1}A.

The two estimates use the same measured scatter and differ only in their divisor. Dividing by n solves the likelihood problem. Dividing by n-1 corrects the expectation. Hotelling’s statistic below uses S=A/(n-1) because the stated F calibration is derived for that convention. Substituting a different divisor without adjusting the calibration changes the test.

A single mean is the wrong centre when the observations fall into G groups that may have different means and are assumed to share one covariance \Sigma. Let group g have n_g observations \mathbf{x}_{ig}, i=1,\ldots,n_g, with n=\sum_g n_g. Independence across groups lets the one-sample argument run inside each group, so the maximum likelihood estimator of \boldsymbol{\mu}_g is the group mean \bar{\mathbf{x}}_g. The unbiased estimator of the common \Sigma pools the corrected cross-product matrices. It is the pooled sample covariance

S=\frac{1}{n-G}\sum_{g=1}^G\sum_{i=1}^{n_g}(\mathbf{x}_{ig}-\bar{\mathbf{x}}_g)(\mathbf{x}_{ig}-\bar{\mathbf{x}}_g)^\top =\frac{1}{n-G}\sum_{g=1}^G A_g.

Each estimated group mean consumes one degree of freedom, which is why the divisor is n-G rather than n-1. The one-sample matrix is the case G=1.

Bias and the sample covariance

Used in

See also