Maximum likelihood for a covariance matrix
Section of Estimation and inference for the multivariate normal, part of Statistical learning
Maximum likelihood for a covariance matrix
| Type | Section |
| Taught in | Estimation and inference for the multivariate normal |
| Part | Statistical learning |
The inverse covariance is a convenient optimisation variable. Set \Theta=\Sigma^{-1}. The quadratic part of the log-likelihood is linear in \Theta, and the determinant term becomes \log|\Theta|. Together they form a concave function on the positive definite matrices. The maximiser can be found and then inverted. The invariance result says why the inverted matrix is the maximum likelihood estimator of \Sigma. The general statement is the invariance of maximum likelihood. The argument used for the covariance is the following.
Proposition 1 (Invariance of the maximum likelihood estimator) Let the distribution of the data depend on \theta=(\theta_1,\ldots,\theta_k), with maximum likelihood estimator \widehat\theta. Let \tau(\theta)=(\tau_1(\theta),\ldots,\tau_r(\theta)), 1\le r\le k, be a transformation of the parameter. Then the maximum likelihood estimator of \tau_l(\theta) is \widehat\tau_l=\tau_l(\widehat\theta).
Let M(\tau;\mathbf{x})=\sup_{\theta:\tau(\theta)=\tau}L(\theta;\mathbf{x}) be the likelihood induced on the transformed parameter. For any value \tau in the range of the transformation,
\sup_{\theta:\tau(\theta)=\tau}L(\theta;\mathbf{x}) \le\sup_{\theta}L(\theta;\mathbf{x}) =L(\widehat\theta;\mathbf{x}) =\sup_{\theta:\tau(\theta)=\tau(\widehat\theta)}L(\theta;\mathbf{x}) =M\bigl(\tau(\widehat\theta);\mathbf{x}\bigr).
The last equality holds because \widehat\theta itself satisfies the constraint \tau(\theta)=\tau(\widehat\theta), so the supremum over that constraint set is attained at \widehat\theta. Hence M(\tau(\widehat\theta))\ge M(\tau) for every \tau, and \tau(\widehat\theta) maximises M.
Apply the lemma with \theta=\Sigma^{-1}=:\Theta. The free coordinates are the k=p(p+1)/2 distinct entries of the precision, and the transformed coordinates are the distinct entries of \Sigma=\Theta^{-1}. Substitute \Sigma^{-1}=\Theta and \log|\Sigma|=-\log|\Theta|:
\ell(\boldsymbol{\mu},\Theta) =-\frac12 np\log(2\pi)+\frac12 n\log|\Theta| -\frac12\sum_{i=1}^n(\mathbf{x}_i-\boldsymbol{\mu})^\top\Theta(\mathbf{x}_i-\boldsymbol{\mu}).
The derivative with respect to a symmetric matrix uses \partial\log|\Theta|/\partial\Theta=\Theta^{-1} and \partial(\mathbf{v}^\top\Theta\mathbf{v})/\partial\Theta=\mathbf{v}\mathbf{v}^\top. Those two identities give
\frac{\partial\ell}{\partial\Theta} =\frac n2\,\Theta^{-1}-\frac12\sum_{i=1}^n(\mathbf{x}_i-\boldsymbol{\mu})(\mathbf{x}_i-\boldsymbol{\mu})^\top.
The mean has already been maximised. Set \boldsymbol{\mu}=\widehat{\boldsymbol{\mu}}=\bar{\mathbf{x}} and set the derivative equal to zero:
\frac n2\,\widehat\Theta^{-1}-\frac12\sum_{i=1}^n(\mathbf{x}_i-\bar{\mathbf{x}})(\mathbf{x}_i-\bar{\mathbf{x}})^\top=0.
Multiply by 2. The sum is A, so n\widehat\Theta^{-1}-A=0 and \widehat\Theta^{-1}=A/n. Invariance, applied to \Theta\mapsto\Theta^{-1}, says that the maximum likelihood estimator of \Sigma is
\widehat\Sigma=\frac1n A.
There is a condition behind that answer: A must be positive definite. Then strict concavity in the precision makes the stationary point the unique maximum. If A is singular, the unrestricted positive definite Gaussian covariance has no finite maximum likelihood estimator in the interior of the parameter space. Shrinking the variance in an unobserved direction increases the likelihood without bound. An algebraic expression such as A/n is not automatically an admissible parameter. When the estimator does exist, its divisor is n.
Used in
See also
Invariance of the maximum likelihood estimator · Wishart distribution · Sampling law of the normal mean and scatter · Multivariate central limit theorem · The limit of Hotelling’s statistic · Null distribution of Hotelling’s · Two-sample Hotelling test · Likelihood of a multivariate normal sample