Multivariate normal density
Section of Covariance and the multivariate normal, part of Statistical learning
Multivariate normal density
| Type | Section |
| Taught in | Covariance and the multivariate normal |
| Part | Statistical learning |
The univariate normal density is
\phi(x;\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\Bigl\{-\frac{(x-\mu)^2}{2\sigma^2}\Bigr\}, \qquad x\in\mathbb{R}.
If the coordinates of \mathbf{X}=(X_1,\ldots,X_p)^\top are independent with X_j\sim\operatorname{N}(\mu_j,\sigma_j^2), the mean vector is \boldsymbol{\mu}=(\mu_1,\ldots,\mu_p)^\top and the covariance is the diagonal matrix \Sigma=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_p^2). The joint density is the product of the marginals,
\begin{aligned} f(\mathbf{x}) &=\prod_{j=1}^p\frac{1}{\sqrt{2\pi}\,\sigma_j}\exp\Bigl\{-\frac{(x_j-\mu_j)^2}{2\sigma_j^2}\Bigr\}\\ &=\frac{1}{(2\pi)^{p/2}\prod_{j=1}^p\sigma_j} \exp\Bigl\{-\frac12\sum_{j=1}^p\Bigl(\frac{x_j-\mu_j}{\sigma_j}\Bigr)^2\Bigr\}. \end{aligned}
For this diagonal \Sigma, |\Sigma|^{1/2}=\prod_j\sigma_j and
(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})=\sum_{j=1}^p\frac{(x_j-\mu_j)^2}{\sigma_j^2}.
The product formula can therefore be written without mentioning the diagonal structure. That rewriting is the definition in every dimension.
Definition 1 (Multivariate normal distribution) Let \boldsymbol{\mu}\in\mathbb{R}^p and let \Sigma be a symmetric positive definite p\times p matrix. A random vector \mathbf{X} has the p-variate normal distribution \operatorname{N}_p(\boldsymbol{\mu},\Sigma) if it has density
f(\mathbf{x})=\frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \exp\Bigl\{-\frac12(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\Bigr\}, \qquad\mathbf{x}\in\mathbb{R}^p.
The mean vector is \boldsymbol{\mu} and the covariance matrix is \Sigma. The subscript p is dropped when the dimension is clear.
When \Sigma is positive definite, f(\mathbf{x})>0 for every \mathbf{x} and the integral of f equals 1, so f is a density. The representation \boldsymbol{\mu}+B\mathbf{Z} with \mathbf{Z} standard normal in \mathbb{R}^p and BB^\top=\Sigma is the same family, and it remains available when \Sigma is only positive semi-definite and no density exists. The second slot of \operatorname{N}(\mu,\sigma^2) and of \operatorname{N}_p(\boldsymbol{\mu},\Sigma) is a variance or a covariance matrix.
In two dimensions the surface height is a density, and probability is the volume under that surface. A normal model assigns some probability to every open region, including negative lengths. Whether that matters depends on how much probability falls there.
The bivariate normal has five parameters: two means, two variances, and a correlation \rho. In matrix form
\boldsymbol{\mu}=\begin{pmatrix}\mu_1\\ \mu_2\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}\sigma_1^2 & \rho\sigma_1\sigma_2\\ \rho\sigma_1\sigma_2 & \sigma_2^2\end{pmatrix}.
The determinant and inverse are
|\Sigma|=\sigma_1^2\sigma_2^2(1-\rho^2), \qquad \Sigma^{-1}=\frac{1}{\sigma_1^2\sigma_2^2(1-\rho^2)} \begin{pmatrix}\sigma_2^2 & -\rho\sigma_1\sigma_2\\ -\rho\sigma_1\sigma_2 & \sigma_1^2\end{pmatrix}.
Write z_j=(x_j-\mu_j)/\sigma_j. Substituting into the density gives
f(x_1,x_2)=\frac{1}{2\pi\sigma_1\sigma_2\sqrt{1-\rho^2}} \exp\Bigl\{-\frac{1}{2(1-\rho^2)}\bigl(z_1^2-2\rho z_1 z_2+z_2^2\bigr)\Bigr\}.
Level sets are ellipsoids
Main article: Level sets are ellipsoids
An ellipse drawn through a data cloud describes the spread of observations. A confidence region for the mean describes uncertainty about the centre. The second region contracts as more independent observations are collected. The population cloud does not. The contours below hold \boldsymbol{\mu} and \Sigma fixed.
Fix c>0 and consider \{\mathbf{x}:f(\mathbf{x})=c\}. If c exceeds the maximum of f, the set is empty. Otherwise divide f(\mathbf{x})=c by the leading constant and take logarithms:
-\frac12(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}) =\log\bigl(c\,(2\pi)^{p/2}|\Sigma|^{1/2}\bigr),
which is
(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})=C
for a constant C determined by c. For C>0 this is an ellipsoid centred at \boldsymbol{\mu}. In the bivariate case the level sets are ellipses. Correlation rotates and stretches them: the contour for \rho=0 is axis-aligned, the contour for \rho=0.75 leans up, and the contour for \rho=-0.75 leans down.
To find the axes, use \Sigma=\Gamma\Lambda\Gamma^\top with orthonormal columns \boldsymbol{\gamma}_1,\ldots,\boldsymbol{\gamma}_p and \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p). Then \Sigma^{-1}=\Gamma\Lambda^{-1}\Gamma^\top. In the rotated coordinates \mathbf{u}=\Gamma^\top(\mathbf{x}-\boldsymbol{\mu}),
(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})=\mathbf{u}^\top\Lambda^{-1}\mathbf{u}=\sum_{j=1}^p\frac{u_j^2}{\lambda_j}=C.
This is an ellipsoid whose jth semi-axis points along \boldsymbol{\gamma}_j and has length \sqrt{C\lambda_j}. Those directions are the principal component directions derived on Principal component analysis.
Definition 2 (Squared Mahalanobis distance) The squared Mahalanobis distance of \mathbf{X} from its mean is
(\mathbf{X}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{X}-\boldsymbol{\mu}).
If \mathbf{X}\sim\operatorname{N}_p(\boldsymbol{\mu},\Sigma), set \mathbf{Z}=\Sigma^{-1/2}(\mathbf{X}-\boldsymbol{\mu}), where \Sigma^{-1/2}=\Gamma\Lambda^{-1/2}\Gamma^\top is the symmetric inverse square root. The linear-image formula for covariances gives
\operatorname{Var}(\mathbf{Z})=\Sigma^{-1/2}\Sigma\Sigma^{-1/2}=I.
The next proposition says \mathbf{Z} is normal, so its coordinates are independent standard normals. The squared Mahalanobis distance is therefore
\mathbf{Z}^\top\mathbf{Z}=\sum_{j=1}^p Z_j^2\sim\chi^2_p,
the chi-squared distribution with p degrees of freedom. A density contour with C=\chi^2_{p;\alpha}, the upper \alpha point of that law, encloses probability 1-\alpha. The same upper-tail convention, with the upper probability in the first subscript, is used for the t and F critical values on Estimation and inference for the multivariate normal.
Used in
See also
Joint distribution and density · Mean vector and covariance matrix · Correlation · Multivariate normal distribution · Squared Mahalanobis distance · Linear images of a normal vector · Conditional distribution of a multivariate normal · Partial correlation