Gaussian graphical models

Section of Graphical models, part of Statistical learning

Gaussian graphical models

Type Section
Taught in Graphical models
Part Statistical learning

A marginal correlation can be nonzero for every pair while most conditional correlations, given the rest, are zero. The Gaussian case turns that distinction into an entry of a matrix. Consider a p-dimensional random vector X=(X_1,\ldots,X_p)^\top with V=\{1,\ldots,p\}, distributed as multivariate normal with mean \mu and covariance \Sigma. The covariance matrix is the one defined for a random vector on joint distributions, and the density is the multivariate normal density. The inverse of the covariance is the concentration matrix, or precision matrix, K=\Sigma^{-1}, with entries k_{jk}. The covariance is assumed regular enough to be invertible.

The proof that a zero in K is conditional independence reorders the variables so that the pair of interest is the first block, then reads the conditional covariance from the block inverse. That block inverse is the Schur complement of the conditional normal distribution.

Theorem 1 (A zero precision entry is conditional independence) Let X\sim\operatorname{N}_p(\mu,\Sigma) with \Sigma regular. For j\neq k,

X_j\perp X_k\mid X_{V\setminus\{j,k\}} \iff k_{jk}=0.

The zero entries of the precision matrix indicate conditional independence between the corresponding variables, with the conditioning on all the remaining variables.

Article: A zero precision entry is conditional independence →

The variables can be put in any order, so without loss of generality j=1, k=2, A=\{1,2\} and B=\{3,\ldots,p\}. From the block inverse, writing

\Sigma^{-1}=K=\begin{pmatrix} K_{AA} & K_{AB} \\ K_{AB}^\top & K_{BB} \end{pmatrix},

one has K_{AA}=\Sigma_{A\mid B}^{-1}. The conditional law is X_A\mid X_B\sim\operatorname{N}(\mu_{A\mid B},\,\Sigma_{A\mid B})=\operatorname{N}(\mu_{A\mid B},\,K_{AA}^{-1}). If

K_{AA}=\begin{pmatrix} k_{11} & k_{12} \\ k_{12} & k_{22} \end{pmatrix}, \qquad \Sigma_{A\mid B}=K_{AA}^{-1} =\frac{1}{|K_{AA}|} \begin{pmatrix} k_{22} & -k_{12} \\ -k_{12} & k_{11} \end{pmatrix},

then k_{12}=0 makes the conditional covariance of X_1 and X_2 given X_B equal to 0. Conditional on X_B, the pair is bivariate normal with zero covariance, so X_1 and X_2 are conditionally independent. Conversely, conditional independence gives zero conditional covariance, so k_{12}=0. The argument applies to any pair.

There are two separate steps. Matrix inversion identifies the conditional covariance. Gaussianity turns a zero conditional covariance into conditional independence. Drop the second assumption and the last inference can fail. For jointly normal coordinates, zero covariance is independence, which is the special property of the multivariate normal density. The negative sign also deserves attention: after normalisation, a negative precision entry represents a positive partial correlation.

Several consequences follow from the same block formula.

\rho_{jk\mid V\setminus\{j,k\}} =\frac{-k_{jk}}{\sqrt{k_{jj}k_{kk}}},

since \Sigma_{A\mid B} has diagonal k_{22}/|K_{AA}| and k_{11}/|K_{AA}| and off-diagonal -k_{12}/|K_{AA}|.

By the partial-correlation formula, a negative off-diagonal of the precision corresponds to positive conditional dependence. Gaussian graphical models have the pairwise, local, and global Markov properties with respect to the graph whose edges are the nonzero entries of K.

A matrix of marginal correlations can be dense when the conditional dependence graph is sparse. Partial correlation is the dependence that remains after the other variables are held fixed. The chain below is that contrast in four variables.

Example 1 (Zeros in K, not in \Sigma) Question. Let X be a stationary Gaussian autoregressive process observed at four times, with \operatorname{corr}(X_j,X_k)=0.6^{|j-k|}. Read the conditional independences from K=\Sigma^{-1}, and compute the partial correlations.

Solution.

\Sigma=\begin{pmatrix} 1 & 0.6 & 0.36 & 0.216 \\ 0.6 & 1 & 0.6 & 0.36 \\ 0.36 & 0.6 & 1 & 0.6 \\ 0.216 & 0.36 & 0.6 & 1 \end{pmatrix}, \qquad K=\Sigma^{-1}=\begin{pmatrix} 1.5625 & -0.9375 & 0 & 0 \\ -0.9375 & 2.125 & -0.9375 & 0 \\ 0 & -0.9375 & 2.125 & -0.9375 \\ 0 & 0 & -0.9375 & 1.5625 \end{pmatrix}.

Every pair of the four variables is marginally correlated, so the covariance graph is complete, but K is tridiagonal:

X_1\perp X_3\mid (X_2,X_4), \qquad X_1\perp X_4\mid (X_2,X_3), \qquad X_2\perp X_4\mid (X_1,X_3).

The graph is the chain 1-2-3-4. Each variable depends on the past only through the most recent value. The partial correlations are

\rho_{12\mid 34}=\frac{0.9375}{\sqrt{1.5625\times 2.125}}=0.515, \qquad \rho_{23\mid 14}=\frac{0.9375}{2.125}=0.441, \qquad \rho_{34\mid 12}=0.515.

As a check of the diagonal formula, k_{11}=1.5625=1/0.64 and the variance of X_1 given (X_2,X_3,X_4) is 1-0.6^2=0.64, since X_1 depends on the rest only through X_2. And k_{22}=2.125=1/0.4706, the reciprocal of the variance of X_2 given the others. The same pattern on six variables has a dense covariance, a tridiagonal precision, and partial correlations only between neighbours.

The log density

Main article: Log density of a Gaussian graphical model

Write the multivariate normal density in terms of K:

\begin{aligned} f(x) &=\frac{1}{(2\pi)^{p/2}\,|\Sigma|^{1/2}} \exp\Bigl(-\tfrac12(x-\mu)^\top\Sigma^{-1}(x-\mu)\Bigr)\\ &=\frac{|K|^{1/2}}{(2\pi)^{p/2}} \exp\Bigl(-\tfrac12(x-\mu)^\top K(x-\mu)\Bigr). \end{aligned}

Treating \mu and K as constants, the logarithm of the density, up to constants c and c_1, is

\begin{aligned} \log f(x) &=c-\tfrac12(x-\mu)^\top K(x-\mu) =c-\tfrac12 x^\top Kx+x^\top K\mu-\tfrac12\mu^\top K\mu\\ &=c_1-\tfrac12\sum_{j\in V} k_{jj} x_j^2 -\sum_{j\in V}\sum_{k<j} k_{jk} x_j x_k +\sum_{j\in V} x_j\Bigl(\sum_{k\in V} k_{jk}\mu_k\Bigr)\\ &=c_1-\tfrac12\sum_{j\in V} k_{jj} x_j^2 +\sum_{j\in V} h_j x_j -\sum_{j\in V,\,k<j} k_{jk} x_j x_k, \end{aligned}

with h=K\mu. The log density is a sum of quadratic main effects with coefficients -k_{jj}/2, linear main effects with coefficients h_j, and quadratic interactions with coefficients -k_{jk}. Taking the exponential, f is a product over single variables and over pairs, with a pair appearing only if k_{jk}\neq 0. Each pair with k_{jk}\neq 0 is an edge, so the factors involve complete subsets of size at most 2, singletons or edges. Such subsets need not be maximal cliques. Their factors can be absorbed into containing clique potentials. This is an explicit instance of the factorisation, and of the Hammersley–Clifford theorem, with \psi_{jk}(x_j,x_k)=\exp(-k_{jk} x_j x_k).

Used in

See also

Undirected graph · Hammersley–Clifford · A zero precision entry is conditional independence · Maximum likelihood for a known graph · Graphical lasso · Markov graphs · Graph terminology · Conditional independence

Glossary and index