Conditional-mean line and least squares
Section of Covariance and the multivariate normal, part of Statistical learning
Conditional-mean line and least squares
| Type | Section |
| Taught in | Covariance and the multivariate normal |
| Part | Statistical learning |
Least squares of a response Y on a predictor X minimises \sum_i(y_i-a-bx_i)^2 and treats the observed x_i as fixed. The unconstrained bivariate normal, fitted to the same complete pairs, estimates (\widehat\mu_X,\widehat\mu_Y) and a covariance with one divisor in every entry, whether that divisor is n or n-1. Its conditional mean of Y given X=x is
\widehat\mu_Y+\frac{s_{YX}}{s_{XX}}(x-\widehat\mu_X).
The slope s_{YX}/s_{XX} and the intercept \widehat\mu_Y-(s_{YX}/s_{XX})\widehat\mu_X are exactly the least-squares slope and intercept. The Gaussian fit and the least-squares fit draw the same conditional-mean line. They do not estimate the same object. Least squares does not model the marginal distribution of X, and it does not by itself supply the conditional variance s_{YY}-s_{YX}^2/s_{XX}. The major axis of the fitted ellipse is a different line, and so is the line obtained by regressing X on Y and solving for Y. The laboratory checks the agreement of the first two lines on Iris setosa.
Used in
See also
Joint distribution and density · Mean vector and covariance matrix · Correlation · Multivariate normal distribution · Squared Mahalanobis distance · Linear images of a normal vector · Conditional distribution of a multivariate normal · Partial correlation