Least squares
Section of Linear regression, part of Statistical inference
Least squares
| Type | Section |
| Taught in | Linear regression |
| Part | Statistical inference |
| Questions using it | 2 |
Definition 1 (Least-squares line) For points (x_1,y_1),\ldots,(x_n,y_n) with the x_i not all equal, the least-squares line y=b_0+b_1x minimises
L(b_0,b_1)=\sum_{i=1}^n (y_i-b_0-b_1 x_i)^2.
The minimiser is
b_1=\frac{\sum_{i=1}^n(x_i-\overline x)(y_i-\overline y)}{\sum_{i=1}^n(x_i-\overline x)^2}, \qquad b_0=\overline y-b_1\overline x.
Setting \partial L/\partial b_0=0 and \partial L/\partial b_1=0 gives the two normal equations \sum(y_i-b_0-b_1x_i)=0 and \sum x_i(y_i-b_0-b_1x_i)=0. The first gives b_0=\overline y-b_1\overline x, so the line passes through (\overline x,\overline y). Substituting it into the second gives the formula for b_1. The worked answer to the exercise below fills in the algebra.
Example 1 (Advertising and sales) A company records monthly advertising spending x and sales y, both in units of 10 000 dollars.
| Month | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Spending x | 1.2 | 0.8 | 1.0 | 1.3 | 0.7 | 0.8 | 1.0 | 0.6 | 0.9 | 1.1 |
| Sales y | 101 | 92 | 110 | 120 | 90 | 82 | 93 | 75 | 91 | 105 |
Question. Find the least-squares line.
Solution. \overline x=0.94 and \overline y=95.9. The sums are \sum(x_i-\overline x)^2=0.444 and \sum(x_i-\overline x)(y_i-\overline y)=23.34, so b_1=23.34/0.444=52.6 and b_0=95.9-52.6\times0.94=46.5. The fitted line is y=46.5+52.6\,x: each extra 10 000 dollars of advertising goes with about 526 000 dollars of extra sales, in these data. The line fits a scatter of past months. It does not say that spending more would cause more sales.
The line is best only for squared vertical deviations. Minimising absolute deviations gives a different line, less sensitive to outliers. Minimising horizontal deviations, to predict x from y, gives yet another. Squared deviations are used because the minimiser has a closed form, and because, under a normal model, least squares is maximum likelihood. If the line must pass through the origin, y=bx, the same calculation gives b=\sum x_iy_i/\sum x_i^2.
Curved relationships
Main article: Curved relationships
Not every relationship is a straight line. Two remedies keep the least-squares machinery.
The first fits a polynomial, y=b_0+b_1x+\cdots+b_kx^k. The sum of squares is still quadratic in the coefficients, so setting its k+1 partial derivatives to zero gives k+1 linear equations. The model is still linear in its parameters, even though the curve is not a line.
The second transforms the variables. If theory says y=ae^{bx}, then \log y=\log a+bx is linear in x, and a line is fitted to the points (x_i,\log y_i). The same idea linearises y=ax^b, on the log–log scale, and a logistic curve, on the logit scale. The fitted slope is then a slope on the transformed scale, and so are the errors.
Example 2 (Melanoma and latitude) The melanoma rate per 100 000 white males in nine US locations is
| Latitude | 32.8 | 33.9 | 34.1 | 37.9 | 40.0 | 40.8 | 41.7 | 42.2 | 45.0 |
|---|---|---|---|---|---|---|---|---|---|
| Rate | 9.0 | 5.9 | 6.6 | 5.8 | 5.5 | 3.0 | 3.4 | 3.1 | 3.8 |
Question. Compare a straight line, a quadratic and a line for the log rate.
Solution. The straight line is y=20.6-0.399\,x, but the points bend: the southernmost location is far above it. A quadratic fits the bend, y=66.4-2.81\,x+0.0314\,x^2, at the cost of a coefficient with no simple meaning. The log rate is close to linear, \log y=4.51-0.0761\,x, which says the rate falls by a factor e^{-0.0761}=0.93, about 7%, for each degree further north. A constant percentage change per degree is often the more natural reading for a rate.
Used in
- Taught in Linear regression
- Also used in Functions of random variables and limit theorems · Regression examples · Bayesian linear regression · Introduction to generalised linear models · Covariance and the multivariate normal · Logistic regression · High-dimensional analysis · Graphical models
- Questions that use it (2) — Ridge makes the matrix invertible (answer) · C19
- All 5 questions on Linear regression
See also
Ordinary least squares · Least-squares line · Simple linear regression · Multiple linear regression · Gaussian linear model · Normal linear model · High-dimensional data · Ridge regression · Least-squares estimators are unbiased · Sampling distribution of the regression estimators