Jackknife
Section of Estimating error rates, part of Statistical learning
Jackknife
| Type | Section |
| Taught in | Estimating error rates |
| Part | Statistical learning |
| Questions using it | 1 |
Leave-one-out cross-validation scores a classifier. The jackknife is the older leave-one-out idea, built to estimate the bias (Quenouille, 1949, 1956) and the standard error (Tukey, 1958) of a parameter estimator rather than an error rate. The difficulty it addresses is that the bias of \hat\theta is \mathbb{E}(\hat\theta)-\theta, and \theta is unknown, so the bias cannot be computed by subtracting the truth. The jackknife manufactures a comparison inside the sample.
The ith jackknife sample leaves out observation i. Estimate \theta on each of these samples, giving \hat\theta_{(i)} for i=1,\ldots,n, and estimate it on the full sample, giving \hat\theta. The average of the leave-one-out estimates is
\hat\theta_{(\cdot)}=\frac1n\sum_{i=1}^n\hat\theta_{(i)}.
Definition 1 (Jackknife bias and standard error) \widehat{\mathrm{bias}}_{\mathrm{jack}} =(n-1)\bigl(\hat\theta_{(\cdot)}-\hat\theta\bigr), \qquad \widehat{\mathrm{se}}_{\mathrm{jack}} =\Biggl(\frac{n-1}{n}\sum_{i=1}^n\bigl(\hat\theta_{(i)}-\hat\theta_{(\cdot)}\bigr)^2\Biggr)^{1/2}.
The bias-corrected estimate is \hat\theta-\widehat{\mathrm{bias}}_{\mathrm{jack}}, which rearranges to n\hat\theta-(n-1)\hat\theta_{(\cdot)}.
The factor n-1 comes from a leading bias term a/n. Suppose
\mathbb{E}\hat\theta=\theta+\frac{a}{n}+o(n^{-1}).
Each leave-one-out estimator is the same functional on n-1 observations, so \mathbb{E}\hat\theta_{(i)}=\theta+a/(n-1)+o(n^{-1}) and the same expansion holds for the average \hat\theta_{(\cdot)}. The gap between the two expectations is
\mathbb{E}\bigl(\hat\theta_{(\cdot)}-\hat\theta\bigr) =\frac{a}{n-1}-\frac{a}{n}+o(n^{-1}) =\frac{a}{n(n-1)}+o(n^{-1}).
Multiplying by n-1 recovers a/n, the leading term of the bias of the full-sample estimator. This explains the first-order correction. It does not mean that every realised jackknife bias estimate equals the true bias. Smooth estimators are the natural setting. Nonsmooth statistics can behave differently.
Example 1 (Jackknife bias of the maximum likelihood variance) Let the data be 2,4,4,6,9, so n=5, and let
\hat\theta=\frac1n\sum_{i=1}^n(x_i-\bar x)^2,
the maximum likelihood variance estimator. For a normal sample this estimator is biased downward by the factor (n-1)/n: if s^2=\sum(x_i-\bar x)^2/(n-1) is the unbiased sample variance, then \mathbb{E}\hat\theta=((n-1)/n)\sigma^2.
Question. Compute the jackknife bias, the bias-corrected estimate, and the jackknife standard error.
Solution. The sample mean is \bar x=(2+4+4+6+9)/5=5. The squared deviations are 9,1,1,1,16, which sum to 28, so \hat\theta=28/5=5.6. The unbiased variance is 28/4=7.
Deleting 2 leaves 4,4,6,9, with mean 5.75. The squared deviations from 5.75 sum to 2(1.75)^2+(0.25)^2+(3.25)^2=16.75, and 16.75/4=4.1875. Deleting 9 leaves 2,4,4,6, with mean 4 and squared deviations 4,0,0,4, so the estimator is 8/4=2. Deleting 4, the other 4, or 6 each gives 6.6875. The five leave-one-out values, in the order of the deleted points 2,4,4,6,9, are
\hat\theta_{(i)}=4.1875,\;6.6875,\;6.6875,\;6.6875,\;2.0.
Their average is \hat\theta_{(\cdot)}=26.25/5=5.25. The jackknife bias is
\widehat{\mathrm{bias}}_{\mathrm{jack}}=4(5.25-5.6)=4(-0.35)=-1.4,
negative, as the factor (n-1)/n predicts. The bias-corrected estimate is
\hat\theta-\widehat{\mathrm{bias}}_{\mathrm{jack}}=5.6-(-1.4)=7.0,
which equals the unbiased sample variance 28/4=7. Equivalently, n\hat\theta-(n-1)\hat\theta_{(\cdot)}=5(5.6)-4(5.25)=28-21=7.
The deviations of the leave-one-out values from 5.25 are -1.0625, 1.4375, 1.4375, 1.4375, and -3.25. The sum of their squares is 17.890625. Then
\widehat{\mathrm{se}}_{\mathrm{jack}} =\sqrt{\frac45\times 17.890625} =\sqrt{14.3125} =3.78.
Used in
- Taught in Estimating error rates
- Questions that use it (1) — Jackknife of the mean
- All 7 questions on Estimating error rates
See also
Fixed-rule error · Confusion matrix · Apparent error · Leave-one-out cross-validation · -fold cross-validation · Bootstrap error and the correction · Jackknife bias and standard error · A held-out test set