A held-out test set

Section of Estimating error rates, part of Statistical learning

A held-out test set

Type Section
Taught in Estimating error rates
Part Statistical learning
Questions using it 10

The apparent rate fails because the cases used to estimate the error helped to build the rule. Holding labelled cases out of the fit removes that dependence. The training set builds the classifier and the test set estimates the error. This is the test-set, or holdout, method. It raises four questions.

  1. Which classifier is the error rate being estimated for?
  2. What proportion of the data is allocated to the test set?
  3. How is that proportion implemented?
  4. How can a confidence interval for the error rate be provided?

An independent test sample evaluates the particular fit trained without it. If the rule is subsequently refit on the training data plus the test data, the rule has changed. The old score is not an independent assessment of that new fit. It may be a useful guide to a procedure trained on slightly more data, and that requires a judgement about how performance changes with training size.

Choosing the split is a trade-off. More training observations may improve the rule, while more test observations make its estimated error more precise. Improvement with training size is not a universal monotone theorem. Choose a split large enough to assess the errors that matter, especially inside rare classes, and report the training and test sizes alongside the score.

If the test cases are independent draws and the trained rule is held fixed, the number of errors on the test set is a binomial count. A confidence interval for a binomial proportion then covers the error of that particular trained rule. Its width is proportional to 1/\sqrt{n_{\mathrm{test}}}, which is why a small test set, however carefully held out, still gives a coarse estimate. The interval does not cover the extra uncertainty that comes from having trained the rule on one sample rather than another.

For independent identically distributed observations, a random split keeps a file’s sorting order from determining the assessment. It does not repair dependence. Repeated measurements from the same person require group-based splits when the target is a new person. Prediction of a future time requires a time-ordered split. The sampling unit and the deployment target determine what must be held out together.

Under the independent identically distributed setup, the empirical test error is

\widehat{\mathrm{Err}}_{\mathrm{test}} =\frac{1}{n_{\mathrm{test}}} \sum_{i=1}^{n_{\mathrm{test}}} Q\bigl(y_i,\eta_{(X_{\mathrm{train}},y_{\mathrm{train}})}(x_i)\bigr),

and the class-specific rates divide the errors inside class g by the number of test cases from that class,

\widehat{\mathrm{Err}}_{\mathrm{test},g} =\frac{\sum_{i=1}^{n_{\mathrm{test}}} Q\bigl(y_i,\eta_{(X_{\mathrm{train}},y_{\mathrm{train}})}(x_i)\bigr)\,I(y_i,g)} {\sum_{l=1}^{n_{\mathrm{test}}} I(y_l,g)}.

Example 1 (How many test observations?) Suppose a classifier makes 18 errors on n_{\mathrm{test}}=120 test observations, so \widehat{\mathrm{Err}}_{\mathrm{test}}=0.15.

Question. Give an approximate 95\% interval, and find how many test observations would bring the half-width down to 0.03 if the error stays near 0.15.

Solution. Treating the count as binomial, the standard error is

\sqrt{\frac{0.15\times 0.85}{120}}=\sqrt{0.0010625}=0.0326.

The approximate 95\% interval is 0.15\pm 1.96(0.0326)=(0.086,0.214), far from precise. The half-width formula 1.96\sqrt{\hat p(1-\hat p)/n} equals 0.03 when

n=\frac{1.96^2\,(0.15)(0.85)}{0.03^2} =\frac{3.8416\times 0.1275}{0.0009} \approx 544,

so about 545 test observations. Halving the half-width multiplies the required test size by about four, because the width scales with 1/\sqrt{n_{\mathrm{test}}}.

Used in

See also

Confusion matrix · Overfitting · Validation set · Fixed-rule error · Apparent error · Leave-one-out cross-validation · -fold cross-validation · Bootstrap error and the correction

Glossary and index