Validation set
Topic of Networks in practice, part of Statistical learning · Also in Estimating error rates
Validation set
| Type | Topic |
| Taught in | Networks in practice · Estimating error rates |
| Part | Statistical learning |
| Questions using it | 10 |
Data held out from fitting and used to choose tuning parameters or to decide when to stop training. Because it is used for choices, it is not an honest estimate of final performance; a separate test set, untouched until the end, is.
In Networks in practice
From Networks in practice
To control generalisation the data are divided into a training set, a validation set, and a test set.
In Estimating error rates
Cross-validation as written above estimates the error of a rule whose form is already fixed. Choosing a tuning constant \lambda by the same errors, and then reporting the smallest of those errors as the error of the chosen rule, uses each case twice: once to pick \lambda, and again to advertise the pick. The advertised number is optimistically biased.
As a numerical sketch, the test errors across \lambda=1,\ldots,10 may have a best value of 0.18. That 0.18 was the criterion that selected \lambda, so among noisy estimates it is the one that fell furthest below the truth. A validation set may choose \lambda. A test set that did not enter the choice estimates the error. The optimism is a tendency over repeated samples, not an inequality that every split obeys: one test score can still come out smaller than the selected validation score. Nested cross-validation is this separation when the sample is too small for three permanent blocks.
Nested cross-validation places the complete selection procedure inside an outer assessment loop. This is the separation studied by Cawley and Talbot (2010). The inner winner can differ between outer folds. In the construction below, the remaining m-1 outer folds also serve as inner folds.
- Split the data into m folds. Of those, m-2 are for training, one is for validation, and one is for testing.
- Using the training and validation data, perform (m-1)-fold cross-validation for each candidate k=1,\ldots,K. Estimate the overall error for each k from the cross-validation results, and choose the k that gives the lowest error rate.
- With this k=k_1^*, train the classifier on the training and validation data and test it on the test fold.
- Repeat for the other choices of test fold, m times in all, getting possibly different values k_2^*,\ldots,k_m^* each time. Average the results to estimate the error rate of the procedure on future observations.
- For the final classifier, choose a new k=k_{\mathrm{final}}^* by a similar procedure that uses only training and validation, with no test fold held out of the selection.
- Train the final classifier on all the data using k=k_{\mathrm{final}}^*.
The cases that select k are not the cases that assess the selected model. If one cross-validation error both picks the best of many candidates and is then reported as that candidate’s accuracy, the report is optimistic, because the minimum of many noisy estimates is systematically one of the estimates that happened to be low. Step 5 rebuilds the choice of k without a test fold, and step 6 trains the final classifier on every observation at that chosen k. The nested average estimates the error of this whole procedure, including the selection, not the error of whichever k looked best inside one split.
The same pattern appears when k for nearest neighbours is chosen on one sample and then scored on another. Training error falls toward zero as k falls to 1. A ten-fold cross-validation curve, and the error on a large independent draw, stay above that training curve and prefer a larger neighbourhood. The laboratory at the end of this page is that separation on a small scale.
Used in
- Taught in Networks in practice · Estimating error rates
- Also used in Principal component analysis · Hierarchical clustering and kernel density estimation · Estimating error rates · Ensembles, random forests, and boosting
- Questions that use it (10) — A tempting wrong turn (answer) · Costs of cross-validation (answer) · Separate tuning from evaluation · A tempting wrong turn (answer) · Normalisation and nearest neighbours · Tuning a nearest-neighbour classifier · A tempting wrong turn (answer) · Decision trees and validation · Sum of squares versus accuracy · Complexity and early stopping
- All 7 questions on Networks in practice
See also
A held-out test set · Early stopping · Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network