Training set
Topic of Estimating error rates, part of Statistical learning · Also in Networks in practice
Training set
| Type | Topic |
| Taught in | Estimating error rates · Networks in practice |
| Part | Statistical learning |
| Questions using it | 15 |
The data used to fit a model. Error measured on the same data, the apparent error, is optimistic because those cases helped choose the rule. An honest assessment uses cases excluded from every decision that built the predictor, including preprocessing and tuning.
In Estimating error rates
The classifier has been fitted. The question is how often it fails on a new observation from the population that matters. Scoring the training sample is easy, and those observations helped choose the rule. An honest assessment scores observations that were excluded from every decision that built the predictor, including preprocessing and tuning.
In Networks in practice
From Networks in practice
To control generalisation the data are divided into a training set, a validation set, and a test set.
- The training set is used to optimise the weights.
- The validation set is monitored during or after training to choose when to stop, the number of nodes or weights, the learning rate, and the other tuning choices.
- The test set is used to assess how well the final model performs.
This is the same discipline as nested cross-validation. Anything chosen by looking at data is part of model building and must be assessed on data that played no part. Early stopping uses the validation error: stop at the epoch where it is smallest.
Used in
- Taught in Estimating error rates · Networks in practice
- Also used in Principal component analysis · Hierarchical clustering and kernel density estimation · Statistical classifiers and discriminant analysis · Nearest neighbours · Classification trees · Ensembles, random forests, and boosting · Backpropagation · Networks in practice · Support vector machines · High-dimensional analysis
- Questions that use it (15) — Principal components as a modelling step (answer) · A tempting wrong turn · Parameter counts · The fitted LDA model · Costs of cross-validation (answer) · A tempting wrong turn (answer) · Separate tuning from evaluation (answer) · Neighbours by hand · Normalisation and nearest neighbours (answer) · Tuning a nearest-neighbour classifier (answer) · What an accuracy table does and does not say (answer) · Decision trees and validation (answer) · Counting for NETtalk · A tempting wrong turn (answer) · Principal components before linear discrimination (answer)
- All 7 questions on Estimating error rates
See also
Apparent error · AdaBoost · Bagging · Bootstrap error and the correction · Cover–Hart bound · Epoch · Mini-batch · Overfitting