Choosing an error-rate estimator
Section of Estimating error rates, part of Statistical learning
Choosing an error-rate estimator
| Type | Section |
| Taught in | Estimating error rates |
| Part | Statistical learning |
Choose an assessment design by naming its target and respecting the sampling structure. An untouched independent test set gives a direct estimate for a fixed fitted rule, and the binomial interval above covers that conditional error. Cross-validation uses the available observations more extensively, at the cost of dependent scores from several fits. If model selection is part of training, repeat it inside the assessment loop. No choice of five, ten, or n folds compensates for leaking the assessed cases into that selection.
Used in
See also
Fixed-rule error · Confusion matrix · Apparent error · Leave-one-out cross-validation · -fold cross-validation · Bootstrap error and the correction · Jackknife bias and standard error · A held-out test set