Classification accuracy
Topic of Estimating error rates, part of Statistical learning
Classification accuracy
| Type | Topic |
| Taught in | Estimating error rates |
| Part | Statistical learning |
| Questions using it | 10 |
The proportion of cases classified correctly, one minus the error rate. With unbalanced classes it can be high for a rule that never predicts the rare class, and the accuracy of a model chosen as the best of many is optimistic unless it is assessed on separate data.
In Estimating error rates
The cases that select k are not the cases that assess the selected model. If one cross-validation error both picks the best of many candidates and is then reported as that candidate’s accuracy, the report is optimistic, because the minimum of many noisy estimates is systematically one of the estimates that happened to be low. Step 5 rebuilds the choice of k without a test fold, and step 6 trains the final classifier on every observation at that chosen k. The nested average estimates the error of this whole procedure, including the selection, not the error of whichever k looked best inside one split.
Used in
- Taught in Estimating error rates
- Also used in Model selection for stochastic models · Statistical classifiers and discriminant analysis · Nearest neighbours · Ensembles, random forests, and boosting · Neural units · Networks in practice · Support vector machines · High-dimensional analysis · Graphical models
- Questions that use it (10) — Separate tuning from evaluation · Tuning a nearest-neighbour classifier · What an accuracy table does and does not say · Affine invariance (answer) · A tempting wrong turn (answer) · Online versus batch (answer) · Sum of squares versus accuracy · A tempting wrong turn · When ordinary linear discrimination is not defined (answer) · Principal components before linear discrimination (answer)
- All 7 questions on Estimating error rates
See also
Fixed-rule error · Confusion matrix · Apparent error · Leave-one-out cross-validation · -fold cross-validation · Bootstrap error and the correction · Jackknife bias and standard error · A held-out test set