Overfitting
Topic of Networks in practice, part of Statistical learning · Also in Estimating error rates
Overfitting
| Type | Topic |
| Taught in | Networks in practice · Estimating error rates |
| Part | Statistical learning |
| Questions using it | 2 |
Fitting the training data so closely that noise is learned as if it were structure: the training error keeps falling while the error on new data rises. Regularisation, early stopping and tuning on held-out data guard against it.
In Networks in practice
From Networks in practice
Overfitting. The model is a very good fit to the training data. It may have modelled some useful structure and has also modelled many irrelevant aspects. This is common with overly complex models.
In Estimating error rates
Definition 1 (Apparent error) The apparent error scores a rule on the cases that trained it:
\mathrm{err} =\frac1n\sum_{i=1}^n Q\bigl(y_i,\eta_{(X,y)}(x_i)\bigr),
where \eta_{(X,y)} is the classifier fitted to the whole observed sample (X,y)=\{(x_i,y_i)\}_{i=1}^n.
As an estimator of the fixed-rule target \mathrm{Err}, the bias is
\mathbb{E}(\mathrm{err}-\mathrm{Err}).
The same pairs (x_i,y_i) enter the fitted rule and the average that scores it, so the summands are not draws from the conditional law of Q(Y,\eta(X)) given that rule. The dependence pulls the average down. The apparent error describes the fit to the training cases, and it is usually optimistic as an estimate of future error. Optimism is a repeated-sampling tendency. It is not a promise that every observed training error sits below every test error.
The bias is exact for one neighbour. Suppose the inputs x_i are distinct and \eta is 1-nearest neighbour. The nearest training point to x_i is x_i itself, so the rule returns y_i, the loss Q(y_i,\eta(x_i)) is 0, and \mathrm{err}=0 on every sample. Then
\mathbb{E}(\mathrm{err}-\mathrm{Err})=\mathbb{E}(0-\mathrm{Err})=-\mathbb{E}(\mathrm{Err})\le 0.
The apparent error is biased downward by the expected true error of the fitted rule, regardless of how useful the neighbours of a new point will be. That is why a training error of zero, for this rule or for a tree grown until every leaf is pure, still needs an assessment on cases the rule did not use.
Parameter count alone does not give the size of the optimism. Stability, regularisation, signal strength, and the geometry of the predictors all matter. A low-dimensional fit can be unstable when the information is weak. A zero training error needs an independent assessment.
Used in
- Taught in Networks in practice · Estimating error rates
- Also used in Statistical classifiers and discriminant analysis · Estimating error rates · Ensembles, random forests, and boosting · Neural units · High-dimensional analysis
- Questions that use it (2) — Choosing model complexity · Counting for NETtalk
- All 7 questions on Networks in practice
See also
Training set · Regularisation · Early stopping · A held-out test set · Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation