Sensitivity and specificity
Topic of Estimating error rates, part of Statistical learning
Sensitivity and specificity
| Type | Topic |
| Taught in | Estimating error rates |
| Part | Statistical learning |
Sensitivity is the proportion of truly positive cases that a classifier labels positive; specificity is the proportion of truly negative cases it labels negative. Both are read off a confusion matrix, and moving a decision threshold trades one against the other.
In Estimating error rates
Definition 1 (Confusion matrix) A confusion matrix has, in position (i,j), the number of observations of true class i that the classifier predicted as class j. Dividing the matrix by the number of observations in the test set, the sum of the off-diagonal elements is \widehat{\mathrm{Err}}. Dividing row g by its own total, the sum of that row excluding the diagonal element is \widehat{\mathrm{Err}}_g.
Example 1 (Reading a confusion matrix) A three-class classifier is evaluated on 150 test observations, 50 from each class.
| predicted 1 | predicted 2 | predicted 3 | row total | |
|---|---|---|---|---|
| true 1 | 40 | 5 | 5 | 50 |
| true 2 | 4 | 38 | 8 | 50 |
| true 3 | 2 | 6 | 42 | 50 |
Question. Compute the overall and class-specific error rates.
Solution. The off-diagonal total is 5+5+4+8+2+6=30, so \widehat{\mathrm{Err}}=30/150=0.20. By rows, \widehat{\mathrm{Err}}_1=10/50=0.20, \widehat{\mathrm{Err}}_2=12/50=0.24, and \widehat{\mathrm{Err}}_3=8/50=0.16. The class sizes are equal, so
\widehat{\mathrm{Err}}=\tfrac13(0.20+0.24+0.16)=0.20,
which is \sum_g\hat\pi_g\widehat{\mathrm{Err}}_g with \hat\pi_g=1/3. The largest confusion is class 2 predicted as class 3, eight cases.
Used in
See also
Confusion matrix · Fixed-rule error · Apparent error · Leave-one-out cross-validation · -fold cross-validation · Bootstrap error and the correction · Jackknife bias and standard error · A held-out test set