A confusion matrix

Topic: Estimating error rates · more questions on this topic
Test-set size ›

A two-class classifier tested on 200 observations (120 from class 1, 80 from class 2) gives the confusion matrix

\begin{pmatrix} 108 & 12 \\ 16 & 64 \end{pmatrix}

(rows true, columns predicted). Compute \widehat{\mathrm{Err}}, \widehat{\mathrm{Err}}_1, \widehat{\mathrm{Err}}_2, and check \widehat{\mathrm{Err}}=\sum_g\hat\pi_g\widehat{\mathrm{Err}}_g using the class proportions in the test set.

Worked answer (click to show)

Overall error uses the full sample size. A class-specific error uses its own class denominator. The off-diagonal total is 12+16=28, so \widehat{\mathrm{Err}}=28/200=0.14. By rows, \widehat{\mathrm{Err}}_1=12/120=0.10 and \widehat{\mathrm{Err}}_2=16/80=0.20. The test-set class proportions are 120/200=0.6 and 80/200=0.4, and

0.6(0.10)+0.4(0.20)=0.06+0.08=0.14,

which equals \widehat{\mathrm{Err}}.