Test-set size

Topic: Estimating error rates · more questions on this topic
‹ A confusion matrix The 0.632 rule ›

For an error rate near 0.10, how many test observations are needed so that the approximate 95\% binomial interval has half-width 0.02?

Worked answer (click to show)

The half-width equation, for independent test outcomes and an error probability near the planning value, is

1.96\sqrt{\frac{p(1-p)}{n}}=0.02.

With p=0.10,

n=\frac{1.96^2\,(0.10)(0.90)}{0.02^2} =\frac{3.8416\times 0.09}{0.0004} =864.36,

so about 865 observations. The requirement grows like the inverse square of the half-width.