Fisher’s direction

Topic: Statistical classifiers and discriminant analysis · more questions on this topic
‹ Parameter counts The fitted LDA model ›

For the data of the two-class LDA example, compute Fisher’s direction S^{-1}(\widehat{\boldsymbol\mu}_1-\widehat{\boldsymbol\mu}_2) and the value of the separation ratio for it and for the naive direction \widehat{\boldsymbol\mu}_1-\widehat{\boldsymbol\mu}_2. Which is larger?

Worked answer (click to show)

The optimal direction uses the covariance to reweight the mean difference. Compare the same scale-invariant separation ratio for each proposed direction. Here \mathbf{d}=\widehat{\boldsymbol\mu}_1-\widehat{\boldsymbol\mu}_2=(-3,-2)^\top and

S^{-1}\mathbf{d}=(-1.5,-1.5)^\top\propto(1,1)^\top.

For \mathbf{a}=(1,1)^\top, one has \mathbf{a}^\top\mathbf{d}=-5 and

\mathbf{a}^\top S\mathbf{a} =\tfrac43+2\cdot\tfrac23+\tfrac23 =\tfrac{10}{3}, \qquad \frac{(-5)^2}{2\cdot 10/3} =\frac{25}{20/3} =3.75.

For the naive direction \mathbf{a}=\mathbf{d}, one has \mathbf{a}^\top\mathbf{d}=13 and

\mathbf{a}^\top S\mathbf{a} =9\cdot\tfrac43+2\cdot 6\cdot\tfrac23+4\cdot\tfrac23 =22.67, \qquad \frac{169}{2\times 22.67} =\frac{169}{45.33} =3.728.

Fisher’s direction attains the maximum \mathbf{d}^\top S^{-1}\mathbf{d}/2=7.5/2=3.75. The naive direction is slightly lower because the mean difference is already nearly along the leading axis of S.