LDA and principal components

Topic: Statistical classifiers and discriminant analysis · more questions on this topic
‹ The fitted LDA model

Give mathematical explanations, defining the quantities you use.

  1. Explain the two-class LDA model and how its parameters and classification rule are obtained from labelled observations.

  2. Explain the main concepts and mathematical construction of principal component analysis, including principal directions, scores, and variance explained. The derivation is on Principal component analysis.

  1. State \mathbb{P}(Y=k)=\pi_k and X\mid Y=k\sim\operatorname{N}_p(\boldsymbol\mu_k,\Sigma) with a shared covariance. Estimate the means from the class samples, the common covariance from the pooled within-class scatter, and the priors from suitable class proportions or from an external prevalence. Apply Bayes’ rule and choose the largest posterior under equal losses. Equivalent discriminants are

\delta_k(\mathbf{x}) =\mathbf{x}^\top\widehat\Sigma^{-1}\widehat{\boldsymbol\mu}_k -\tfrac12\widehat{\boldsymbol\mu}_k^\top\widehat\Sigma^{-1}\widehat{\boldsymbol\mu}_k +\log\widehat\pi_k.

The common quadratic terms cancel, so the boundary is linear.

  1. Centre the predictors, and standardise them when the units require it. Compute the eigendecomposition of the covariance, or of the correlation matrix if the columns were standardised. The principal directions are orthonormal eigenvectors ordered by decreasing eigenvalue. They arise by maximising projected variance subject to a unit-length constraint and to orthogonality with the directions already chosen. The scores are the projections of the centred observations onto these directions. The eigenvalues are the variances of those scores, so the ratio of an eigenvalue to the trace is the proportion of variance explained by that component. The construction uses the predictors alone. It does not use a response label.