Bayes classifier
Section of Statistical classifiers and discriminant analysis, part of Statistical learning
Bayes classifier
| Type | Section |
| Taught in | Statistical classifiers and discriminant analysis |
| Part | Statistical learning |
| Questions using it | 3 |
The class density f_g(\mathbf{x}) answers how the measurements are distributed once the class is known. The measurements are observed and the class is what is wanted. A rare class can have a high density at \mathbf{x} and still be unlikely. Bayes’ theorem is the conversion, with the class in the role of the discrete variable and the measurement vector in the role of the continuous one. If the joint distribution of (X,Y) were known, the conversion would be exact:
\mathbb{P}(Y=g\mid X=\mathbf{x}) =\frac{f_g(\mathbf{x})\,\pi_g}{f_X(\mathbf{x})},
where f_g(\mathbf{x}) is the density of class g and f_X(\mathbf{x}) is the marginal density of the measurements. The marginal is the mixture of the class densities,
f_X(\mathbf{x})=\sum_{g=1}^{G}\pi_g f_g(\mathbf{x}).
That is the mixture on Mixtures and the EM algorithm, with one difference that matters for estimation: here the component label is observed on every training case, so the fit does not have to infer which component produced which point. Each class density below is a multivariate normal distribution, possibly after a mixture or a kernel has been substituted for that normal.
Definition 1 (Posterior class probability) The posterior probability of class g given \mathbf{x} is
\tau_g(\mathbf{x}) =\frac{\pi_g f_g(\mathbf{x})}{\sum_{g'=1}^{G}\pi_{g'} f_{g'}(\mathbf{x})}.
The class that this posterior says is most probable is
B(\mathbf{x})=\arg\max_{g\in\{1,\ldots,G\}}\tau_g(\mathbf{x}) =\arg\max_{g\in\{1,\ldots,G\}}\pi_g f_g(\mathbf{x}).
The second equality holds because the marginal density f_X(\mathbf{x}) is the same for every class, so it cannot change which class wins. When the densities and the priors are the true ones, this rule has the lowest expected zero–one risk. The plug-in rule keeps the same comparison and replaces each unknown quantity by an estimate from the training sample:
\eta^{(X_{\mathrm{all}},\mathbf{y}_{\mathrm{all}})}(\mathbf{x}) =\arg\max_{g}\widehat\tau_g(\mathbf{x}) =\arg\max_{g}\widehat\pi_g\widehat f_g(\mathbf{x}).
Used in
- Taught in Statistical classifiers and discriminant analysis
- Also used in Estimating error rates · Logistic regression · Nearest neighbours · Classification trees · Ensembles, random forests, and boosting
- Questions that use it (3) — A Bayes rule in one dimension · Unequal costs · The factor of two
- All 8 questions on Statistical classifiers and discriminant analysis
See also
Bayes’ rule · Posterior class probability · Posterior membership · Bayes-optimal rule · Bayes action for two hypotheses · Bayes consistency of -nearest neighbours · Cover–Hart bound · Training data and classifier · Zero–one loss and risk