Empirical Bayes
Definition taught in Hierarchical Bayes, part of Bayesian statistics
Empirical Bayes
| Type | Definition |
| Taught in | Hierarchical Bayes |
| Part | Bayesian statistics |
| Properties | 1 |
| Worked examples | 1 |
An empirical Bayes analysis estimates prior or population-level hyperparameters from data and then treats those estimates as fixed in a lower-level Bayesian update. A full Bayesian analysis instead assigns the hyperparameters a prior and propagates their posterior uncertainty.
Properties
Normal hierarchical shrinkage
Main article: Normal hierarchical shrinkage
The final term is constant in \theta_j. The remaining kernel is normal, so \theta_j\mid\overline y_j,\mu,\tau^2,\sigma^2 \sim\operatorname{N}\!\left( \frac{B_j}{A_j},\frac1{A_j}\right). Its variance is \begin{align*} \frac1{A_j} &=\frac1{n_j/\sigma^2+1/\tau^2}\\ &=\frac{\sigma^2\tau^2}{\sigma^2+n_j\tau^2}, \end{align*} and its mean can be rearranged without omitting any terms: \begin{align*} \frac{B_j}{A_j} &=\frac{n_j\overline y_j/\sigma^2+\mu/\tau^2} {n_j/\sigma^2+1/\tau^2}\\ &=\frac{n_j\tau^2\overline y_j+\sigma^2\mu} {\sigma^2+n_j\tau^2}\\ &=\frac{n_j\tau^2}{\sigma^2+n_j\tau^2}\overline y_j +\frac{\sigma^2}{\sigma^2+n_j\tau^2}\mu\\ &=w_j\overline y_j+(1-w_j)\mu, \end{align*} where w_j=\frac{n_j\tau^2}{\sigma^2+n_j\tau^2}, \qquad 1-w_j=\frac{\sigma^2}{\sigma^2+n_j\tau^2}. Thus
\mathbb{E}(\theta_j\mid\overline y_j,\mu,\tau^2,\sigma^2) =w_j\overline y_j+(1-w_j)\mu.
Examples
Large and small tutorial classes
Worked example from Hierarchical Bayes
Suppose the overall mean is \mu=70, the within-class standard deviation is \sigma=10, and the between-class standard deviation is \tau=5. Class A has 100 students and \overline y_A=75, while Class B has only two students and \overline y_B=95.
Question. Compute the two shrinkage weights and posterior means, treating \mu, \sigma^2, and \tau^2 as fixed, and explain why the more extreme sample mean need not produce the more extreme posterior mean.
Solution. For Class A, \begin{align*} w_A&=\frac{100(25)}{100+100(25)}=\frac{25}{26}\approx0.962,\\ \mathbb{E}(\theta_A\mid\overline y_A,\mu,\tau^2,\sigma^2) &=\frac{25}{26}(75)+\frac1{26}(70)\approx74.81. \end{align*} For Class B, \begin{align*} w_B&=\frac{2(25)}{100+2(25)}=\frac13,\\ \mathbb{E}(\theta_B\mid\overline y_B,\mu,\tau^2,\sigma^2) &=\frac13(95)+\frac23(70)\approx78.33. \end{align*} The extreme average from the small class is not ignored, but it is recognised as much less precise.
Used in
- Taught in Hierarchical Bayes, where the derivation and examples are.
- Also used in Reference formulas and problem-solving workflow
- All 9 questions on Hierarchical Bayes
See also
Hyperparameter · Hierarchical Bayesian model · Normal hierarchical shrinkage · Exchangeability · Study-rate full conditional · Marginal hyperposterior · Uniformity depends on a domain · Grid Monte Carlo approximation