Likelihood and reference prior
Section of Bayesian linear regression, part of Bayesian statistics
Likelihood and reference prior
| Type | Section |
| Taught in | Bayesian linear regression |
| Part | Bayesian statistics |
| Questions using it | 1 |
Conditional independence in Equation (1) in Bayesian linear regression gives \begin{aligned} p({\boldsymbol{y}}\mid{\boldsymbol{\beta}},\sigma^2,{\color{blue}\boldsymbol{X}}) &=\prod_{i=1}^n \frac1{\sqrt{2\pi\sigma^2}} \exp\!\left\{-\frac{(y_i-{\boldsymbol{x}}_i^\top{\boldsymbol{\beta}})^2} {2\sigma^2}\right\}\\ &=(2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac{\operatorname{SSE}({\boldsymbol{\beta}})} {2\sigma^2}\right\}. \end{aligned} The standard improper reference prior is p({\boldsymbol{\beta}},\sigma^2)\propto\frac1{\sigma^2}, \qquad {\boldsymbol{\beta}}\in\mathbb{R}^p,\quad\sigma^2>0. It is flat in {\boldsymbol{\beta}} and flat in \log\sigma^2, since \,\mathrm{d}(\log\sigma^2)=\,\mathrm{d}\sigma^2/\sigma^2. Neither factor integrates to one, so “uninformative” is a modelling convention, not the absence of assumptions.
An improper reference prior is usable only when the posterior is proper. Here that requires enough information to identify all p regression coefficients, including full column rank and, in the ordinary derivation, n>p. The prior is a limiting device rather than a population from which parameters could literally be simulated, so prior prediction is unavailable under it even when posterior inference is valid.
Bayes’ rule gives the joint posterior kernel p({\boldsymbol{\beta}},\sigma^2\mid{\boldsymbol{y}},{\color{blue}\boldsymbol{X}}) \propto (\sigma^2)^{-(n/2+1)} \exp\!\left\{-\frac{\operatorname{SSE}({\boldsymbol{\beta}})} {2\sigma^2}\right\}. This kernel defines a proper posterior when
\operatorname{rank}({\color{blue}\boldsymbol{X}})=p, \qquad n>p, \qquad \operatorname{SSE}>0.
For continuous Gaussian data, the last condition holds with probability one under full rank and n>p.
Used in
- Taught in Bayesian linear regression
- Questions that use it (1) — Bayesian regression and prediction
- All 8 questions on Bayesian linear regression
See also
Marginal Student posterior · Gaussian linear model · Error and residual · Ordinary least squares · Sampling distribution of the OLS estimator · Studentised coefficient pivot · Least-squares decomposition · Conditional coefficient posterior