Kernel of a density

Topic of Bayesian foundations, part of Bayesian statistics · Also in Conjugate analysis for normal models

Kernel of a density

Type Topic
Taught in Bayesian foundations · Conjugate analysis for normal models
Part Bayesian statistics
Questions using it 59

The part of a density that depends on the variable, with every constant factor dropped. A posterior is computed as a kernel, p(\theta\mid y)\propto p(y\mid\theta)\,p(\theta), and recognising the kernel of a named family, such as \theta^{a-1}(1-\theta)^{b-1} for a Beta, identifies the posterior without computing its normalising constant.

In Bayesian foundations

From Bayesian foundations

Key idea

Bayesian calculation is the controlled multiplication of a likelihood by a prior, followed by normalisation or recognition of a known density kernel. The proportionality symbol removes constants in the variable being analysed, not arbitrary terms that still depend on that variable.

In Conjugate analysis for normal models

From Conjugate analysis for normal models

It is more useful to see how one might discover the prior than simply to be given its name. The unknown \sigma^2 is positive and continuous, and the likelihood kernel contains two distinctive pieces:

(\sigma^2)^{-n/2} \quad\text{and}\quad \exp\!\left\{-\frac{S_\mu/2}{\sigma^2}\right\}.

We therefore want a positive continuous density that already contains a power of \sigma^2 and an exponential in 1/\sigma^2.

The natural candidates can be rejected or repaired one at a time. An exponential density is positive and continuous, but its kernel e^{-\lambda\sigma^2} has \sigma^2, rather than 1/\sigma^2, in the exponent; with one rate parameter it also has its mode fixed at zero. A \chi^2 distribution is only a special Gamma distribution. A Gamma density for \sigma^2 contains (\sigma^2)^{a-1}e^{-b\sigma^2}, so both appearances of \sigma^2 point the wrong way. A Gamma density for the precision \tau=1/\sigma^2, however, has exactly the required structure. Transforming back gives the inverse-Gamma family.

Definition 1 (Inverse-Gamma distribution) In the shape–rate convention,

\sigma^2\sim\operatorname{InvGamma}(\alpha_0,\beta_0),

means that \sigma^2>0 has density

p(\sigma^2) =\frac{\beta_0^{\alpha_0}}{\Gamma(\alpha_0)} (\sigma^2)^{-\alpha_0-1} \exp\!\left(-\frac{\beta_0}{\sigma^2}\right).

Article: Inverse-Gamma distribution →

This follows directly from the transformation \tau=1/\sigma^2\sim\operatorname{Gamma}(\alpha_0,\beta_0). Indeed,

\frac{\,\mathrm{d}\tau}{\,\mathrm{d}\sigma^2}=-\frac1{(\sigma^2)^2}, \qquad \left|\frac{\,\mathrm{d}\tau}{\,\mathrm{d}\sigma^2}\right| =\frac1{(\sigma^2)^2},

and therefore \begin{align*} p(\sigma^2) &=p_\tau(1/\sigma^2) \left|\frac{\,\mathrm{d}\tau}{\,\mathrm{d}\sigma^2}\right|\\ &=\frac{\beta_0^{\alpha_0}}{\Gamma(\alpha_0)} (1/\sigma^2)^{\alpha_0-1} e^{-\beta_0/\sigma^2} \frac1{(\sigma^2)^2}\\ &=\frac{\beta_0^{\alpha_0}}{\Gamma(\alpha_0)} (\sigma^2)^{-\alpha_0-1}e^{-\beta_0/\sigma^2}. \end{align*}

The posterior kernel is \begin{align*} p(\sigma^2\mid{\boldsymbol{x}}) &\propto p({\boldsymbol{x}}\mid\sigma^2)p(\sigma^2)\\ &\propto (\sigma^2)^{-n/2} e^{-S_\mu/(2\sigma^2)} (\sigma^2)^{-\alpha_0-1} e^{-\beta_0/\sigma^2}\\ &=(\sigma^2)^{-(\alpha_0+n/2)-1} \exp\!\left[-\frac{\beta_0+S_\mu/2}{\sigma^2}\right]. \end{align*}

Proposition 1 (Normal–inverse-Gamma variance update) Hence \boxed{ \sigma^2\mid{\boldsymbol{x}} \sim\operatorname{InvGamma}\!\left( \alpha_0+\frac n2, \beta_0+\frac12S_\mu \right)}.

Article: Normal–inverse-Gamma variance update →

For T\sim\operatorname{InvGamma}(a,b),

\mathbb{E}(T)=\frac b{a-1}\quad(a>1), \qquad \operatorname{mode}(T)=\frac b{a+1}.

The posterior mode is therefore \begin{align*} \widehat{\sigma^2}_{\mathrm{MAP}} &=\frac{\beta_0+S_\mu/2}{\alpha_0+n/2+1}\\ &=\frac{2\beta_0+S_\mu}{2(\alpha_0+1)+n}\\ &=\frac{2(\alpha_0+1)}{2(\alpha_0+1)+n} \frac{\beta_0}{\alpha_0+1} +\frac{n}{2(\alpha_0+1)+n} \frac{S_\mu}{n}. \end{align*} Thus, for the mode, the prior is equivalent to 2(\alpha_0+1) squared-deviation observations with total sum of squares 2\beta_0.

Always check the parameterisation before trusting a Gamma or inverse-Gamma formula. Shape–rate, shape–scale, variance, standard deviation, and precision conventions can produce expressions that look nearly identical while meaning different things. This is not a notational nuisance to wave away: it changes the numerical posterior.

Used in

See also

Kernel · Kernel density estimator · Gaussian kernel · A proper approximation to the flat prior · Conjugacy is optional · Marginal hyperposterior · Normal hierarchical shrinkage · Bayesian credible interval

Glossary and index