Generic exponential-family conjugate update

Proposition taught in Exponential families and conjugacy, part of Bayesian statistics

Generic exponential-family conjugate update

Type Proposition
Taught in Exponential families and conjugacy
Part Bayesian statistics
Worked examples 3
Questions using it 11

The hyperparameters update as \boxed{ \alpha\leftarrow \alpha+S({\boldsymbol{x}}), \qquad \beta\leftarrow\beta+n.}

Examples

Poisson natural form and Gamma conjugacy

Worked example from Exponential families and conjugacy

Question. For iid Poisson observations with mean \theta, write the model in the a(\theta), b(x), c(\theta), T(x) form, identify the sufficient statistic, and derive the corresponding conjugate prior and posterior update.

Solution. For one Poisson observation, \begin{align*} p(x\mid\theta) &=\frac{e^{-\theta}\theta^x}{x!}\\ &=\frac1{x!}\exp\{x\log\theta\}e^{-\theta}\\ &=\frac{b(x)\exp\{c(\theta)T(x)\}}{a(\theta)}, \end{align*} with

b(x)=\frac1{x!}, \qquad c(\theta)=\log\theta, \qquad T(x)=x, \qquad a(\theta)=e^\theta.

For the Poisson model, the definition Equation (4) in Exponential families and conjugacy gives S({\boldsymbol{x}})=\sum_i x_i. The joint likelihood is \begin{align*} p({\boldsymbol{x}}\mid\theta) &=\prod_{i=1}^n\frac{e^{-\theta}\theta^{x_i}}{x_i!}\\ &=\left(\prod_{i=1}^n\frac1{x_i!}\right) \exp\{(\log\theta)S({\boldsymbol{x}})-n\theta\}. \end{align*} The factorisation theorem shows that S({\boldsymbol{X}}) is sufficient for \theta. It is also minimal sufficient by the likelihood-ratio criterion. Because the natural parameter c(\theta)=\log\theta ranges over the open set \mathbb{R}, S({\boldsymbol{X}}) is complete as well.

The generic prior Equation (6) in Exponential families and conjugacy becomes \begin{align*} p(\theta\mid\alpha,\beta) &\propto \frac{\exp\{\alpha\log\theta\}}{(e^\theta)^\beta}\\ &=\theta^\alpha e^{-\beta\theta}. \end{align*} This is the kernel of \operatorname{Gamma}(\alpha+1,\beta), not \operatorname{Gamma}(\alpha,\beta). Using the sufficient statistic S({\boldsymbol{x}}), \begin{align*} p(\theta\mid{\boldsymbol{x}}) &\propto \theta^{S({\boldsymbol{x}})} e^{-n\theta}\theta^\alpha e^{-\beta\theta}\\ &=\theta^{S({\boldsymbol{x}})+\alpha}e^{-(n+\beta)\theta}, \end{align*} so \boxed{ \theta\mid{\boldsymbol{x}} \sim\operatorname{Gamma}(\alpha+S({\boldsymbol{x}})+1,\beta+n)}. Its posterior mean is \begin{align*} \mathbb{E}(\theta\mid{\boldsymbol{x}}) &=\frac{\alpha+S({\boldsymbol{x}})+1}{\beta+n}\\ &=\frac{(\alpha+1)+n\overline x}{\beta+n}\\ &=\frac\beta{\beta+n}\frac{\alpha+1}{\beta} +\frac n{\beta+n}\overline x. \end{align*} The prior therefore behaves, for the mean, as \beta prior observations with total count \alpha+1.

Q1. Rayleigh conjugate prior

Question page: Q1. Rayleigh conjugate prior

Suppose X_1,\ldots,X_n\mid\theta \overset{\mathrm{iid}}{\sim}\operatorname{Rayleigh}(\theta), with density f(x\mid\theta) =\frac{x}{\theta}\exp\left(-\frac{x^2}{2\theta}\right), \qquad x>0,\quad \theta>0.

Using the general exponential-family conjugacy result, find a conjugate prior for \theta and derive the posterior distribution.

Definitions. For observed {\boldsymbol{x}}=(x_1,\ldots,x_n), L(\theta;{\boldsymbol{x}}) is the sampling density viewed as a function of \theta. A kernel omits only factors constant in that parameter. Conjugacy means that prior and posterior belong to the same family. We use \operatorname{InvGamma}(a,b):\quad p(\theta)\propto\theta^{-a-1}e^{-b/\theta}, \qquad \theta>0,\quad a,b>0, with shape a and inverse-Gamma scale b. The Rayleigh observations are conditionally iid and positive.

The general exponential-family form used here is f(x\mid\theta) =b(x)\frac{\exp\{c(\theta)T(x)\}}{a(\theta)}, for which a conjugate prior has kernel \pi(\theta) \propto \frac{\exp\{\alpha c(\theta)\}}{a(\theta)^\beta}.

For the Rayleigh density, f(x\mid\theta) =x\, \frac{\exp\left\{-\dfrac{1}{\theta}\dfrac{x^2}{2}\right\}} {\theta}. Thus, we identify b(x)=x, \qquad T(x)=\frac{x^2}{2}, \qquad c(\theta)=-\frac{1}{\theta}, \qquad a(\theta)=\theta.

Substitution into the conjugate-prior kernel gives \pi(\theta) \propto \frac{\exp(-\alpha/\theta)}{\theta^\beta} =\theta^{-\beta}\exp\left(-\frac{\alpha}{\theta}\right). Matching this to the inverse-Gamma kernel in Definitions gives a=\beta-1, \qquad b=\alpha. Therefore, a conjugate prior is \boxed{\theta\sim\operatorname{InvGamma}(\beta-1,\alpha)}, with \alpha>0 and \beta>1 for a proper prior.

For an iid sample, the likelihood is \begin{aligned} L(\theta;{\boldsymbol{x}}) &=\prod_{i=1}^n \frac{x_i}{\theta} \exp\left(-\frac{x_i^2}{2\theta}\right)\\ &\propto \theta^{-n} \exp\left(-\frac{1}{2\theta}\sum_{i=1}^n x_i^2\right). \end{aligned} Multiplying the likelihood by the prior gives \begin{aligned} \pi(\theta\mid{\boldsymbol{x}}) &\propto L(\theta;{\boldsymbol{x}})\pi(\theta)\\ &\propto \theta^{-n} \exp\left(-\frac{\sum_{i=1}^n x_i^2}{2\theta}\right) \theta^{-\beta} \exp\left(-\frac{\alpha}{\theta}\right)\\ &\propto \theta^{-(\beta+n)} \exp\left[ -\frac{\alpha+\frac12\sum_{i=1}^n x_i^2}{\theta} \right]. \end{aligned} Recognising the inverse-Gamma kernel, \boxed{ \theta\mid{\boldsymbol{x}} \sim \operatorname{InvGamma}\left( \beta+n-1, \alpha+\frac12\sum_{i=1}^n x_i^2 \right) }.

Equivalently, if the prior is written directly as \theta\sim\operatorname{InvGamma}(a,b), then

Key result. \boxed{ \theta\mid{\boldsymbol{x}} \sim \operatorname{InvGamma}\left( a+n, b+\frac12\sum_{i=1}^n x_i^2 \right) }. This is the form worth remembering after working through the derivation.

Q12. Pareto likelihood and Gamma posterior

Question page: Q12. Pareto likelihood and Gamma posterior

Suppose X_1,\ldots,X_n are conditionally iid with density p(x_i\mid\theta) =\theta m^\theta x_i^{-(\theta+1)}, \qquad x_i\geq m, where m>0 is known and \theta>0 is unknown. Use the prior p(\theta)=\theta e^{-\theta}, \qquad \theta>0. You may use the fact that a \operatorname{Gamma}(A,B) distribution in shape–rate form has kernel \theta^{A-1}e^{-B\theta}, mean A/B, and mode (A-1)/B when A>1.

[8 marks]

(a) Show that the likelihood can be written L(\theta;{\boldsymbol{x}})\propto\theta^n e^{-\theta S}, \qquad S=\sum_{i=1}^n\log\left(\frac{x_i}{m}\right). [3 marks]

(b) Derive the posterior distribution of \theta. Identify its distribution, parameters, and support. [3 marks]

(c) State the posterior mean and MAP estimate. [2 marks]

Start with part (a). The target expression is supplied, so the task is to show how multiplying the densities produces it.

Definitions. The known lower endpoint is m>0, and the observed data satisfy x_i\ge m. We use the shape–rate Gamma kernel \theta^{A-1}e^{-B\theta} for \theta>0, A,B>0, with mean A/B and mode (A-1)/B when A>1. The MAP estimate is this posterior mode. The prior \theta e^{-\theta} is \operatorname{Gamma}(2,1).

Since m is known and the observed sample is valid, the support indicator does not depend on \theta.

(a) Likelihood

By conditional independence, \begin{aligned} L(\theta;{\boldsymbol{x}}) &=\prod_{i=1}^n \left[\theta m^\theta x_i^{-(\theta+1)}\right]\\ &=\theta^n m^{n\theta} \prod_{i=1}^n x_i^{-(\theta+1)}. \end{aligned} The factor m^{n\theta} must be retained because it depends on \theta. Using a^b=e^{b\log a}, \begin{aligned} L(\theta;{\boldsymbol{x}}) &=\theta^n \exp\left(n\theta\log m\right) \exp\left[-(\theta+1)\sum_{i=1}^n\log x_i\right]\\ &=\theta^n \exp\left[-\sum_{i=1}^n\log x_i\right] \exp\left[-\theta\left\{ \sum_{i=1}^n\log x_i-n\log m \right\}\right]. \end{aligned} The first exponential in the final line does not depend on \theta and may be absorbed into the proportionality constant. Also, \sum_{i=1}^n\log x_i-n\log m =\sum_{i=1}^n\log\left(\frac{x_i}{m}\right) =S. Therefore, \boxed{ L(\theta;{\boldsymbol{x}})\propto\theta^n e^{-\theta S}, \qquad \theta>0. }

(b) Posterior distribution

Multiplying the likelihood by the prior gives \begin{aligned} p(\theta\mid{\boldsymbol{x}}) &\propto L(\theta;{\boldsymbol{x}})p(\theta)\\ &\propto \theta^n e^{-\theta S}\, \theta e^{-\theta}\\ &=\theta^{n+1}e^{-(S+1)\theta}, \qquad \theta>0. \end{aligned} Comparing this with the shape–rate Gamma kernel \theta^{A-1}e^{-B\theta}, we obtain A-1=n+1, \qquad B=S+1. Hence \boxed{ \theta\mid{\boldsymbol{x}} \sim\operatorname{Gamma}(n+2,S+1) \quad\text{in shape--rate form}, \qquad \theta>0. } Because x_i\geq m, each \log(x_i/m)\geq0, so S+1>0 as required.

(c) Posterior mean and MAP estimate

For a shape–rate \operatorname{Gamma}(A,B) distribution, the mean is A/B. Therefore, \boxed{ \mathbb{E}(\theta\mid{\boldsymbol{x}})=\frac{n+2}{S+1}. } The MAP estimate is the posterior mode. Since A=n+2>1, \boxed{ \widehat{\theta}_{\mathrm{MAP}} =\frac{A-1}{B} =\frac{n+1}{S+1}. }

Used in

See also

Hyperparameter · Conjugacy is optional · Dirichlet distribution · Normal–inverse-Gamma prior · Statistic and estimator · Sufficient statistic · Fisher–Neyman factorisation · Minimal sufficiency

Glossary and index