Conjugacy is optional

Remark taught in The Beta–Binomial model, part of Bayesian statistics

Conjugacy is optional

Type	Remark
Taught in	The Beta–Binomial model
Part	Bayesian statistics
Worked examples	2

Conjugacy is convenient, not compulsory. A prior concentrated around 1/2 could be obtained by truncating a Laplace density:

p(\theta)\propto \exp\!\left(-\frac{|\theta-1/2|}{b}\right) \mathbf{1}\{0\leq\theta\leq1\}.

The posterior kernel is then

p(\theta\mid x) \propto \theta^x(1-\theta)^{n-x} \exp\!\left(-\frac{|\theta-1/2|}{b}\right) \mathbf{1}\{0\leq\theta\leq1\}.

If equality rather than proportionality is needed, symmetry supplies the normalising constant without numerical integration: \begin{align*} \int_0^1\exp\!\left(-\frac{|\theta-1/2|}{b}\right)\,\mathrm{d}\theta &=2\int_0^{1/2}\exp\!\left(-\frac{1/2-\theta}{b}\right)\,\mathrm{d}\theta\\ &=2\left[b\exp\!\left(-\frac{1/2-\theta}{b}\right)\right]_{\theta=0}^{1/2}\\ &=2b\left(1-e^{-1/(2b)}\right). \end{align*} Thus the proper truncated-Laplace prior has multiplier ([2b{1-e^{-1/(2b)}}]^{-1}). The resulting posterior is not a named standard density, so its normalising integral and summaries must be evaluated numerically. The later MCMC chapters provide a general route when direct integration becomes difficult.

Examples

Q42. A nonconjugate prior and a change of variable

Question page: Q42. A nonconjugate prior and a change of variable

[12 marks]

Suppose X\mid\theta\sim\operatorname{Binomial}(n,\theta). Define Z=\log\frac{\theta}{1-\theta}, and assign the prior Z\sim N(1,1).

(a) Derive the prior density of \theta, including the Jacobian. [3 marks]

(b) Find the posterior kernel p(\theta\mid x). Is this prior conjugate? [2 marks]

(c) Derive the posterior kernel p(z\mid x). [3 marks]

(d) Construct a random-walk Metropolis–Hastings algorithm targeting p(z\mid x), using Z^*\mid Z^{(t)}=z\sim N(z,s^2). Explain how its output can estimate \mathbb{E}(\theta\mid x). [4 marks]

Worked answer (click to show)

Definitions. The sampling parameter is 0<\theta<1, and its logit coordinate is z=\log\{\theta/(1-\theta)\}. The prior is Z\sim N(1,1), where the second argument is variance. For an invertible transformation, densities obey p_\theta(\theta)=p_Z(z(\theta))|dz/d\theta|. A kernel is always defined relative to its coordinate and reference measure; here these are d\theta or dz.

(a) Prior density of \theta. The inverse transformation is \theta=g(z)=\frac{e^z}{1+e^z},\qquad z\in\mathbb R, and \frac{dz}{d\theta} =\frac{d}{d\theta}\{\log\theta-\log(1-\theta)\} =\frac1{\theta(1-\theta)}. Since Z\sim N(1,1), the change-of-variable formula gives \boxed{ p(\theta) =\frac{1}{\sqrt{2\pi}\,\theta(1-\theta)} \exp\left[-\frac12\left\{\log\frac{\theta}{1-\theta}-1\right\}^2\right], \quad 0<\theta<1. } The density is zero outside (0,1). This is a logit-normal prior.

(b) Posterior kernel for \theta. For an observed count x\in\{0,\ldots,n\}, L(\theta;x)=\binom nx\theta^x(1-\theta)^{n-x}. Multiplication by the prior gives \boxed{ p(\theta\mid x)\propto \theta^{x-1}(1-\theta)^{n-x-1} \exp\left[-\frac12\left\{\log\frac{\theta}{1-\theta}-1\right\}^2\right], \quad 0<\theta<1. } This prior is not conjugate: the posterior does not in general remain logit-normal. In particular, its logit-transformed posterior is not Normal, as the following kernel shows.

(c) Posterior kernel for z. Substituting \theta=g(z) in the likelihood, \theta^x(1-\theta)^{n-x} =\frac{e^{xz}}{(1+e^z)^n}. Multiplying by the Normal prior density for z yields \boxed{ p(z\mid x)\propto \exp\left\{xz-n\log(1+e^z)-\frac12(z-1)^2\right\}, \quad z\in\mathbb R. } Alternatively, transforming the posterior from part (b) requires |d\theta/dz|=\theta(1-\theta), which cancels the prior’s Jacobian factor. Simply substituting \theta=g(z) into a density in \theta without this Jacobian would give the wrong density in z.

(d) Random-walk Metropolis–Hastings sampler. Define the log posterior kernel h(z)=xz-n\log(1+e^z)-\frac12(z-1)^2. The proposal density is symmetric because q(z^*\mid z) =\frac1{\sqrt{2\pi s^2}}\exp\left\{-\frac{(z^*-z)^2}{2s^2}\right\} =q(z\mid z^*). Thus the acceptance probability is A(z,z^*)=\min\{1,\exp[h(z^*)-h(z)]\}. Equivalently, the log of the untruncated ratio is \begin{aligned} \Delta=h(z^*)-h(z) &=x(z^*-z)-n\{\log(1+e^{z^*})-\log(1+e^z)\}\\ &\quad-\frac12\{(z^*-1)^2-(z-1)^2\}. \end{aligned}

Pseudocode. Choose any z^{(0)}\in\mathbb R and a fixed s>0. For t=0,\ldots,T-1:

1. Propose z^*\sim N(z^{(t)},s^2).

2. Calculate \Delta=h(z^*)-h(z^{(t)}).

3. Independently draw U\sim\operatorname{Uniform}(0,1).

4. If \log U\le\min\{0,\Delta\}, set z^{(t+1)}=z^*; otherwise set z^{(t+1)}=z^{(t)}.

This Markov chain has p(z\mid x) as its equilibrium density. Initial draws from an arbitrary start are not automatically posterior draws, and successive draws are generally dependent.

Estimating the requested posterior mean. After discarding a suitable initial burn-in of B iterations, transform each retained state and average: \theta^{(t)}=g(z^{(t)}), \qquad \boxed{ \widehat{\mathbb{E}(\theta\mid x)} =\frac1{T-B}\sum_{t=B+1}^T\frac{e^{z^{(t)}}}{1+e^{z^{(t)}}} }. With adequate convergence and a sufficiently long run, the ergodic average estimates the posterior expectation. Assess mixing and Monte Carlo error; the retained values are not independent.

For stable numerical evaluation, use \log(1+e^z)=\max\{0,z\}+\log(1+e^{-|z|}) and a stable logistic function.

Key result. Average the transformed draws. In general, \mathbb{E}\{g(Z)\mid x\}\ne g\{\mathbb{E}(Z\mid x)\}, so transforming the average of the z draws does not estimate the same quantity.

A logit-normal prior for a coin

Question page: A logit-normal prior for a coin

Suppose X\mid\theta\sim\operatorname{Bin}(n,\theta) and prior knowledge favours heads. Consider the logit-normal prior

p(\theta)= \frac{1}{\sqrt{2\pi}\,\theta(1-\theta)} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right], \qquad 0<\theta<1,

where \operatorname{logit}(\theta)=\log\{\theta/(1-\theta)\}.

1. Plot the prior on a grid and describe its shape.

2. Generate 100{,}000 prior draws by taking Y\sim\operatorname{N}(1,1) and \theta=\exp(Y)/\{1+\exp(Y)\}. Add a kernel-density estimate from the draws to the exact prior curve.

3. Explain how the draws approximate the prior mean.

4. After observing x heads in n tosses, show that

p(\theta\mid x)\propto \theta^{x-1}(1-\theta)^{n-x-1} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right].

Explain why this posterior is not Beta.

Worked answer (click to show)

Question. Suppose X\mid\theta\sim\operatorname{Bin}(n,\theta) and prior knowledge favours heads. Consider the logit-normal prior

p(\theta)= \frac{1}{\sqrt{2\pi}\,\theta(1-\theta)} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right], \qquad 0<\theta<1,

where \operatorname{logit}(\theta)=\log\{\theta/(1-\theta)\}.

1. Plot the prior on a grid and describe its shape.

2. Generate 100{,}000 prior draws by taking Y\sim\operatorname{N}(1,1) and \theta=\exp(Y)/\{1+\exp(Y)\}. Add a kernel-density estimate from the draws to the exact prior curve.

3. Explain how the draws approximate the prior mean.

4. After observing x heads in n tosses, show that

p(\theta\mid x)\propto \theta^{x-1}(1-\theta)^{n-x-1} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right].

Explain why this posterior is not Beta.

Solution. Let Y=\operatorname{logit}(\theta). Then

\theta=\frac{e^Y}{1+e^Y}, \qquad \frac{\,\mathrm{d} Y}{\,\mathrm{d}\theta} =\frac1\theta+\frac1{1-\theta} =\frac1{\theta(1-\theta)}.

Transforming Y\sim\operatorname{N}(1,1) therefore produces exactly the stated prior.

1. The density is unimodal, concentrated above 1/2, and has a longer left tail than right tail. Its mode is near 0.84. The exact curve and a simulation check can be produced together by the following code.

dlogitnormal <- function(theta) {
  ok <- theta > 0 & theta < 1
  ans <- numeric(length(theta))
  ans[ok] <- dnorm(qlogis(theta[ok]), mean = 1, sd = 1) /
             (theta[ok] * (1 - theta[ok]))
  ans
}

set.seed(3500)
theta_grid <- seq(0.001, 0.999, length.out = 1000)
theta_draw <- plogis(rnorm(100000, mean = 1, sd = 1))

plot(theta_grid, dlogitnormal(theta_grid), type = "l",
     xlab = expression(theta), ylab = expression(p(theta)))
lines(density(theta_draw, from = 0, to = 1), col = "steelblue")

2. The overlaid kernel-density estimate should follow the exact curve. Agreement is a useful check both of the transformation and of the Jacobian in the displayed density.

3. By the law of large numbers,

\widehat{\mathbb{E}(\theta)} =\frac1L\sum_{l=1}^L\theta^{(l)} \longrightarrow \mathbb{E}(\theta).

For L=100{,}000 the answer varies slightly with the seed; numerical integration gives \mathbb{E}(\theta)\approx0.6967.

4. The binomial likelihood kernel is

p(x\mid\theta)\propto\theta^x(1-\theta)^{n-x}.

Multiplying by the prior gives \begin{align*} p(\theta\mid x) &\propto p(x\mid\theta)p(\theta)\\ &\propto \theta^x(1-\theta)^{n-x} \frac1{\theta(1-\theta)} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right]\\ &=\boxed{ \theta^{x-1}(1-\theta)^{n-x-1} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right]}. \end{align*} A Beta kernel consists only of powers of \theta and 1-\theta. The extra factor involving the squared logit cannot be absorbed into those powers, so the posterior is not Beta.

Used in

• Taught in The Beta–Binomial model, where the derivation and examples are.
• All 11 questions on The Beta–Binomial model

See also

Generic exponential-family conjugate update · Markov chain · Beta distribution · Beta–Binomial conjugacy · Two pseudo-count conventions · Sequential coherence · Beta–Binomial posterior prediction

Glossary and index