A logit-normal prior for a coin

Topic: The Beta–Binomial model · more questions on this topic
‹ A geometric observation and a uniform prior Q6 ›

Suppose X\mid\theta\sim\operatorname{Bin}(n,\theta) and prior knowledge favours heads. Consider the logit-normal prior

p(\theta)= \frac{1}{\sqrt{2\pi}\,\theta(1-\theta)} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right], \qquad 0<\theta<1,

where \operatorname{logit}(\theta)=\log\{\theta/(1-\theta)\}.

  1. Plot the prior on a grid and describe its shape.

  2. Generate 100{,}000 prior draws by taking Y\sim\operatorname{N}(1,1) and \theta=\exp(Y)/\{1+\exp(Y)\}. Add a kernel-density estimate from the draws to the exact prior curve.

  3. Explain how the draws approximate the prior mean.

  4. After observing x heads in n tosses, show that

p(\theta\mid x)\propto \theta^{x-1}(1-\theta)^{n-x-1} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right].

Explain why this posterior is not Beta.

Question. Suppose X\mid\theta\sim\operatorname{Bin}(n,\theta) and prior knowledge favours heads. Consider the logit-normal prior

p(\theta)= \frac{1}{\sqrt{2\pi}\,\theta(1-\theta)} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right], \qquad 0<\theta<1,

where \operatorname{logit}(\theta)=\log\{\theta/(1-\theta)\}.

  1. Plot the prior on a grid and describe its shape.

  2. Generate 100{,}000 prior draws by taking Y\sim\operatorname{N}(1,1) and \theta=\exp(Y)/\{1+\exp(Y)\}. Add a kernel-density estimate from the draws to the exact prior curve.

  3. Explain how the draws approximate the prior mean.

  4. After observing x heads in n tosses, show that

p(\theta\mid x)\propto \theta^{x-1}(1-\theta)^{n-x-1} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right].

Explain why this posterior is not Beta.

Solution. Let Y=\operatorname{logit}(\theta). Then

\theta=\frac{e^Y}{1+e^Y}, \qquad \frac{\,\mathrm{d} Y}{\,\mathrm{d}\theta} =\frac1\theta+\frac1{1-\theta} =\frac1{\theta(1-\theta)}.

Transforming Y\sim\operatorname{N}(1,1) therefore produces exactly the stated prior.

  1. The density is unimodal, concentrated above 1/2, and has a longer left tail than right tail. Its mode is near 0.84. The exact curve and a simulation check can be produced together by the following code.
dlogitnormal <- function(theta) {
  ok <- theta > 0 & theta < 1
  ans <- numeric(length(theta))
  ans[ok] <- dnorm(qlogis(theta[ok]), mean = 1, sd = 1) /
             (theta[ok] * (1 - theta[ok]))
  ans
}

set.seed(3500)
theta_grid <- seq(0.001, 0.999, length.out = 1000)
theta_draw <- plogis(rnorm(100000, mean = 1, sd = 1))

plot(theta_grid, dlogitnormal(theta_grid), type = "l",
     xlab = expression(theta), ylab = expression(p(theta)))
lines(density(theta_draw, from = 0, to = 1), col = "steelblue")

  1. The overlaid kernel-density estimate should follow the exact curve. Agreement is a useful check both of the transformation and of the Jacobian in the displayed density.

  2. By the law of large numbers,

\widehat{\mathbb{E}(\theta)} =\frac1L\sum_{l=1}^L\theta^{(l)} \longrightarrow \mathbb{E}(\theta).

For L=100{,}000 the answer varies slightly with the seed; numerical integration gives \mathbb{E}(\theta)\approx0.6967.

  1. The binomial likelihood kernel is

p(x\mid\theta)\propto\theta^x(1-\theta)^{n-x}.

Multiplying by the prior gives \begin{align*} p(\theta\mid x) &\propto p(x\mid\theta)p(\theta)\\ &\propto \theta^x(1-\theta)^{n-x} \frac1{\theta(1-\theta)} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right]\\ &=\boxed{ \theta^{x-1}(1-\theta)^{n-x-1} \exp\!\left[-\frac12\{\operatorname{logit}(\theta)-1\}^2\right]}. \end{align*} A Beta kernel consists only of powers of \theta and 1-\theta. The extra factor involving the squared logit cannot be absorbed into those powers, so the posterior is not Beta.