A geometric observation and a uniform prior

Topic: The Beta–Binomial model · more questions on this topic
‹ A future run of tosses A logit-normal prior for a coin ›

Let X be the number of coin tosses up to and including the first head. Thus

\mathbb{P}(X=x\mid\theta)=\theta(1-\theta)^{x-1}, \qquad x=1,2,\ldots,

where \theta is the probability of a head. Use the prior \theta\sim\operatorname{Unif}(0,1).

  1. Find and identify p(\theta\mid x).

  2. Discuss whether the prior was intended to be informative and whether it is conjugate.

  3. Find the posterior mode.

  4. Find \mathbb{E}(\theta\mid x).

  5. Express the posterior mean as a convex combination of the prior mean and the maximum-likelihood estimate of \theta.

  6. Give an equivalent-prior-observations interpretation, and state which pseudo-count convention you are using.

Question. Let X be the number of coin tosses up to and including the first head. Thus

\mathbb{P}(X=x\mid\theta)=\theta(1-\theta)^{x-1}, \qquad x=1,2,\ldots,

where \theta is the probability of a head. Use the prior \theta\sim\operatorname{Unif}(0,1).

  1. Find and identify p(\theta\mid x).

  2. Discuss whether the prior was intended to be informative and whether it is conjugate.

  3. Find the posterior mode.

  4. Find \mathbb{E}(\theta\mid x).

  5. Express the posterior mean as a convex combination of the prior mean and the maximum-likelihood estimate of \theta.

  6. Give an equivalent-prior-observations interpretation, and state which pseudo-count convention you are using.

Solution.

  1. The prior density is one on (0,1), so Bayes’ rule gives \begin{align*} p(\theta\mid x) &\propto \mathbb{P}(X=x\mid\theta)p(\theta)\\ &=\theta(1-\theta)^{x-1}\\ &=\theta^{2-1}(1-\theta)^{x-1}. \end{align*} The normalising integral is

\int_0^1\theta(1-\theta)^{x-1}\,\mathrm{d}\theta =B(2,x) =\frac{\Gamma(2)\Gamma(x)}{\Gamma(x+2)} =\frac{1}{x(x+1)}.

Consequently

p(\theta\mid x)=x(x+1)\theta(1-\theta)^{x-1}, \qquad \boxed{\theta\mid x\sim\operatorname{Beta}(2,x)}.

  1. The uniform prior was chosen to give equal density to equal-length intervals of \theta, so it was intended as a non-informative prior on this particular scale. That description is not invariant under a transformation of \theta. Since \operatorname{Unif}(0,1)=\operatorname{Beta}(1,1) and the Beta family is conjugate to this geometric likelihood, the prior is conjugate.

  2. A \operatorname{Beta}(a,b) density has interior mode (a-1)/(a+b-2) when a,b>1. Here this gives

\operatorname{mode}(\theta\mid x) =\frac{2-1}{2+x-2}=\boxed{\frac1x}.

For x=1 this is the boundary mode at one.

  1. The posterior mean is

\boxed{\mathbb{E}(\theta\mid x)=\frac{2}{x+2}}.

  1. The prior mean is 1/2. To obtain the maximum-likelihood estimator, differentiate the log likelihood: \begin{align*} \ell(\theta)&=\log\theta+(x-1)\log(1-\theta),\\ \ell'(\theta)&=\frac1\theta-\frac{x-1}{1-\theta}. \end{align*} Setting \ell'(\theta)=0 gives 1-\theta=(x-1)\theta, hence \widehat\theta_{\mathrm{MLE}}=1/x. With w=2/(x+2), \begin{align*} w\mathbb{E}(\theta)+(1-w)\widehat\theta_{\mathrm{MLE}} &=\frac{2}{x+2}\frac12 +\left(1-\frac{2}{x+2}\right)\frac1x\\ &=\frac1{x+2}+\frac{x}{x+2}\frac1x\\ &=\frac2{x+2}\\ &=\mathbb{E}(\theta\mid x). \end{align*}

  2. Under the posterior-mean concentration convention, \operatorname{Beta}(1,1) has total prior weight two with one prior head and one prior tail: it behaves like one head among two prior tosses. Under the kernel exponent convention, however, the prior contributes \alpha-1=0 powers of \theta and \beta-1=0 powers of 1-\theta. These are two different pseudo-count conventions; the first is the one used in the requested linear combination.