Dirichlet moments

Proposition taught in Categorical data and the Dirichlet distribution, part of Bayesian statistics

Dirichlet moments

Type Proposition
Taught in Categorical data and the Dirichlet distribution
Part Bayesian statistics
Worked examples 1

Standard Dirichlet integrals give \begin{align*} \mathbb{E}(P_k)&=\frac{\alpha_k}{\alpha_0},\\ \mathbb{E}(P_k^2)&=\frac{\alpha_k(\alpha_k+1)} {\alpha_0(\alpha_0+1)},\\ \mathbb{E}(P_jP_k)&=\frac{\alpha_j\alpha_k} {\alpha_0(\alpha_0+1)},\qquad j\neq k. \end{align*}

Examples

Q22. Advertising responses and Dirichlet inference

Question page: Q22. Advertising responses and Dirichlet inference

[10 marks]

An advertising campaign records responses from 60 independently sampled people:

Negative Neutral Positive
Male 8 6 16
Female 12 4 14

Let \mathbf p=(p_{MN},p_{MU},p_{MP},p_{FN},p_{FU},p_{FP}) be the six population cell probabilities, with U denoting a neutral response. Model the counts as multinomial and use \mathbf p\sim\operatorname{Dirichlet}(1,1,1,1,1,1). You may use the Dirichlet kernel f(\mathbf p)\propto\prod_{k=1}^{K}p_k^{\alpha_k-1}, \qquad p_k>0,\quad \sum_{k=1}^{K}p_k=1.

(a) [2 marks] Derive the posterior distribution of \mathbf p, stating its parameters and support.

(b) [2 marks] Find the posterior mean of p_{MP} and its component of the joint MAP estimate of \mathbf p. Explain why that MAP component equals the observed cell proportion here.

(c) [2 marks] Find the posterior predictive probability that one new person responds positively, regardless of gender. Express it as a posterior expectation before calculating it.

(d) [2 marks] For that new person, calculate \mathbb{P}(\text{positive response}\mid\text{female},\text{data}). Explain why this differs from \mathbb{P}(\text{female and positive response}\mid\text{data}).

(e) [2 marks] Are the six components of \mathbf p independent under the posterior? Explain using their constraint.

Definitions. Let D be the observed table and \mathbf p its six population cell probabilities. The labels M,F denote male and female; N,U,P denote negative, neutral, and positive. With parameters a_1,\ldots,a_K>0, a Dirichlet density has kernel \prod_k p_k^{a_k-1} on the simplex p_k>0, \sum_kp_k=1. The joint MAP maximises this density over the entire simplex, not a one-component marginal density.

(a) Posterior distribution. Write the counts in the stated order as \mathbf x=(8,6,16,12,4,14),\qquad n=60. The multinomial likelihood is L(\mathbf p;D)=\frac{60!}{\prod_{k=1}^{6}x_k!} \prod_{k=1}^{6}p_k^{x_k} \propto\prod_{k=1}^{6}p_k^{x_k}. The Dirichlet prior has kernel \prod_k p_k^{1-1}=1 on the simplex. Consequently, p(\mathbf p\mid D)\propto \prod_{k=1}^{6}p_k^{x_k+1-1}, which is the Dirichlet kernel with parameters x_k+1. Thus \boxed{\mathbf p\mid D\sim\operatorname{Dirichlet}(9,7,17,13,5,15).} Its density is supported on the simplex p_k>0, \sum_{k=1}^6p_k=1; the boundary has probability zero. The posterior concentration is A=9+7+17+13+5+15=66.

(b) Posterior mean and MAP. For a Dirichlet vector with parameters a_1,\ldots,a_K and A=\sum_k a_k, \mathbb{E}(p_k)=\frac{a_k}{A},\qquad \widehat p_{k,\mathrm{MAP}}=\frac{a_k-1}{A-K} \quad\text{when every }a_k>1. Here every posterior parameter exceeds one, so \boxed{\mathbb{E}(p_{MP}\mid D)=\frac{17}{66}\approx0.2576,} \qquad \boxed{\widehat p_{MP,\mathrm{MAP}}= \frac{17-1}{66-6}=\frac{16}{60}\approx0.2667.} The MAP component is taken from the mode of the joint posterior for \mathbf p. Since the prior density is constant on the simplex, maximising the posterior is equivalent to maximising the likelihood, giving the observed cell proportions.

(c) Predicting a positive response. Conditional on \mathbf p, the probability of a positive response is p_{MP}+p_{FP}. Averaging over posterior uncertainty gives \begin{aligned} \mathbb{P}(\text{positive}\mid D) &=\int (p_{MP}+p_{FP})p(\mathbf p\mid D)\,d\mathbf p\\ &=\mathbb{E}(p_{MP}+p_{FP}\mid D)\\ &=\frac{17}{66}+\frac{15}{66} =\boxed{\frac{16}{33}\approx0.4848.} \end{aligned} Prediction uses posterior averaging, not substitution of the MAP estimate.

(d) Conditioning on the new person being female. First calculate the two relevant posterior predictive probabilities: \mathbb{P}(\text{female and positive}\mid D)=\mathbb{E}(p_{FP}\mid D)=\frac{15}{66}, \mathbb{P}(\text{female}\mid D) =\mathbb{E}(p_{FN}+p_{FU}+p_{FP}\mid D) =\frac{13+5+15}{66}=\frac{33}{66}. By the definition of conditional probability, \boxed{ \mathbb{P}(\text{positive}\mid\text{female},D) =\frac{15/66}{33/66}=\frac5{11}\approx0.4545. } By contrast, the joint probability is \boxed{\mathbb{P}(\text{female and positive}\mid D) =\frac{15}{66}=\frac5{22}\approx0.2273.} The conditional probability restricts attention to the female row; the joint probability refers to both events occurring for a person sampled from the whole population.

(e) Posterior dependence. No. The probabilities must satisfy \sum_k p_k=1 exactly. Increasing one component leaves less total probability for the others. More formally, if they were independent, then 0=\operatorname{Var}\!\left(\sum_{k=1}^{6}p_k\,\middle|\,D\right) =\sum_{k=1}^{6}\operatorname{Var}(p_k\mid D)>0, a contradiction, since each component has positive posterior variance. Indeed, distinct Dirichlet components have \operatorname{Cov}(p_j,p_k\mid D) =-\frac{a_ja_k}{A^2(A+1)}<0,\qquad j\ne k.

Key result. Add counts to the Dirichlet parameters; average probabilities for prediction; and distinguish a joint cell probability from a probability conditional on a row.

Used in

See also

Dirichlet distribution · Categorical and multinomial models · The Beta distribution is the two-category Dirichlet · Multinomial central limit theorem · Interior Dirichlet mode · Dirichlet–Multinomial conjugacy

Glossary and index