Gamma distribution

Definition taught in Random variables and distributions, part of Probability · Also on the distribution sheet

Gamma distribution

The fruit posterior \operatorname{Gamma}(189,21), with the region \theta\leq10 shaded. From Bayesian hypotheses and model comparison.
Notation x\sim\operatorname{Gamma}(\alpha,\beta)
Parameters \alpha,\beta>0, shape–rate
Support x>0
Density p(x)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}
Mean \mathbb{E}(x)=\alpha/\beta
Mode (\alpha-1)/\beta for \alpha\ge 1
Variance \operatorname{Var}(x)=\alpha/\beta^{2}
Conjugate prior for X_i\mid\theta\sim\operatorname{Poisson}(\theta), giving \operatorname{Gamma}(\alpha+\sum_i x_i,\beta+n)
Taught in Random variables and distributions
Part Probability
Properties 1
Worked examples 4
Questions using it 10

From the distribution sheet; mode and conjugate pairs from the formula lookup.

X has the gamma distribution with shape \alpha>0 and rate \lambda>0, written X\sim\operatorname{Gamma}(\alpha,\lambda), when its density is

\frac{\lambda^\alpha x^{\alpha-1}e^{-\lambda x}}{\Gamma(\alpha)},\qquad x>0,

where \Gamma(\alpha)=\int_0^\infty u^{\alpha-1}e^{-u}\,\mathrm{d} u. Then \mathbb{E} X=\alpha/\lambda and \operatorname{Var}(X)=\alpha/\lambda^2.

Properties

Generic exponential-family conjugate update

Main article: Generic exponential-family conjugate update

The hyperparameters update as \boxed{ \alpha\leftarrow \alpha+S({\boldsymbol{x}}), \qquad \beta\leftarrow\beta+n.}

Examples

Poisson natural form and Gamma conjugacy

Worked example from Exponential families and conjugacy

Question. For iid Poisson observations with mean \theta, write the model in the a(\theta), b(x), c(\theta), T(x) form, identify the sufficient statistic, and derive the corresponding conjugate prior and posterior update.

Solution. For one Poisson observation, \begin{align*} p(x\mid\theta) &=\frac{e^{-\theta}\theta^x}{x!}\\ &=\frac1{x!}\exp\{x\log\theta\}e^{-\theta}\\ &=\frac{b(x)\exp\{c(\theta)T(x)\}}{a(\theta)}, \end{align*} with

b(x)=\frac1{x!}, \qquad c(\theta)=\log\theta, \qquad T(x)=x, \qquad a(\theta)=e^\theta.

For the Poisson model, the definition Equation (4) in Exponential families and conjugacy gives S({\boldsymbol{x}})=\sum_i x_i. The joint likelihood is \begin{align*} p({\boldsymbol{x}}\mid\theta) &=\prod_{i=1}^n\frac{e^{-\theta}\theta^{x_i}}{x_i!}\\ &=\left(\prod_{i=1}^n\frac1{x_i!}\right) \exp\{(\log\theta)S({\boldsymbol{x}})-n\theta\}. \end{align*} The factorisation theorem shows that S({\boldsymbol{X}}) is sufficient for \theta. It is also minimal sufficient by the likelihood-ratio criterion. Because the natural parameter c(\theta)=\log\theta ranges over the open set \mathbb{R}, S({\boldsymbol{X}}) is complete as well.

The generic prior Equation (6) in Exponential families and conjugacy becomes \begin{align*} p(\theta\mid\alpha,\beta) &\propto \frac{\exp\{\alpha\log\theta\}}{(e^\theta)^\beta}\\ &=\theta^\alpha e^{-\beta\theta}. \end{align*} This is the kernel of \operatorname{Gamma}(\alpha+1,\beta), not \operatorname{Gamma}(\alpha,\beta). Using the sufficient statistic S({\boldsymbol{x}}), \begin{align*} p(\theta\mid{\boldsymbol{x}}) &\propto \theta^{S({\boldsymbol{x}})} e^{-n\theta}\theta^\alpha e^{-\beta\theta}\\ &=\theta^{S({\boldsymbol{x}})+\alpha}e^{-(n+\beta)\theta}, \end{align*} so \boxed{ \theta\mid{\boldsymbol{x}} \sim\operatorname{Gamma}(\alpha+S({\boldsymbol{x}})+1,\beta+n)}. Its posterior mean is \begin{align*} \mathbb{E}(\theta\mid{\boldsymbol{x}}) &=\frac{\alpha+S({\boldsymbol{x}})+1}{\beta+n}\\ &=\frac{(\alpha+1)+n\overline x}{\beta+n}\\ &=\frac\beta{\beta+n}\frac{\alpha+1}{\beta} +\frac n{\beta+n}\overline x. \end{align*} The prior therefore behaves, for the mean, as \beta prior observations with total count \alpha+1.

Q48. Conjugate Bayesian inference for a Gamma rate

Question page: Q48. Conjugate Bayesian inference for a Gamma rate

[10 marks]

Let X_1,\ldots,X_n be conditionally independent given \lambda, with density f(x_i\mid\lambda) =\frac{\lambda^\alpha}{\Gamma(\alpha)} x_i^{\alpha-1}e^{-\lambda x_i},\qquad x_i>0, where \alpha>0 is known and \lambda>0 is unknown.

Suppose \lambda\sim\operatorname{Gamma}(a,b), where a,b>0, with prior density p(\lambda)=\frac{b^a}{\Gamma(a)}\lambda^{a-1}e^{-b\lambda}.

(a) Derive the posterior distribution of \lambda\mid{\boldsymbol{x}}, identifying its parameters. Show the likelihood and posterior kernels. [4 marks]

(b) State the posterior mean \mathbb{E}[\lambda\mid{\boldsymbol{x}}]. [1 mark]

(c) Derive the maximum likelihood estimator \widehat\lambda_{\mathrm{ML}}. [2 marks]

(d) Express the posterior mean in the form \mathbb{E}[\lambda\mid{\boldsymbol{x}}] =w_n\mathbb{E}[\lambda]+(1-w_n)\widehat\lambda_{\mathrm{ML}}. Identify w_n, verify that this is a convex combination, and determine the limits of the weights as n\to\infty, assuming \bar x_n\to m>0 and fixed prior parameters. [3 marks]

Definitions. \operatorname{Gamma}(A,B) uses shape A and rate B, with kernel \lambda^{A-1}e^{-B\lambda} on \lambda>0 and mean A/B. The sampling shape \alpha>0 is known, while the rate \lambda is unknown. Write T_n=\sum_i x_i=n\bar x_n>0. The prior parameters a,b>0 are fixed; the asymptotic statement uses the specified \bar x_n\to m>0.

(a) Likelihood and posterior. Conditional independence gives \begin{aligned} L(\lambda;{\boldsymbol{x}}) &=\prod_{i=1}^n \left\{\frac{\lambda^\alpha}{\Gamma(\alpha)} x_i^{\alpha-1}e^{-\lambda x_i}\right\}\\ &=\frac{\prod_{i=1}^n x_i^{\alpha-1}}{\Gamma(\alpha)^n} \lambda^{n\alpha}e^{-\lambda T_n}. \end{aligned} The leading factor does not involve \lambda, so \boxed{L(\lambda;{\boldsymbol{x}})\propto\lambda^{n\alpha}e^{-T_n\lambda}, \qquad\lambda>0.} Multiplying by the prior kernel, \begin{aligned} p(\lambda\mid{\boldsymbol{x}}) &\propto\lambda^{n\alpha}e^{-T_n\lambda} \lambda^{a-1}e^{-b\lambda}\\ &=\lambda^{a+n\alpha-1}e^{-(b+T_n)\lambda}. \end{aligned} Matching \lambda^{A-1}e^{-B\lambda} gives \boxed{\lambda\mid{\boldsymbol{x}}\sim \operatorname{Gamma}(a+n\alpha,\ b+T_n) \quad\text{(shape--rate)},\qquad\lambda>0.} Thus the Gamma family is conjugate for the unknown rate.

(b) Posterior mean. A shape–rate Gamma distribution has mean shape divided by rate: \boxed{\mathbb{E}[\lambda\mid{\boldsymbol{x}}]=\frac{a+n\alpha}{b+T_n}.}

(c) Maximum likelihood estimator. The log-likelihood, up to a constant, is \ell(\lambda)=n\alpha\log\lambda-T_n\lambda+C. Its derivatives are \ell'(\lambda)=\frac{n\alpha}{\lambda}-T_n, \qquad \ell''(\lambda)=-\frac{n\alpha}{\lambda^2}<0. Solving \ell'(\lambda)=0 therefore gives the unique maximiser \boxed{\widehat\lambda_{\mathrm{ML}}=\frac{n\alpha}{T_n} =\frac{\alpha}{\bar x_n}.}

(d) Convex combination and limiting weights. The prior mean is \mathbb{E}[\lambda]=a/b. Split the posterior mean as follows: \begin{aligned} \frac{a+n\alpha}{b+T_n} &=\frac{b}{b+T_n}\frac ab +\frac{T_n}{b+T_n}\frac{n\alpha}{T_n}\\ &=w_n\mathbb{E}[\lambda]+(1-w_n)\widehat\lambda_{\mathrm{ML}}, \end{aligned} where \boxed{w_n=\frac{b}{b+T_n}=\frac{b}{b+n\bar x_n}, \qquad 1-w_n=\frac{T_n}{b+T_n}.} Because b>0 and T_n>0, both weights are strictly between zero and one, and \frac{b}{b+T_n}+\frac{T_n}{b+T_n}=1. Hence this is a convex combination of the prior mean and the MLE. Notice that the weights depend on the observed total T_n, not just on n.

Under \bar x_n\to m>0, we have T_n=n\bar x_n\to\infty. With fixed prior parameters, \boxed{w_n\longrightarrow0,\qquad1-w_n\longrightarrow1.} The prior-mean contribution consequently vanishes and the data receive asymptotically all the weight. In fact, both estimates have the same limit: \widehat\lambda_{\mathrm{ML}}=\frac\alpha{\bar x_n} \longrightarrow\frac\alpha m, \qquad \mathbb{E}[\lambda\mid{\boldsymbol{x}}] =\frac{a/n+\alpha}{b/n+\bar x_n} \longrightarrow\frac\alpha m.

Key result. For this rate parameter, the prior weight is b/(b+\sum_i x_i). Derive the weights from the actual posterior mean; they are not always “prior sample size divided by total sample size.”

Q29. Comparing Exponential and Gamma lifetime models

Question page: Q29. Comparing Exponential and Gamma lifetime models

[20 marks]

Researchers observed lifetimes {\boldsymbol{x}}=(x_1,\ldots,x_n), with each x_i>0, for a sample of n medical devices. Two models are proposed, with conditionally iid observations under each model: M_1:\quad X_i\mid\lambda\sim\operatorname{Exp}(\lambda), \qquad M_2:\quad X_i\mid\theta\sim\operatorname{Gamma}(3,\theta). Prior experiments suggest the parameter priors \lambda\mid M_1\sim\operatorname{Gamma}(1,3), \qquad \theta\mid M_2\sim\operatorname{Gamma}(1,1). Parameterisation: all Gamma distributions use shape–rate notation, and the exponential parameter is its rate.

(i) [6 marks] Find p({\boldsymbol{x}}\mid M_1), the marginal likelihood of the data under model M_1.

(ii) [6 marks] Find p({\boldsymbol{x}}\mid M_2), the marginal likelihood of the data under model M_2.

(iii) [4 marks] Obtain an expression for BF_{12}, the Bayes factor comparing M_1 with M_2. For the observed data, BF_{12}=1.347. Briefly interpret this number. Based on this evidence alone, which model is preferred?

(iv) [4 marks] Suppose the researchers’ prior belief is that M_2 is three times as likely as M_1. What are the posterior model odds? Which model is now preferred?

Definitions. Write S=\sum_i x_i and Q=\prod_i x_i^2. \operatorname{Exp}(\lambda) uses rate \lambda; \operatorname{Gamma}(a,b) uses shape a and rate b. For model M_j, the marginal likelihood is m_j({\boldsymbol{x}})=\int L_j(\vartheta;{\boldsymbol{x}})p_j(\vartheta)\,d\vartheta and BF_{12}=m_1/m_2. Use the same positive, conditionally independent lifetime sample and the stated proper priors in both models.

A marginal likelihood averages the fully normalised likelihood over the parameter prior. Unlike posterior kernel matching, this calculation must retain all multiplicative constants, including factors involving only the data.

Both calculations use the Gamma integral \int_0^\infty t^{A-1}e^{-Bt}\,dt =\frac{\Gamma(A)}{B^A},\qquad A,B>0.

(i) Marginal likelihood under M_1. For an exponential distribution with rate \lambda, p(x_i\mid\lambda,M_1)=\lambda e^{-\lambda x_i},\qquad x_i>0. Therefore the likelihood and normalised prior are p({\boldsymbol{x}}\mid\lambda,M_1)=\lambda^n e^{-\lambda S}, \qquad p(\lambda\mid M_1)=3e^{-3\lambda},\quad\lambda>0. Integrating their product, \begin{aligned} p({\boldsymbol{x}}\mid M_1) &=\int_0^\infty p({\boldsymbol{x}}\mid\lambda,M_1) p(\lambda\mid M_1)\,d\lambda\\ &=3\int_0^\infty\lambda^n e^{-(S+3)\lambda}\,d\lambda\\ &=\boxed{\frac{3\Gamma(n+1)}{(S+3)^{n+1}}.} \end{aligned} The power \lambda^n corresponds to Gamma shape n+1, not n. The leading 3 comes from the normalised prior and cannot be omitted.

(ii) Marginal likelihood under M_2. For a Gamma distribution with shape 3 and rate \theta, p(x_i\mid\theta,M_2) =\frac{\theta^3}{\Gamma(3)}x_i^2e^{-\theta x_i} =\frac{\theta^3}{2}x_i^2e^{-\theta x_i},\qquad x_i>0. Thus p({\boldsymbol{x}}\mid\theta,M_2)=\frac{Q}{2^n}\theta^{3n}e^{-\theta S}, \qquad p(\theta\mid M_2)=e^{-\theta},\quad\theta>0. Integrating gives \begin{aligned} p({\boldsymbol{x}}\mid M_2) &=\int_0^\infty p({\boldsymbol{x}}\mid\theta,M_2) p(\theta\mid M_2)\,d\theta\\ &=\frac{Q}{2^n}\int_0^\infty\theta^{3n}e^{-(S+1)\theta}\,d\theta\\ &=\boxed{\frac{Q\,\Gamma(3n+1)}{2^n(S+1)^{3n+1}}.} \end{aligned} Although Q/2^n does not involve \theta, it remains part of the marginal likelihood. Dropping it would give the wrong model comparison.

(iii) Bayes factor and its interpretation. By definition, \begin{aligned} BF_{12} &=\frac{p({\boldsymbol{x}}\mid M_1)}{p({\boldsymbol{x}}\mid M_2)}\\ &=\frac{3\Gamma(n+1)}{(S+3)^{n+1}} \frac{2^n(S+1)^{3n+1}}{Q\,\Gamma(3n+1)}. \end{aligned} Hence \boxed{BF_{12} =\frac{3\cdot2^n\Gamma(n+1)(S+1)^{3n+1}} {\Gamma(3n+1)(S+3)^{n+1}\prod_{i=1}^{n}x_i^2}.} The observed value BF_{12}=1.347 means that the marginal likelihood of the data under M_1 is 1.347 times that under M_2, using the specified parameter priors. The data therefore favour M_1, but only slightly. This is not a posterior probability of either model.

Based on the marginal evidence alone, M_1 is preferred. If the prior model probabilities were equal, its posterior probability would be \mathbb{P}(M_1\mid{\boldsymbol{x}})=\frac{1.347}{1+1.347}\approx0.574. Different prior model odds can reverse that posterior preference.

(iv) Unequal prior model probabilities. The prior belief in the question gives \frac{\mathbb{P}(M_1)}{\mathbb{P}(M_2)}=\frac13. Posterior odds equal the Bayes factor times prior odds, so \boxed{ \frac{\mathbb{P}(M_1\mid{\boldsymbol{x}})}{\mathbb{P}(M_2\mid{\boldsymbol{x}})} =1.347\left(\frac13\right)=0.449. } Thus the posterior odds are 0.449:1 in favour of M_1 relative to M_2, or approximately 2.227:1 in favour of M_2. With these two models as the alternatives, \mathbb{P}(M_1\mid{\boldsymbol{x}})=\frac{0.449}{1+0.449}\approx0.310, \mathbb{P}(M_2\mid{\boldsymbol{x}})=\frac1{1+0.449}\approx0.690. Model M_2 is now preferred. The Bayes factor moves the odds toward M_1, but the evidence is too weak to overcome the prior three-to-one preference for M_2.

Key result. Keep the normalising constants when integrating likelihoods. A Bayes factor updates model odds; it is not itself a posterior model probability.

Ten successive exponential components

Question page: Ten successive exponential components

A component has an exponential lifetime with expectation 1/2 year. On failure it is replaced at once by an identical new component. Ten components are used this way, and T is the time at which the tenth fails.

  1. Find \mathbb{E} T and \operatorname{Var}(T).

  2. Approximate \mathbb{P}(T>6) with the central limit theorem.

  3. Name the exact distribution of T.

  1. T=X_1+\cdots+X_{10} with the X_i independent \operatorname{Exp}(2), each with mean 1/2 and variance 1/4. So \mathbb{E} T=5 years and \operatorname{Var}(T)=10/4=2.5.

  2. By the central limit theorem,

\mathbb{P}(T>6)\approx1-\Phi\Bigl(\frac{6-5}{\sqrt{2.5}}\Bigr)=1-\Phi(0.632)=0.264.

  1. T is exactly \operatorname{Gamma}(10,2), a sum of ten independent exponentials with a common rate. The exact probability is \mathbb{P}(T>6)=\mathbb{P}(\operatorname{Poisson}(12)\le9)=0.242; the normal approximation is a little high because the gamma law is skewed to the right.

Used in

See also

Random variable · Cumulative distribution function · Discrete distribution · Continuous distribution · Expectation · Expectation of a function · Variance · Probability generating function

Glossary and index