Natural exponential family

Definition taught in Exponential families and conjugacy, part of Bayesian statistics

Natural exponential family

Type Definition
Taught in Exponential families and conjugacy
Part Bayesian statistics
Properties 3
Worked examples 3
Questions using it 1

In the a–b–c–T notation used here, the natural or canonical form of an exponential-family density is p(x\mid\theta) =\frac{b(x)\exp\{c(\theta)^{\mathsf T}T(x)\}} {a(\theta)}, where c(\theta) is the natural or canonical parameter, T(x) is the natural statistic, b(x) does not depend on \theta, and a(\theta) is the normalising function.

Properties

Log-partition moment identities

Main article: Log-partition moment identities

Whenever c(\theta) can be used as a local parameter and differentiation under the normalising integral is valid,

\nabla_c\log a(\theta) =\mathbb{E}_\theta\{T(X)\}, \qquad \nabla_c^2\log a(\theta) =\operatorname{Cov}_\theta\{T(X)\}.

Here derivatives are taken with respect to c=c(\theta), so a(\theta) is regarded as a function of c through the chosen parametrisation. Consequently \log a(\theta) is convex as a function of c, and the one-observation Fisher information in the natural parameter is \nabla_c^2\log a(\theta).

Generic exponential-family conjugate update

Main article: Generic exponential-family conjugate update

The hyperparameters update as \boxed{ \alpha\leftarrow \alpha+S({\boldsymbol{x}}), \qquad \beta\leftarrow\beta+n.}

Pitman–Koopman–Darmois, informal form

Main article: Pitman–Koopman–Darmois, informal form

For regular iid models with parameter-independent support, the theorem says, roughly, that the existence for every n of a sufficient statistic whose dimension stays fixed essentially characterises exponential families. The support condition matters: models such as \operatorname{Uniform}(0,\theta) can have a fixed-dimensional sufficient statistic without having the regular form Equation (3) in Exponential families and conjugacy.

Examples

Poisson natural form and Gamma conjugacy

Worked example from Exponential families and conjugacy

Question. For iid Poisson observations with mean \theta, write the model in the a(\theta), b(x), c(\theta), T(x) form, identify the sufficient statistic, and derive the corresponding conjugate prior and posterior update.

Solution. For one Poisson observation, \begin{align*} p(x\mid\theta) &=\frac{e^{-\theta}\theta^x}{x!}\\ &=\frac1{x!}\exp\{x\log\theta\}e^{-\theta}\\ &=\frac{b(x)\exp\{c(\theta)T(x)\}}{a(\theta)}, \end{align*} with

b(x)=\frac1{x!}, \qquad c(\theta)=\log\theta, \qquad T(x)=x, \qquad a(\theta)=e^\theta.

For the Poisson model, the definition Equation (4) in Exponential families and conjugacy gives S({\boldsymbol{x}})=\sum_i x_i. The joint likelihood is \begin{align*} p({\boldsymbol{x}}\mid\theta) &=\prod_{i=1}^n\frac{e^{-\theta}\theta^{x_i}}{x_i!}\\ &=\left(\prod_{i=1}^n\frac1{x_i!}\right) \exp\{(\log\theta)S({\boldsymbol{x}})-n\theta\}. \end{align*} The factorisation theorem shows that S({\boldsymbol{X}}) is sufficient for \theta. It is also minimal sufficient by the likelihood-ratio criterion. Because the natural parameter c(\theta)=\log\theta ranges over the open set \mathbb{R}, S({\boldsymbol{X}}) is complete as well.

The generic prior Equation (6) in Exponential families and conjugacy becomes \begin{align*} p(\theta\mid\alpha,\beta) &\propto \frac{\exp\{\alpha\log\theta\}}{(e^\theta)^\beta}\\ &=\theta^\alpha e^{-\beta\theta}. \end{align*} This is the kernel of \operatorname{Gamma}(\alpha+1,\beta), not \operatorname{Gamma}(\alpha,\beta). Using the sufficient statistic S({\boldsymbol{x}}), \begin{align*} p(\theta\mid{\boldsymbol{x}}) &\propto \theta^{S({\boldsymbol{x}})} e^{-n\theta}\theta^\alpha e^{-\beta\theta}\\ &=\theta^{S({\boldsymbol{x}})+\alpha}e^{-(n+\beta)\theta}, \end{align*} so \boxed{ \theta\mid{\boldsymbol{x}} \sim\operatorname{Gamma}(\alpha+S({\boldsymbol{x}})+1,\beta+n)}. Its posterior mean is \begin{align*} \mathbb{E}(\theta\mid{\boldsymbol{x}}) &=\frac{\alpha+S({\boldsymbol{x}})+1}{\beta+n}\\ &=\frac{(\alpha+1)+n\overline x}{\beta+n}\\ &=\frac\beta{\beta+n}\frac{\alpha+1}{\beta} +\frac n{\beta+n}\overline x. \end{align*} The prior therefore behaves, for the mean, as \beta prior observations with total count \alpha+1.

C4. Exponential family

Question page: C4. Exponential family

What is an exponential family? Identify its natural parameter and statistic.

Definitions/notation. An exponential family has density or mass function f(x\mid\theta)=b(x)\exp\{c(\theta)^\top T(x)\}/a(\theta), where c(\theta) is the natural parameter, T(x) the natural statistic, b(x) a parameter-free base factor and a(\theta) the normalising function. In the regular form used here, the support is independent of \theta.

Q57. Negative-binomial canonical link

Question page: Q57. Negative-binomial canonical link

[6 marks]

Let Y have a negative-binomial distribution with probability mass function f(y;\theta) =\binom{y+r-1}{r-1}\theta^r(1-\theta)^y, \qquad y=0,1,2,\ldots, where r>0 is known and 0<\theta<1.

(a) Show that this distribution belongs to the linear exponential family by expressing its probability mass function in the form f(y;\theta) =\exp\left\{ \frac{y\,b(\theta)+c(\theta)}{e(\phi)}+d(y;\phi) \right\}. Identify the natural parameter b(\theta). [3 marks]

(b) Given that \mu=\mathbb{E}(Y)=\frac{r(1-\theta)}{\theta}, derive the canonical link function g(\mu). [3 marks]

Definitions. Here Y counts failures, with support \{0,1,2,\ldots\}, known r>0, and 0<\theta<1. For non-integer r, interpret the binomial coefficient as \binom{y+r-1}{r-1}=\Gamma(y+r)/\{\Gamma(r)\Gamma(y+1)\}. In the stated linear exponential-family form, the natural parameter is b(\theta); the canonical link is g(\mu)=b\{\theta(\mu)\}. We use scale convention e(\phi)=1.

(a) Exponential-family representation. The support \{0,1,2,\ldots\} does not depend on \theta. Take logarithms of the supplied mass function: \begin{aligned} \log f(y;\theta) &=\log\left\{\binom{y+r-1}{r-1}\theta^r(1-\theta)^y\right\}\\ &=y\log(1-\theta)+r\log\theta +\log\binom{y+r-1}{r-1}. \end{aligned} Since r is known, there is no free dispersion parameter to estimate here. Choose e(\phi)=1 and identify \boxed{ b(\theta)=\log(1-\theta),\qquad c(\theta)=r\log\theta,\qquad e(\phi)=1, } \boxed{d(y;\phi)=\log\binom{y+r-1}{r-1}.} The symbol \phi is redundant in this choice: d depends on the known r, not on the unknown \theta. Substitution gives f(y;\theta) =\exp\left\{y\log(1-\theta)+r\log\theta+ \log\binom{y+r-1}{r-1}\right\}, exactly the required form. In particular, \boxed{\eta=b(\theta)=\log(1-\theta)<0.} The natural parameter is \log(1-\theta), not \log\theta: it is the coefficient multiplying the observation y.

The Gamma-function convention in Definitions makes this representation valid for every stated r>0.

(b) Canonical link. Start with the supplied mean and solve for the original parameter: \begin{aligned} \mu&=\frac{r(1-\theta)}{\theta},\\ \mu\theta&=r-r\theta,\\ (\mu+r)\theta&=r,\\ \theta&=\frac r{\mu+r},\qquad 1-\theta=\frac{\mu}{\mu+r}. \end{aligned} Now substitute into the natural parameter from part (a): \boxed{ g(\mu)=b\{\theta(\mu)\} =\log\left(\frac{\mu}{\mu+r}\right) =\log\mu-\log(\mu+r),\qquad\mu>0. } Thus a canonical-link regression would set \eta_i=\mathbf X_i^\top\boldsymbol\beta=g(\mu_i), with \eta_i<0 to keep the means valid. Solving for the mean gives e^\eta=\frac{\mu}{\mu+r} \quad\Longrightarrow\quad \mu(1-e^\eta)=re^\eta \quad\Longrightarrow\quad \boxed{\mu=\frac{re^\eta}{1-e^\eta},\qquad\eta<0.}

Independent check from the normalising term. Write the log mass function as \log f(y;\eta)=y\eta-A(\eta)+d(y), \qquad A(\eta)=-r\log(1-e^\eta). Differentiating the identity \sum_y f(y;\eta)=1 gives 0=\sum_y\{y-A'(\eta)\}f(y;\eta), so \mathbb{E}(Y)=A'(\eta). Here A'(\eta)=\frac{re^\eta}{1-e^\eta}, which is exactly the inverse link just derived.

A log link can be chosen for a negative-binomial regression, but it is not the canonical link derived here.

Used in

See also

Log-partition moment identities · Statistic and estimator · Sufficient statistic · Fisher–Neyman factorisation · Minimal sufficiency · Complete statistic · Ancillary statistic · Basu’s theorem

Glossary and index