Exponential family

Definition taught in Introduction to generalised linear models, part of Generalised linear models

Exponential family

Type Definition
Taught in Introduction to generalised linear models
Part Generalised linear models
Properties 4
Worked examples 2
Questions using it 7

A random variable Y whose distribution depends on a parameter \theta belongs to a (one-parameter) exponential family, with dispersion \phi, if its density or mass function can be written f(y;\theta)=\exp\left\{\frac{a(y)\,b(\theta)+c(\theta)}{e(\phi)}+d(y;\phi)\right\}, where a, b, c, d and e are known functions. The dispersion \phi is treated as a nuisance: often known, and at least constant across observations. It is the family’s variance parameter.

Properties

Mean and variance of an exponential family

Main article: Mean and variance of an exponential family

If Y has density or mass function Equation (7) in Introduction to generalised linear models, and if differentiation in \theta may be passed under the integral or the sum, then \mathrm{E}(Y)=-c'(\theta), \qquad \mathrm{Var}(Y)=-c''(\theta)\,e(\phi).

Log-partition moment identities

Main article: Log-partition moment identities

Whenever c(\theta) can be used as a local parameter and differentiation under the normalising integral is valid,

\nabla_c\log a(\theta) =\mathbb{E}_\theta\{T(X)\}, \qquad \nabla_c^2\log a(\theta) =\operatorname{Cov}_\theta\{T(X)\}.

Here derivatives are taken with respect to c=c(\theta), so a(\theta) is regarded as a function of c through the chosen parametrisation. Consequently \log a(\theta) is convex as a function of c, and the one-observation Fisher information in the natural parameter is \nabla_c^2\log a(\theta).

Generic exponential-family conjugate update

Main article: Generic exponential-family conjugate update

The hyperparameters update as \boxed{ \alpha\leftarrow \alpha+S({\boldsymbol{x}}), \qquad \beta\leftarrow\beta+n.}

Pitman–Koopman–Darmois, informal form

Main article: Pitman–Koopman–Darmois, informal form

For regular iid models with parameter-independent support, the theorem says, roughly, that the existence for every n of a sufficient statistic whose dimension stays fixed essentially characterises exponential families. The support condition matters: models such as \operatorname{Uniform}(0,\theta) can have a fixed-dimensional sufficient statistic without having the regular form Equation (3) in Exponential families and conjugacy.

Examples

Q45. Gamma regression from the exponential-family form

Question page: Q45. Gamma regression from the exponential-family form

[12 marks]

Independent positive responses have density f(y_i\mid\mu_i,k) =\frac{k^k}{\Gamma(k)\mu_i^k} y_i^{k-1}\exp\left(-\frac{ky_i}{\mu_i}\right), \qquad y_i>0, where \mu_i>0 and k>0.

(a) Express the density in the form f(y;\theta,\phi) =\exp\left\{ \frac{a(y)b(\theta)+c(\theta)}{e(\phi)}+d(y;\phi) \right\}, taking \theta=\mu and \phi=1/k. Identify all five functions. [4 marks]

(b) Find \mathbb{E}(Y_i) and \operatorname{Var}(Y_i). [2 marks]

(c) Under the model \log\mu_i=\beta_0+\beta_1x_i, interpret \beta_1 and derive the ratio of expected responses at x+1 and x. [2 marks]

(d) Derive the score vector for \boldsymbol\beta, treating k as known. [3 marks]

(e) Explain why this model does not assert that \log Y_i is Normally distributed. [1 mark]

Definitions. A Gamma variable with shape k and rate k/\mu has mean \mu and variance \mu^2/k; set \phi=1/k. In the exponential-family form of part (a), b(\theta) is the coefficient paired with the statistic a(y) after division by e(\phi). For regression, \mathbf X_i=(1,x_i)^\top, \boldsymbol\beta=(\beta_0,\beta_1)^\top, and \mu_i=e^{\mathbf X_i^\top\boldsymbol\beta}. Responses are independent and covariates fixed; the score calculation holds k>0 known.

(a) Exponential-family representation. For one observation, taking logarithms gives \log f(y\mid\mu,k) =k\log k-\log\Gamma(k)-k\log\mu+(k-1)\log y-\frac{ky}{\mu}. With \theta=\mu and \phi=1/k, rearrange this as \log f(y;\theta,\phi) =\frac{-y/\theta-\log\theta}{\phi} +\frac1\phi\log\frac1\phi-\log\Gamma\!\left(\frac1\phi\right) +\left(\frac1\phi-1\right)\log y. One valid identification of the five functions is \boxed{ a(y)=y,\qquad b(\theta)=-\frac1\theta,\qquad c(\theta)=-\log\theta,\qquad e(\phi)=\phi, } \boxed{ d(y;\phi)=\frac1\phi\log\frac1\phi -\log\Gamma\!\left(\frac1\phi\right) +\left(\frac1\phi-1\right)\log y. } The support is y>0, with \theta>0 and \phi>0.

(b) Mean and variance. The given density is Gamma with shape k and rate k/\mu_i. Therefore \boxed{\mathbb{E}(Y_i)=\frac{k}{k/\mu_i}=\mu_i,\qquad \operatorname{Var}(Y_i)=\frac{k}{(k/\mu_i)^2} =\frac{\mu_i^2}{k}=\phi\mu_i^2.}

(c) Interpretation of the coefficient. The log link gives \mu(x)=e^{\beta_0+\beta_1x}, so \boxed{\frac{\mathbb{E}(Y\mid x+1)}{\mathbb{E}(Y\mid x)} =\frac{e^{\beta_0+\beta_1(x+1)}}{e^{\beta_0+\beta_1x}} =e^{\beta_1}.} Thus a one-unit increase in x multiplies the expected response by e^{\beta_1}, corresponding to a percentage change 100(e^{\beta_1}-1)\%. The coefficient \beta_1 itself is the change in the log of the mean, not an additive change in the response.

(d) Score vector. Let \mathbf X_i=(1,x_i)^\top and \boldsymbol\beta=(\beta_0,\beta_1)^\top, so that \mu_i=e^{\mathbf X_i^\top\boldsymbol\beta}. Treating k as known, \ell(\boldsymbol\beta) =-k\sum_{i=1}^n\left(\log\mu_i+\frac{y_i}{\mu_i}\right)+C, where C does not depend on \boldsymbol\beta. Differentiate each summand with respect to its mean and apply the chain rule: \frac{\partial\ell_i}{\partial\mu_i} =-\frac{k}{\mu_i}+\frac{ky_i}{\mu_i^2}, \qquad \frac{\partial\mu_i}{\partial\boldsymbol\beta}=\mu_i\mathbf X_i. It follows that \begin{aligned} S(\boldsymbol\beta) &=\sum_{i=1}^n \left(-\frac{k}{\mu_i}+\frac{ky_i}{\mu_i^2}\right)\mu_i\mathbf X_i\\ &=\boxed{k\sum_{i=1}^n\mathbf X_i\left(\frac{y_i}{\mu_i}-1\right)}. \end{aligned} In components, S(\boldsymbol\beta) =k\begin{pmatrix} \sum_i(y_i/\mu_i-1)\\[3pt] \sum_i x_i(y_i/\mu_i-1) \end{pmatrix}.

(e) Why this is not a lognormal model. The model specifies a Gamma distribution for Y_i and a linear predictor for \log\mu_i=\log\mathbb{E}(Y_i). It does not specify a Normal distribution for \log Y_i. Applying a log link to a conditional mean is not the same operation as transforming the response and assuming Normal errors.

Key result. Keep the response distribution and the link separate: Gamma response plus log link does not mean lognormal response.

C4. Exponential family

Question page: C4. Exponential family

What is an exponential family? Identify its natural parameter and statistic.

Definitions/notation. An exponential family has density or mass function f(x\mid\theta)=b(x)\exp\{c(\theta)^\top T(x)\}/a(\theta), where c(\theta) is the natural parameter, T(x) the natural statistic, b(x) a parameter-free base factor and a(\theta) the normalising function. In the regular form used here, the support is independent of \theta.

Used in

See also

Canonical link · Generalised linear model · Pitman–Koopman–Darmois, informal form · Mean and variance of an exponential family

Glossary and index