Exponential families and conjugacy
Part of Bayesian statistics · 18 questions on this topic · What links here
Exponential families and conjugacy
| Part | Bayesian statistics |
| Chapter | 7 of 9 in this part |
| Previous | Bayesian hypotheses and model comparison |
| Next | Hierarchical Bayes |
| Questions | 18 |
| Definitions | Statistic and estimator · Sufficient statistic · Minimal sufficiency · Complete statistic · Ancillary statistic · Natural exponential family |
| Results | Fisher–Neyman factorisation · Basu’s theorem · Log-partition moment identities · Pitman–Koopman–Darmois, informal form · Generic exponential-family conjugate update |
The conjugate calculations in the Beta–Binomial, Normal–Normal, and Gamma–Poisson models are not isolated coincidences. They are instances of a general pattern: the likelihood compresses the observations into sufficient statistics, and a matching prior allows those statistics to update a fixed number of hyperparameters. This page isolates that algebraic mechanism. Linking related groups through a hierarchy is a different idea: the prior parameters become uncertain population parameters, and the update has to carry that extra uncertainty.1
Statistics and inferential reduction
Definition 1 (Statistic and estimator) Write the complete sample as {\boldsymbol{X}}=(X_1,\ldots,X_n) and its realised value as {\boldsymbol{x}}=(x_1,\ldots,x_n). A statistic is any function
T=T({\boldsymbol{X}})
of the random data alone. It cannot contain an unknown parameter. An estimator is a statistic used to estimate a parameter or a function of one, so every estimator is a statistic but not every statistic is introduced for estimation. For example, \overline X, \max_iX_i, and \sum_i(X_i-\overline X)^2 are all statistics.
The same mathematical expression p({\boldsymbol{x}}\mid\theta) is viewed in two different ways:
as a sampling density, {\boldsymbol{x}} varies and \theta is fixed;
as a likelihood, the observed {\boldsymbol{x}} is fixed and \theta varies.
The likelihood is therefore a function of \theta; it is not generally a probability density in \theta. A prior is required before it can be normalised to form a posterior.
Statistical reduction asks whether the full data vector can be replaced by a lower-dimensional statistic without losing information about \theta under the assumed model. This is the role of sufficiency.
Sufficiency is not a claim that the discarded features are useless for every purpose. It says they add no information about the named parameter within the assumed model. Residual patterns can still be vital for checking that model. The parameter, the model, and the inferential purpose therefore belong in any careful statement that a statistic is sufficient.
Sufficiency and the Fisher–Neyman theorem
Definition 2 A statistic T({\boldsymbol{X}}) is sufficient for \theta if the conditional distribution of {\boldsymbol{X}} given T({\boldsymbol{X}})=t does not depend on \theta.
Once T is known, the remaining details of the sample may still be useful for checking the model or answering other questions, but they contain no further information about \theta within that model. Sufficiency is therefore model-relative. A statistic sufficient when a nuisance parameter is known need not remain sufficient when that parameter is also unknown.
The conditional definition is conceptually clean but awkward to verify directly. The standard practical tool is the following theorem.
Theorem 1 (Fisher–Neyman factorisation) Suppose the family \{p({\boldsymbol{x}}\mid\theta):\theta\in\Omega\} is dominated by a common measure. Then T({\boldsymbol{X}}) is sufficient for \theta if and only if there are nonnegative functions g_\theta and h such that p({\boldsymbol{x}}\mid\theta) =g_\theta\{T({\boldsymbol{x}})\}h({\boldsymbol{x}}), \qquad \theta\in\Omega, \qquad(1) where h does not depend on \theta.
The factorisation says that every part of the likelihood that depends on \theta must pass through T({\boldsymbol{x}}). For a discrete sample, the easy direction of the theorem can be seen directly. If T({\boldsymbol{x}})=t, then \begin{align*} \mathbb{P}_\theta({\boldsymbol{X}}={\boldsymbol{x}}\mid T=t) &=\frac{p({\boldsymbol{x}}\mid\theta)} {\sum_{{\boldsymbol{u}}:T({\boldsymbol{u}})=t} p({\boldsymbol{u}}\mid\theta)}\\ &=\frac{g_\theta(t)h({\boldsymbol{x}})} {\sum_{{\boldsymbol{u}}:T({\boldsymbol{u}})=t} g_\theta(t)h({\boldsymbol{u}})}\\ &=\frac{h({\boldsymbol{x}})} {\sum_{{\boldsymbol{u}}:T({\boldsymbol{u}})=t}h({\boldsymbol{u}})}. \end{align*} The final expression contains no \theta. The measure-theoretic theorem extends the same cancellation to continuous and mixed models.
The Bayesian consequence is immediate. If the prior density is \pi(\theta), then \begin{align*} p(\theta\mid{\boldsymbol{x}}) &\propto p({\boldsymbol{x}}\mid\theta)\pi(\theta)\\ &=g_\theta\{T({\boldsymbol{x}})\}h({\boldsymbol{x}})\pi(\theta)\\ &\propto g_\theta\{T({\boldsymbol{x}})\}\pi(\theta), \end{align*} because h({\boldsymbol{x}}) is constant in \theta. Thus two samples having the same sufficient statistic produce the same posterior under the same prior. This is why conjugate updates can often be written using only a count, a sum, or a pair of sums.
Worked factorisation: a normal mean
Example 1 (Sufficiency for a normal mean) Suppose
X_1,\ldots,X_n\mid\mu \stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2), \qquad \sigma^2\text{ known}.
Question. Use the Fisher–Neyman factorisation theorem to find a sufficient statistic for \mu. Then identify a sufficient statistic when both \mu and \sigma^2 are unknown.
Solution. The joint density is \begin{align*} p({\boldsymbol{x}}\mid\mu) &=(2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac1{2\sigma^2} \sum_{i=1}^n(x_i-\mu)^2\right\}. \end{align*} Expand the quadratic: \begin{align*} \sum_{i=1}^n(x_i-\mu)^2 &=\sum_{i=1}^n(x_i^2-2\mu x_i+\mu^2)\\ &=\sum_{i=1}^nx_i^2 -2\mu\sum_{i=1}^nx_i +n\mu^2. \end{align*} Substitution gives \begin{align*} p({\boldsymbol{x}}\mid\mu) &=\underbrace{ (2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac1{2\sigma^2}\sum_{i=1}^nx_i^2\right\}} _{h({\boldsymbol{x}})}\\ &\quad\times \underbrace{ \exp\!\left\{ \frac{\mu}{\sigma^2}\sum_{i=1}^nx_i -\frac{n\mu^2}{2\sigma^2}\right\}} _{g_\mu\{T({\boldsymbol{x}})\}}. \end{align*} All dependence on \mu passes through
T({\boldsymbol{x}})=\sum_{i=1}^nx_i.
The factorisation theorem therefore proves that \sum_iX_i is sufficient for \mu. Since \overline X=T/n is a one-to-one function of T when n is fixed, \overline X is equivalent and is also sufficient.
If both \mu and \sigma^2 are unknown, the displayed factorisation no longer isolates all parameter dependence through \sum_iX_i. A sufficient statistic for (\mu,\sigma^2) is instead
\left(\sum_{i=1}^nX_i,\sum_{i=1}^nX_i^2\right),
or equivalently
\left(\overline X,\sum_{i=1}^n(X_i-\overline X)^2\right).
This is the nuisance-parameter warning in concrete form.
Minimal sufficiency, completeness, and ancillarity
Definition 3 (Minimal sufficiency) Sufficiency alone does not guarantee the greatest possible compression: the entire sample is trivially sufficient. A statistic T is minimal sufficient if it is a function of every other sufficient statistic.
Under common-support regularity conditions, a useful likelihood-ratio criterion is \frac{p({\boldsymbol{x}}\mid\theta)}{p({\boldsymbol{y}}\mid\theta)} \text{ is constant in }\theta \quad\Longleftrightarrow\quad T({\boldsymbol{x}})=T({\boldsymbol{y}}). \qquad(2)
For the known-variance normal model, \begin{align*} \frac{p({\boldsymbol{x}}\mid\mu)}{p({\boldsymbol{y}}\mid\mu)} &=\exp\!\left\{ -\frac{\sum_i x_i^2-\sum_i y_i^2}{2\sigma^2} +\frac{\mu}{\sigma^2} \left(\sum_i x_i-\sum_i y_i\right) \right\}. \end{align*} The first term is independent of \mu. The second is independent of \mu if and only if \sum_i x_i=\sum_i y_i. Therefore \sum_iX_i, or equivalently \overline X, is minimal sufficient for \mu.
Definition 4 A statistic T is complete for a family if
\mathbb{E}_\theta\{q(T)\}=0,\qquad \theta\in\Omega,
implies q(T)=0 almost surely for every \theta\in\Omega.
Sufficiency concerns lossless data reduction; completeness concerns whether a nonzero function of the reduced data can have mean zero for every parameter value. A complete sufficient statistic is minimal sufficient. The Lehmann–Scheffé theorem further says that an unbiased estimator which is a function of a complete sufficient statistic is the unique uniformly minimum-variance unbiased estimator of its target.
Completeness must not be claimed merely because a model has been written in exponential-family notation. For a full-rank exponential family whose natural parameter space contains an open set of the appropriate dimension, the natural sufficient statistic is complete under standard integrability conditions. Curved or restricted subfamilies need not inherit this property.
Definition 5 (Ancillary statistic) A statistic A({\boldsymbol{X}}) is ancillary if its sampling distribution does not depend on \theta.
Theorem 2 (Basu’s theorem) A complete sufficient statistic is independent of every ancillary statistic.
In the normal model with known \sigma^2, \overline X carries the information about \mu while the centred residual pattern (X_1-\overline X,\ldots,X_n-\overline X) is ancillary for \mu. The distinction helps separate parameter information from model-diagnostic variation.
Why exponential families keep appearing
Definition 6 (Natural exponential family) In the a–b–c–T notation used here, the natural or canonical form of an exponential-family density is p(x\mid\theta) =\frac{b(x)\exp\{c(\theta)^{\mathsf T}T(x)\}} {a(\theta)}, \qquad(3) where c(\theta) is the natural or canonical parameter, T(x) is the natural statistic, b(x) does not depend on \theta, and a(\theta) is the normalising function.
Remark 1 (Translation to another common notation). Some books write the same density as
p(x\mid{\boldsymbol{\eta}}) =h(x)\exp\{{\boldsymbol{\eta}}^{\mathsf T}T(x)-A({\boldsymbol{\eta}})\}.
This is only a change of names: {\boldsymbol{\eta}}=c(\theta), h(x)=b(x), and A({\boldsymbol{\eta}})=\log a(\theta). We retain the a(\theta)–b(x)–c(\theta)–T(x) notation from this point onward.
For an independent sample, define once and for all S({\boldsymbol{X}})=\sum_{i=1}^nT(X_i). \qquad(4) Then \begin{align*} p({\boldsymbol{x}}\mid\theta) &=\prod_{i=1}^n \frac{b(x_i)\exp\{c(\theta)^{\mathsf T}T(x_i)\}} {a(\theta)}\\ &=\left\{\prod_{i=1}^n b(x_i)\right\} \frac{\exp\{c(\theta)^{\mathsf T}S({\boldsymbol{x}})\}} {a(\theta)^n}. \end{align*} The Fisher–Neyman theorem therefore gives the fixed-dimensional sufficient statistic S({\boldsymbol{X}}). Its dimension does not grow with the sample size.
This is why exponential families keep reappearing in elementary Bayesian calculations. They turn an ever-growing data set into a fixed collection of updated totals, and conjugate priors update those totals by addition. The pattern explains the familiar formulas; it does not imply that every useful model must be exponential-family or that a nonconjugate prior is somehow illegitimate.
Proposition 1 (Log-partition moment identities) Whenever c(\theta) can be used as a local parameter and differentiation under the normalising integral is valid,
\nabla_c\log a(\theta) =\mathbb{E}_\theta\{T(X)\}, \qquad \nabla_c^2\log a(\theta) =\operatorname{Cov}_\theta\{T(X)\}.
Here derivatives are taken with respect to c=c(\theta), so a(\theta) is regarded as a function of c through the chosen parametrisation. Consequently \log a(\theta) is convex as a function of c, and the one-observation Fisher information in the natural parameter is \nabla_c^2\log a(\theta).
The density must integrate to one, so the normalising function is
a(\theta) =\int b(x)\exp\{c(\theta)^{\mathsf T}T(x)\}\,\mathrm{d} x.
Assuming differentiation under the integral is valid, \begin{align*} \nabla_c\log a(\theta) &=\frac{\nabla_ca(\theta)}{a(\theta)}\\ &=\frac{ \int T(x)b(x) \exp\{c(\theta)^{\mathsf T}T(x)\}\,\mathrm{d} x} {a(\theta)}\\ &=\mathbb{E}_\theta\{T(X)\}. \end{align*} Differentiating once more, \begin{align*} \nabla_c^2\log a(\theta) &=\mathbb{E}_\theta\{T(X)T(X)^{\mathsf T}\} -\mathbb{E}_\theta\{T(X)\} \mathbb{E}_\theta\{T(X)\}^{\mathsf T}\\ &=\operatorname{Cov}_\theta\{T(X)\}. \end{align*} Consequently \log a(\theta) is convex as a function of c. For an iid sample the log-likelihood, score, and Fisher information in the natural parameter are \begin{align*} \ell(\theta;{\boldsymbol{x}}) &=c(\theta)^{\mathsf T}S({\boldsymbol{x}}) -n\log a(\theta)+\text{constant},\\ \nabla_c\ell(\theta;{\boldsymbol{x}}) &=S({\boldsymbol{x}})-n\nabla_c\log a(\theta),\\ \mathcal I_n(c) &=-\mathbb{E}_\theta \{\nabla_c^2\ell(\theta;{\boldsymbol{X}})\}\\ &=n\nabla_c^2\log a(\theta)\\ &=n\operatorname{Cov}_\theta\{T(X)\}. \end{align*} The likelihood equations therefore match the observed sufficient statistic to its model expectation:
S({\boldsymbol{x}})=n\mathbb{E}_{\widehat\theta}\{T(X)\}.
Theorem 3 (Pitman–Koopman–Darmois, informal form) For regular iid models with parameter-independent support, the theorem says, roughly, that the existence for every n of a sufficient statistic whose dimension stays fixed essentially characterises exponential families. The support condition matters: models such as \operatorname{Uniform}(0,\theta) can have a fixed-dimensional sufficient statistic without having the regular form Equation 3.
Conjugate priors from the exponential-family form
Not every likelihood has a useful finite-dimensional conjugate family. A large and important class that does is the exponential family. Keeping the same notation, one observation has density or mass function p(x\mid\theta) =\frac{b(x)\exp\{c(\theta)^{\mathsf T}T(x)\}} {a(\theta)}, \qquad(5) where T(x) is the one-observation natural statistic, c(\theta) is the canonical parameter, b(x) does not depend on \theta, and a(\theta) is the normalising function from Equation 3.
Choose the prior kernel, where \alpha has the same dimension as c(\theta) and \beta is scalar, p(\theta\mid\alpha,\beta) \propto \frac{\exp\{c(\theta)^{\mathsf T}\alpha\}} {a(\theta)^\beta}. \qquad(6) For independent data x_1,\ldots,x_n, the likelihood is \begin{align*} p({\boldsymbol{x}}\mid\theta) &=\prod_{i=1}^n \frac{b(x_i)\exp\{c(\theta)^{\mathsf T}T(x_i)\}} {a(\theta)}\\ &=\left(\prod_{i=1}^nb(x_i)\right) \frac{ \exp\{c(\theta)^{\mathsf T}S({\boldsymbol{x}})\}} {a(\theta)^n}. \end{align*} Multiplying by the prior gives \begin{align*} p(\theta\mid{\boldsymbol{x}}) &\propto \frac{\exp\{c(\theta)^{\mathsf T}S({\boldsymbol{x}})\}} {a(\theta)^n} \frac{\exp\{c(\theta)^{\mathsf T}\alpha\}} {a(\theta)^\beta}\\ &=\frac{ \exp\{c(\theta)^{\mathsf T} (\alpha+S({\boldsymbol{x}}))\}} {a(\theta)^{\beta+n}}. \end{align*}
Proposition 2 (Generic exponential-family conjugate update) Thus the hyperparameters update as \boxed{ \alpha\leftarrow \alpha+S({\boldsymbol{x}}), \qquad \beta\leftarrow\beta+n.} \qquad(7)
Poisson example and Gamma conventions
Example 2 (Poisson natural form and Gamma conjugacy) Question. For iid Poisson observations with mean \theta, write the model in the a(\theta), b(x), c(\theta), T(x) form, identify the sufficient statistic, and derive the corresponding conjugate prior and posterior update.
Solution. For one Poisson observation, \begin{align*} p(x\mid\theta) &=\frac{e^{-\theta}\theta^x}{x!}\\ &=\frac1{x!}\exp\{x\log\theta\}e^{-\theta}\\ &=\frac{b(x)\exp\{c(\theta)T(x)\}}{a(\theta)}, \end{align*} with
b(x)=\frac1{x!}, \qquad c(\theta)=\log\theta, \qquad T(x)=x, \qquad a(\theta)=e^\theta.
For the Poisson model, the definition Equation 4 gives S({\boldsymbol{x}})=\sum_i x_i. The joint likelihood is \begin{align*} p({\boldsymbol{x}}\mid\theta) &=\prod_{i=1}^n\frac{e^{-\theta}\theta^{x_i}}{x_i!}\\ &=\left(\prod_{i=1}^n\frac1{x_i!}\right) \exp\{(\log\theta)S({\boldsymbol{x}})-n\theta\}. \end{align*} The factorisation theorem shows that S({\boldsymbol{X}}) is sufficient for \theta. It is also minimal sufficient by the likelihood-ratio criterion. Because the natural parameter c(\theta)=\log\theta ranges over the open set \mathbb{R}, S({\boldsymbol{X}}) is complete as well.
The generic prior Equation 6 becomes \begin{align*} p(\theta\mid\alpha,\beta) &\propto \frac{\exp\{\alpha\log\theta\}}{(e^\theta)^\beta}\\ &=\theta^\alpha e^{-\beta\theta}. \end{align*} This is the kernel of \operatorname{Gamma}(\alpha+1,\beta), not \operatorname{Gamma}(\alpha,\beta). Using the sufficient statistic S({\boldsymbol{x}}), \begin{align*} p(\theta\mid{\boldsymbol{x}}) &\propto \theta^{S({\boldsymbol{x}})} e^{-n\theta}\theta^\alpha e^{-\beta\theta}\\ &=\theta^{S({\boldsymbol{x}})+\alpha}e^{-(n+\beta)\theta}, \end{align*} so \boxed{ \theta\mid{\boldsymbol{x}} \sim\operatorname{Gamma}(\alpha+S({\boldsymbol{x}})+1,\beta+n)}. \qquad(8) Its posterior mean is \begin{align*} \mathbb{E}(\theta\mid{\boldsymbol{x}}) &=\frac{\alpha+S({\boldsymbol{x}})+1}{\beta+n}\\ &=\frac{(\alpha+1)+n\overline x}{\beta+n}\\ &=\frac\beta{\beta+n}\frac{\alpha+1}{\beta} +\frac n{\beta+n}\overline x. \end{align*} The prior therefore behaves, for the mean, as \beta prior observations with total count \alpha+1.
In this exponent convention, letting \alpha\to0 and \beta\to0 produces the improper flat kernel p(\theta)\propto1. The Poisson Jeffreys prior is p(\theta)\propto\theta^{-1/2}, corresponding to \alpha=-1/2,\beta=0. The scale-invariant kernel p(\theta)\propto1/\theta corresponds to \alpha=-1,\beta=0. These three improper priors are different and lead to Gamma shapes S+1, S+1/2, and S, respectively.
Improper priors can be useful limiting devices, but they create enough risks that posterior propriety must be checked rather than admired.
Conjugacy in an exponential family updates a fixed number of hyperparameters with sufficient statistics. Hierarchical Bayes, developed next, addresses a separate modelling problem: how to represent and learn from variation among related groups.
Questions on this topic
Open a question to attempt it. Its worked answer is folded away at the bottom of the page.
What links here
Pages (8)
- Estimation — via sufficient statistic
- The Beta–Binomial model — via Conjugacy
- Conjugate analysis for normal models — via natural parameter
- Categorical data and the Dirichlet distribution — via conjugate prior
- Hierarchical Bayes — via conjugacy
- Bayesian linear regression — via conjugacy
- Introduction to generalised linear models — via sufficient statistic
- How to use the final preparation — via conjugacy
Questions and answers (18)
- Find the prior rather than recalling it · Factorisation and data reduction · Sufficiency and the posterior · Exponential observations parameterised by rate · Exponential observations parameterised by mean · Q1 · Q13 · Q23 (answer) · Q24 (answer) · Q28 · Q58 · Question 2 (answer) · Question 4 · Question 5 (answer) · Question 7 · C2 · C3 · C4
See also
- Articles from this chapter: Statistic and estimator · Sufficient statistic · Fisher–Neyman factorisation · Minimal sufficiency · Complete statistic · Ancillary statistic · Basu’s theorem · Natural exponential family · Translation to another common notation · Log-partition moment identities · Pitman–Koopman–Darmois, informal form · Generic exponential-family conjugate update
- All bayesian statistics chapters
- Glossary and index
- Distribution sheet and formula lookup
Footnotes
Conjugacy is not needed for Bayesian inference. Metropolis–Hastings mostly asks for ratios of target densities; Gibbs sampling benefits more directly from named full conditional distributions. That computational distinction is a better reason to value conjugacy than mere nostalgia for closed forms.↩︎