Sufficient statistic

Definition taught in Exponential families and conjugacy, part of Bayesian statistics

Sufficient statistic

Type Definition
Taught in Exponential families and conjugacy
Part Bayesian statistics
Properties 3
Worked examples 3
Questions using it 4

A statistic T({\boldsymbol{X}}) is sufficient for \theta if the conditional distribution of {\boldsymbol{X}} given T({\boldsymbol{X}})=t does not depend on \theta.

Properties

Fisher–Neyman factorisation

Main article: Fisher–Neyman factorisation

Suppose the family \{p({\boldsymbol{x}}\mid\theta):\theta\in\Omega\} is dominated by a common measure. Then T({\boldsymbol{X}}) is sufficient for \theta if and only if there are nonnegative functions g_\theta and h such that p({\boldsymbol{x}}\mid\theta) =g_\theta\{T({\boldsymbol{x}})\}h({\boldsymbol{x}}), \qquad \theta\in\Omega, where h does not depend on \theta.

Basu’s theorem

Main article: Basu’s theorem

A complete sufficient statistic is independent of every ancillary statistic.

Pitman–Koopman–Darmois, informal form

Main article: Pitman–Koopman–Darmois, informal form

For regular iid models with parameter-independent support, the theorem says, roughly, that the existence for every n of a sufficient statistic whose dimension stays fixed essentially characterises exponential families. The support condition matters: models such as \operatorname{Uniform}(0,\theta) can have a fixed-dimensional sufficient statistic without having the regular form Equation (3) in Exponential families and conjugacy.

Examples

Sufficiency for a normal mean

Worked example from Exponential families and conjugacy

Suppose

X_1,\ldots,X_n\mid\mu \stackrel{\mathrm{iid}}{\sim}\operatorname{N}(\mu,\sigma^2), \qquad \sigma^2\text{ known}.

Question. Use the Fisher–Neyman factorisation theorem to find a sufficient statistic for \mu. Then identify a sufficient statistic when both \mu and \sigma^2 are unknown.

Solution. The joint density is \begin{align*} p({\boldsymbol{x}}\mid\mu) &=(2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac1{2\sigma^2} \sum_{i=1}^n(x_i-\mu)^2\right\}. \end{align*} Expand the quadratic: \begin{align*} \sum_{i=1}^n(x_i-\mu)^2 &=\sum_{i=1}^n(x_i^2-2\mu x_i+\mu^2)\\ &=\sum_{i=1}^nx_i^2 -2\mu\sum_{i=1}^nx_i +n\mu^2. \end{align*} Substitution gives \begin{align*} p({\boldsymbol{x}}\mid\mu) &=\underbrace{ (2\pi\sigma^2)^{-n/2} \exp\!\left\{-\frac1{2\sigma^2}\sum_{i=1}^nx_i^2\right\}} _{h({\boldsymbol{x}})}\\ &\quad\times \underbrace{ \exp\!\left\{ \frac{\mu}{\sigma^2}\sum_{i=1}^nx_i -\frac{n\mu^2}{2\sigma^2}\right\}} _{g_\mu\{T({\boldsymbol{x}})\}}. \end{align*} All dependence on \mu passes through

T({\boldsymbol{x}})=\sum_{i=1}^nx_i.

The factorisation theorem therefore proves that \sum_iX_i is sufficient for \mu. Since \overline X=T/n is a one-to-one function of T when n is fixed, \overline X is equivalent and is also sufficient.

If both \mu and \sigma^2 are unknown, the displayed factorisation no longer isolates all parameter dependence through \sum_iX_i. A sufficient statistic for (\mu,\sigma^2) is instead

\left(\sum_{i=1}^nX_i,\sum_{i=1}^nX_i^2\right),

or equivalently

\left(\overline X,\sum_{i=1}^n(X_i-\overline X)^2\right).

This is the nuisance-parameter warning in concrete form.

Sufficiency and the posterior

Question page: Sufficiency and the posterior

Let {\boldsymbol{X}}=(X_1,\ldots,X_n)^{\mathsf T} have likelihood p({\boldsymbol{x}}\mid\theta) and prior p(\theta). If U({\boldsymbol{X}}) is sufficient for \theta, use the Fisher–Neyman factorisation theorem and Bayes’ rule to show that the posterior depends on the observed sample only through U({\boldsymbol{x}}). Retain the normalising integral in the derivation.

Question. Let {\boldsymbol{X}}=(X_1,\ldots,X_n)^{\mathsf T} have likelihood p({\boldsymbol{x}}\mid\theta) and prior p(\theta). If U({\boldsymbol{X}}) is sufficient for \theta, use the Fisher–Neyman factorisation theorem and Bayes’ rule to show that the posterior depends on the observed sample only through U({\boldsymbol{x}}). Retain the normalising integral in the derivation.

Solution. By the Fisher–Neyman factorisation theorem, sufficiency of U({\boldsymbol{X}}) implies that there are nonnegative functions g and h such that

p({\boldsymbol{x}}\mid\theta)=g\{U({\boldsymbol{x}}),\theta\}h({\boldsymbol{x}}),

where h does not depend on \theta. Bayes’ rule, including its normalising integral, gives \begin{align*} p(\theta\mid{\boldsymbol{x}}) &=\frac{p({\boldsymbol{x}}\mid\theta)p(\theta)} {\displaystyle\int p({\boldsymbol{x}}\mid t)p(t)\,\mathrm{d} t}\\ &=\frac{g\{U({\boldsymbol{x}}),\theta\}h({\boldsymbol{x}})p(\theta)} {\displaystyle\int g\{U({\boldsymbol{x}}),t\}h({\boldsymbol{x}})p(t)\,\mathrm{d} t}\\ &=\boxed{ \frac{g\{U({\boldsymbol{x}}),\theta\}p(\theta)} {\displaystyle\int g\{U({\boldsymbol{x}}),t\}p(t)\,\mathrm{d} t}}. \end{align*} Every occurrence of the observed sample is through U({\boldsymbol{x}}); the factor h({\boldsymbol{x}}) cancels because it is constant with respect to the integration variable and the parameter.

C3. Sufficiency

Question page: C3. Sufficiency

What is a sufficient statistic, using the conditional-distribution definition?

Definitions/notation. A statistic T(X) is a function of the sample alone and is sufficient for \theta if the conditional distribution of X given T(X) does not depend on \theta. Thus, within the assumed model, the remaining sample details add no information about \theta once T(X) is known.

Used in

See also

Basu’s theorem · Minimal sufficiency · Pitman–Koopman–Darmois, informal form · Statistic and estimator · Fisher–Neyman factorisation · Complete statistic · Ancillary statistic · Natural exponential family

Glossary and index