Probability spaces and events

Part of Stochastic processes · 3 questions on this topic · What links here

Probability spaces and events

Part Stochastic processes
Chapter 9 of 12 in this part
Previous Birth–death chains and queues
Next Random variables and laws
Questions 3
Definitions Probability space
Results Borel–Cantelli lemmas

Probability has to say which outcomes and which events are being discussed. On a finite set every subset can be an event. On an uncountable set, a sigma-algebra is a collection large enough for the countable operations the later arguments use.

Finite sample spaces, equally likely outcomes, the elementary axioms, conditional probability, and independence of events are on Random experiments and probability. This page keeps those identities and adds the sigma-algebra, countable additivity in the form needed for tail events, continuity along monotone sequences, and the Borel–Cantelli lemmas. The random variables built on this space are on Random variables and laws.

Outcomes and events

A sample space \Omega is the set of elementary outcomes. An event is a subset of \Omega. The complement A^c is the event that A does not occur, the union A\cup B is the event that at least one occurs, and the intersection A\cap B is the event that both occur. Disjoint events have empty intersection. A partition is a family of disjoint events whose union is \Omega. Conditioning on the cells of a partition turns one probability into a sum of simpler ones, which is the law of total probability.

Definition 1 (Probability space) A sigma-algebra \mathcal{F} on \Omega contains \Omega, is closed under complements, and is closed under countable unions. It is then closed under countable intersections, by taking complements. A probability measure \mathrm{P}:\mathcal{F}\to[0,1] satisfies \mathrm{P}(\Omega)=1 and is countably additive: for pairwise disjoint events A_n\in\mathcal{F},

\mathrm{P}\Bigl(\bigcup_{n=1}^{\infty} A_n\Bigr)=\sum_{n=1}^{\infty}\mathrm{P}(A_n).

The triple (\Omega,\mathcal{F},\mathrm{P}) is a probability space. The probability of an event A is written \mathbb{P}(A) as well as \mathrm{P}(A).

Article: Probability space →

The three axioms on Random experiments and probability are this definition specialised to the events that page uses. Closure under countable unions is what makes an event such as “infinitely many of the A_n occur” an event. That event is

\limsup_{n\to\infty} A_n=\bigcap_{m\ge 1}\bigcup_{n\ge m} A_n,

the event that belongs to every tail union. It is the event in the Borel–Cantelli lemmas.

Generated and Borel sigma-algebras

The sigma-algebra generated by a collection \mathcal{C}, written \sigma(\mathcal{C}), is the smallest sigma-algebra containing \mathcal{C}. For a single set A it is \{\emptyset, A, A^c, \Omega\}, with duplicates removed if A is empty or all of \Omega. For two sets A and B, the four atoms

A\cap B,\qquad A\cap B^c,\qquad A^c\cap B,\qquad A^c\cap B^c

form a partition of \Omega. Every set in \sigma(A,B) is a union of some of these atoms.

On \mathbb{R}, the Borel sigma-algebra \mathcal{B}(\mathbb{R}) is the sigma-algebra generated by the open intervals. It contains the closed intervals, the half-open intervals, and every countable set, because each singleton is the intersection of a shrinking sequence of open intervals. Calling a set a Borel event says that it belongs to this sigma-algebra. It does not say that every subset of \mathbb{R} is Borel.

Consequences of the axioms

Countable additivity implies the properties already listed for a finite experiment, and two limit properties that need the countable axiom.

\mathrm{P}(\emptyset)=0, because \emptyset,\emptyset,\ldots is a disjoint sequence whose union is \emptyset, so \mathrm{P}(\emptyset)=\sum\mathrm{P}(\emptyset) forces the value 0. Finite additivity is countable additivity with the later sets empty. If A\subset B, then B=A\cup(B\setminus A) is a disjoint union, so \mathrm{P}(A)\le\mathrm{P}(B). The inclusion-exclusion identity

\mathrm{P}(A\cup B)=\mathrm{P}(A)+\mathrm{P}(B)-\mathrm{P}(A\cap B)

is the same rearrangement as on the experiments page: add the two probabilities, and put back the overlap that was counted twice.

The union bound replaces an overlapping union by a larger disjoint one. Define B_1=A_1 and B_n=A_n\setminus(A_1\cup\cdots\cup A_{n-1}). The B_n are disjoint, their union equals \bigcup A_n, and B_n\subset A_n, so

\mathrm{P}\Bigl(\bigcup_{n=1}^{\infty} A_n\Bigr)=\sum_{n=1}^{\infty}\mathrm{P}(B_n)\le\sum_{n=1}^{\infty}\mathrm{P}(A_n).

If A_n\uparrow A, meaning A_n\subset A_{n+1} and \bigcup_n A_n=A, continuity from below says \mathrm{P}(A_n)\uparrow\mathrm{P}(A). The increments A_1, A_2\setminus A_1, A_3\setminus A_2, \ldots are disjoint and sum to A, so \mathrm{P}(A)=\mathrm{P}(A_1)+\sum_{n\ge 2}\mathrm{P}(A_n\setminus A_{n-1}), and the partial sum of that series is \mathrm{P}(A_n).

If A_n\downarrow A, continuity from above says \mathrm{P}(A_n)\downarrow\mathrm{P}(A). Apply continuity from below to the complements: A_n^c\uparrow A^c, so \mathrm{P}(A_n^c)\uparrow\mathrm{P}(A^c), and subtract from 1. For a probability the first set automatically has finite measure, which is the extra hypothesis the same argument needs for a general measure.

Borel–Cantelli

Theorem 1 (Borel–Cantelli lemmas) If \sum_n\mathrm{P}(A_n)<\infty, then \mathrm{P}(A_n\text{ infinitely often})=0. If the events A_n are independent and \sum_n\mathrm{P}(A_n)=\infty, then \mathrm{P}(A_n\text{ infinitely often})=1.

Article: Borel–Cantelli lemmas →

For the first statement, the union bound on the tail gives

\mathrm{P}\Bigl(\bigcup_{n\ge m} A_n\Bigr)\le\sum_{n\ge m}\mathrm{P}(A_n).

The right-hand side tends to 0 as m\to\infty whenever the series converges. The event that infinitely many A_n occur is contained in every tail union, so its probability is 0. Equivalently, the tail unions decrease to that event, and continuity from above passes the limit inside \mathrm{P}. Independence is not used.

For the second statement, independence turns the probability of a long run of failures into a product. For N\ge m,

\mathrm{P}\Bigl(\bigcap_{n=m}^{N} A_n^c\Bigr)=\prod_{n=m}^{N}\bigl(1-\mathrm{P}(A_n)\bigr)\le\exp\Bigl(-\sum_{n=m}^{N}\mathrm{P}(A_n)\Bigr),

because 1-x\le e^{-x} for x\in[0,1]. If the series of probabilities diverges, the exponent tends to -\infty as N\to\infty, and the probability that every A_n with n\ge m fails is 0. Every tail therefore contains at least one occurrence, with probability 1. That is the event of infinitely many occurrences.

Divergence without independence does not give the second conclusion. Repeat one event A of probability 1/2 at every n. The probabilities sum to infinity, and \{A_n\text{ infinitely often}\}=A still has probability 1/2.

Conditional probability and independence

When \mathrm{P}(B)>0,

\mathrm{P}(A\mid B)=\frac{\mathrm{P}(A\cap B)}{\mathrm{P}(B)}.

This is the definition on Random experiments and probability. If (B_i) is a finite or countable partition into events of positive probability,

\mathrm{P}(A)=\sum_i\mathrm{P}(A\mid B_i)\,\mathrm{P}(B_i).

Bayes’ rule divides one term of that sum by the whole sum:

\mathrm{P}(B_j\mid A)=\frac{\mathrm{P}(A\mid B_j)\,\mathrm{P}(B_j)}{\sum_i\mathrm{P}(A\mid B_i)\,\mathrm{P}(B_i)}.

The formulas are identities for events. They do not require a density. The density form used to update a prior is on Bayesian foundations.

Two events are independent when \mathrm{P}(A\cap B)=\mathrm{P}(A)\mathrm{P}(B), the definition on Random experiments and probability. For a collection, pairwise independence is weaker than mutual independence. For three events, the three pairwise product formulas do not imply \mathrm{P}(A\cap B\cap C)=\mathrm{P}(A)\mathrm{P}(B)\mathrm{P}(C).

Example 1 (Pairwise, and not mutual) Let U and V be independent fair bits, and set A=\{U=1\}, B=\{V=1\}, and C=\{U\neq V\}.

Question. Check pairwise independence, and check mutual independence.

Solution. Each of A, B, and C has probability 1/2. Also \mathrm{P}(A\cap B)=\mathrm{P}(U=V=1)=1/4=\mathrm{P}(A)\mathrm{P}(B). The same count gives \mathrm{P}(A\cap C)=\mathrm{P}(U=1,V=0)=1/4 and \mathrm{P}(B\cap C)=1/4. Every pair is independent. The triple intersection A\cap B\cap C requires U=V and U\neq V, so it is empty and has probability 0, not 1/8. The three events are not mutually independent.

The second Borel–Cantelli lemma assumes independence of the family used in the product, not only pairwise products. A pairwise condition would not justify writing the probability of the intersection as the product over the whole tail.

Countable operations on a sequence of events

For a sequence of events A_n:

  • at least one occurs on \bigcup_n A_n;
  • all of them occur on \bigcap_n A_n;
  • infinitely many occur on \limsup_n A_n=\bigcap_m\bigcup_{n\ge m} A_n;
  • all but finitely many occur on \liminf_n A_n=\bigcup_m\bigcap_{n\ge m} A_n.

The limsup asks for an occurrence in every tail. The liminf asks for one tail in which every event occurs. Almost-sure convergence is a statement that a limsup of deviation events has probability 0, which is why the first Borel–Cantelli lemma is the bridge from a summable series of probabilities to a pathwise limit on Expectation and convergence.

Key idea

A probability space is a sample space, a sigma-algebra of events, and a countably additive probability. Countable unions make tail events measurable. If the probabilities of A_n sum to a finite number, only finitely many occur, with probability 1. If the events are independent and the probabilities sum to infinity, infinitely many occur, with probability 1.

Questions on this topic

Open a question to attempt it. Its worked answer is folded away at the bottom of the page.

Question Answer
The algebra generated by two sets ✓
Two series of events ✓
Continuity from above ✓

See also