Jensen’s inequality
Topic of Expectation and convergence, part of Stochastic processes · Also in Nearest neighbours, Classification trees, Dependence along a series
Jensen’s inequality
| Type | Topic |
| Taught in | Expectation and convergence · Nearest neighbours · Classification trees · Dependence along a series |
| Part | Stochastic processes · Statistical learning · Time series |
If \varphi is convex and X and \varphi(X) are integrable, then \varphi(\mathbb{E}X)\le\mathbb{E}\varphi(X); for concave \varphi the inequality reverses. With \varphi(x)=x^2 it says \operatorname{Var}(X)\ge0.
In Expectation and convergence
Expectation is linear on integrable variables, and it is monotone: X\le Y implies \mathbb{E} X\le\mathbb{E} Y when both expectations exist. If \varphi is convex and X is integrable, with \varphi(X) integrable as well, Jensen’s inequality says
\varphi(\mathbb{E} X)\le\mathbb{E}\varphi(X).
In Nearest neighbours
From Nearest neighbours
where R^*=\mathbb{E}[r(X)] is the Bayes risk, because 1-r\le 1. Jensen’s inequality, using the concavity of u\mapsto u(1-u) on [0,1] together with \mathbb{E}[r(X)]=R^*, gives the sharper Cover–Hart bound
\mathbb{E}\bigl[2r(1-r)\bigr]\le 2R^*(1-R^*).
In Classification trees
From Classification trees
Here m_L and m_R are the children of node m, and \hat p_L(m) is the proportion of the cases at node m that take the left branch. For the Gini index and for entropy, concavity of I as a function of the class proportions gives \Delta I(m)\ge 0: the child proportions average, with weights \hat p_L(m) and 1-\hat p_L(m), back to the parent proportions, so Jensen’s inequality says the parent impurity is at least the weighted average of the child impurities. A split cannot increase the weighted impurity. The misclassification rate is not strictly concave, and a split can leave it unchanged. The search described next maximises \Delta I(m) over variables and cut-points. It is greedy. It does not score the tree that the children will eventually become.
In Dependence along a series
Every weakly stationary series is a moving average of infinite order, plus a deterministic piece. In practice the mean is subtracted first and put back at the end. The modelling choices are then which of those pieces we keep: is the dependence Markov, or seasonal, or both; is the season only in the mean; are the lags added or multiplied? A multiplicative series is often logged, and then the forecast is on the log scale. Jensen’s inequality says \mathrm{E}(\log X)\neq\log\mathrm{E}(X). Exponentiating a forecast of the log is not a forecast of the mean on the original scale.
Used in
See also
Markov’s inequality · Chebyshev’s inequality · Cauchy–Schwarz inequality · Monotone convergence, Fatou, and dominated convergence · Modes of convergence · Strong law · Uniform integrability · Inequalities and limits under the expectation