Exponential loss
Section of Ensembles, random forests, and boosting, part of Statistical learning
Exponential loss
| Type | Section |
| Taught in | Ensembles, random forests, and boosting |
| Part | Statistical learning |
The factor \varepsilon_t/(1-\varepsilon_t) was a choice in the original algorithm. Friedman, Hastie, and Tibshirani (2000) supplied the loss that produces it. The population criterion is the exponential loss
J(f)=\mathbb{E}\bigl[e^{-Y f(X)}\bigr],
where (X,Y) is a fresh draw, Y\in\{-1,+1\}, and f(x) is a real-valued score, positive when the classifier leans toward class +1. Exponential loss charges e^{-yf}. A negative margin yf is a mistake, and a large negative margin is punished exponentially, while a large positive margin is almost free. That emphasis on the worst mistakes is why the loss matches a procedure that keeps raising the weights of the cases it gets wrong.
With \tau_1(x)=\mathbb{P}(Y=1\mid x), the conditional expected loss is an average of two exponentials:
\mathbb{E}\bigl[e^{-Y f(x)}\bigm|x\bigr] =\tau_1(x)\,e^{-f(x)}+\bigl(1-\tau_1(x)\bigr)\,e^{f(x)}.
The score wanted minimises this conditional loss at every x. Differentiate in the scalar f=f(x):
\frac{\partial}{\partial f}\mathbb{E}\bigl[e^{-Yf}\bigm|x\bigr] =-\tau_1 e^{-f}+(1-\tau_1)e^{f}.
The second derivative is \tau_1 e^{-f}+(1-\tau_1)e^{f}>0, so the single stationary point is a minimum. Set the first derivative to zero:
(1-\tau_1)e^{f}=\tau_1 e^{-f}, \qquad e^{2f}=\frac{\tau_1}{1-\tau_1}, \qquad f^*(x)=\frac12\log\frac{\tau_1(x)}{1-\tau_1(x)}.
The minimiser is half the conditional log odds of class 1. The posterior \tau_1(x) did the work: it is the only feature of the conditional distribution of Y that enters the expected exponential loss, because Y takes only two values. When the log odds are 0, one has \tau_1(x)=1/2 and f^*(x)=0. The sign of f^* is therefore the equal-cost Bayes rule, which is why the ensemble classifies by \operatorname{sign}(f). Inverting the half-logit recovers the probability,
\tau_1(x)=\frac{1}{1+e^{-2f^*(x)}}, \qquad 1-\tau_1(x)=\frac{1}{1+e^{2f^*(x)}}.
Exponential loss does not estimate the class densities. It estimates the same posterior comparison that logistic regression writes as a logit, up to the factor 1/2.
Used in
See also
Gradient boosting · Squared error of an average · Random forest · AdaBoost · The reweighted error is one half · Why averaging helps · Ensemble error decomposition · Where disagreement comes from