KL divergence

Topic: Model selection for stochastic models · more questions on this topic
‹ The Laplace integral in one dimension A tempting wrong turn ›

Show that D(f\Vert\widehat f)\ge 0, with equality only when \widehat f=f, using \ln u\le u-1. Show that D(f\Vert\widehat f)=\int f\ln f-\mathbb{E}[\ln\widehat f(X)], and explain why selecting the model with the largest expected log-likelihood on new data selects the model closest to f in this sense.

Worked answer (click to show)

Set u=\widehat f/f. The inequality \ln u\le u-1 gives

D(f\Vert\widehat f) =-\int f\ln(\widehat f/f) \ge -\int f\bigl(\widehat f/f-1\bigr) =-\int\widehat f+\int f =0,

because both densities integrate to one. Equality in \ln u\le u-1 holds only at u=1, so \widehat f=f almost everywhere. Splitting the logarithm, \ln(f/\widehat f)=\ln f-\ln\widehat f, gives

D(f\Vert\widehat f)=\int f\ln f-\mathbb{E}\bigl[\ln\widehat f(X)\bigr].

The first term does not depend on the fitted model, so maximising \mathbb{E}[\ln\widehat f(X)] over models is the same as minimising D. The comparison selects the fitted density closest to f in Kullback–Leibler divergence.