Softmax gradient

Topic: Networks in practice · more questions on this topic
‹ Initial weights Weight decay ›

Verify \partial E/\partial z_j=s_j-y_j for two classes and show that it reduces to \partial E/\partial z=o-y for the sigmoid output with cross-entropy loss, where o=g(z_1-z_2).

Worked answer (click to show)

For two classes, E=-y_1\ln s_1-y_2\ln s_2 with

s_1=\frac{e^{z_1}}{e^{z_1}+e^{z_2}}=g(z_1-z_2)=:o, \qquad s_2=1-o.

The softmax gradient gives \partial E/\partial z_1=s_1-y_1=o-y_1 and

\frac{\partial E}{\partial z_2}=s_2-y_2=-(o-y_1),

since y_2=1-y_1. With the single logit z=z_1-z_2, \partial E/\partial z=o-y_1, which is the cross-entropy gradient for a sigmoid output o=g(z) with target y=y_1. Directly,

E=-y\ln g(z)-(1-y)\ln\bigl(1-g(z)\bigr)

has

\frac{\partial E}{\partial z}=-y(1-g)+(1-y)g=g-y.