Linear hidden units

Topic: Neural units · more questions on this topic
‹ An XOR network of your own Why XOR needs an extra layer ›

Suppose every hidden unit has the identity activation g(u)=u. Show that the network is equivalent to a single-layer network, and that therefore it cannot compute XOR with a threshold output.

Worked answer (click to show)

With u_q=w_{0q}+\sum_j w_{jq} x_j and h_q=u_q, the output net input is

z_k=w_{0k}+\sum_q w_{qk} u_q =w_{0k}+\sum_q w_{qk} w_{0q}+\sum_j\Biggl(\sum_q w_{qk} w_{jq}\Biggr) x_j.

This is an affine function of x. The network is a single-layer network with weights \tilde w_{jk}=\sum_q w_{qk} w_{jq} and a threshold, or sigmoid, output. No such unit computes XOR. Hidden layers enlarge the represented class only if the hidden activation is nonlinear.