Changes that lead to deep networks
Section of Networks in practice, part of Statistical learning
Changes that lead to deep networks
| Type | Section |
| Taught in | Networks in practice |
| Part | Statistical learning |
Deep networks compose many learned transformations. The rectifier g(z)=\max(0,z) has derivative one for z>0 and zero for z<0. That avoids sigmoid saturation on its positive branch, while allowing inactive units on the negative branch. Convolutional networks share local filters. Recurrent networks reuse a state update across a sequence. Autoencoders learn to reconstruct their inputs. These are changes to the architecture and the objective. They are not exemptions from validation.
For a centred data matrix, a linear autoencoder with a bottleneck of dimension q and squared reconstruction loss has a global optimum spanning a leading q-dimensional principal subspace, provided no additional constraints change the problem. This follows from the same best-rank-q approximation argument as principal components. The weights need not be the individual principal directions, and tied eigenvalues make the optimal subspace non-unique. Nonlinear autoencoders solve a broader problem. Calling their output a principal-component analysis would lose the distinction.
Used in
See also
Softmax is multinomial logistic regression · What has to be settled before a step · Weight initialisation · Variance argument for weight initialisation · Batch and online learning · Preparing data for a network · Output coding, softmax, and cross-entropy · Constructing and training a network