Universal approximation
Section of Neural units, part of Statistical learning
Universal approximation
| Type | Section |
| Taught in | Neural units |
| Part | Statistical learning |
XOR is a finite construction. An approximation theorem asks about a whole function class and states how the error is measured.
Theorem 1 (One hidden layer approximates continuous functions) For a continuous target f on [0,1]^p and any \varepsilon>0, there is a finite sum
F(x)=\sum_{q=1}^{l} a_q\,\sigma(w_q^\top x+b_q)
with a continuous sigmoidal activation such that \sup_{x\in[0,1]^p}|F(x)-f(x)|<\varepsilon (Cybenko, 1989, Theorem 2). The output here is a linear combination, not a bounded sigmoid forced to represent arbitrary real values. Results for measurable targets use weaker notions of approximation. They do not assert uniform approximation of every discontinuous function by continuous networks.
Article: One hidden layer approximates continuous functions →
Suitable weights and a finite width exist for each continuous target and accuracy. The theorem supplies neither a training algorithm nor a sample-size guarantee. Approximation error concerns the best function available in the model class. Optimisation error concerns finding it. Statistical error concerns learning from finite observations. These can behave differently.
One hidden layer can approximate a wide class of functions arbitrarily closely. The theorem is silent on the number of units and on whether one hidden layer is an efficient architecture. A deeper network can be smaller for the same approximation. The tradeoff among target complexity, sample size, and width is the overfitting problem on networks in practice.
Used in
- Taught in Neural units
- All 7 questions on Neural units
See also
Artificial neuron · Perceptron learning rule · A single threshold unit cannot compute XOR · One hidden layer approximates continuous functions · History of the neural unit · Neural unit · Linear separability · Threshold unit without a bias