Activation function
Topic of Neural units, part of Statistical learning
Activation function
| Type | Topic |
| Taught in | Neural units |
| Part | Statistical learning |
| Questions using it | 3 |
The fixed function a neural unit applies to its weighted input: the step function of the threshold unit, or the logistic sigmoid used when the weights are trained by gradient descent. A non-linear activation is what lets a hidden layer represent functions a single linear score cannot.
In Neural units
From Neural units
Two activations are used here. The threshold, the McCulloch–Pitts decision, and the logistic sigmoid are
f(\mathrm{net})= \begin{cases} 0, & \mathrm{net}<0,\\ 1, & \mathrm{net}\ge 0, \end{cases} \qquad g(z)=\frac{1}{1+e^{-z}}.
In Neural units, continued
From Neural units
An artificial neuron is that hyperplane followed by an activation. It combines a linear score with a fixed function of the score, and it is the right object for the gates that are linearly separable. XOR forces the next object: a hidden layer of such neurons, whose outputs are new coordinates. In those coordinates the same four points can sit so that one further linear score separates them.
Used in
- Taught in Neural units
- Also used in Backpropagation · Networks in practice
- Questions that use it (3) — Linear hidden units · Why XOR needs an extra layer · One backpropagation update (answer)
- All 7 questions on Neural units
See also
Artificial neuron · Perceptron · Sigmoid function · Gradient descent · Hidden layer · One hidden layer approximates continuous functions · Perceptron learning rule · A single threshold unit cannot compute XOR