Neural unit
Section of Neural units, part of Statistical learning
Neural unit
| Type | Section |
| Taught in | Neural units |
| Part | Statistical learning |
Definition 1 (Artificial neuron) An artificial neuron, or unit, has inputs x_1,\ldots,x_p, weights w_1,\ldots,w_p, and a bias w_0 (also written \beta or b). The net input, also called the weighted input or the activation, is
\mathrm{net}=\sum_{j=0}^{p} w_j x_j=w_0+w^\top x,\qquad x_0\equiv 1.
The output is o=f(\mathrm{net}) for an activation function f. The bias is the weight on an extra input that is permanently 1, so every later derivative has the same form.
Two activations are used here. The threshold, the McCulloch–Pitts decision, and the logistic sigmoid are
f(\mathrm{net})= \begin{cases} 0, & \mathrm{net}<0,\\ 1, & \mathrm{net}\ge 0, \end{cases} \qquad g(z)=\frac{1}{1+e^{-z}}.
The threshold is a hard cut. Its derivative is zero on each open half-space, so a gradient method has nothing to follow. The sigmoid is a smooth stand-in for the same cut. It satisfies
g'(z)=g(z)\bigl(1-g(z)\bigr)=e^{-z}g(z)^2.
The identity is what the chain rule multiplies by on the backpropagation page. The derivative is largest at z=0, where it equals 1/4, and it becomes small once |z| is large. A unit that has already saturated is slow to correct.
Place m units side by side with the same inputs and different weights. That computes m scalar features, each a function of a linear score. A later layer can combine those features. If they are used only as separate threshold outputs, each output still defines a half-space in the original coordinates.
A sigmoid unit with output o=1/(1+e^{-w^\top x-w_0}) is the same log-odds map as logistic regression: the output is the logistic probability of that linear score. A threshold unit is the hard classification rule attached to the same score. A single unit is a linear classifier of the same kind as linear discriminant analysis and logistic regression. This page adds hidden layers, so that one unit receives the outputs of others. The gradient of the resulting loss is calculated on the backpropagation page. Those two steps are the ones a single logistic regression does not take.
Used in
- Taught in Neural units
- All 7 questions on Neural units
See also
Artificial neuron · Perceptron learning rule · A single threshold unit cannot compute XOR · One hidden layer approximates continuous functions · History of the neural unit · Linear separability · Threshold unit without a bias · Threshold unit with a bias