History of the neural unit
Section of Neural units, part of Statistical learning
History of the neural unit
| Type | Section |
| Taught in | Neural units |
| Part | Statistical learning |
A neuron has a cell body containing the nucleus, branching input fibres, and a long output fibre that ends in synapses onto other cells. The comparison below is a historical illustration of scale. The numerical entries were not verified independently, and neurons and transistors are different physical objects. The counts do not establish equivalent computing capacity.
| Human brain | 2007 machine | 2018 machine | Units | |
|---|---|---|---|---|
| Computational units | 10^{11} | 3\times 10^{8} | 3\times 10^{9} | neurons / transistors |
| Storage units | 10^{11} | 10^{10} | 3\times 10^{10} | neurons / bits of memory |
| 10^{12} | 2\times 10^{12} | bits of disk | ||
| Cycle time | 10^{-3} | 3\times 10^{-10} | 3\times 10^{-10} | seconds |
| Bandwidth | 10^{14} | 10^{10} | 5\times 10^{11} | bits / second |
The dates below identify selected publications. They do not assign a single invention date to the whole field.
- 1943: McCulloch and Pitts publish a logical neuron model with binary output.
- 1958: Rosenblatt creates the perceptron, a single binary neuron, with a learning algorithm. He also considered multilayer perceptrons, without a learning algorithm for them.
- 1960: Widrow and Hoff introduce the ADALINE neuron, with a real-valued response, trained by stochastic gradient descent.
- 1969: Minsky and Papert publish Perceptrons, which analyses the perceptron and points out its failure on problems that are not linearly separable, including XOR.
- 1986: Rumelhart, Hinton and Williams publish the backpropagation algorithm for multilayer perceptrons with differentiable activations. Werbos had derived the same chain-rule calculation in 1974.
- 1998: LeCun and colleagues publish gradient-based document recognition, building on earlier convolutional networks.
- 2006: Hinton, Osindero and Teh publish a fast learning algorithm for deep belief networks. That is one development in deep learning. Many other networks, algorithms, and applications sit between these dates.
Adding hidden units changes what the model can represent. A learning algorithm changes how the weights are searched for. The XOR construction below is the first of those two obstacles. Backpropagation is the second.
Used in
- Taught in Neural units
- All 7 questions on Neural units
See also
Artificial neuron · Perceptron learning rule · A single threshold unit cannot compute XOR · One hidden layer approximates continuous functions · Neural unit · Linear separability · Threshold unit without a bias · Threshold unit with a bias