Convolution

At each position, each filter weight is multiplied by the input value under it and the products are added, giving one number. Repeating this at every position produces a new grid, called a feature map, that shows where the input contains the pattern the filter responds to. For example, a filter of five weights slid along a series of daily returns would produce one value per day, and after training it might respond most strongly to one short sequence, such as three rising days followed by a sharp fall. Because the same weights are used at every position, a pattern is detected wherever it occurs, and the network has far fewer weights to learn than if each position had its own.

A convolutional neural network (CNN) stacks layers of filters, so that later layers detect more complex patterns built from the simpler ones found by earlier layers. In LeCun’s 1989 design, each filter layer is followed by a layer that averages neighboring values and shrinks the feature map. The filter weights are not set by hand; they are learned from examples with backpropagation.

Kunihiko Fukushima’s neocognitron (Biological Cybernetics, 1980) arranged cells in planes whose members all had “input synapses of the same spatial distribution”, so each plane detected one feature at every position, and it learned without labeled examples. Yann LeCun and colleagues at AT&T Bell Laboratories trained shared-weight layers of this kind with backpropagation in Handwritten Digit Recognition with a Back-Propagation Network (NIPS 1989). They described each layer as “equivalent to a convolution with a small size kernel”, and on zip code digits from the U.S. Postal Service their network had a 1 percent error rate with about 9 percent of digits rejected. AlexNet (Krizhevsky, Sutskever and Hinton, NIPS 2012), a network with five convolutional layers trained on two GPUs, won the ILSVRC-2012 image recognition contest with a top-5 error of 15.3 percent against 26.2 percent for the second-best entry. Top-5 error is the share of test images whose correct label is not among the model’s five most probable labels. The Computer History Museum writes that before AlexNet almost none of the leading computer vision papers used neural networks, and after it almost all of them did.