What is backpropagation, and why did a 1986 Nature paper about it matter so much?

Every time a phone recognizes your voice, a photo app finds faces, or a language model like GPT-2 finishes a sentence, it is using a network whose connections were set by the same basic procedure: backpropagation. The idea is simple to say. Make a guess, measure the error, then trace the blame for that error backward through the network and adjust each connection a little. Repeat millions of times.

The method was not brand new in 1986, but a short letter in the journal Nature made it famous. Titled "Learning representations by back-propagating errors," it appeared in volume 323, pages 533 to 536, on 9 October 1986. Its authors were David E. Rumelhart and Ronald J. Williams of the Institute for Cognitive Science at the University of California, San Diego, and Geoffrey E. Hinton of Carnegie-Mellon University in Pittsburgh. The journal received it on 1 May and accepted it on 31 July 1986. You can find it through its DOI.

The problem it solved

A neural network is made of simple units connected by weighted links. Some units take in the input, some give the output, and in between there can be hidden units, which are neither. Earlier learning rules, such as the perceptron procedure of the 1960s, could only adjust connections that led straight to the output. Marvin Minsky and Seymour Papert's 1969 book Perceptrons, which the paper cites, had proved that some simple-looking tasks, such as spotting symmetry, cannot be done without units in between. Hidden units could in principle fix that, but nobody had a practical general way to decide what they should learn.

The letter's answer was that the network can work it out for itself. With backpropagation, the hidden units come to stand for important features of the problem, features no one programmed in. The authors stressed that this ability to create new features is what set the method apart from older, simpler ones.

How backpropagation works

Training happens in two passes.

1. Forward pass. An input is set on the bottom layer. Each unit adds up its weighted inputs and squashes the total into a number between 0 and 1 with a smooth S-shaped curve called the sigmoid. Layer by layer, the signal reaches the output.
2. Measure the error. The output is compared with the right answer, and the squared differences are added up.
3. Backward pass. Using the chain rule from calculus, the network works out how much a small change in each output unit's input would change the error. It then passes that sensitivity backward along every connection to the layer below, and so on down. At the end, every weight has a number saying which way, and how strongly, it pushed the error.
4. Adjust. Each weight moves a small step in the direction that reduces the error. This is called gradient descent, like walking downhill in fog by feeling the slope under your feet.

A worked example: one weight, one step

Take a single unit with one input of 1 and a weight of 0.5. The right answer is 1.

In a real network the same arithmetic runs for every weight at once, and a hidden unit's blame is the sum of the blame flowing back to it from all the units above.

What the paper demonstrated

The letter showed the method on two small tasks. In the first, a network with only two hidden units learned to tell whether a row of six on/off inputs was a mirror image around its middle, something no single layer can do. It needed 1,425 passes through all 64 possible inputs, and the solution it found used neat, balanced weights.

The second task was more striking. The authors drew two family trees, one English and one Italian, with 12 people each, and gave a five-layer network facts such as "Colin's aunt is Jennifer." After training on 100 of the 104 possible facts, it answered the 4 unseen ones correctly. When they inspected the hidden units, some had come to encode features like a person's nationality, generation or branch of the family, none of which appeared explicitly in the input.

The authors were frank about limits. The error landscape can contain dips that are not the lowest point, though in practice their networks rarely got stuck. They noted that David Parker and Yann Le Cun had found variants independently, and they cautioned that, as it stood, the method was unlikely to be the way real brains learn.

Why it mattered

The idea had a long prehistory. Control engineers such as Henry Kelley and Arthur Bryson derived related methods around 1960, Seppo Linnainmaa published the general technique of automatic differentiation in 1970, and Paul Werbos proposed using it for neural networks in his 1974 dissertation. What the 1986 letter added was a clear demonstration that hidden layers learn meaningful internal representations, delivered to a wide scientific audience alongside the influential book Parallel Distributed Processing, which the letter also cites.

The letter made backpropagation the best-known way to train neural networks. Interest faded again in the 2000s, then returned strongly in the 2010s, when cheap graphics processors made it practical to train much larger networks.

Is backpropagation still how neural networks learn today?

Yes. In 2020 almost every deep learning system is trained with backpropagation and gradient descent: image classifiers, speech recognizers, AlphaGo's networks and language models such as GPT-2 and Google's BERT. Modern software libraries compute the backward pass automatically, so engineers rarely write the calculus by hand.

The details have changed. Networks now have dozens or hundreds of layers instead of five, units often use simpler functions than the sigmoid, and training uses refined versions of gradient descent that adapt the step size. But the core loop, forward pass, error, backward pass, small update, is the one described in those four pages of Nature.

The question the authors left open is also still open. Researchers continue to search for learning rules the brain could plausibly use, since, as the authors themselves said, backpropagation in this form is probably not how real neurons learn.