Learning by backpropagation

The event and the narrow claim

On 9 October 1986, Nature published a four-page letter titled "Learning representations by back-propagating errors," by David Rumelhart, Geoffrey Hinton, and Ronald Williams. The paper described a procedure that repeatedly adjusts the connection weights of a layered network of neuron-like units so as to reduce the difference between the network's actual output and a desired output. Its stated novelty was not the arithmetic of gradients but a consequence of it: the "hidden" units, the ones that are neither input nor output, "come to represent important features of the task domain," and this ability to create useful new features is what the authors said "distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure."

The precise historical claim is narrow and worth stating carefully. The 1986 letter did not first invent the reverse-mode chain rule, nor did it prove that deep networks would one day classify images or translate languages. It provided a clear, reproducible demonstration, in the most visible journal in science, that gradient descent through several layers could discover internal representations no one had hand-designed. Separating that documented result from the broad later judgment (that this is how modern AI is trained) is the whole task of placing the event correctly.

The wall the perceptron had hit

To see what constraint the paper addressed, start with its named foil. Frank Rosenblatt's 1958 perceptron was a trainable pattern classifier whose convergence procedure adjusted weights on only a single layer of modifiable connections. It came with a guarantee: if a data set was linearly separable, the perceptron rule would find a separating boundary. The trouble was everything that was not linearly separable.

Marvin Minsky and Seymour Papert's 1969 book Perceptrons made the limit rigorous, showing that a single-layer perceptron cannot compute functions as simple as the exclusive-or (XOR) of two inputs. The obvious escape was to stack layers, letting intermediate units transform the input into a representation where the problem became separable. But no one had a general, practical rule for training those intermediate weights: the perceptron rule needed a target for each unit it adjusted, and there is no supplied target for a hidden unit. That missing training signal for interior layers was the specific obstacle backpropagation removed. Jürgen Schmidhuber's history of the field traces how central this credit-assignment problem was to a generation of research.

Who computed the gradient first

The 1986 paper is famous, but it was not first, and its authors did not claim otherwise. Reverse-mode differentiation, the exact computational trick backpropagation relies on, was described by Seppo Linnainmaa in the early 1970s as a way to accumulate rounding error through a sequence of elementary operations. Applying that idea specifically to training neural networks was written out in Paul Werbos's 1974 Harvard doctoral thesis and later developed in his 1990 account of backpropagation through time. David Parker and Yann LeCun arrived at closely related formulations independently in the mid-1980s.

So the honest attribution is co-discovery and, above all, popularization. What Rumelhart, Hinton, and Williams supplied was a lucid statement, a compelling demonstration, and the reach of Nature, at a moment when the field was ready to listen. Historical credit for making a method usable and known is real credit, and it is simply different from credit for first derivation. Conflating the two produces the recurring priority disputes that still attach to this event, and it also obscures a genuinely useful lesson: an idea's diffusion through a field depends as much on the clarity and timing of its presentation as on its originality. Werbos's thesis sat largely unread for over a decade. Linnainmaa's technique was framed as a numerical-analysis tool for tracking rounding error, not as a training method, so the neural-network community did not connect it to their own unsolved credit-assignment problem until years later. The 1986 letter succeeded partly because it named the destination (useful learned features in hidden units) rather than only the machinery, and partly because it arrived with worked examples a reader could inspect and believe.

How the procedure actually works

The mechanism is the chain rule of calculus applied in a particular order. Each unit computes a weighted sum of its inputs and passes it through a smooth, nonlinear squashing function; the 1986 paper used the logistic sigmoid, whose smoothness matters because gradient descent needs derivatives. The network's error is a single number, typically half the summed squared difference between output and target. Training proceeds in two sweeps. A forward pass sends an input through the network to produce an output and its error. A backward pass then propagates that error from the output layer toward the input, using the chain rule at each step to compute how much each weight contributed to the error, that is, the partial derivative of the error with respect to that weight.

Once every weight has its gradient, all weights are nudged a small step in the direction that reduces error. The key economy is that the backward pass reuses the quantities computed for downstream layers when it reaches upstream layers, so the cost of computing all the gradients is comparable to a single forward pass rather than growing with the number of weights. The paper also added a momentum term, blending each update with a fraction of the previous one to smooth the descent across the error surface. Crucially, every quantity a unit needs during the backward pass is local to that unit and its own connections, which is what later made the procedure mechanical enough to automate. Nothing here supplies a target for a hidden unit; instead the hidden units receive a share of blame for the output error, and that shared blame is the signal that lets interior layers learn.

It is worth being explicit about why this is economical, because the efficiency is the whole practical point. A naive way to estimate how a single weight affects the error is to nudge that weight slightly and rerun the entire network to see how the error changed, then repeat that for every weight separately. For a network with many thousands of weights, that numerical approach costs one full forward pass per weight, which is prohibitive. Backpropagation instead computes every weight's gradient in one backward sweep by reusing intermediate results, so the total cost of computing all the derivatives is a small constant multiple of a single forward pass, regardless of how many weights the network has. That is the property, sometimes called reverse-mode accumulation, that made training networks with many layers and many parameters computationally realistic at all, long before those networks were large by today's standards.

What the hidden units actually learned

The letter's persuasive core was its worked examples. One was the mirror-symmetry task: a network had to output whether the pattern of activity across its input units was symmetric about its centre. The reported result was that a small set of hidden units developed a systematic, interpretable weight pattern, with weights scaling in roughly power-of-two steps along the input, that let them detect symmetry, a solution the designers had not specified. A companion and much longer treatment in the 1986 Parallel Distributed Processing volumes carried further demonstrations. The most cited was a network Hinton trained on two structurally identical families, one English and one Italian, comprising 24 people related by 12 kinship terms such as father, aunt, and nephew. Given a person and a relationship, the network had to name the correct relative, and its hidden units spontaneously came to encode properties the input never stated: one unit tracked nationality, another a person's generation, another which branch of the family a person belonged to. Nobody specified those features; gradient descent discovered them because they made the mapping learnable. That is exactly the phenomenon the Nature letter advertised, now on a task with obvious semantic structure.

Two caveats stated in the paper itself matter for an honest reading. First, gradient descent can settle into local minima rather than the best possible solution, and the authors reported that in their experiments this was rarely a serious problem, which is an empirical observation on small tasks, not a theorem. Second, the demonstrations were tiny by any modern measure: a handful of input units, a handful of hidden units, and training sets a person could enumerate by hand. The evidence in 1986 was that the method worked and produced sensible internal features on toy problems, not that it would scale to the millions or billions of parameters that later networks would use. That gap between a persuasive small demonstration and a proof of scalability is exactly what the following decades had to close, largely through means the original letter did not anticipate.

The competing programs of the 1980s

Backpropagation did not arrive into an empty field; it arrived into one dominated by a rival paradigm. Through the 1980s the mainstream of artificial intelligence was symbolic: expert systems, hand-built rules, and logic, approaches that represented knowledge explicitly rather than learning it from examples. Connectionism, the neural-network research program the 1986 paper belonged to, was a minority position and, after the pessimism that followed Perceptrons, a somewhat marginal one.

Even within neural approaches there were serious alternatives. John Hopfield's 1982 associative-memory networks and the Boltzmann machine, a stochastic network trained with a different, physics-derived learning rule that Hinton co-developed, were prominent contemporaries. That lineage is exactly what the 2024 Nobel Prize in Physics, awarded to Hopfield and Hinton "for foundational discoveries and inventions that enable machine learning with artificial neural networks," recognized. Placing backpropagation among these competitors, rather than treating it as the only game, explains why its eventual dominance was a contingent outcome and not a foregone conclusion. Expert systems, for their part, scaled poorly: every new domain of knowledge required a fresh round of manual rule-writing by domain specialists, and the resulting systems were brittle outside the narrow cases their rules anticipated. That contrast, rules that had to be written by hand versus features that could be learned from data, is the deeper reason the connectionist program eventually displaced the symbolic one as computing power and data both grew, even though in 1986 that outcome was far from obvious to most of the field.

Two decades of comparative dormancy

If the method worked in 1986, why did the deep-learning era not visibly begin until around 2012? The gap is one of the most instructive facts about the event. Part of the answer is mechanical: the error signal tends to shrink, or occasionally explode, as it is multiplied backward through layer after layer. Yoshua Bengio and colleagues characterized this difficulty in 1994, showing that learning long-range dependencies with gradient descent is hard precisely because the propagated gradients decay. Sigmoids made this worse, since their derivatives are small over much of their range, so a deep stack of them could shrink a gradient toward zero well before it reached the earliest layers, leaving those layers to learn extremely slowly if at all.

The other part of the answer was resources. Backpropagation is data-hungry and compute-hungry, and in the 1980s and 1990s there were neither the large labeled data sets nor the fast parallel hardware needed to train large networks. Two concrete fixes later illustrate what actually changed. In 2006, Geoffrey Hinton and Ruslan Salakhutdinov showed in Reducing the Dimensionality of Data with Neural Networks that greedily pretraining a deep network one layer at a time, using unsupervised restricted Boltzmann machines, gave gradient descent a starting point good enough to train networks that had been considered untrainable, which briefly reignited the field. A few years later, Xavier Glorot and Yoshua Bengio diagnosed the problem more directly, showing that the standard sigmoid saturates and that weights had been initialized at the wrong scale, then prescribing an initialization that let signals and gradients flow through many layers. Replacing sigmoids with the non-saturating rectified linear unit removed much of the remaining vanishing-gradient problem outright. The core algorithm sat as a known but underpowered tool while symbolic AI, and later support-vector machines and other statistical methods, drew more attention. What ultimately changed was not the backward pass but its surroundings: better activation functions and initialization, far more data, and graphics processors that made the arithmetic cheap.

It is worth naming the hardware shift explicitly, since it is easy to understate. Graphics processing units were designed to perform the same simple arithmetic operation across huge arrays of pixels simultaneously, and the matrix multiplications that dominate backpropagation's forward and backward passes have exactly that shape: the same operation repeated across many independent elements. Repurposing that hardware for neural-network training, alongside frameworks that made it straightforward to express a network as a sequence of matrix operations, turned a computation that had been impractical at scale in the 1980s into one that could run in hours rather than months. None of that required changing the learning rule itself. The chain-rule computation the 1986 letter described is, in its mathematical essence, identical to the one running inside a modern accelerator; only the scale of the arithmetic and the machinery around it changed.

From zip codes to ImageNet

The line from the 1986 letter to contemporary systems runs through concrete engineering. In 1989, a team including Yann LeCun applied backpropagation to a convolutional network that read handwritten digits, reported in "Backpropagation Applied to Handwritten Zip Code Recognition". This was a working, real-scale demonstration that gradient descent could train a structured deep network on a genuine perception task, and it seeded the convolutional architectures used for decades afterward, including the systems that would later read cheques for banks. The convolutional design itself mattered because it constrained the network to reuse the same learned weights across every position in an image, which reduced the number of free parameters and made the learning problem tractable with the data and compute available at the time; backpropagation supplied the mechanism that adjusted those shared weights.

The decisive public turn came in 2012, when AlexNet, a deep convolutional network with about 60 million parameters trained on GPUs by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, cut the top-5 error on the ImageNet classification benchmark far below the previous state of the art. Its training was backpropagation with stochastic gradient descent. By 2015 the same core method underpinned the survey "Deep learning" that LeCun, Bengio, and Hinton wrote for Nature, and in 2019 the three received the 2018 ACM A.M. Turing Award for the body of work that made deep neural networks central to computing. Backpropagation remained the engine underneath all of it, and the 2012 result is the clearest single marker of the moment the wider computing industry, not just the connectionist research community, began treating gradient-trained deep networks as the default tool for perception tasks rather than a curiosity.

The optimization backbone of the modern era

By the 2010s backpropagation had stopped being a formula a researcher derived by hand and become infrastructure. Modern deep-learning frameworks such as PyTorch, TensorFlow, and JAX are, at their core, general reverse-mode automatic differentiation engines: a practitioner writes only the forward computation, the framework records it as a graph of elementary operations, and it replays that graph backward to produce every gradient automatically, exactly the two-pass economy the 1986 letter described, now applied to arbitrary architectures rather than one hand-derived network. The architectures themselves kept changing while the optimizer did not. The transformer, introduced in 2017, discarded the recurrence and convolution of earlier networks in favour of attention, yet it is trained by backpropagation with stochastic gradient descent, in practice through adaptive variants such as Adam. The large language models of the 2020s, with hundreds of billions of parameters spread across thousands of accelerators, learn through the same backward pass, parallelized and scaled but unchanged in principle. This is the strongest honest sense in which the event matters: the specific 1986 demonstration is history, but the credit-assignment procedure it popularized is still the mechanism by which nearly every deployed neural network learns.

This generality is itself a legacy of the 1986 formulation. Because the paper framed the method as a general procedure operating on any layered network of differentiable units, rather than as a technique tailored to one architecture, it transferred cleanly to convolutional networks, recurrent networks, transformers, and diffusion models alike, none of which existed when the letter was written. The procedure did not need to be reinvented for each new architecture; only the architecture's forward computation changed, and the automatic differentiation machinery derived the corresponding backward computation on its own. That transferability is a large part of why the 1986 paper, rather than any of the specific architectures built on top of it, is the event usually cited as the origin of the modern training paradigm.

The honest limits and caveats

Several qualifications keep the significance from tipping into myth. The method optimizes whatever objective it is given on whatever data it sees; good gradients do not fix bad data, a poorly chosen architecture, or a mis-specified loss. It offers no guarantee of a global optimum, and it does not by itself confer generalization, calibration, or reliability outside the training distribution. It is a way to assign credit for an error, not a theory of what a system should learn, and a network trained by gradient descent will happily learn a spurious correlation in the training data as readily as it learns the pattern its designers intended.

There is also a long-running scientific question about whether the brain does anything like backpropagation. The algorithm requires each connection to know, during the backward pass, the weights used elsewhere in the forward pass, the so-called weight-transport problem, which has no obvious biological implementation. A 2020 review, "Backpropagation and the brain," surveys this debate and the more biologically plausible approximations proposed in response. For a timeline, the point is that backpropagation's success as an engineering method is a separate claim from any assertion that it explains learning in real nervous systems, and the two claims should not be allowed to blur into one another just because both use the word "learning."

How to place backpropagation in AI history

The defensible reading is that the 1986 Nature letter is a landmark of demonstration and diffusion rather than of first invention. The reverse-mode gradient predates it; what the event contributed was a clear demonstration, stated persuasively enough to convince a wide audience, that a multilayer network could learn its own internal representations. Its practical payoff arrived decades later, once data, hardware, and a handful of architectural fixes caught up.

A useful way to navigate outward from this node is to hold three questions apart. Which earlier idea supplied the mechanism (reverse-mode differentiation, from Linnainmaa and Werbos)? Which later work removed a barrier the method still faced (the vanishing-gradient analyses, layerwise pretraining, better activations and initialization, GPUs, and large data sets)? And which downstream result actually depended on training deep networks by gradient descent (convolutional digit recognition, AlexNet, and the transformer-era systems built on the same optimization backbone)? Keeping those levels distinct prevents the common error of collapsing forty years of separate discoveries into a single 1986 origin story, while still crediting that paper with what it genuinely did: it made learning by backpropagation something the field could see, reproduce, and build on, and it gave the field a shared vocabulary and worked demonstration to organize four subsequent decades of engineering progress around.

References

• Rumelhart, Hinton and Williams (1986). Learning representations by back-propagating errors.
• Rosenblatt (1958). The perceptron: a probabilistic model for information storage and organization in the brain.
• Linnainmaa (1976). Taylor expansion of the accumulated rounding error.
• Werbos (1990). Backpropagation through time: what it does and how to do it.
• LeCun et al. (1989). Backpropagation Applied to Handwritten Zip Code Recognition.
• Bengio, Simard and Frasconi (1994). Learning long-term dependencies with gradient descent is difficult.
• Schmidhuber (2015). Deep learning in neural networks: an overview.
• Hinton and Salakhutdinov (2006). Reducing the Dimensionality of Data with Neural Networks.
• Glorot and Bengio (2010). Understanding the difficulty of training deep feedforward neural networks.
• Krizhevsky, Sutskever and Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks.
• Vaswani et al. (2017). Attention Is All You Need.
• LeCun, Bengio and Hinton (2015). Deep learning.
• Lillicrap et al. (2020). Backpropagation and the brain.
• The Royal Swedish Academy of Sciences (2024). The Nobel Prize in Physics 2024.
• Association for Computing Machinery (2019). Fathers of the Deep Learning Revolution Receive ACM A.M. Turing Award.