LeNet (convolutional networks): Learned filters read the mail

A seven-author paper and its narrow claim

In the final 1989 issue of Neural Computation, seven researchers at AT&T Bell Laboratories in Holmdel, New Jersey, published a short paper titled Backpropagation Applied to Handwritten Zip Code Recognition. Yann LeCun was first author, joined by Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard and Lawrence Jackel. The article records that the manuscript was received on 7 July 1989 and accepted on 12 September 1989. Its claim was specific: a single multilayer network, trained by backpropagation, could perform the whole recognition step for handwritten digits, from a size-normalized image to a final class, with no hand-designed feature extractor in between.

The measured result was modest by later standards and important in kind. On 2,007 held-out digits cut from real envelopes, the network misclassified 102, an error rate of 5.0 percent, after making 10 mistakes on its 7,291 training digits. To reach 1 percent error on the digits it did accept, it had to reject 12.1 percent of the test set. Nothing in the paper calls the system LeNet. That family name was attached later, as the 1998 Proceedings of the IEEE paper on document recognition shows when it labels its comparison models LeNet-1, LeNet-4 and LeNet-5. Nor does the paper claim to have invented convolution. Its argument was that constraints built into the wiring of a network, rather than features built by hand, were what let learning generalize on a messy real task. The larger historical claim, that this design became the template for modern computer vision, rests on what happened over the following twenty-three years and has to be argued separately.

Zip codes from Buffalo and a hand-built predecessor

The data came from the United States Postal Service. As the paper describes it, the set held 9,298 segmented numerals digitized from handwritten zip codes on mail passing through the Buffalo, New York, post office, written by many people with many instruments and widely varying care. Postal Service contractors had already located each zip code on the envelope and separated the digits, a hard problem the Bell Labs group did not attempt. Each digit, typically around 40 by 60 pixels, was scaled into a 16 by 16 grayscale image. The authors stressed that both the training and test partitions contained ambiguous, unclassifiable and even mislabeled examples, the normal condition of operational data.

The same group had attacked this problem a year earlier with a very different philosophy. Their 1988 conference paper, Neural Network Recognizer for Hand-Written Zip Code Digits, described a pipeline that deskewed and thinned each digit and then scanned it with 49 feature-extractor templates of 7 by 7 pixels, with important parts of the work handled by the lab's custom analog neural-network chip. Those templates were convolution kernels, but people designed them, one stroke ending or arc at a time, and combined their outputs by hand into 18 feature maps. Only the final stage learned: a two-layer network with 40 hidden units. That system rejected 14 percent of postal digits to hold errors on the rest to 1 percent. The authors also reported that feeding raw bitmaps to the same classifiers multiplied the error rate at least fivefold, and concluded that a direct adaptive attack on the pixels would not succeed without careful engineering.

The 1989 paper is a reversal of that conclusion within a single year. It kept the convolutional shape of the old front end and handed its coefficients to backpropagation. The authors wrote that the new system performed slightly better than the previous one even though much less specific knowledge about digits had been built into it, and they named the approach "constrained backpropagation."

How weight sharing turned backpropagation into convolution

The mechanism is easiest to see in the first hidden layer. It held 12 feature maps, each an 8 by 8 grid of units. Every unit looked at a 5 by 5 patch of the input, and neighboring units looked at patches two pixels apart, so each map was a subsampled view of the whole image. The decisive constraint was that all 64 units in a map used the same 25 weights, with only their biases left free. A map therefore computed the same local detector at every position, which the paper interprets as a nonlinear, subsampled convolution. If a stroke ending mattered in the upper left of a digit, the same detector would find it in the lower right. The paper credits the weight-sharing device to a 1986 chapter by Rumelhart, Hinton and Williams, the same authors whose Nature letter on back-propagating errors made gradient training of hidden units widely known, and it reads the device as an equality constraint: many connections, one adjustable number.

The arithmetic shows why this mattered. The first layer had 768 units and 19,968 connections but only 1,068 free parameters. A second layer of 12 maps, each 4 by 4, combined eight of the first-layer maps through 5 by 5 kernels. Above that sat 30 fully connected units and 10 outputs, one per digit. In total the network had 1,256 units and 64,660 connections, but just 9,760 independent parameters. The authors argued that this reduced the capacity of the model, which they discussed in terms of entropy and Vapnik-Chervonenkis dimension, and with it the amount of data needed to generalize. They tested the argument directly. A fully connected network with one hidden layer of 40 units and 10,690 connections reached 1.6 percent training error but 8.1 percent test error, and needed 19.4 percent rejections to reach 1 percent error. The constrained network fit its training set more tightly and generalized better at the same time.

Training was ordinary by modern lights. Units used a scaled hyperbolic tangent, the loss was mean squared error, and weights were updated after every example by stochastic gradient descent, with a diagonal approximation of second derivatives setting a step size for each weight. The run used the SN simulator written by Léon Bottou and LeCun on a Sun-4/260 workstation, took 23 passes through the training set (167,693 pattern presentations), and by the paper's own account about three days. The authors observed that some learned kernels resembled feature detectors found in animal vision, citing Hubel and Wiesel's 1962 study of the cat visual cortex, while others resembled detectors that engineers had designed for earlier character recognizers.

From a workstation to a signal-processing board

The paper did not stop at a simulation. Because almost all of the computation is multiply and accumulate, the trained network ran on an off-the-shelf board built around an AT&T DSP-32C chip, rated at 12.5 million multiply-add operations per second. A personal computer with a camera captured, binarized and segmented the image, and the DSP normalized and classified each digit. The whole pipeline handled 10 to 12 digits per second, limited mainly by normalization, and the DSP alone classified more than 30 normalized digits per second. A footnote reported that a larger variant, with about 100,000 connections and 2,600 free parameters, reached 1 percent error at 9 percent rejection, roughly 30 percent better than the best networks with hand-coded kernels.

A network matching that description appears in a companion paper from the 1989 NIPS conference, Handwritten Digit Recognition with a Back-Propagation Network. Its architecture is closer to the textbook convolutional network: shared-weight feature maps alternate with separate averaging and subsampling layers, which the authors compare to the neocognitron while stressing that every layer is trained by backpropagation. With 4,635 units, 98,442 connections and 2,578 independent parameters, it had been pruned from a version with four times more free parameters using Optimal Brain Damage, a method by LeCun, Denker and Sara Solla that uses second-derivative information to delete unimportant weights.

AT&T also protected the design. US Patent 5,067,164, filed on 30 November 1989 and granted on 19 November 1991 to Denker, Howard, Jackel and LeCun, claims a layered character-recognition network of constrained feature maps and undersampling maps whose kernels are learned by constrained backpropagation. The patent record shows the rights assigned to NCR in 1996, which matters for the commercial story below.

Who got there first

The paper sits in a crowded neighborhood, and an honest history names the neighbors. Kunihiko Fukushima's neocognitron, published in 1980, already stacked layers of local feature detectors with layers that tolerated shifts in position. The 1998 paper itself says the convolution and subsampling combination, inspired by Hubel and Wiesel's simple and complex cells, was implemented in the neocognitron, though no globally supervised procedure like backpropagation was available to train it. The Deep Learning textbook by Ian Goodfellow, Yoshua Bengio and Aaron Courville makes the same distinction: the neocognitron had most of the architectural elements of the modern convolutional network but learned through layer-wise unsupervised clustering.

On the training side, weight sharing along one dimension had already been joined to backpropagation for speech. Alex Waibel and colleagues, Geoffrey Hinton among them, published Phoneme Recognition Using Time-Delay Neural Networks in March 1989. The zip code paper discusses a large time-delay network of Waibel's for consonant recognition, with about 18,000 connections and 1,800 free parameters, whose training took 18 days on an Alliant mini-supercomputer, which is why he proposed building such networks from smaller, separately trained ones. The Bell Labs network trained in one piece in three days. The textbook frames the 1989 work as applying the same training algorithm to two-dimensional convolution over images after time-delay networks had succeeded.

The sharpest priority claim comes from Jürgen Schmidhuber, who has long contested how credit is assigned in deep learning. His Annotated History of Modern AI and Deep Learning credits Wei Zhang and colleagues in Japan with a backpropagation-trained two-dimensional convolutional network for character recognition presented in 1988, and with a journal submission he dates to 13 April 1989, about twelve weeks before the Bell Labs manuscript was received. His account is an argument as well as a record, but it shows that the 1989 paper was not alone. The Bell Labs work also had a prehistory of its own. LeCun had been a postdoctoral researcher with Hinton in Toronto, as the Computer History Museum notes in its account of AlexNet, and the zip code paper describes its architecture as a direct extension of his earlier 1989 study on simple digit images. What set the Holmdel paper apart was not one component but the combination: two-dimensional shared-weight layers, end-to-end gradient training, real postal data at the scale of thousands of examples, and a working path to hardware.

A decade of reading checks

The program continued through most of the 1990s, and its mature statement is the 1998 paper Gradient-Based Learning Applied to Document Recognition by LeCun, Bottou, Bengio and Patrick Haffner. Its LeNet-5 had seven layers with trainable parameters, a 32 by 32 input, 340,908 connections and about 60,000 trainable parameters. The paper also describes the Modified NIST benchmark, or MNIST, built by mixing two NIST databases into 60,000 training digits and 10,000 test digits. LeNet-5 reached 0.95 percent test error, or 0.8 percent when the training set was enlarged with artificially distorted copies, and a boosted combination of three LeNet-4 networks reached 0.7 percent. The same comparison carries a useful corrective: a support vector machine with a ninth-degree polynomial kernel, trained with distorted examples, also reached 0.8 percent, and the paper puts the uncertainty of its quoted error rates at about 0.1 percentage points. LeNet-1, a small early embodiment of the architecture with about 2,600 free parameters, scored 1.7 percent.

The commercial story is told in the same paper. A check-reading system built on convolutional character recognizers, and trained as a whole with graph transformer networks, read the numeric courtesy amount. The authors set the threshold of economic viability for banks at 50 percent of checks read with under 1 percent error, the rest going to human operators. On 646 business checks the system read 82 percent correctly with 1 percent errors and 17 percent rejects, against 68 percent correct for the group's previous system. An independent test by systems integrators in 1995 favored it over other commercial readers. It was integrated into NCR's line of check-reading systems, had been fielded in several US banks since June 1996, and was reading millions of checks per day. The Deep Learning textbook adds that by the end of the decade the NCR-deployed system was reading over 10 percent of all checks in the United States. That figure comes from authors close to the work, since Bengio wrote both texts, and neither cites an independent audit of it.

Why the idea waited two decades

Commercial success did not move convolutional networks to the center of the field. The Computer History Museum's history of AlexNet observes that before 2012 neural networks could not consistently outperform other machine learning methods, and the MNIST tie with support vector machines illustrates the point. Digits are small, centered, monochrome strokes drawn from ten classes. Natural photographs offered none of those conveniences, and the labeled data and arithmetic needed to train much larger convolutional networks on them did not yet exist.

Andrej Karpathy made the scale gap concrete in 2022, when he reimplemented the 1989 network in PyTorch. Because the original postal dataset is believed to be lost, he approximated it by shrinking MNIST digits to 16 by 16 and drawing the same numbers of training and test images. His baseline reached 4.09 percent test error, close to the paper's figure but not identical. Training that took about three days on the Sun workstation took about 90 seconds on the CPU of an M1 MacBook Air, which he called roughly a 3,000-fold naive speedup. Later refinements, namely cross-entropy loss, the AdamW optimizer, data augmentation, dropout and rectified linear units, cut test error to 1.59 percent, and adding more training data brought it to 1.25 percent. His reading was that the macro-level recipe, a differentiable layered network trained by backpropagation and stochastic gradient descent, had barely changed. What changed was size, by many orders of magnitude.

That is exactly where the line resumed in 2012. The AlexNet paper cites the Bell Labs digit work among convolutional networks that make strong and mostly correct assumptions about images, namely that their statistics are stationary and their pixel dependencies local. It also names the obstacle: such networks had been too expensive to apply at scale to high-resolution images until fast GPU implementations of convolution and large labeled datasets like ImageNet arrived. The Deep Learning textbook puts the longer arc in one phrase, saying that convolutional networks carried the torch for the rest of deep learning.

The honest limits

The 1989 evidence is narrow. The test set held 2,007 digits, so the headline rests on 102 errors, and the authors acknowledged that the rejection thresholds behind the 12.1 percent figure were chosen using performance on the test set itself, which makes that number optimistic. The paper reports no separate validation set. Segmentation was done upstream by contractors, and the paper attributes most of its misclassifications to segmentation errors it did not have to solve. Karpathy reports that the original dataset is believed to be lost, so even a careful reproduction can match the result only approximately.

The retrospective record also inflates the event. Guinness World Records lists the 1989 network as the first neural network to identify handwritten characters, a claim contradicted by the Bell Labs group's own 1988 recognizer, whose final stage was a trained network, and by the 1986 neocognitron demonstration on handwritten digits that Schmidhuber's history records. The IEEE Milestone dedicated at Nokia Bell Labs on 21 October 2025, Convolutional Neural Networks, 1989, bundles under one year a research program that, by its own supporting text, spanned LeCun's time at Bell Labs from 1988 to 1996. Its citation credits methods from pruning to self-supervised learning, and its supporting text describes accuracy validated on MNIST, a benchmark first described in the group's later work. A milestone is a reasonable way to honor a program, but it is not evidence of what one paper showed. The priority questions above are also real. Calling LeNet the first convolutional network overstates the case. Calling it an early and widely seen demonstration that backpropagation could learn convolutional features from raw pixels on an operational task, and then ship, is defensible.

How to place LeNet in AI history

The cleanest reading is as a bridge. Backpropagation had shown in 1986 that hidden units could learn internal representations, mostly on small, synthetic problems. The zip code paper showed that the same algorithm, given an architecture that encoded locality and translation invariance through shared weights, could learn its own features from pixels on a real task and then run on commodity signal-processing hardware. The authors themselves noted that the scaling was far better than results on small artificial problems would have predicted. That design principle, putting prior knowledge into the wiring rather than into hand-written features, is the one AlexNet scaled up in 2012 and that residual networks pushed to far greater depth in 2015.

Its institutional trail is equally specific: a patent, a commercial check reader sold through NCR, the MNIST benchmark that Karpathy could still use in 2022 as a stand-in for the lost postal data, and much later the 2018 ACM Turing Award, which the milestone record lists among the work's recognitions. LeNet did not end the skepticism toward neural networks in its own decade, and others had built pieces of it first. Its lasting contribution was a working proof, on mail and on money, that learned convolution was practical, kept alive until data and computation caught up with it.

References

• Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard and Lawrence Jackel, Neural Computation (1989). Backpropagation Applied to Handwritten Zip Code Recognition.
• Yann LeCun, Bernhard Boser, John Denker et al., Advances in Neural Information Processing Systems 2 (1989). Handwritten Digit Recognition with a Back-Propagation Network.
• John Denker, W. R. Gardner, Hans Peter Graf et al., Advances in Neural Information Processing Systems 1 (1988). Neural Network Recognizer for Hand-Written Zip Code Digits.
• Yann LeCun, John Denker and Sara Solla, Advances in Neural Information Processing Systems 2 (1989). Optimal Brain Damage.
• Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner, Proceedings of the IEEE (1998). Gradient-Based Learning Applied to Document Recognition.
• John Denker, Richard Howard, Lawrence Jackel and Yann LeCun, AT&T Bell Laboratories, via Google Patents (1991). US Patent 5,067,164: Hierarchical constrained automatic learning neural network for character recognition.
• Kunihiko Fukushima, Biological Cybernetics (1980). Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position.
• David Hubel and Torsten Wiesel, The Journal of Physiology (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex.
• David Rumelhart, Geoffrey Hinton and Ronald Williams, Nature (1986). Learning representations by back-propagating errors.
• Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano and Kevin Lang, IEEE Transactions on Acoustics, Speech, and Signal Processing (1989). Phoneme Recognition Using Time-Delay Neural Networks.
• Ian Goodfellow, Yoshua Bengio and Aaron Courville, MIT Press (2016). Deep Learning, Chapter 9: Convolutional Networks.
• Jürgen Schmidhuber, IDSIA (2022). Annotated History of Modern AI and Deep Learning.
• Hansen Hsu, Computer History Museum (2025). CHM Releases AlexNet Source Code.
• Andrej Karpathy (2022). Deep Neural Nets: 33 years ago and 33 years from now.
• Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks.
• Guinness World Records (2023). First neural network to identify handwritten characters.
• IEEE History Center, Engineering and Technology History Wiki (2025). Milestones: Convolutional Neural Networks, 1989.