Neocognitron: Shift invariance built into the wiring

A broadcaster's laboratory and a narrow claim

In April 1980, Biological Cybernetics published a ten-page paper by Kunihiko Fukushima of the NHK Broadcasting Science Research Laboratories in Tokyo, part of the research arm of Japan's public broadcaster. Titled Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position, it proposed a layered network that, after repeated exposure to a handful of patterns and with no labels at all, came to respond to each pattern through a single output cell regardless of where on its input array the pattern appeared. The claim was deliberately bounded. Fukushima presented the model as a working hypothesis about visual recognition in the brain, demonstrated by computer simulation on a few hand-drawn characters, and he suggested it could also help engineers build character readers that did not need every input centered first.

The date needs a word of explanation. The journal records the manuscript as received on 28 October 1979, and a footnote states that preliminary reports had already appeared. One was a three-page paper at the Sixth International Joint Conference on Artificial Intelligence, held in Tokyo from 20 to 23 August 1979, where the IJCAI-79 proceedings carry Fukushima's "Self-organization of a neural network which gives position-invariant response" with the same seven-layer simulation. A Japanese journal article followed in the October 1979 Transactions of the IECE, which Fukushima's own 2019 review of the neocognitron and Jürgen Schmidhuber's Annotated History of Modern AI and Deep Learning both list as the first full account. Chronicle uses 1980 because the Biological Cybernetics paper is the complete English description that the international field read and cited. The 1979 papers establish priority, and nothing in the story turns on the months between them.

The setting matters too. Fukushima, born in 1936, joined NHK in 1958 after graduating from Kyoto University and spent his first years on efficient coding of television signals, according to the profile NEC published when he received the 2021 C&C Prize. In 1965 he moved to the Broadcasting Science Research Laboratories, where he worked with neurophysiologists and psychologists. As the historian Hansun Hsiung describes in The Conversation, NHK had created a laboratory that year for the "bionics of vision", rooted in studies of the psychology and physiology of television audiences. The neocognitron came out of a broadcaster's curiosity about how people see, not out of an artificial intelligence program.

The problem the cognitron could not solve

Fukushima had already built one self-organizing network. His 1975 cognitron, also published in Biological Cybernetics, was a multilayered network that learned through competition among its cells, without a teacher. By his own account in the 1980 paper, it had a decisive weakness: the same pattern shown at two different positions was treated as two different patterns. The same was true, he wrote, of most earlier recognition models, from Frank Rosenblatt's perceptrons to proposals by Kabrisky and Giebel. Engineering practice had the same problem. Optical character readers of the time had to normalize the position of each character before recognizing it, which became difficult when characters were noisy or distorted.

The way out came from neurophysiology. David Hubel and Torsten Wiesel's 1962 study of receptive fields in the cat's visual cortex and their later papers described a hierarchy of cell types. In the simplified reading that the Deep Learning textbook by Ian Goodfellow, Yoshua Bengio and Aaron Courville still teaches, simple cells respond to a specific feature, such as an oriented bar, at a precise place, while complex cells respond to similar features but tolerate small shifts in their position. Fukushima's paper follows this chain from the lateral geniculate body through simple, complex and hypercomplex cells, then extends it upward by analogy to cells in monkey inferotemporal cortex that respond to whole shapes, such as the silhouette of a hand, largely regardless of position or size. The neuroscience was at the height of its prestige: Hubel and Wiesel shared the 1981 Nobel Prize in Physiology or Medicine for their discoveries about information processing in the visual system, a year after the neocognitron paper appeared.

Fukushima was candid that the model went beyond the data. He wrote that the Hubel and Wiesel hierarchy did not hold in its original form, citing contrary evidence such as direct connections from the geniculate body to complex cells, and he adopted it only as the main stream of visual information flow. The neocognitron was an engineering abstraction of a disputed biological picture, and its author said so.

How S-cells and C-cells divide the work

The network has an input layer of photoreceptors followed by a cascade of modules. Each module has two layers. The first is made of S-cells, named after simple cells, whose input connections are modifiable and which act as feature detectors. The second is made of C-cells, named after complex cells, whose input connections are fixed. Every layer is divided into what Fukushima called cell-planes. All cells in one cell-plane have the same pattern of input connections, shifted from cell to cell so that each looks at a different location. A cell-plane therefore applies one detector across the whole visual field, which is what a modern convolutional layer does with a shared kernel, although Fukushima did not use the word convolution. If a feature moves, a different cell in the same plane responds to it.

An S-cell weighs its excitatory input against a signal from an accompanying inhibitory cell that computes a root-mean-square of the same inputs. The excitation is divided by the inhibition, a form of shunting inhibition, and the result passes through a function that outputs zero for negative values and the value itself otherwise. That function is the rectifier now called ReLU, which Schmidhuber's history traces to Fukushima's 1969 work on visual feature extraction. A parameter sets how strongly inhibition sharpens selectivity, trading the ability to tell similar patterns apart against tolerance for distortion.

A C-cell receives fixed connections from a small neighborhood of S-cells in the corresponding plane, weighted more heavily near the center, and responds strongly whenever at least one of them does. Its output saturates, and each C-layer holds fewer cells per plane than the S-layer before it. This is the operation later called pooling and subsampling. In the neocognitron it is a weighted spatial average, not a maximum. In his 2019 review Fukushima stresses that C-cells pool by spatial averaging rather than by a max operation, and argues that the averaging also smooths noise in the S-cell responses.

Stacking modules makes receptive fields grow and tolerance accumulate. The paper describes recognition as piecewise template matching in many small windows, repeated stage by stage, with the allowance for shift widened a little at each step until a final cell sees the whole input. Because tolerance is granted locally, a character whose upper half matches a stored pattern in one place and whose lower half matches it slightly elsewhere can still be recognized. That idea, gradual invariance through alternating detection and pooling, is the architectural core that later convolutional networks inherited.

Learning without a teacher

The training procedure is where the neocognitron differs most from its descendants. Fukushima called it "learning without a teacher": the network sees stimulus patterns repeatedly and receives no information about which category each belongs to. Each time a pattern is shown, the S-cells in each layer are grouped into overlapping S-columns, sets of cells from every plane whose receptive fields sit at nearly the same position, an idea the paper compares to Hubel and Wiesel's hypercolumns. The most active cell in each column becomes a candidate, and within each cell-plane only the strongest candidate is chosen as the representative. The representative's excitatory connections are strengthened in proportion to the activity feeding them, and its inhibitory connection grows as well.

The crucial step comes next. Every other cell in the representative's plane has its connections strengthened by exactly the same amounts. This keeps the shared connection pattern intact, so a feature learned at one position becomes detectable at every position. Planes that supply no representative are left unchanged, which pushes different planes toward different features and keeps two planes from learning the same one. The initial connections were not random: they were small positive values chosen to give each plane a weak preference for a different orientation. The scheme is competitive and local, and it computes no error signal at the output. Nothing in the procedure asks whether the network's final responses are useful for classification. They come out distinct only because competition at every layer tends to separate patterns that differ.

What the 1980 simulation actually showed

The evidence was a single simulation, described precisely. The network had seven layers: a 16 by 16 input array and three S and C modules, with 24 cell-planes in every layer after the input. The planes shrank with depth, from 16 by 16 in the first S-layer to 10 by 10, 8 by 8, 6 by 6 and 2 by 2, ending in a final C-layer of 24 single cells whose receptive fields covered the entire input. Each S-cell could connect to a 5 by 5 window of the layer below, across all 24 planes after the first module. Fukushima reported his parameter settings, including the inhibition strengths and learning rates for each layer, so the experiment could in principle be repeated.

Training used five patterns, the digits 0 to 4, each presented 20 times at randomly shifted positions. After that, each digit drove exactly one of the final C-cells, no final cell responded to more than one digit, and the response did not change when the pattern moved. A figure shows shifted, distorted, resized and partly incomplete versions of each digit that still produced the right output. A second experiment trained the network on X, Y, T and Z, letters that share strokes, and it learned to tell them apart. A footnote adds that training was actually easier with patterns at a fixed position, so the shifted presentations were the harder test.

The third experiment is the most revealing. With all ten digits, the network could be made to recognize every one correctly only if the parameters were properly adjusted and the patterns skillfully presented, and small changes to either critically damaged the result. Fukushima concluded that 24 cell-planes per layer were not enough, and that larger networks had not been simulated because his computer lacked the memory. There was no held-out test set, no error rate, and no comparison against a rival method. The paper demonstrates a mechanism; it does not measure a recognizer.

From self-organization to backpropagation

The lineage from here to modern vision is well documented, and it runs through a change of training method. Fukushima continued the work at NHK with colleagues including S. Miyake and Takayuki Ito, and Schmidhuber's history records a 1986 NHK film of a neocognitron recognizing handwritten digits. Meanwhile others joined weight sharing to gradient training. Schmidhuber credits Alex Waibel's time-delay networks of 1987 with one-dimensional convolutions trained by backpropagation for speech, and Wei Zhang and colleagues in Japan with a backpropagation-trained two-dimensional convolutional network for character recognition in 1988.

The best-known version came from Bell Labs. In Handwritten Digit Recognition with a Back-Propagation Network, presented at the 1989 NIPS conference, Yann LeCun and colleagues described modules made of shared-weight feature maps followed by averaging and subsampling layers. They called the arrangement reminiscent of the neocognitron, citing Fukushima and Miyake's 1982 paper, and named the key difference: they trained with backpropagation instead of unsupervised learning, which they judged better suited to classification. In other words, the Bell Labs network kept Fukushima's layout and replaced his learning rule. The Deep Learning textbook reaches the same verdict in general terms: the neocognitron already had most of the design features of today's convolutional networks, but it trained them by unsupervised clustering, one layer at a time.

The C-cell kept changing too. Schmidhuber's history dates the replacement of Fukushima's spatial averaging by max pooling to 1990 for one-dimensional networks and 1993 for two-dimensional ones. Max pooling became standard in the networks that later dominated computer vision, including the 2012 AlexNet paper, which placed overlapping max-pooling layers after several of its convolutional layers. In neuroscience the line continued separately. Maximilian Riesenhuber and Tomaso Poggio's 1999 hierarchical model of object recognition in cortex named the neocognitron as the starting point for hierarchical models of invariant recognition, borrowed Fukushima's S and C notation for its template-matching and pooling units, and made the pooling a max operation. The neocognitron thus seeded two lineages: an engineering one that dropped its learning rule and a biological one that kept asking how the cortex achieves invariance.

The credit question

Retrospective honors make large claims. The Franklin Institute's 2021 Bower Award citation credits Fukushima with inventing the "first deep convolutional neural network", and NEC's C&C Prize profile calls the neocognitron the first step toward the convolutional networks behind the 2012 breakthrough. Schmidhuber's history assigns him the basic architecture of alternating convolutional and downsampling layers. On architecture, these claims are well supported by the 1980 paper itself. The cell-plane is a shared convolutional filter, the C-layer is pooling with subsampling, and the hierarchy is deep in the modern sense. Precedents existed for parts of the design: the Deep Learning textbook notes parameter sharing across positions in earlier connectionist vision models such as Marr and Poggio's of 1976, and Fukushima himself cited Giebel and Kabrisky. None of the sources consulted here, however, identifies an earlier system that combined these parts into a trainable, multi-stage, shift-tolerant recognizer.

The record is less generous in practice. The AlexNet paper's reference list cites LeCun's work but not Fukushima's, a point Schmidhuber's history makes explicitly. LeCun's 1989 conference paper does cite the neocognitron, yet for the weight-sharing technique it cites Rumelhart, Hinton and Williams in 1986, although cell-planes with shared connections are explicit in Fukushima's 1980 paper. Hsiung's account in The Conversation sets this in a wider pattern, arguing that Japanese pioneers have been written out of the history and that Fukushima and Shun-ichi Amari could have been considered for the 2024 Nobel Prize in Physics, which went to John Hopfield and Geoffrey Hinton. The awards also simplify. The Franklin Institute says Fukushima trained the network to read handwritten digits from zero to nine, which is true of later versions but not of the 1980 paper, where ten digits worked only under careful tuning.

There is also a difference of aims that the credit debate tends to flatten. Hsiung reports that Fukushima regarded his networks as simulations of the brain rather than as artificial intelligence, that he did not pursue scaling the neocognitron up with bigger datasets as NHK's demand for applied technology grew, and that he resigned from NHK in 1988. NEC's profile lists his professorship at Osaka University from 1989. In his 2019 review, written in his eighties, he argues that deep learning is not the only path to networks that rival the brain and that the field should keep learning from biology. For its author, the success of its descendants did not settle the question the neocognitron was built to ask.

The honest limits

Several limits follow directly from the primary source. The network was demonstrated on a handful of patterns drawn on a 16 by 16 grid, with no independent test data and no error rate, so the 1980 result says nothing quantitative about recognition accuracy. Its learning was fragile at ten classes and depended on hand-tuned parameters and on the way patterns were presented. The unsupervised competitive rule has no way to align internal features with the categories that matter, which is the problem backpropagation later solved, and the textbook judgment that the architecture survived while the training algorithm did not is fair. NEC's profile acknowledges that limited computing power kept the neocognitron in basic research for years.

The biological claims are also weaker than the analogy suggests. The S and C distinction rests on a simplified reading of Hubel and Wiesel, and the Deep Learning textbook warns that simple and complex cells may be points on a continuum rather than two kinds of cell. Fukushima himself flagged the contradictory physiology. The tolerance the network achieved covered shifts and small distortions within the image plane, and Riesenhuber and Poggio describe the neocognitron's invariance as limited to such image-plane transformations, whereas later models aimed at invariance to other kinds of change as well. Finally, the line from the neocognitron to modern convolutional networks is a line of architecture, not of code or of continuous development. The networks that won benchmarks were rebuilt around backpropagation, max pooling and large labeled datasets, by researchers who sometimes cited Fukushima and sometimes did not.

How to place the neocognitron in AI history

The neocognitron is best read as the moment the architecture of convolutional vision was first written down as a working, simulated system. Its lasting contributions are structural: shared detectors applied at every position, fixed pooling layers that trade positional precision for tolerance, and a deep hierarchy that builds invariance a little at a time. Its training method, competitive self-organization without labels, was the part the field set aside. Joining the structure to backpropagation, as Zhang and colleagues and the Bell Labs group did by the end of the 1980s, produced the trainable convolutional network, and scaling that with graphics hardware and labeled data produced AlexNet in 2012.

Three questions help in tracing connections from this event. What did the neocognitron take from biology, and how much of that biology still holds? Which of its components survived intact (weight sharing, hierarchy), which were modified (spatial averaging became max pooling), and which were replaced (self-organization gave way to gradient descent)? And how should credit be shared between the person who proposed an architecture and the people who found a way to train it at scale? Answered carefully, these questions place Fukushima's paper where the evidence puts it: not as a working character reader, which it was not in 1980, but as the blueprint that the most successful family of vision models has followed since, designed at a broadcaster's laboratory by a researcher who was trying to understand how people see.

References

• Kunihiko Fukushima, Biological Cybernetics (1980). Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position.
• Kunihiko Fukushima, Proceedings of IJCAI-79, Volume 1 (1979). Self-organization of a neural network which gives position-invariant response, pages 291 to 293 of the collected technical papers.
• Kunihiko Fukushima, Nonlinear Theory and Its Applications, IEICE (2019). Recent advances in the deep CNN neocognitron.
• Kunihiko Fukushima, Biological Cybernetics (1975). Cognitron: A self-organizing multilayered neural network.
• Jürgen Schmidhuber, IDSIA (2022). Annotated History of Modern AI and Deep Learning.
• Ian Goodfellow, Yoshua Bengio and Aaron Courville, MIT Press (2016). Deep Learning, Chapter 9: Convolutional Networks.
• David Hubel and Torsten Wiesel, The Journal of Physiology (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex.
• NobelPrize.org (1981). The Nobel Prize in Physiology or Medicine 1981.
• Yann LeCun, Bernhard Boser, John Denker et al., Advances in Neural Information Processing Systems 2 (1989). Handwritten Digit Recognition with a Back-Propagation Network.
• Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks.
• Maximilian Riesenhuber and Tomaso Poggio, Nature Neuroscience (1999). Hierarchical models of object recognition in cortex.
• The Franklin Institute (2021). Kunihiko Fukushima, 2021 Bower Award and Prize for Achievement in Science.
• NEC Corporation (2021). Profile and Detailed Achievements of Dr. Kunihiko Fukushima, Recipient of the 2021 C&C Prize.
• Hansun Hsiung, The Conversation (2024). Japanese scientists were pioneers of AI, yet they're being written out of its history.