ResNet: Depth without the degradation

The event and the narrow claim

On 10 December 2015, four researchers at Microsoft Research Asia, Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, posted a paper describing a training method for very deep neural networks called residual learning. Deep Residual Learning for Image Recognition made one narrow, testable claim: reformulating a stack of layers so that they learn a residual relative to their own input, rather than an unreferenced mapping, makes substantially deeper networks easier to optimize. The authors demonstrated the idea on image classification and reported that an ensemble of residual networks reached 3.57% top-5 error on the ImageNet test set and won first place in the ILSVRC 2015 classification task. These are task-specific benchmark results on one dataset, not measurements of visual understanding in general.

The historically useful question is not whether this paper "solved depth," but which specific obstacle it removed, which ingredients it inherited, and which later results depended on it. The paper's own framing is disciplined in exactly this way: it isolates an optimization phenomenon, proposes a mechanism, and then tests it. Separating that documented contribution from the sweeping claim that ResNet made deep learning possible is the first step to placing it correctly. The conference version went on to receive the CVPR 2016 Best Paper Award.

The degradation problem, stated precisely

By late 2015, depth had become the dominant lever in computer vision. VGG had shown that stacking many small 3x3 convolutions into sixteen or nineteen weight layers improved large-scale image recognition, and GoogLeNet had reached comparable accuracy with a twenty-two-layer inception design. The intuition was straightforward: a deeper network is a strict superset of a shallower one, since the extra layers could in principle learn identity mappings and do no harm.

Experiment refused to cooperate. The ResNet authors documented a degradation problem: as plain networks grew deeper, accuracy saturated and then got worse, and crucially the extra error appeared in training, not only in testing. A 34-layer plain network had higher training error than an 18-layer plain network. This was not overfitting, which would show as low training error paired with high test error. It was an optimization failure: the deeper model could represent the shallower one but stochastic gradient descent did not find that solution. This is the constraint the paper attacked, and stating it precisely matters, because it is often misremembered as a vanishing-gradient story, a conflation worth untangling. Earlier training difficulties in deep networks were frequently attributed to gradients shrinking toward zero as they propagate backward. By 2015 that particular problem was largely mitigated by normalized initialization and by batch normalization, which keeps forward activations and backward gradients at healthy scales throughout a deep stack. The ResNet authors used batch normalization and verified that their plain networks had non-vanishing gradients, yet the degradation persisted. Their conclusion was that the difficulty lay in optimization dynamics, the practical hardness of driving a deep stack of nonlinear layers toward an identity or near-identity mapping, not simply in gradient magnitude. Getting this right changes what the fix is for: it is a reparameterization that makes the easy solutions easy to reach, not a patch on gradient flow alone.

Highway networks and the gating alternative

ResNet did not appear in a vacuum. Months earlier, Highway Networks by Rupesh Srivastava, Klaus Greff and Jürgen Schmidhuber had attacked the same problem from a different angle. Borrowing the gating idea from long short-term memory, they added learned transform and carry gates that let information bypass a layer, and reported that networks hundreds of layers deep could be optimized with plain gradient descent, nearly independent of depth.

The comparison sharpens what ResNet contributed. Highway gates are data-dependent and carry their own parameters; when a gate closes, the shortcut is suppressed. ResNet's shortcut is an unconditional, parameter-free identity add. That choice sounds minor but has consequences: the identity path is always open, it adds no parameters or meaningful computation, and it makes the "do nothing" solution the default rather than something the network must learn to switch on. Historically this is a case where simplifying a predecessor, rather than adding to it, was the productive move.

How a residual block actually works

The mechanism is compact. Suppose a few stacked layers are meant to compute some desired mapping H(x). Instead of asking those layers to produce H(x) directly, ResNet has them compute a residual function F(x) = H(x) - x, and then recovers the target as F(x) + x. The "plus x" term is implemented by a shortcut connection that carries the block's input forward and adds it, element by element, to the block's output. When input and output dimensions match, this shortcut is a pure identity with no parameters; when they differ, a 1x1 convolution projects the input to the right shape.

The hypothesis behind the reparameterization is that residual mappings are easier for optimization to reach than unreferenced ones. In the limiting case where the optimal transformation for a block is close to the identity, the network only has to push the weights of F toward zero, which is far easier than coaxing a stack of nonlinear layers into approximating identity from scratch. The shortcut also gives gradients a direct additive route back through the network. None of this changes which functions the network can represent; it changes which ones training actually finds. That separation, between representational capacity and optimization reachability, is the intellectual core of the paper.

The depth ladder and the bottleneck block

ResNet was released as a family, and the specific configurations became reference architectures still named daily. The paper defined 18-, 34-, 50-, 101- and 152-layer variants. The shallower two use a basic block of two 3x3 convolutions. The deeper three use a bottleneck block that sandwiches a 3x3 convolution between two 1x1 convolutions, the first reducing channel dimension and the last restoring it, so the expensive spatial convolution operates on fewer channels. This is what made extreme depth affordable: the 152-layer network, roughly eight times deeper than VGG-19, still required fewer floating-point operations, about 11.3 billion versus VGG-19's roughly 19.6 billion. The bottleneck was not cosmetic. By shrinking the channel count before the 3x3 convolution and expanding it again afterward, it held the per-block cost roughly constant even as blocks were stacked by the dozen, which is a large part of why ResNet-50 became the field's default vision backbone for the better part of a decade.

The result held under direct comparison. Where the 34-layer plain network was worse than its 18-layer counterpart, the 34-layer residual network clearly beat the 18-layer residual network, and its training error was lower too. The reparameterization, not additional capacity, flipped the sign of the depth effect. The authors pushed the point further on CIFAR-10, training a network of over a thousand layers; it optimized without trouble, though a 1202-layer version did not beat a 110-layer one, which they attributed to overfitting on a small dataset rather than any optimization failure.

The evidence, and where the numbers came from

The headline evidence was the ImageNet Large Scale Visual Recognition Challenge, the annual benchmark on roughly 1.28 million training images across 1000 categories that had organized the field's progress since 2012. Top-5 error on that task had fallen from about 15% for the 2012 convolutional network to around 6.7% for GoogLeNet in 2014. ResNet's 3.57% ensemble result in 2015 continued that trajectory, and sat below the roughly 5% top-5 error that a careful human annotator was estimated to reach on the same labeling task. That comparison is specific to this benchmark's closed set of categories and its top-5 protocol; it is not a statement that machine vision had matched human perception in general. A single 152-layer model, with no ensembling, already reached about 4.5% top-5 error on the validation set, so it was the depth itself and not merely committee averaging that carried the gain.

The win was not confined to classification. The same residual backbones took first place in ImageNet detection and localization and in the COCO 2015 detection and segmentation tracks, with the paper reporting a 28% relative improvement on COCO detection. The breadth mattered: it showed the contribution was a general-purpose feature extractor, not a trick tuned to one scoreboard.

After the paper: refinements and rivals

Within months the same group returned with Identity Mappings in Deep Residual Networks, which analyzed exactly why the shortcut helps and how to make it help more. By keeping the shortcut a clean identity and moving batch normalization and activation before the convolutions, a design they called pre-activation, they created a path along which signals propagate directly from any block to any other, forward and backward. The refined design trained more easily at extreme depth: they reported a 1001-layer network reaching 4.62% error on CIFAR-10. This follow-up is a good example of how a strong empirical result invites a second, more mechanistic paper that explains and tightens it, rather than treating the original as final.

ResNet did not end architectural debate; it reframed it. DenseNet took the connectivity idea to an extreme by concatenating, rather than adding, the outputs of all previous layers, arguing this improved feature reuse and gradient flow with fewer parameters. ResNeXt, from an overlapping set of authors that again included He, kept the residual skeleton but replaced each block with many parallel transformations of the same topology, introducing "cardinality" as a new axis alongside depth and width. Wider but shallower residual networks were shown to match very deep ones, and the inception line continued in parallel.

The lesson for a timeline is that skip connections became a shared substrate on which several distinct design philosophies competed. Concatenation versus addition, depth versus width versus cardinality: these were live disagreements, and the fact that they all assumed some form of shortcut is itself the measure of how quickly the core idea was absorbed.

Downstream influence beyond vision

The residual connection outgrew image recognition. Its most consequential migration was into sequence modeling: the Transformer wraps every attention and feed-forward sub-layer in a residual connection followed by normalization, the step that lets gradients flow through dozens of stacked blocks. Essentially every large language model in use today inherits that pattern, which means it inherits ResNet's shortcut. Within vision, the residual backbone became the standard feature extractor beneath the detection and segmentation systems of the following years, from region-proposal detectors to feature-pyramid and instance-segmentation frameworks, so a large share of applied computer vision quietly ran on a ResNet. The broad review of the field by LeCun, Bengio and Hinton, published in Nature the same year, had already framed depth as the engine of representation learning; residual learning is a large part of why that depth became routinely trainable. The paper itself became one of the most cited in all of science, a rough proxy for how thoroughly the design pattern diffused.

The honest limits

Several caveats keep the claim proportionate. First, residual learning is a statement about optimization, not a guarantee that more layers help; architecture, compute and task still decide whether added depth pays, and the 1202-layer CIFAR result is the authors' own reminder. Second, the mechanism enables training but does not by itself explain what deep networks learn or make them interpretable; a network that optimizes cleanly can still be brittle, biased by its data, or wrong in ways a benchmark does not surface. Third, benchmark supremacy on ImageNet's fixed categories is narrow evidence: it says nothing about robustness to distribution shift, adversarial inputs or the open-ended visual world. Fourth, the intuition that identity is easy to learn is a hypothesis supported by strong evidence, not a theorem; later work kept debating why residual networks train so well, including the view that they behave like ensembles of many shorter paths. Fifth, the identity shortcut carries design choices of its own: when dimensions change, the projection shortcut adds parameters, and the size of the gain from depth varied with dataset, block design and training budget rather than following a single universal curve.

How to place ResNet in AI history

The defensible reading is that ResNet is a foundational engineering idea with a precisely documented origin and an unusually wide blast radius. The narrow claim, that residual reparameterization removes the degradation barrier to depth, was demonstrated cleanly in 2015 and refined in 2016. The broad claim, that skip connections became a default component of deep learning, rests not on assertion but on the Highway predecessor it simplified, the vision descendants that assumed it, and the Transformer generation that carried it into language.

To navigate outward from this event, three questions help. Which constraint did it remove, and how do we know it was that constraint and not another? Which later systems reused the shortcut while changing something else, from DenseNet's concatenation to the Transformer's attention? And which limits remained for others to address, from interpretability to robustness? Answered together, they place ResNet as what it was: not the invention of deep learning, and not a measure of machine intelligence, but the change that made training very deep networks a normal, reliable thing to do.

References

• He, Zhang, Ren and Sun (2015). Deep Residual Learning for Image Recognition.
• He, Zhang, Ren and Sun (2016). Deep Residual Learning for Image Recognition, CVPR conference version.
• Srivastava, Greff and Schmidhuber (2015). Highway Networks.
• Simonyan and Zisserman (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition.
• Szegedy et al. (2014). Going Deeper with Convolutions.
• Ioffe and Szegedy (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.
• He, Zhang, Ren and Sun (2016). Identity Mappings in Deep Residual Networks.
• Huang, Liu, van der Maaten and Weinberger (2016). Densely Connected Convolutional Networks.
• Xie, Girshick, Dollar, Tu and He (2016). Aggregated Residual Transformations for Deep Neural Networks.
• Russakovsky et al. (2015). ImageNet Large Scale Visual Recognition Challenge.
• Vaswani et al. (2017). Attention Is All You Need.
• LeCun, Bengio and Hinton (2015). Deep learning.