# What is a residual network (ResNet), and why did it let neural networks get so deep?

Date: 2021-08-15 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-are-residual-networks-resnet/

**Answer:** A residual network, or ResNet, is a neural network built from blocks that each learn only a small change to their input and then add the original input back through a shortcut. Introduced by Kaiming He and three Microsoft Research colleagues in a paper posted on 10 December 2015, the design made networks of 152 layers trainable, won the 2015 ImageNet competition, and became a standard building block of modern AI.

By 2015, the lesson of deep learning seemed clear: deeper networks see more. A network with more layers can build up from edges to textures to parts to whole objects, and each jump in depth had brought better results in image recognition. Yet researchers kept running into a wall. Past a certain depth, adding layers made networks worse, not better, even on the very images they were trained on.

On 10 December 2015, four researchers at Microsoft Research posted a paper on arXiv called "Deep Residual Learning for Image Recognition" that went straight through that wall. Their trick was a shortcut so simple it fits in one line: let each block of layers pass its input straight through and add its own small correction on top.

## What happened, and who was involved

The authors were Kaiming He and three colleagues: Xiangyu Zhang, Shaoqing Ren and Jian Sun. Their networks, which they called **residual nets** or **ResNets**, were tested on ImageNet, a collection of about 1.28 million training photos sorted into 1,000 categories, and the basis of an annual contest known as the ImageNet challenge.

The results were striking. The team trained networks up to 152 layers deep, about eight times deeper than the well-known VGG networks, yet cheaper to run. A group of these networks working together made a mistake on only 3.57 percent of test images, counting an answer as right if the correct label was in its top five guesses. That won first place in the 2015 ImageNet classification task. ResNets also took first place in ImageNet's detection and localization tasks and in the COCO competition's detection and segmentation tasks.

## The problem: deeper got worse

The paper opens with a puzzle. The team trained two ordinary "plain" networks, one with 20 layers and one with 56. The deeper one had *higher* error, not only on new images but on the training images themselves. That rules out overfitting (memorizing the training set). The deeper network was simply harder to train.

This makes little sense on paper. Take a good 20-layer network and add 36 layers that do nothing but copy their input forward unchanged; the result should be exactly as good. So a deeper network can always do at least as well, in theory. In practice, the standard training method, which nudges every connection a little at a time, could not find that solution. Learning to "do nothing" turned out to be surprisingly hard for a stack of layers.

## How residual learning works

He and his colleagues changed what each block is asked to learn. Instead of producing a whole new output, a block learns only the **residual**, the difference between what it should output and what came in. A **shortcut connection** then carries the original input around the block and adds it back:

> output = input + correction

If a block has nothing useful to add, it only needs to push its correction toward zero, which is easy, and the input passes through untouched. The shortcuts add no extra numbers to learn and almost no extra computation.

An everyday analogy is editing a document with "track changes." Rather than retyping the whole page at every round of review, each editor marks only what should change. If an editor has nothing to say, the page goes through as it was. With fifty editors in a row, the tracked-changes process is far less likely to garble the text than fifty full retypes.

Here is a toy example with one number flowing through three blocks:

| Block | Input | Correction learned | Output (input + correction) |
|---|---|---|---|
| 1 | 5.0 | +0.8 | 5.8 |
| 2 | 5.8 | 0.0 (nothing useful to add) | 5.8 |
| 3 | 5.8 | −0.3 | 5.5 |

Block 2 has learned to stay out of the way, and the signal survives intact. The same shortcut also helps during training, when error signals travel backward through the network: they can flow along the shortcuts instead of fading out as they pass through dozens of layers.

> **Try it:** Take a sentence and pass it along a chain of five friends. In round one, each friend rewrites it from memory. In round two, each friend receives the written sentence and may only cross out or add a word or two. Compare how close each final version is to the original.

The paper also tested the idea at extremes. On a smaller image set called CIFAR-10, the team trained a network of more than 1,000 layers. It trained successfully, though it scored slightly worse on new images than a 110-layer version, likely because it was far bigger than such a small dataset needed.

## Why it mattered

Residual connections were not invented from nothing. Earlier researchers had tried "skip" connections, and the LSTM networks used for language, along with the "highway networks" published in 2015, contain related ideas. But ResNet showed, convincingly and at scale, that a plain identity shortcut is enough to make very deep networks trainable.

The design spread fast. Follow-up work such as Wide ResNets and DenseNet in 2016 and ResNeXt in 2017 built on it, and ResNet models became a common starting point for image recognition work.

## Where do residual connections show up in today's AI models?

Almost everywhere. The 2017 Transformer, the design behind today's large language models, wraps a residual connection around each of its sub-layers and cites the ResNet paper for the idea. That means BERT, GPT-2 and GPT-3 all rely on the same "input plus correction" structure, stacked dozens of times. Very deep Transformers would be extremely hard to train without it.

Residual connections also appear in game-playing and science systems. DeepMind's AlphaGo Zero, from 2017, is built on a residual network, and the AlphaFold system for predicting protein structures uses them too.

If you look inside almost any modern neural network, you will find the one-line idea from December 2015: keep what you have, and learn only what to change.

## Sources

- Deep Residual Learning for Image Recognition: https://arxiv.org/abs/1512.03385
- Residual neural network: https://en.wikipedia.org/w/index.php?title=Residual_neural_network&oldid=1038140520
- Attention Is All You Need: https://arxiv.org/abs/1706.03762