What was AlexNet, and why did its 2012 ImageNet win change AI?

In the autumn of 2012, a small team from the University of Toronto entered a contest to teach computers to recognize objects in photos. Their entry, under the team name SuperVision, did not just win. It beat the best of the rest by more than ten percentage points. The network behind it soon became known as AlexNet, after its first author, Alex Krizhevsky.

The result mattered because of how it was achieved. Every other leading team used carefully hand-designed image features. AlexNet learned its features directly from raw pixels, and the team made that practical by training on two consumer graphics cards.

The contest and the result

The contest was the ImageNet Large Scale Visual Recognition Challenge 2012, usually shortened to ILSVRC. ImageNet is a huge collection of labelled photos; for the challenge, entrants trained on about 1.2 million images sorted into 1,000 categories. Their systems then had to label images they had never seen.

Scoring used the "top-5 error": a system may make five guesses per picture, and it counts as wrong only if the correct label is not among them. According to the encyclopedic record, AlexNet's entry was submitted on 30 September 2012. The official results list SuperVision first with an error of 0.153, that is, 15.3 percent. The runner-up, a University of Tokyo team called ISI, scored 26.2 percent, and teams from Oxford (VGG) and XRCE/INRIA came in around 27 percent, all three relying on hand-designed features fed to simpler classifiers. SuperVision also won the harder localization task, which asks where in the image the object is.

The three authors, Krizhevsky, Ilya Sutskever and Geoffrey Hinton, all at the University of Toronto, described the work in a paper for the NIPS 2012 conference, "ImageNet Classification with Deep Convolutional Neural Networks."

How AlexNet worked

AlexNet is a convolutional neural network, or CNN. A neural network is a stack of simple number-crunching units whose connections are adjusted during training. A convolutional network adds one idea that suits pictures: instead of every unit looking at the whole image, small filters slide across it, each one scanning for a pattern such as an edge or a patch of color. Layers higher up combine those patterns into textures, parts and finally whole objects.

Here is a simplified walk through what happens when AlexNet looks at a photo of a cat:

Layer group	What it detects (roughly)	Cat example
First convolutional layers	Edges, lines, color blobs	Outline of ears, stripes
Middle convolutional layers	Textures and simple parts	Fur, an eye, a nose shape
Last convolutional layers	Larger object parts	A cat-like face
Three fully connected layers	Combines everything into a decision	Scores for all 1,000 classes
Final output	Picks the top guesses	"tabby cat", "tiger cat", ...

The network had 60 million adjustable parameters and 650,000 units. Its eight learned layers split into five convolutional layers followed by three fully connected ones. Three practical choices made it work. It used ReLU units (rectified linear units, which simply pass positive numbers and zero out negatives), which the authors found trained several times faster than older units. It used "dropout," which randomly switches off units during training so the network cannot lean too heavily on any one of them. And it was split across two NVIDIA GTX 580 graphics cards with 3 GB of memory each, where it trained for five to six days.

Try it: Take one photo and write the top five labels you would guess for it, as a top-5 system would. Then ask a friend to do the same. If neither list contains the "right" answer, think about why: is the category ambiguous, or is the object partly hidden?

Why it mattered

AlexNet was not the first neural network on a graphics card, nor the first to win an image contest. Dan Cireșan and colleagues at the Swiss lab IDSIA had won several in 2011 and 2012 with similar GPU networks, and the whole design descends from the convolutional networks Yann LeCun introduced in 1989. What AlexNet did was win the field's flagship benchmark by a margin nobody could ignore.

The authors drew the lesson themselves: their paper predicts better results simply from bigger datasets and quicker graphics cards. Researchers took the hint, and a wave of papers followed that used convolutional networks trained on GPUs. The recipe of big data, big networks and GPU training has since spread well beyond images. By 2020 the paper had been cited more than 70,000 times. Today's large models, from GPT-3 to OpenAI's new image models DALL-E and CLIP announced last month, rest on that recipe.

Why did graphics cards turn out to be so good for training neural networks?

Graphics cards, or GPUs, were designed to draw video games. Drawing a frame means doing the same small calculation for millions of pixels at once, so a GPU contains thousands of simple processors working in parallel. A regular processor (CPU) has only a few, more powerful cores built to do one thing after another.

Training a neural network turns out to be the same kind of job. Each layer multiplies large grids of numbers together, and nearly all of those multiplications can happen at the same time. A convolution, in particular, applies the same filter at every position in an image, which is exactly the repeated, parallel work a GPU is built for.

The AlexNet team wrote their own highly optimized code for these operations. Even so, memory was the tight limit: one card's 3 GB could not hold the whole network, which is why they split it across two cards that talk to each other only at certain layers. That is also why they saw hardware as the main brake: they wrote that the network's size was limited mainly by GPU memory and by how long they were willing to wait.