What is ImageNet, and why did a collection of labelled photos change AI?

If you want to teach a child what a dog is, you show them dogs: big ones, small ones, dogs in the park, dogs on sofas, dogs half hidden behind a fence. A computer learning to recognise images needs the same thing, only far more of it. In the mid-2000s, the image collections researchers used held a few dozen or a few hundred categories. ImageNet set out to cover nearly every noun you could photograph.

The project was led by Li Fei-Fei, who began working on the idea in 2006, and grew into a team at Princeton University. Its first formal description, a paper by six Princeton authors with Jia Deng first and Li Fei-Fei last, was presented in June 2009 at CVPR, the main yearly conference on computer vision, held that year in Florida.

What ImageNet is

ImageNet borrows its structure from WordNet, a large dictionary of English built at Princeton that groups words by meaning. In WordNet, a set of words that mean the same thing, such as "dog, domestic dog, Canis familiaris," is called a synset. Synsets are arranged in a tree: "dog" sits under "canine," which sits under "carnivore," then "mammal," and so on. WordNet has about 80,000 noun synsets.

Fei-Fei Li met Christiane Fellbaum, one of WordNet's creators, to discuss the plan; Wikipedia dates the meeting to 2007 in its main text, while an article it cites places it in 2006. The goal they settled on was ambitious: for each noun synset, collect 500 to 1,000 clean, full-resolution photos. Complete, that would mean tens of millions of images.

The 2009 paper reported the first stage: 12 branches of the tree, including mammals, birds, fish, vehicles, furniture, musical instruments, tools, flowers and fruit, covering 5,247 synsets and 3.2 million images. It was already much larger and more varied than the datasets researchers had been using.

How the photos were collected and checked

Building a database this big by hand in a lab would take decades. The team used two tools: internet search engines and crowds of paid online workers.

Here is the pipeline, step by step:

1. Search. For each synset, query several image search engines with all its synonyms. To get more results, add a parent word (a search for "whippet" also tries "whippet dog" and "whippet greyhound").
2. Translate. Run the same searches in Chinese, Spanish, Dutch and Italian to widen the pool.
3. Remove duplicates. On average, each synset ends up with more than 10,000 candidate images.
4. Verify. Only about 10% of raw search results actually show the right thing, so each candidate is shown to workers on Amazon Mechanical Turk, an online marketplace for small paid tasks, along with the synset's definition. They answer one question: does this image contain this object?
5. Vote. Several workers look at each image, and the answers are combined, with harder categories needing more agreement before an image is accepted.

The authors checked a random sample of 80 synsets and found the labels were correct about 99.7% of the time. According to Wikipedia, labelling ran from July 2008 to April 2010 and involved about 49,000 workers in 167 countries, who sorted more than 160 million candidate pictures.

From database to contest

A dataset is most useful when everyone tests on it the same way. In 2010, the ImageNet team, working with the organisers of the smaller PASCAL visual recognition challenge, launched the ImageNet Large Scale Visual Recognition Challenge. It used a trimmed set of 1,000 categories, including many dog breeds that are hard even for people to tell apart.

The results tell the story of the decade:

"Top-5" means the system gets five guesses per image and counts as right if the correct label is among them.

Why ImageNet mattered

The 2012 result is the one that changed the field. AlexNet, trained on graphics processors, beat every traditional method by a wide margin, and researchers across computer vision switched to deep neural networks almost overnight. Wikipedia quotes The Economist saying that people across the whole technology industry started paying attention. Much of the deep learning used for image tasks in 2020, from photo search to medical scans, traces back to that moment.

ImageNet also changed how AI research is done. It showed that a large, carefully built, public dataset with a shared benchmark could push a whole field forward faster than any single clever algorithm.

It has also become a case study in the risks of large datasets. Between 2018 and 2020, the ImageNet team reviewed the 2,832 categories describing people, found more than half of them potentially offensive, and kept only a small fraction. Researchers have shown how the labels can carry the biases of the words and workers behind them.

Why does the data matter as much as the algorithm?

Because a learning system can only learn what its examples show it. A clever algorithm trained on a few thousand photos will still fail on the variety of the real world; a fairly ordinary one trained on millions of varied, correctly labelled photos can do surprisingly well. Li Fei-Fei's bet, at a time when most of the field focused on designing better models, was that the data itself was the bottleneck.

The same lesson holds for the language models in the news in 2020. GPT-3 is impressive partly because of its design, but mostly because it learned from an enormous amount of text. And just as with ImageNet, what is in that data, including its mistakes and biases, shows up in what the model produces.

For anyone building with AI, the practical takeaway is simple: before blaming the model, look at the examples it learned from.