How did Google's neural network learn to recognize cats on its own in 2012?

In June 2012, Google published a blog post with an unusual result. Its researchers had shown a giant artificial neural network still frames from YouTube videos for days, without telling it what any picture contained. When they looked inside afterwards, they found that one of its artificial neurons fired strongly whenever a cat appeared. Nobody had shown it a single image marked "cat."

The story spread quickly, partly because cats and YouTube make a good headline. Underneath was a serious question: can a machine discover meaningful concepts by itself, just by looking at enough raw data?

Who did it, and when

The work came from a project inside Google that became known as Google Brain. It began in 2011, when Jeff Dean and Greg Corrado of Google teamed up part-time with Andrew Ng, a Stanford professor interested in deep learning, to build it at very large scale. They built software called DistBelief to spread a single neural network across many machines in Google's data centers.

The paper, "Building High-level Features Using Large Scale Unsupervised Learning," lists eight authors led by Quoc V. Le of Stanford. A first version went online in December 2011, and the team presented it at the International Conference on Machine Learning (ICML) in Edinburgh, Scotland, in June 2012. Google announced it to the public on 26 June 2012 in a post by Dean and Ng, and the New York Times covered the result the same month.

What they built

The scale was the headline. Most neural networks in use had between 1 million and 10 million connections, according to the Google post. This one had about 1 billion. It ran on 1,000 machines with 16 cores each, 16,000 cores in all, for three days.

The training data was 10 million images, each 200 by 200 pixels, taken from randomly chosen YouTube videos. Crucially, none of them carried a label. The network had nine layers and was a type called a sparse autoencoder.

An autoencoder is easiest to grasp as a compression game. The network squeezes each image into a short internal summary, then tries to rebuild the original image from that summary alone. To rebuild well, it must keep what matters and throw away noise. "Sparse" means only a few internal units are allowed to be active for any given image, which pushes each unit to specialize in one kind of pattern.

The steps looked like this:

1. Feed in a YouTube frame, with no label.
2. The network encodes it layer by layer into a compact set of active units.
3. It decodes that summary back into an image and measures how far off the copy is.
4. It adjusts its connections slightly to make the next copy better.
5. Repeat millions of times across 16,000 cores.
6. Afterwards, show the trained network test pictures, some with faces or cats and some without, and look for any single unit that lights up for one group and not the other.

What it found

The best "face neuron" could tell faces from non-faces with 81.7 percent accuracy, and it kept working when faces were shifted, resized or turned. The same network contained units tuned to human bodies and to cat faces; the best cat unit scored 74.8 percent. The researchers had been inspired by an old idea in neuroscience, sometimes nicknamed "grandmother neurons": single brain cells that respond to one specific thing, such as your grandmother's face.

The team then tested how useful these self-taught features were. Using them as a starting point, and only then adding labeled examples, they reached 15.8 percent accuracy on a version of ImageNet with about 22,000 categories. That sounds low, but with so many choices it was a 70 percent relative improvement over the previous best.

Why it mattered

The cat result showed that very large networks, given enough computing power, could learn useful concepts from unlabeled data. It also proved that Google's infrastructure could train networks far bigger than anyone had used before. Google said at once that it was applying the approach to speech recognition and language, and in March 2013 it hired Geoffrey Hinton, whose Toronto team had just won the 2012 ImageNet contest with AlexNet.

The project itself moved out of the experimental Google X lab into Google's main research organization. And the idea of learning from huge amounts of unlabeled data sits at the heart of today's language models, such as GPT-3, which learn by reading raw text with no human labels.

What is the difference between supervised and unsupervised learning?

In supervised learning, every training example comes with the right answer attached. You show the system a photo and the label "cat," a thousand times over, and it learns to map pictures to labels. AlexNet's win on ImageNet was supervised learning: 1.2 million photos, each labeled by people.

In unsupervised learning, there are no answers. The system gets only the raw data and must find structure on its own, by grouping similar items, compressing the data, or predicting missing pieces. The Google cat network was unsupervised: it saw only pixels, and its "cat" concept emerged from the data.

Labels are expensive, because people must create them by hand, while unlabeled data is everywhere. That is why researchers keep looking for ways to learn as much as possible without labels, then use a small labeled set at the end, exactly as the Google team did. GPT-3 follows a related approach: it learns by predicting the next word in ordinary text, so the text itself supplies the answers and no one has to label anything.