What were Hinton's deep belief networks, and how did they restart deep learning?

By the early 2000s, neural networks had a reputation problem. The idea of stacking many layers of artificial neurons, each finding more abstract patterns than the last, had been around for decades. In practice, networks with more than one or two hidden layers were slow and frustrating to train, and many researchers had moved on to other methods, such as support vector machines, that gave reliable results with less fuss.

In July 2006, a short paper titled "A fast learning algorithm for deep belief nets" changed the mood. Its authors were Geoffrey Hinton and Simon Osindero of the University of Toronto and Yee-Whye Teh of the National University of Singapore. It appeared in Neural Computation, volume 18, issue 7, and it offered a practical recipe for training a deep network in stages.

The problem the paper solved

A neural network learns by adjusting the strength of the connections between its units, often called weights. With one hidden layer, the standard method, backpropagation, works well: it measures the network's error at the output and passes corrections backward through the layers. With many layers, those corrections tend to become weak or erratic by the time they reach the bottom, and training can take forever or settle into poor solutions.

Hinton's team took a different route. Instead of training all the layers together from random starting weights, they trained one layer at a time, without labels, and only then fine-tuned the whole network.

How a deep belief net is built

The building block is a restricted Boltzmann machine (RBM). It has two rows of units: a visible row that receives data, such as the pixels of an image, and a hidden row that learns to describe that data. Every visible unit is linked to every hidden unit, but units in the same row are not linked to each other. That restriction is what makes the machine quick to train.

An RBM learns by trying to reconstruct its input. It looks at an image, turns on hidden units to describe it, then uses those hidden units to redraw the image. Whenever the redrawing differs from the original, it adjusts its weights slightly so that the next attempt is closer. Over many images, the hidden units become detectors for common features, such as strokes and curves in handwriting.

The trick is to stack them. Here is the recipe from the paper, as steps:

1. Train an RBM on raw pixel images. Its hidden units learn simple features.
2. Freeze those weights. Feed every image through and record what the hidden units do.
3. Train a second RBM, treating those recorded activities as if they were the data.
4. Repeat for as many layers as you want. Each one learns patterns in the layer below.
5. Finally, fine-tune the whole stack together with a slower procedure that the authors call the "up-down" algorithm.

The authors show that adding each new layer in this way improves a mathematical bound on how well the whole model describes the data. In other words, the greedy approach is not just a hack; it has a sound basis.

The term "belief net" refers to a kind of probability model in which connections express how likely some units are to be on, given the state of others. Because the model is generative, meaning it can produce new examples as well as recognise them, the authors could run it in reverse and display images of what the network "had in mind" for each digit.

The handwriting test

To show the method worked, the team used MNIST, a standard collection of handwritten digits: 60,000 examples for learning and a separate 10,000 for testing, each 28 by 28 pixels. Their network had two layers of 500 units, a top layer of 2,000 units, and 10 extra units representing the digit labels 0 to 9.

Here is how it compared with other methods on the version of the task where the algorithm is given no built-in knowledge that the inputs are images:

Methods designed specifically for images, or given extra distorted training examples, did better still. But on a level playing field, a deep network trained this way beat the best alternatives. Each layer took a few hours to pretrain in Matlab on a single 3 GHz processor, and the full training took about a week.

Why the paper mattered

The result itself was a fraction of a percentage point on a digit test. Its significance was what it proved: deep networks were not hopeless. Wikipedia describes the layer-by-layer training trick as the source of one of the earliest deep learning methods that actually worked well.

The paper also showed the value of learning without labels. Most of the work happens before the network is ever told which digit is which. That matters because unlabelled data, such as images or text, is far more plentiful than carefully labelled data.

The years that followed brought a wave of research on deep networks, the spread of the term "deep learning," and, from 2012, a string of wins in image and speech recognition. As of late 2020, deep learning is behind most of the AI news: language models such as GPT-3, and DeepMind's AlphaFold system, whose protein-structure predictions impressed biologists at the CASP14 assessment this week.

Do today's deep learning systems still use deep belief networks?

Mostly not. Researchers soon found that, with more data, faster graphics processors and some simple changes to how artificial neurons respond, very deep networks could be trained directly with backpropagation, skipping the layer-by-layer pretraining. Large modern systems are usually trained that way.

The ideas, though, live on. Pretraining on unlabelled data and then fine-tuning for a specific task is exactly how many language models work today: GPT-3, for instance, first learns from a huge amount of text and can then be adapted to particular jobs. The notion that each layer builds more abstract features on top of the one below is still the standard way to explain what a deep network does.

So deep belief networks are less a tool in everyday use than a turning point. They showed that depth could work, and that renewed confidence brought people, funding and computing power back to neural networks.