What are LeNet-5 and the MNIST dataset, and why does every deep learning course start with them?

Open almost any beginner's tutorial on neural networks and you will meet the same small, blurry pictures: handwritten 0s, 1s and 7s, each 28 pixels square. They come from MNIST, a collection of digits built at AT&T in the mid-1990s. The network that made them famous, LeNet-5, is the template that many image-recognition systems still follow.

Both were laid out in detail in "Gradient-Based Learning Applied to Document Recognition," a long paper by four researchers, Yann LeCun, Léon Bottou, Patrick Haffner and Yoshua Bengio, in the November 1998 issue of the Proceedings of the IEEE. The authors worked at AT&T Labs-Research in Red Bank, New Jersey. The work itself is older than the paper: the MNIST database dates from 1994, and some accounts date LeNet-5 to 1995. The 1998 paper is where it all appears together.

What the paper contained

The paper had a big goal: to show that learning from data beats hand-built rules for reading documents. It compared many methods on one shared test, presented LeNet-5, introduced a way to train whole multi-part recognition systems at once (called graph transformer networks), and described two systems in commercial use, including one that read bank checks.

How LeNet-5 works

LeNet-5 builds on the convolutional network LeCun's group had used for ZIP codes in 1989. It has seven layers with trainable weights, not counting the input, and takes a 32 by 32 pixel image. It alternates two kinds of layers:

1. Convolution (C1): six small pattern detectors, each with its own set of shared weights, slide across the image and produce six feature maps, such as maps of edges at different angles.
2. Subsampling (S2): each map is shrunk to half its width and height by combining every 2 by 2 block of values into one, so exact positions matter less.
3. Convolution (C3): sixteen detectors combine the earlier maps into more complex shapes, such as corners and loops.
4. Subsampling (S4): the maps are shrunk again.
5. Convolution and full connections (C5, F6): the features are combined into 84 numbers describing the whole digit.
6. Output: ten units, one per digit, each measuring how closely those 84 numbers match its digit; the best match is the answer.

Because detectors share weights across positions, the network has 340,908 connections but only about 60,000 numbers to learn. The input is larger than the digits themselves, which fit in a 20 by 20 box, so that features near the edges, such as the end of a stroke, still fall in the middle of a detector's view.

How MNIST was built

Before MNIST, the obvious source of digit images was the US National Institute of Standards and Technology (NIST). But NIST's training set, Special Database 3, came from Census Bureau employees, while its test set, Special Database 1, came from high school students, whose writing is much messier. Results depended on which set you trained on.

The authors mixed the two. They split Special Database 1 by writer, 250 writers for training and 250 for testing, and topped both halves up with examples from Special Database 3. The result, "Modified NIST" or MNIST, has 60,000 training images and 10,000 test images. Each digit was scaled to fit a 20 by 20 box and centered in a 28 by 28 grid.

The paper then compared classifiers on it:

The distortions were small, random stretches and squeezes of the training images, a trick that adds 540,000 extra examples without collecting any new handwriting.

Reading checks

LeNet-5 was not only a laboratory result. A check-reading system built around it was integrated into NCR's check-processing products. It was fielded in several US banks from June 1996, and the paper reports it reading millions of checks per day. An independent test by systems integrators in 1995 had found it more accurate than other commercial systems for reading the amount written on a check.

Why it matters now

Convolutional networks did not take over right away. Hardware was limited, and support vector machines could match or beat them on many tasks. That changed after the success of the AlexNet network in 2012, when convolutional networks became the default choice for computer vision. The modern designs are much deeper and larger, but their pattern of convolution, pooling and fully connected layers comes straight from LeNet.

MNIST, meanwhile, became the "hello world" of machine learning: small enough to train on a laptop in minutes, hard enough to show the difference between methods. Researchers now compare their strongest models on larger, harder image collections, but MNIST is still where most students begin.

How can you train your own digit reader on MNIST?

The quickest route is a free online notebook, such as Google Colab, with a library like TensorFlow or PyTorch, both of which can download MNIST in one line. A simple plan:

1. Load the 60,000 training and 10,000 test images and scale the pixel values to between 0 and 1.
2. Build a small network like LeNet: two convolution layers with pooling, then one or two fully connected layers and ten outputs.
3. Train for a few passes over the data and measure the error on the test set.
4. Look at the digits it gets wrong. Many will be ones you would hesitate over too.

A small network of this kind typically gets around 99 percent of the test digits right. Try removing the convolution layers and using only fully connected ones, and compare. You will see for yourself the advantage LeCun's team demonstrated in 1998.