What is a Hopfield network, and how can a network of simple units remember things?

Think of how a few notes of a song can bring back the whole tune, or how a blurry photo of a friend is still enough to recognize them. Your memory does not look things up by an address, the way a filing cabinet does. It works from content: a piece of the thing calls up the rest. In 1982 the physicist John Hopfield showed how a network of very simple artificial neurons could do something similar, and the design now carries his name.

His paper, "Neural networks and physical systems with emergent collective computational abilities," appeared in PNAS, the journal of the US National Academy of Sciences, volume 79, issue 8, pages 2554 to 2558, in April 1982. It was short and mathematical, written by a physicist rather than a computer scientist, and it appeared at a moment when neural network research had very few followers.

What Hopfield proposed

The paper's central claim is in its title. Useful computing abilities, Hopfield argued, can emerge as collective properties of a system with a great many simple, identical parts. No single part knows anything; the knowledge lives in the pattern of connections among them. He called the parts neurons and noted that the model, loosely based on the brain, could also be built as an electronic chip.

The ability he focused on is called content-addressable memory, also known as associative memory. You give the network part of a stored pattern, and it returns the whole pattern. The abstract also lists some ability to generalize, to recognize a familiar input, to sort inputs into categories and to correct errors, and it notes that the behavior hardly changes if some individual units fail. An ordinary computer, by contrast, can be stopped by one broken wire.

How the network works

Each unit in the original network has only two states, on or off (written 1 and 0, or +1 and -1 in many textbooks). Every unit is connected to every other unit, and each connection has a strength, or weight, that can be positive (encouraging the two units to agree) or negative (pushing them to disagree). Two rules keep things tidy: the connection from unit A to unit B has the same weight as the one from B to A, and no unit connects to itself.

The network runs one unit at a time, in no fixed order. A unit adds up the signals arriving from the others, each multiplied by its weight. If the total is above zero, the unit switches on; otherwise it switches off. Then another unit takes its turn, and so on, until nothing changes anymore.

Hopfield's key insight came from physics. He defined a single number for the whole network, which he called its energy, and showed that with symmetric weights every update either lowers that number or leaves it unchanged. The network therefore always comes to rest somewhere, the way a ball rolling across hilly ground ends up in a hollow. If you shape the ground so that each memory sits at the bottom of its own valley, then a clue that starts near a memory rolls down into it. The math is closely related to the Ising model, which physicists use to describe tiny magnets that tend to line up with their neighbors.

To carve those valleys, the usual method follows Donald Hebb's 1949 idea that cells active together strengthen their link: for each memory, raise the weight between any two units that share the same state and lower it between units that differ.

A worked example: storing one pattern

Take four units and store the pattern +1, -1, +1, -1. Using the Hebbian rule, each weight is the product of the two units' values in the pattern. Then feed the network a damaged clue, with unit 2 flipped by mistake: +1, +1, +1, -1.

The error is fixed, and the network has rebuilt the stored pattern from a corrupted copy.

Why it mattered

The paper arrived during a long lull. After the late 1960s, most AI funding and attention went to symbolic programs built from hand-written rules, and few researchers worked on neural networks. Hopfield showed that such networks could be analyzed with the well-tested tools of physics, which gave the field a solid mathematical footing and renewed interest. The idea was not entirely new: W. A. Little had described a similar network in 1974, and Hebb's learning rule dates from 1949. What Hopfield added was the energy picture that made the behavior predictable.

Work followed quickly. A 1984 paper extended the model to units with smoothly varying activity, closer to real neurons. In 1985 Hopfield and David Tank used these networks to search for short routes in the traveling salesman problem, the classic puzzle of visiting many cities by the shortest path. The Boltzmann machine, another influential design of the 1980s, is essentially a Hopfield network whose units switch on and off with a bit of randomness.

The network also has clear limits. It can settle into a false memory, a valley nobody meant to create. And it fills up fast: with the simple Hebbian rule, it reliably holds only about 138 patterns per 1,000 units.

How do Hopfield networks relate to the neural networks used today?

They share the basic material: simple units, weighted connections and memories stored in the weights rather than in any single place. Hopfield networks are still used as a model of how human memory might recall a whole from partial cues.

The architecture is different, though. A Hopfield network is recurrent: signals loop around until the whole network settles. Most networks in use in early 2020, such as the image classifiers behind photo search or the language models BERT and GPT-2, are arranged in layers that pass information mostly in one direction, and they learn by gradually adjusting millions of weights with backpropagation rather than by a single Hebbian step.

Hopfield's deeper legacy is a way of thinking: a network's behavior pictured as an energy landscape, with learning as the shaping of hills and valleys. When researchers talk about a model getting stuck in a local minimum, they are using that picture.