What is long short-term memory (LSTM), and how does it help neural networks remember?

Long short-term memory (LSTM) is a design for recurrent neural networks, published by Sepp Hochreiter and Jürgen Schmidhuber in the journal Neural Computation in November 1997, that lets a network hold on to information across hundreds or thousands of steps. Each LSTM memory cell keeps a stored value that changes only when learned "gates" allow it, which solves the vanishing-error problem that kept earlier recurrent networks from learning long-range patterns.

What is long short-term memory (LSTM), and how does it help neural…

Read the sentence "The cat that the neighbors' children kept feeding all summer was finally adopted." To know who was adopted, you must remember "the cat" across a dozen words. People do this without effort. For neural networks, holding a fact over a long stretch of input was, for years, one of the hardest problems in the field.

Long short-term memory, or LSTM, was the fix that worked. Sepp Hochreiter, then at the Technische Universität München, and Jürgen Schmidhuber, at the Swiss AI lab IDSIA in Lugano, described it in a long paper in Neural Computation, volume 9, issue 8, dated 15 November 1997. The journal had received it in August 1995. An early version had appeared as a 1995 technical report and at the NIPS conference in 1996.

The problem: memories that fade

A recurrent neural network (RNN) reads a sequence one item at a time, such as words in a sentence or sound samples in speech, and passes a running summary forward from each step to the next. In principle, that summary can carry information as far as needed.

In practice, training failed. Networks learn by backpropagation: they compare their output with the right answer and send the error backward to adjust their weights. In a recurrent network, the error has to travel backward through every time step. At each step it is multiplied again, so it tends either to blow up or to shrink toward zero. Hochreiter had analyzed this in his 1991 diploma thesis, supervised by Schmidhuber. When the error vanishes, the network simply cannot learn that something it saw 100 steps ago matters now.

How LSTM works

Hochreiter and Schmidhuber's answer was a unit that, by design, passes its error backward without shrinking or growing. They called it the constant error carousel: a simple unit connected to itself with a fixed weight of one, so whatever it stores stays put from step to step.

A carousel alone would be useless, because every new input would disturb it. So they wrapped it in a memory cell with two gates. A gate is a small unit whose output, between 0 and 1, multiplies a signal, acting like a valve:

The input gate decides when new information may be written into the cell.
The output gate decides when the cell's contents may affect the rest of the network.

The network learns when to open and close each gate, just as it learns its other weights. Here is a small worked example of a single cell in a task where the network must remember the first symbol of a long sequence:

Step	Input	Input gate	Stored value	Output gate
1	"X" (the key symbol)	open	X	closed
2 to 999	random noise	closed	X	closed
1,000	"?" (time to answer)	closed	X	open: answer "X"

Try it: Write a list of 30 random letters, but make the first one a capital. Ask a friend to read the list aloud and then tell you the first letter. Now ask them to read it again while holding a coin they may only set down for the letter worth keeping. The coin works like the input gate: it decides what gets stored.

What the paper showed

The paper compared LSTM with the main recurrent methods of the day, including backpropagation through time, real-time recurrent learning and Elman networks, on a series of artificial tasks: learning a tricky grammar, remembering symbols across long gaps of noise, adding or multiplying two numbers marked somewhere in a long sequence, and classifying sequences by the order of widely separated symbols. LSTM learned to bridge gaps of more than 1,000 time steps, solved tasks no earlier recurrent method had solved, and learned much faster. The authors planned to try it next on real data: time series, music and speech.

What followed

The design kept evolving. In 1999, Felix Gers, Schmidhuber and Fred Cummins added a third gate, the forget gate, which lets a cell clear its memory when it is no longer needed. Most LSTMs used today include it.

Then came the results. In 2009, an LSTM-based system led by Alex Graves won an international competition for connected handwriting recognition, the first time a recurrent network had won such a contest. In 2013, a team of Graves, Abdel-rahman Mohamed and Geoffrey Hinton built LSTMs into a system that set a record on a classic speech benchmark called TIMIT. LSTMs became a standard tool for speech recognition, machine translation, time series forecasting and robot control.

Why are Transformers replacing LSTMs in language models?

An LSTM reads one word after another, and each step depends on the one before. That makes it hard to spread training across many processors at once, and it still has to squeeze everything worth remembering into its cells as it goes.

The Transformer, introduced by Google researchers in 2017, drops the step-by-step reading. Its attention mechanism lets every word look directly at every other word in a passage at the same time. That suits modern hardware, which is built to do many calculations in parallel, and it makes long-range links as easy as short ones. Large language models such as BERT and GPT-3 are built on Transformers.

LSTMs have not disappeared, though. They remain a natural fit for streaming data such as sensor readings or live audio, where information arrives one step at a time anyway. And the core LSTM idea, using learned gates to control what flows through a network, shows up in many newer designs.