What is sequence-to-sequence learning, and why was the 2014 paper so important?

Many useful tasks come down to turning one string of items into another: an English sentence into a French one, a recording into text, a long article into a short summary, a question into an answer. The trouble is that the input and output rarely have the same length, and the words do not line up one to one. Before 2014, deep neural networks were very good at fixed-size problems, like labeling a photo, but had no clean way to handle this kind of job.

"Sequence to Sequence Learning with Neural Networks," often shortened to seq2seq, offered one. Its recipe is almost disarmingly simple: one network reads, another writes.

What happened, and who was involved

The paper's first version went up on arXiv, the open archive for research papers, on 10 September 2014. Its three authors, Ilya Sutskever, Oriol Vinyals and Quoc V. Le, all worked at Google. Sutskever is a familiar name for anyone following GPT-3: in December 2015 he became research director of the newly founded OpenAI.

The test they chose was hard: translating English into French using a public benchmark from the 2014 Workshop on Machine Translation (WMT'14). Statistical "phrase-based" systems, built from many separately tuned parts, had dominated this task for years.

How it works: an encoder and a decoder

The model uses two LSTM networks. An LSTM, or Long Short-Term Memory network, is a kind of recurrent neural network: it reads one word at a time and keeps a running internal memory, deciding at each step what to keep and what to forget.

1. The encoder reads the English sentence word by word. When it reaches the end, its memory holds a single list of numbers, a vector, that stands for the whole sentence.
2. The decoder starts from that vector and writes the French sentence one word at a time. Each new word depends on the vector and on the words it has already written.
3. A special end-of-sentence symbol tells the decoder when to stop, so the output can be shorter or longer than the input.

A good analogy is a relay between two interpreters who share no notes. The first listens to the whole sentence and forms an idea of its meaning; the second hears only that idea and expresses it in another language.

The networks were large for 2014: four layers of 1,000 memory cells each, with an input vocabulary of 160,000 English words and an output vocabulary of 80,000 French words, for 384 million adjustable numbers (parameters) in all. Training used 12 million sentence pairs, ran on a machine with eight graphics processors, and took about ten days.

The surprising trick, and the results

The paper's most memorable finding was a simple trick. The authors fed the English sentences in reverse order while keeping the French in normal order. Here is the idea with a tiny example:

In the reversed version, the first English words ("the cat") are now the last thing the encoder reads, right before the decoder writes "le chat." The distance between matching words shrinks at the start of the sentence, which gives the training process an easier first foothold. Reversal raised the score of a single model from 25.9 to 30.6 BLEU, a standard measure of overlap with human translations, and the authors admitted they were surprised by how much it helped.

An ensemble (a team of five networks voting together) scored 34.8 BLEU on the full test set. A strong phrase-based system scored 33.3 on the same data. It was the first time a purely neural system translating directly had beaten that kind of baseline on a large task. When the networks were used to re-rank the statistical system's top 1,000 candidate translations, the score rose to 36.5, close to the best published result.

The networks also handled long sentences better than the authors expected, and the sentence vectors they learned grouped phrases by meaning: sentences with the same meaning in active and passive voice landed close together.

Why it mattered

Two papers from 2014 are usually credited with starting seq2seq: this one, and a closely related encoder–decoder paper from Kyunghyun Cho and colleagues. Their shared weakness was the single fixed-size vector, which has to carry everything about a sentence. Nine days before this paper appeared, Dzmitry Bahdanau, Cho and Yoshua Bengio had posted their fix: attention, which lets the decoder look back at every input word.

The combination changed real products. In September 2016, Google described its Neural Machine Translation system, a deep LSTM encoder–decoder with eight layers on each side, attention and residual shortcuts, and reported about 60 percent fewer errors than its old phrase-based system in side-by-side human ratings of simple sentences. Google Translate moved to that neural system in 2016.

Where is sequence-to-sequence learning used today?

The encoder–decoder idea is now everywhere a computer turns one sequence into another: translation, speech recognition, image captioning (where the "input sequence" is a picture), text summarization and chatbots. The specific parts have changed. Since 2017, the recurrent LSTMs have largely been replaced by Transformers, which read all the words at once and rely on attention throughout.

You can still see the original shape in today's models. Many translation systems keep a separate encoder and decoder. OpenAI's GPT-3, released last year, keeps only the decoder half: it reads your prompt and then writes one word at a time, each word depending on everything before it, exactly as the 2014 decoder did.

If you want to feel the idea directly, try a translation app on a long sentence, then split the same sentence into short ones. Modern systems handle the long version far better than 2014's networks did, largely thanks to attention, but the underlying job is unchanged: read a sequence, then write a new one.