What was the 2014 attention mechanism, and how did it change machine translation?

When a human translator works on a long sentence, they do not memorize it, close the book and write the translation from memory. They glance back at the original, phrase by phrase, as they go. In 2014, a young researcher named Dzmitry Bahdanau and his supervisors gave neural networks the same habit. They called the result a way to "align and translate" at the same time; everyone else soon called it attention.

Attention sounds like a small fix to one translation system. It turned out to be one of the most useful ideas in modern AI. The language models making headlines this year, from Google's BERT to OpenAI's GPT-3, are built from layers of attention.

What happened, and who was involved

On 1 September 2014, the first version of a paper titled "Neural Machine Translation by Jointly Learning to Align and Translate" appeared on arXiv, the free online archive where researchers post papers. Its authors were Dzmitry Bahdanau, then at Jacobs University Bremen in Germany, and Kyunghyun Cho and Yoshua Bengio of the Université de Montréal in Canada. Bengio was already one of the best-known figures in deep learning; Cho had just co-written one of the first neural translation systems. The paper went through several revisions and was presented at the ICLR conference in 2015.

The timing matters. In 2014, a new approach called neural machine translation was emerging. Instead of the older "phrase-based" systems, which stitched together thousands of hand-tuned pieces, one large neural network learned to translate directly from examples. Most of these networks followed an encoder–decoder design: an encoder network reads the source sentence and squeezes it into a single list of numbers, a fixed-length vector; a decoder network then writes the translation from that vector alone.

Bahdanau and his co-authors suspected that this single vector was a bottleneck. A five-word sentence and a fifty-word sentence both had to fit into the same-sized summary. Earlier work had already shown that the quality of a basic encoder–decoder dropped quickly as sentences got longer.

Think of it as being asked to summarize a whole paragraph on one sticky note, then translate the paragraph using only the note. Short paragraphs survive. Long ones lose details.

How attention works

The fix was to keep the encoder's notes on every word instead of only one summary. Their model, which they named RNNsearch, read the sentence in both directions, left to right and right to left, and produced an annotation for each source word: a vector describing that word in the context of its neighbors.

Then, each time the decoder was about to write a word, it did a quick search. It scored every annotation for how relevant it was right now, turned those scores into weights that add up to 1, and blended the annotations using those weights. That blend, called a context vector, is different for every output word. Because the weights are soft percentages rather than a single hard choice, the network can spread its focus over several words when it needs to, and it learns all of this from examples, with no one telling it which words line up.

Here is a simplified picture of what the weights might look like while translating "the European Economic Area" into French:

The paper shows this very example: its model jumped over two words to link "Area" with zone, then worked backward through the adjectives. These weight charts were also a gift to researchers, because for once they could see what the network was looking at.

How well it worked

The team trained on a large English–French collection from the 2014 Workshop on Machine Translation, trimmed to 348 million words, and kept the 30,000 most common words in each language. Each model trained for about five days.

On the full test set, the attention model trained on sentences up to 50 words scored 26.75 on BLEU, a standard measure of how closely machine output matches human translations, compared with 17.82 for the same-size encoder–decoder without attention. Most strikingly, its quality held up on long sentences where the older design collapsed. On sentences with no unknown words, a version trained for longer scored 36.15, slightly ahead of Moses, a well-established phrase-based system, at 35.63.

Why it mattered

Attention spread quickly. In September 2016, Google described its Neural Machine Translation system (GNMT), a deep network that combines attention with stacked LSTM layers and "residual" shortcuts, and reported that it cut translation errors by about 60 percent compared with Google's older phrase-based system on simple test sentences.

Seen from June 2021, the larger story is that attention stopped being an add-on. Researchers began asking what would happen if a network used attention for everything.

How did attention lead to the Transformer?

In 2017, a team at Google answered that question with a paper titled "Attention Is All You Need." Their Transformer dropped the step-by-step recurrent networks entirely. Instead, every word attends to every other word in the same sentence at once, a version called self-attention. The Transformer paper explicitly lists the 2014 approach, under the name "additive attention," as one of the two main kinds it builds on.

Removing the step-by-step reading made training much faster on modern chips, because all the words can be processed in parallel. That speed is what made very large models practical. BERT (2018), GPT-2 (2019) and GPT-3 (2020) are all stacks of Transformer layers, and at their core each layer does what Bahdanau's decoder did: score the other words, turn the scores into weights, and blend.

So when you type a prompt into a GPT-3 app and it picks up on a detail from your first sentence while writing its fifth, you are watching a direct descendant of a translation trick from 2014. The full story of how the Transformer works has its own article in this guide.