What is a Transformer, and why does 'attention' matter?

If you have read about GPT-2 or BERT this year, you have met the word "Transformer." It is the name of a neural network design (a neural network is a program made of many adjustable numbers that learns patterns from examples). The design was introduced in June 2017 by a team of Google researchers in a paper with a memorable title: "Attention Is All You Need."

The key idea is in that title. Older language models read text one word at a time, carrying a running summary forward. A Transformer looks at all the words of a passage together and, for each word, works out how much attention to pay to every other word. That simple-sounding change made models both better at language and much faster to train.

The problem Transformers solved

Before 2017, the leading models for tasks like translation were recurrent neural networks. They process a sentence step by step: word one, then word two, and so on. That has two costs. First, the steps cannot run side by side, so training on large amounts of text is slow. Second, information from early words has to survive many steps to influence later ones, which makes long-distance links hard to learn.

The Transformer removes the step-by-step loop entirely. Every position in the sentence is handled at the same time, and attention connects any word directly to any other, however far apart they are.

What "attention" means, in plain words

Take this sentence: "The trophy didn't fit in the suitcase because it was too big." To understand "it," you need to connect it to "trophy," not "suitcase." Attention is the machinery that makes that connection.

The paper describes it with three ingredients for each word:

The model compares a word's query with every key. Strong matches get high weights, weak ones get low weights, and the word's new representation is a weighted blend of the values. The Transformer runs several of these attention calculations side by side, called "heads" (the original paper uses eight), so one head can track grammar while another tracks meaning.

There is one catch. Because all words are processed at once, the model has no built-in sense of order. The authors fix this by adding "positional encodings," number patterns that mark each word's position, to the word representations.

The original Transformer had two halves, each a stack of six layers: an encoder that reads the input sentence and a decoder that writes the output. On English-to-German and English-to-French translation benchmarks, it beat the best previous systems, and the large version trained in 3.5 days on eight graphics processors, a small fraction of the cost of earlier approaches.

Why this matters for beginners and builders

For anyone using language tools, the Transformer explains a lot about their behavior. Because attention links any two words directly, these models are good at keeping track of who did what across a paragraph. Because they train efficiently in parallel, researchers can feed them far more text than before, and more text has turned out to mean better results.

There is also a practical limit: a Transformer looks at a fixed-size window of text at a time. Anything outside that window is invisible to it. Keep this in mind when you try a model and it seems to "forget" the start of a long document.

How do Transformers power language models like GPT-2 and BERT?

Both models keep the Transformer's attention layers but use them differently. BERT, published by Google researchers in October 2018, uses only the encoder side. During training, some of the words in a sentence (15 percent) are hidden, and BERT learns to guess them using the words on both the left and the right. A second task teaches it to judge whether one sentence follows another. The large version has 340 million parameters. After this general training, BERT can be adapted to a specific job, like answering questions, by adding one small output layer and training briefly on examples. It set new records on eleven language tasks this way.

GPT-2, described by OpenAI in February, is built from Transformer blocks that read text left to right and are trained to predict the next word. Its largest version has 48 layers, about 1.5 billion parameters, and looks at up to 1,024 tokens (words or pieces of words) at once, double the window of the first GPT. Trained on about 40 GB of web text, it can continue a passage in a consistent style and even attempt tasks like summarizing or translating without any task-specific training.

A simple way to remember the difference: BERT is built to understand a passage by looking in both directions, which suits search and classification. GPT-2 is built to write, one word after another, which suits text generation. Both owe their power to attention.