What was GPT-1, and what did "generative pre-training" change?

On 11 June 2018, OpenAI published a paper with a modest title: "Improving Language Understanding by Generative Pre-Training." Its authors, Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, did not give their model a catchy name. The approach became known as GPT, short for generative pre-training, and now that later versions carry numbers, this first model is often called GPT-1.

The idea was simple to state. Let a model read a large pile of ordinary text and learn to predict what comes next. Then take that same model and train it a little on a specific task, such as answering questions. OpenAI showed that this two-step recipe worked better than building a new, custom system for every task.

The problem it tried to solve

Most successful machine learning in 2018 relied on supervised learning: training on examples that people have labeled by hand, such as sentences tagged as positive or negative. Labels are slow and expensive to produce, so most language datasets were small.

Unlabeled text, by contrast, is everywhere. Researchers had already used it to learn word vectors, lists of numbers that capture what each word tends to mean, through methods like word2vec. But a word vector gives each word one fixed meaning, and using unlabeled text to learn anything deeper was hard. Other groups were working on this too: ELMo, ULMFiT and an earlier method called Semi-supervised Sequence Learning all pre-trained on raw text. OpenAI's contribution was to combine pre-training with the Transformer, the attention-based network design Google researchers had introduced in 2017, and to keep the model almost unchanged from one task to the next.

How generative pre-training works

The method has two stages.

Stage 1, pre-training. The model reads BooksCorpus, a collection of over 7,000 unpublished books in genres such as adventure, fantasy and romance. Books were chosen because they contain long, continuous passages, which teach a model to keep track of ideas across many sentences. The training task is language modeling: predict the next token (a word or piece of a word) from the ones before it. This is the "generative" part, because a model that predicts the next word can also generate text.

Stage 2, fine-tuning. The pre-trained model is then trained briefly on a small labeled dataset for one task. Three passes over the data were enough for most tasks.

The clever part is how different tasks fit one model. Rather than adding new machinery for each task, the researchers rearranged the input into a single sequence of text. Here is how that looks:

The model itself was a 12-layer Transformer that only looks backward at earlier words, with 12 attention heads (parts of the network that each focus on different relationships between words). OpenAI's announcement said pre-training took a month on eight graphics processors, a large effort for a language project in 2018 but one that only had to be done once.

What the results showed

The same core model, with very little tuning, improved the best published results on 9 of the 12 tasks studied. Its biggest gains came on tests of commonsense reasoning and reading comprehension, which are thought to need knowledge of the world and reasoning across several sentences. On the Story Cloze test, where a model picks the right ending to a short story, it gained 8.9 percentage points. It did not win everywhere: on a small entailment dataset called RTE, it scored 56% against the previous best of 61.7%.

OpenAI also noticed something intriguing. Even without fine-tuning, the pre-trained model could attempt some tasks. For sentiment, the team added the word "very" after a movie review and checked whether the model found "positive" or "negative" more likely as the next word. That trick reached about 80% accuracy. This kind of test, on a task the model was never trained for, is called zero-shot.

The paper was candid about drawbacks: heavy computing needs, the limits and biases of learning about the world only from text, and models that still break in surprising ways when tested adversarially. OpenAI released its code and the trained model.

Why it mattered

GPT-1 helped establish the pattern that now dominates language AI: pre-train one large model on unlabeled text, then adapt it. Google's BERT, published in October 2018, followed the same broad recipe with a model that reads in both directions, and Google's own announcement named generative pre-training among the work it built on. OpenAI's own conclusion pointed the way forward. Better language modeling went hand in hand with better task results, and the team had used just one eight-GPU machine and about 5 GB of books. More data and more computing power looked like an obvious next step.

How did GPT-1 lead to GPT-2 and GPT-3?

OpenAI took that next step in February 2019 with GPT-2, essentially the same design made much larger: 1.5 billion parameters (the adjustable numbers inside the model), trained on about 40 GB of web pages linked from Reddit. It wrote fluent paragraphs and set the best scores on 7 of 8 language tests it attempted with no fine-tuning at all. Worried about misuse, OpenAI released it in stages, publishing the full model in November 2019.

GPT-3, described in May 2020, scaled the idea again, to 175 billion parameters. It improved on GPT-2's test results by a wide margin. Rather than releasing the full model, OpenAI offered access through a cloud service, starting with a free private beta in June 2020.

Across all three, the core idea stayed the same as in June 2018: teach a Transformer to predict the next word on a large amount of text. What changed was scale, and with scale, less need for task-specific fine-tuning.