How did OpenAI get from GPT-1 to GPT-4?

GPT stands for generative pre-trained transformer. "Generative" means the model produces text. "Pre-trained" means it first learns from a large body of ordinary text by predicting the next word, again and again, before it is used for anything specific. "Transformer" is the type of neural network, a design built around attention, which lets the model weigh every earlier word when choosing the next one.

Every model from GPT-1 to GPT-4 rests on that same foundation. The story of the series is what happened when OpenAI made the models bigger, fed them more text and changed how people ask them for things.

Four generations at a glance

Parameters are the adjustable numbers inside the network that change during training. The GPT-2 paper notes that its smallest model has the same size as the original GPT, which is where the 117 million figure comes from. Between GPT-3 and GPT-4 came GPT-3.5, the model OpenAI uses as its main point of comparison in the GPT-4 report.

One task, four generations

A helpful way to see the change is to follow one job through the series: translating a sentence from English into French. The examples below are illustrative, built from the methods each paper describes.

GPT-1. You could not simply ask. The 2018 approach was to take the pre-trained model and fine-tune it, meaning train it a little more on a labeled dataset made for one task. For translation, you would need a labeled set of English-French pairs and a training run before getting any answer.

GPT-2. The 2019 paper tried something new: no extra training, just a cleverly shaped input. The researchers gave the model example pairs in the form "english sentence = french sentence", then an English sentence followed by "=", and let it continue:

The cat is sleeping. = Le chat dort.
I like apples. = J'aime les pommes.
The library opens at nine. =

It worked only a little. On a standard English-to-French test, GPT-2 scored slightly below a word-for-word dictionary substitution. Still, the model had picked up some translation without being trained for it.

GPT-3. The 2020 paper turned that trick into the main method. You write a short instruction and a handful of examples, called few-shot prompting, and the model infers the task:

Translate English to French.

English: The cat is sleeping.
French: Le chat dort.

English: I like apples.
French: J'aime les pommes.

English: The library opens at nine.
French:

The paper reports that few-shot GPT-3 beat earlier unsupervised translation systems when translating into English, but did less well in the other direction.

GPT-4. By 2023 you can ask the way you would ask a person: "Translate 'The library opens at nine' into French, and give a more formal version too." The model has been fine-tuned with reinforcement learning from human feedback (RLHF), in which people rate answers and the model learns to prefer the ones they rate highly. GPT-4 can also take an image as input, so in principle the sentence could arrive as a photo of a sign.

What changed at each step?

From GPT-1 to GPT-2, the change was scale and data. The model grew more than tenfold, and it learned from WebText, a new collection of millions of web pages that Reddit users had linked to. The surprise was that a bigger model trained only to predict text began to answer questions, summarize and translate without any task training. The paper even got summaries by adding "TL;DR:" after an article, borrowing an internet habit.

From GPT-2 to GPT-3, scale jumped again, to 175 billion parameters trained on about 300 billion tokens. The new idea was in-context learning: show the task in the prompt and the model adapts on the spot, with no change to its weights. That made one model useful for thousands of tasks and gave rise to prompting as a skill.

From GPT-3 to GPT-4, OpenAI stopped publishing the size, the training data and the hardware, citing competition and safety. What it did report was better reasoning and exam results, image input, and training with human feedback so that the model follows plain requests and declines harmful ones more often. The recipe is recognizably the same as in 2018; what changed most is how easy it has become to ask for what you want.