What is a large language model, and why does size matter?

A language model is a program that has learned to guess the next word in a piece of text. Feed it "The cat sat on the" and it gives a high score to "mat" or "floor" and a low one to "elephant." It learns those scores by reading text and adjusting itself every time it guesses wrong. A large language model does the same thing, just at an enormous scale: more internal settings, more text to learn from, and much more computing time.

The word "large" has become a working label for models like OpenAI's GPT-2 and GPT-3. It is not a fixed threshold. What counts as large keeps moving, because each new model tends to be many times bigger than the last.

Three ways to measure "large"

Researchers usually describe size in three ways.

• Parameters. These are the numbers inside the network that get adjusted during training. You can picture them as millions or billions of tiny dials. Together they store everything the model has learned about which words follow which.
• Training data. This is how much text the model reads, often counted in tokens. A token is a word or part of a word.
• Compute. This is the total amount of calculation used for training. It grows with both the number of parameters and the amount of data.

Here is how GPT-2 and GPT-3 compare, using the figures in their papers:

A petaflop/s-day is the work done by a computer performing a thousand trillion operations per second for a full day. The jump from tens to thousands of them shows that GPT-3 is not just a little bigger. It belongs to a different scale altogether.

What extra size buys

Bigger models do better on the basic job of predicting text, but the more striking change shows up in what they can do without retraining. The GPT-3 paper tested eight model sizes, from 125 million to 175 billion parameters, on dozens of tasks. Across many of them, the gap between giving the model no examples and giving it a few examples widened as the models grew. In other words, the largest model made the best use of the examples placed in its prompt. The authors describe this as larger models making more efficient use of information in their context.

Size also has costs. A model with 175 billion parameters is too big for most organizations to run, which is one reason OpenAI offers GPT-3 only through its API. Training uses a great deal of energy. And more data from the web means more of the web's errors and stereotypes.

A worked example: reading a model card for size

When you meet a new language model, you can place it on the size scale with three questions. Try them on GPT-2's smallest version, which has 117 million parameters.

The middle row is the interesting one. GPT-2's four sizes all trained on the same WebText data, and the larger versions did better on the tasks the paper tested. Same text, more parameters, better results. That pattern is exactly what researchers set out to measure.

What are scaling laws?

A scaling law is a simple rule that predicts how well a model will perform from how big it is. In January 2020, a team at OpenAI led by Jared Kaplan and Sam McCandlish published Scaling Laws for Neural Language Models, which measured this carefully across many Transformer models. They found that the model's error on test text falls as a power law in each of the three size measures: parameters, data and compute. A power law means that every time you multiply the size by the same factor, the error drops by a steady proportion. Plotted on a log scale, the trend becomes a straight line, and it held across more than six orders of magnitude.

The paper reports a few practical findings. The shape of the network, such as how deep or how wide it is, matters far less than its overall size. Growing only the model or only the data runs into diminishing returns, so the two need to grow together; by the authors' estimate, an eightfold bigger model needs only about five times more data. Bigger models also learn more from each example. With a fixed budget, the most efficient choice is to train a very large model and stop before it has fully finished learning.

These findings shaped GPT-3 itself. Its paper says it used the scaling-laws analysis to train much larger models on fewer tokens than was common. Scaling laws do not promise that bigger is always better at everything, and the authors note the trend must level off eventually. But for now they give researchers something rare: a way to predict, before spending months of computing time, roughly how much a larger model will improve.