What is a language model, in plain words?

If you have typed a message on a smartphone, you have already met a small language model. As you write "See you", the keyboard offers "tomorrow", "soon" or "later" above the letters. It has no idea what you plan to do this weekend. It has simply learned which words tend to follow "See you" in a lot of text, and it shows you the most likely candidates.

A large language model does the same thing on a much bigger scale. Instead of a few suggestions for a short message, it holds a detailed sense of which words can follow almost any passage, from a recipe to a news story. OpenAI's GPT-2, presented a year ago this week, is the best-known example: it learned from about 40 GB of web text and has 1.5 billion adjustable values inside it.

What "model" means here

In everyday speech a model is a small copy of something, like a model train. In computing, a model is a set of numbers and rules that captures patterns in data so a program can make predictions. A language model captures patterns in language. Its whole job can be stated in one line: given the words so far, estimate how likely each possible next word is.

The GPT-2 paper describes this in the language of probability. A probability is a number between 0 and 1 (or 0% and 100%) that says how likely something is. The model breaks a sentence into a chain of small guesses, each one conditioned on everything that came before. "Conditioned on" just means "taking into account". The guess after "The cat sat on the" depends on all five of those words, not only the last one.

Training: learning from text alone

To learn those patterns, GPT-2 was trained on text gathered from about 8 million web pages. Training is a long process of practice: the model reads a passage, predicts each next word, compares its guess with the word that was really there, and adjusts its internal numbers slightly so the right word becomes more likely next time. Repeat that across billions of words and the numbers come to reflect a great deal about grammar, facts, styles and topics.

No one labeled this data by hand. The text itself provides the answers, because every next word is a free test question. That is why language models can learn from such large amounts of raw writing.

A note on "words": GPT-2 does not actually read whole words. It splits text into pieces called tokens, which may be a full word, part of a word or a punctuation mark. For a first understanding, thinking in words is close enough.

A worked example: one guess at a time

Suppose we give a model the start of a sentence:

Prompt

After a long day at work, I like to relax with a cup of

The model scores every token it knows. The numbers below are invented for illustration, not measured from GPT-2, but they show the shape of what comes back:

Next word	Likelihood
tea	41%
coffee	33%
hot	9%
cocoa	4%
soup	1%
(thousands of others)	the rest

Notice three things. The list is long, because every word in the vocabulary gets some score. The scores add up to 100%. And the top answers make sense because the model has seen countless sentences about relaxing with a drink.

Once a word is picked, say "tea", it is added to the text and the whole process runs again: "After a long day at work, I like to relax with a cup of tea" now becomes the input for the next guess. A paragraph is nothing more than this loop repeated, which is why OpenAI can show GPT-2 writing several pages from a one-sentence start.

Try it: Open a notes app on your phone, type "I am going to the", and tap the middle suggestion ten times in a row. Read the result aloud. You have just watched a tiny language model write by picking one likely word after another, and you will probably notice it drifting or repeating, a small version of the weaknesses large models show too.

What this explains, and what it doesn't

Seeing the model as a next-word guesser explains both its strengths and its odd failures. Because it learned from so much varied text, GPT-2 can continue a news story, a poem or a forum post in a fitting style. OpenAI's announcement describes it as chameleon-like: it picks up the tone and subject of whatever text it is given.

It also explains the mistakes. The model learned what sounds likely, not what is true. OpenAI lists repetition, sudden topic changes and "world modeling" errors among GPT-2's failures, such as writing about fires burning under water. It often takes several tries to get a good sample, and on technical or unusual subjects the results get weaker.

How does a model choose the next word?

Once the model has its list of likelihoods, a separate step decides which word to use. The simplest method, called greedy decoding, always takes the single most likely word. It is predictable, but it tends to get stuck in loops, repeating the same phrase.

The other approach is sampling: picking a word at random, but weighted by the scores, so "tea" is chosen more often than "soup" without "soup" being impossible. This gives livelier text, yet pure sampling sometimes reaches deep into the list for a strange word that throws the sentence off course.

GPT-2's samples use a middle path called top-k sampling. The model keeps only the k most likely words, throws away the rest and samples among those. The GPT-2 paper uses k = 40 in parts of its analysis. Each choice then becomes part of the input for the next guess, which is why the same opening line can lead to quite different stories each time you run it.