Does a language model actually understand what it writes?

When GPT-3 writes a tidy paragraph about volcanoes, it is tempting to assume it knows something about volcanoes. The words are in the right order, the facts are often right, and the tone fits. But a language model is trained on one task only: predicting the next piece of text from the text before it. Whether that task can produce anything we should call understanding is one of the liveliest arguments in AI research right now.

The question became sharper this month with a paper presented at the ACM FAccT conference, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" by the linguist Emily M. Bender, the researcher Timnit Gebru and two co-authors, one writing under a pseudonym. Its title has already given the debate a name.

The case that models only handle form

The skeptical position rests on a distinction from linguistics. Every word has a form (the letters or sounds) and a meaning (what it refers to in the world, and what a speaker intends by it). People learn language by linking the two, while pointing, touching, eating and talking with others. A language model only ever sees the forms.

Bender and Alexander Koller made this case at the ACL conference last year with a thought experiment they call the octopus test. Two people stranded on separate islands chat over an old telegraph cable. A very clever octopus taps the line and, over time, learns to predict their messages with great accuracy. One day it cuts the cable and pretends to be one of them. For small talk it may pass. But when the other person describes a new gadget built from rope and coconuts, or begs for help fending off a bear, the octopus has nothing to go on. It never learned what "rope" or "bear" point to. Any meaning in its replies is supplied by the person reading them.

The new paper applies the same idea to today's large models. Their text, the authors write, is not grounded in an intention to communicate or in any model of the world or of the reader. We find it coherent because humans read meaning into language by habit. Their image for such a system is a "stochastic parrot": something that strings together pieces of language it has seen, guided by probabilities, without reference to what they mean. ("Stochastic" means governed by chance.)

The case that something more is going on

Others are struck by what large models can do without being trained for it. The GPT-3 paper, from OpenAI, reports that the model can do new tasks from a few examples in its prompt. In one test, researchers defined a made-up word, such as "Gigamuru," and asked GPT-3 to use it in a sentence. Across the examples they show, the model produced correct or at least plausible sentences, and once even invented a fitting past tense for a made-up verb. Because the words were invented for the test, the result is hard to explain as simple copying.

The GPT-3 authors are careful, though. They deliberately leave open whether the model truly learns a new task on the spot or simply recognizes patterns it saw during training. And they list clear gaps: the model struggles with "common sense physics," with questions such as whether cheese put in a fridge will melt, and they note that large language models are not grounded in other kinds of experience, such as video or physical interaction. On that point, at least, the two camps agree.

A worked test: change the surface, keep the logic

You can probe this debate yourself. The idea is to ask the same reasoning question twice: once in a familiar form the model has likely seen many times, and once with unusual details that keep the logic identical. Shallow pattern-matching tends to break on the second version.

Q: If I put an ice cube in a hot oven for an hour, what will happen to it?
A:

Q: If I put a small cube of frozen orange juice inside a running clothes dryer for an hour, what will happen to it?
A:

Then try a version where the familiar answer is wrong:

Q: If I put a block of cheese in a fridge overnight, will it melt?
A:

Score each answer on two things: is it correct, and does the explanation actually connect cause and effect? A model that answers the oven question well but stumbles on the dryer or the cheese is probably leaning on familiar wording rather than on any picture of heat.

Is predicting text the same as knowing something?

Not in the ordinary sense of "knowing." When you know that ice melts in an oven, that knowledge is tied to experiences, to other beliefs, and to things you can do, such as taking the ice out before it melts. A language model has none of those links. What it has is a vast, detailed record of how words tend to follow each other. Bender and Koller's point is that this record, however rich, is built from form alone.

Yet the record can contain a lot of information. Text written by people reflects what those people knew, so a model that predicts it well will often produce true statements. The difficulty is that it produces false ones in exactly the same confident voice, because sounding plausible is what it was trained for. That is why the stochastic parrots paper warns about people mistaking fluent output for understanding, and about trusting it too readily.

For a beginner, the practical position sits between the two camps. Treat a language model as a powerful pattern engine that often reflects real knowledge, not as a mind that holds it. Use it for drafting, rewording and brainstorming, and check any fact that matters against a source you trust.