What is John Searle's Chinese room argument, and does it show computers can't understand?

Suppose a computer answers your questions about a story as well as a person would. Does it understand the story? Alan Turing suggested in 1950 that if a machine's answers can't be told apart from a human's, we have good reason to call it intelligent. In 1980, the philosopher John Searle of the University of California, Berkeley, offered a short thought experiment meant to show that this is a mistake. It became known as the Chinese room, and four decades of argument have not settled it.

The argument appeared in Searle's paper "Minds, Brains, and Programs," published in the journal Behavioral and Brain Sciences in 1980. The journal printed it together with commentaries from 27 researchers and Searle's responses. He had already tried the argument out in talks at several universities, and he answered the most common objections in the paper itself.

The thought experiment

Searle asks you to imagine him locked in a room. He knows no Chinese at all; to him, Chinese characters are just shapes. People outside pass in sheets of Chinese writing. Inside, he has a large book of rules written in English, which tell him, purely by the shapes of the symbols, which Chinese symbols to send back out in response.

Unknown to him, the sheets are a story and questions about it, and the symbols he sends out are answers. The rulebook is so good that, to the people outside, his answers look exactly like those of a native Chinese speaker. In Searle's words: "Nobody just looking at my answers can tell that I don't speak a word of Chinese."

Yet he understands nothing. And that, Searle argues, is exactly a computer's situation. A computer follows a program that manipulates symbols by their form. The man in the room does the same. If he gains no understanding by running the program, neither does any computer, since the computer has nothing he lacks.

Here is a tiny version you can picture:

What Searle was arguing against

Searle drew a line between two views. Weak AI says computers are powerful tools for studying the mind, helping researchers state and test theories precisely. He had no quarrel with that. Strong AI says that a computer running the right program literally has a mind and understands, and that the program explains how human understanding works. That was his target.

His concrete example was the work of Roger Schank and colleagues at Yale. Their programs read short stories, such as one about a man who orders a hamburger, finds it burned and storms out without paying, and then answered questions like "Did the man eat the hamburger?" even though the story never says. Supporters claimed such programs understood the stories. Searle said the same argument applied to Terry Winograd's SHRDLU, Joseph Weizenbaum's ELIZA and any computer simulation of the mind.

His key distinction was between syntax, the form of symbols and rules for combining them, and semantics, what the symbols mean. A program, he argued, is defined purely by syntax, and syntax alone never adds up to meaning.

The main replies

Searle listed six objections and answered each. Three of them are still at the center of the debate:

• The systems reply (from Berkeley): the man doesn't understand Chinese, but the whole system, man plus rulebook plus paper, does. Searle answered that the man could memorize the rules and do it all in his head; he would then be the whole system and still understand nothing. This is the most common reply, and the debate over it has never died down.
• The robot reply (from Yale): put the program in a robot with cameras and arms, so its symbols connect to the world. Searle replied that the man would just receive more symbols, now from sensors, and still not know what they meant.
• The brain simulator reply: make the program simulate the firing of a Chinese speaker's neurons. Searle answered that simulating the formal structure of the brain still leaves out the brain's physical, causal powers.

Searle's own positive view is that minds come from biological processes. A machine could think, he said, but only one with causal powers equal to a brain's; a program alone is never enough.

Why it mattered

The Chinese room became perhaps the most discussed argument in cognitive science since the Turing test. The Stanford Encyclopedia of Philosophy notes that by the mid-1990s more than 100 articles had been written on it, and computer scientist Pat Hayes joked that cognitive science was the ongoing project of refuting it. It forced AI researchers to say clearly what they meant by "understanding," and it gave critics a simple, vivid way to question claims that a program grasps what it says.

Does the Chinese room apply to today's language models?

Many people think it fits them closely. A system such as OpenAI's GPT-2, which drew wide attention in 2019, writes fluent paragraphs by predicting likely next words from patterns in a huge amount of text. It has no direct link to the things the words refer to. Searle would say it is a very large rulebook, and fluency is not understanding.

Others answer with versions of the old replies. Supporters of the systems reply say understanding might belong to the whole trained network, not to any single step, just as no single neuron in your head understands English. Researchers working on models that connect text to images, sound or robot sensors are, in effect, pursuing the robot reply.

What the argument offers a beginner is a useful habit. When a program gives a good answer, ask two separate questions: is the answer correct, and does the program understand it? The Chinese room shows why those questions can come apart, and why testing behavior alone may not settle the second.