What happened at the first Loebner Prize, the first real Turing test contest?

In 1950, Alan Turing proposed a thought experiment: if a judge chatting by text cannot reliably tell a machine from a person, we might as well say the machine thinks. For four decades it stayed a thought experiment. Then, on a Friday afternoon in November 1991, someone actually ran it, with prize money, referees, a live audience and the press watching.

That event was the first Loebner Prize competition. It became an annual contest, and it is still the best-known attempt to turn Turing's idea into a real test.

Who organized it, and where

The money came from Hugh Loebner, a New York manufacturer of theater equipment, who launched the prize in 1990 with the Cambridge Center for Behavioral Studies in Massachusetts, a research center devoted to behaviorist psychology. Robert Epstein, the center's president emeritus, organized the first contest. A prize committee chaired by the philosopher Daniel Dennett ran it; members included Joseph Weizenbaum, who had written the famous ELIZA program in the 1960s, and the philosopher Willard Van Orman Quine.

The committee spent almost two years planning. The event took place on 8 November 1991 at the Computer Museum in Boston, where academics, businesspeople, reporters and passers-by filled two floors. Loebner set only two conditions: hold a contest every year, and award a prize every year. The first prize was a modest $1,500. Loebner reportedly set aside $100,000 for the first program to pass a full, unrestricted test.

How the contest worked

The committee knew that no program of the day could survive an open-ended Turing test, so it softened the rules in two ways:

1. One topic per terminal. Each hidden partner, human or machine, chose a single subject, and judges had to stay on it.
2. No tricks. Judges were told to chat naturally, as with another person, rather than set traps.

The judges themselves were chosen to be ordinary. They answered a newspaper job advertisement that asked for little beyond typing ability, and were screened to have no special computer science background. The humans hiding behind some terminals were called "confederates."

Scoring worked like this:

The line in step 3 did not affect the prize. It was there so the organizers could count how often a program was mistaken for a person.

What happened on the day

Things went wrong almost at once. The plan called for ten hidden partners, four of them human. But the rented IBM computers meant for the confederates would not talk to the judges' terminals. After almost two hours of last-minute repairs, the committee started with only two human confederates, and cut each conversation from about fifteen minutes to about seven so reporters could meet their deadlines. In a large auditorium, all the conversations were projected on screens while the writer A. K. Dewdney gave a running commentary.

Six programs competed, on topics fit for a cocktail party: burgundy wine, dry martinis, small talk, problems in relationships, second-grade school subjects, and "whimsical conversation." The two humans talked about Shakespeare and women's clothing.

As expected, the judges ranked the two humans as the most human-like overall. The best program was Joseph Weintraub's whimsical conversation program, which took the $1,500. The surprises came from the dividing lines. Five of the ten judges placed Weintraub's program on the human side, and across all judges there were eight cases of a program being called human. A program by Kenneth Colby, known for PARRY, an older program that imitated a paranoid patient, was judged human twice. Meanwhile, Cynthia Clay, the Shakespeare expert, was taken for a computer three times; one judge reasoned that "[no] human would have that amount of knowledge about Shakespeare."

Why it mattered

The contest made the Turing test concrete, and it drew a lot of attention. It also drew sharp criticism. Stuart Shieber of Harvard, who was a referee in the confederates' room, argued in a 1994 paper that limiting the topic and banning tricky questions removed exactly what makes Turing's test hard. Marvin Minsky called the prize a publicity stunt and jokingly offered a reward to anyone who could stop it.

The rules changed over time. Since 1995, conversations have been unrestricted. Chatbots built from large sets of hand-written patterns, such as Richard Wallace's A.L.I.C.E. and Steve Worswick's Mitsuku, became regular winners; Mitsuku has won five times. In 2019 the format changed again: at Swansea University, members of the public, including schoolchildren, chatted with seventeen bots while knowing they were bots, and no human competitors took part.

Meanwhile, the newest AI language models are trained on huge amounts of text rather than built from scripted replies, which makes the old question of how to judge a conversation partner feel fresh again.

Can a chatbot win the Loebner Prize without understanding anything?

Yes, and the first contest showed why. In one exchange from the transcripts, Weintraub's winning program replied with a remark about the weather followed by the judge's own sentence, pronouns swapped. Under the label "whimsical conversation," a reply that made no sense looked like playfulness rather than failure. Shieber pointed out that whimsy is a style, not a topic, and that it is almost impossible to catch out.

Short time limits, untrained judges and a ban on hard questions all favor tricks over understanding. A program does not need to know anything about wine to deflect, change the subject or flatter. Critics of the prize argue that this rewards deception, which is the opposite of what researchers want to build.

That does not make the contest useless. It is a good measure of how easily people read a mind into text on a screen, a lesson Weizenbaum had already drawn from ELIZA. But passing it, in its restricted form, says more about the judges than about the machine.