# What is the Turing test, and does it still matter?

Date: 2019-03-18 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-the-turing-test/

**Answer:** The Turing test is a typed conversation game proposed by Alan Turing in 1950: if a judge chatting with a hidden machine and a hidden person cannot reliably tell which is which, the machine is said to play the game well. It still matters as a sharp way to think about machine intelligence, but most researchers see it as a test of convincing conversation, not of understanding.

In 1950 the British mathematician Alan Turing published a paper in the philosophy journal *Mind* called "Computing Machinery and Intelligence." Instead of arguing about what "thinking" means, he proposed a game. A judge sits in one room and exchanges typed messages with two hidden players. One is a person, the other is a machine. If the judge cannot tell them apart better than chance, the machine has done something interesting.

That game, now called the Turing test, is close to seventy years old and still comes up whenever a chatbot makes the news. It is worth understanding exactly what Turing proposed, because the popular version is often a little different from the original.

## How the imitation game works

Turing begins with a party game. A man and a woman sit in separate rooms, and an interrogator, who cannot see them, tries to work out who is who by asking questions. Answers are typed, ideally sent by teleprinter, so voice and handwriting give nothing away. Then Turing asks: what happens if a machine takes one of the places? Will the interrogator be wrong as often as before?

In the version most people describe today, the setup is simpler:

| Role | What they do |
|---|---|
| Judge (interrogator) | Types questions and decides which partner is the machine |
| Hidden human | Answers honestly, trying to be recognized as human |
| Hidden machine | Answers in a way meant to pass as human |

Turing also made a prediction. He believed that in about fifty years, computers with a storage capacity of about 10^9 (a billion units of memory) could play so well that an average interrogator would have no better than a 70 percent chance of a correct call after five minutes of questions. In other words, the machine would fool judges about 30 percent of the time.

## Why Turing framed it this way

Turing called the original question, "Can machines think?", too vague to settle. A behavioral test gives you something you can actually run. You do not need to agree on a definition of mind; you only need to watch what happens in the conversation.

He also spent much of the paper answering objections. One he named after Ada Lovelace, who had written in 1843 that the machine could only follow our orders. Turing's reply was that machines might learn, and he suggested building a "child machine" and educating it.

## The main criticisms

The test has been influential and widely criticized. In 1980 the philosopher John Searle described his "Chinese room": a person following a rule book could produce fluent Chinese answers without understanding a word. Searle argued that passing the test would show good symbol handling, not understanding.

Practical contests raised other doubts. The Loebner Prize, funded by Hugh Loebner, has run yearly Turing-style competitions since November 1991. The first winner was a simple program that fooled inexperienced judges, partly by imitating human typing mistakes. That points to a real weakness: the test rewards seeming human, which can include deliberate errors, jokes and changing the subject.

You also meet a reversed version of the test almost every day. A CAPTCHA, the distorted text you type to prove you are not a bot, is a computer trying to tell humans from machines.

> **Try it:** Write five questions you would ask to unmask a chatbot. Mix one fact question, one about something you can see right now, one about feelings, one with a trick of logic, and one that refers back to an earlier answer. Notice which kinds would also trip up a tired human.

## Does it still matter?

As a research goal, the test has lost much of its pull. Most AI researchers today measure systems on specific tasks: translating, answering questions, recognizing images. Fooling a judge in small talk says little about whether a system can do useful work reliably.

As a teaching tool, it is still excellent. It forces you to separate two things that are easy to blur: behavior that looks intelligent, and whatever is going on inside. Every chatbot you try is a small Turing test, and it helps to know you are the judge.

## Could a chatbot pass the Turing test today?

It depends on the rules, and that is exactly the problem. In June 2014, at a Royal Society event run by Kevin Warwick, a researcher at the University of Reading, a chatbot called Eugene Goostman convinced 33 percent of judges, after five-minute chats, that it was human. Warwick called it a pass because it crossed Turing's 30 percent mark. Critics disagreed. Goostman poses as a Ukrainian boy of 13, a character chosen so that odd grammar and gaps in knowledge seem natural, and it often dodges questions with humor. Other chatbots had made similar "passes" before, including Cleverbot at the Techniche festival in 2011.

So with short chats, inexperienced judges and a persona that excuses mistakes, a chatbot can fool a fair share of people. With longer conversations, expert judges and no special excuses, no program has clearly passed.

A newer factor is arriving. Last month OpenAI described GPT-2, a language model whose short samples are often fluent enough to read as human writing, though OpenAI notes it can repeat itself and lose track of the world. Fluent text is not the same as holding up your end of a long, probing conversation. Still, many researchers expect the gap between "sounds human" and "is useful" to become the central question in the years ahead, and the Turing test is a good place to start thinking about it.

## Sources

- Computing Machinery and Intelligence: https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf
- Turing test: https://web.archive.org/web/20190315000000/https://en.wikipedia.org/wiki/Turing_test
- Eugene Goostman: https://web.archive.org/web/20190315000000/https://en.wikipedia.org/wiki/Eugene_Goostman