# Did the chatbot Eugene Goostman really pass the Turing test in 2014?

Date: 2021-05-12 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/did-eugene-goostman-pass-the-turing-test/

**Answer:** On 7 June 2014, at an event at the Royal Society in London, the chatbot Eugene Goostman, posing as a 13-year-old Ukrainian boy, convinced 10 of 30 judges (33 percent) that it was human in five-minute text chats, and the University of Reading declared that the Turing test had been passed for the first time. Most researchers rejected that claim, because the 30 percent bar comes from a prediction Turing made rather than a rule he set, and because the bot relied on a persona and evasive jokes rather than understanding.

On Sunday, 8 June 2014, the University of Reading sent out a press release with a dramatic headline: the Turing test had been passed for the first time. A computer program called Eugene Goostman had fooled a third of the judges at a contest held the day before at the Royal Society in London. News sites around the world repeated the claim.

Within days, it was also being picked apart. Many AI researchers argued that nothing important had happened, and that the announcement said more about publicity than about machine intelligence. The episode is worth knowing in detail, because it shows how easy it is to declare a milestone and how hard it is to define one.

## What happened at the Royal Society

The event, called Turing Test 2014, was organized by Kevin Warwick, who held a visiting professorship at Reading and a senior research post at Coventry University, together with Huma Shah. It took place on Saturday, 7 June 2014, chosen because it was the 60th anniversary of Alan Turing's death. Turing had received a posthumous royal pardon about six months earlier.

Five chatbots competed. According to the university, 30 judges took part, among them the actor Robert Llewellyn, who played the robot Kryten in the comedy series *Red Dwarf*, and Lord Sharkey, who had campaigned for Turing's pardon. Each judge ran five tests, and in each one they typed with two hidden partners at the same time for five minutes, one human and one machine, then decided which was which. That made 300 conversations, with every machine taking part in 30 tests. John Barnden of the University of Birmingham checked the results independently.

Eugene Goostman was judged human in 33 percent of its tests, about 10 of 30. Warwick's team had set the bar at 30 percent, so they announced a pass. Warwick stressed that the conversations were unrestricted, with no preset topics, which he said made this the first true Turing test ever passed.

## Who, or what, was Eugene?

Eugene Goostman was not new and not a supercomputer, despite early headlines; the university later corrected its release to call it a computer program. It had been developed since 2001 in Saint Petersburg, Russia, by Vladimir Veselov, Eugene Demchenko and Sergey Ulasen, and had placed second in the Loebner Prize, another Turing-style contest, in 2005 and 2008. In June 2012, at Bletchley Park, it had already persuaded 29 percent of judges.

The bot's character is Eugene, aged 13, who lives in Odessa in Ukraine and keeps a guinea pig. That choice is a clever piece of design. Veselov explained that a boy that age can claim to know things but is easily forgiven for not knowing others, and a non-native speaker is forgiven for odd grammar.

Here is how Eugene behaved with the computer scientist Scott Aaronson, who published one of his chats:

| Aaronson asked | Eugene's strategy |
|---|---|
| Is a shoebox or Mount Everest bigger? | Dodged, then changed the subject to where Aaronson was from |
| How many legs does a camel have? | Guessed vaguely ("between 2 and 4"), then asked about Aaronson's job |
| How many legs does an ant have? | Repeated the same vague answer word for word, with a smiley |

> **Try it:** Write five questions you would ask to unmask a chatbot in five minutes. Favor questions that need common sense rather than facts, such as "If I put my shoe in the fridge, will it get cold?" Then test them on any chatbot you can find.

## Why experts rejected the claim

Critics raised several objections. The first is about the 30 percent figure. In his 1950 paper, Turing predicted that in about fifty years an average interrogator would have no more than a 70 percent chance of guessing correctly after five minutes. That was a forecast about where computers might be by 2000, not a pass mark, and commentators argued that Warwick had turned it into one.

The second is about method. Gary Marcus, a cognitive scientist, wrote that the bot's success came from constant misdirection and deflection, which an untrained judge might take for wit. Murray Shanahan of Imperial College London called the claim "completely misplaced," saying it made science-fiction AI seem close when it is not. Mike Masnick of Techdirt mocked the "supercomputer" label.

The third is about history. Similar "passes" had been claimed before, by PC Therapist in 1991 and by a modified Cleverbot at the Techniche festival in 2011. Cleverbot's creator, Rollo Carpenter, argued that such tests show only imitation of intelligence. The test itself, how Turing framed it and why it is still debated, has its own article: "What is the Turing test, and does it still matter?"

## What would a convincing Turing test look like?

Most researchers agree that a serious version would need longer conversations, trained and skeptical judges, and no persona that excuses ignorance. Five minutes is very little time, and a judge who does not know how chatbots work can be charmed by jokes. A judge who probes common sense, memory of earlier answers and reasoning step by step is much harder to fool.

It would also need to be repeated, with many sessions and clear rules agreed in advance, rather than one event declared a success by its own organizers. Marcus has suggested a different challenge altogether: a system that can watch a television program and answer questions about what happened in it.

The question has new weight in 2021. Large language models such as GPT-3 write fluent paragraphs on almost any topic, far beyond what Eugene could do. Yet when the French health startup Nabla tested GPT-3 as a medical chatbot, against OpenAI's own advice, it gave a simulated patient in distress advice that could have caused serious harm. Fluency alone is not the same as understanding, which is exactly the lesson the Eugene Goostman affair taught.

## Sources

- Turing Test success marks milestone in computing history: https://web.archive.org/web/2014/http://www.reading.ac.uk/news-and-events/releases/PR583836.aspx
- Computing Machinery and Intelligence: https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf
- Eugene Goostman: https://web.archive.org/web/20190315000000/https://en.wikipedia.org/wiki/Eugene_Goostman