How did IBM's Watson beat the best human players at Jeopardy!?

Jeopardy! is a game of puns, hints and half-hidden meanings. A clue might read "Its largest airport was named for a World War II hero; its second largest, for a World War II battle," and the contestant must respond, in the form of a question, "What is Chicago?" For a computer in the 2000s, that was a far harder problem than chess. Words are slippery, and the show rewards speed as much as knowledge.

In February 2011, IBM showed that a machine could play anyway. Its computer, called Watson after the company's first chief executive, Thomas J. Watson Sr., beat Ken Jennings, who had once won 74 games in a row, and Brad Rutter, the two players IBM describes as the show's greatest champions.

How the challenge came about

After its Deep Blue computer defeated chess world champion Garry Kasparov in 1997, IBM looked for another grand challenge. The two sources tell the origin slightly differently. IBM's own history says computer scientist David Ferrucci pitched the Jeopardy! idea to his managers in 2006. Wikipedia adds an earlier step: in 2004 an IBM Research manager, Charles Lickel, saw a restaurant fall silent to watch Jennings's winning streak and passed the idea on, and in 2005 research executive Paul Horn backed it.

Early results were sobering. In 2006, a first version tried 500 clues from past shows and answered only about 15% correctly; strong human champions get up to 95%. Ferrucci's team, based at IBM's research centre in Yorktown Heights, New York, spent about five years improving it. IBM puts the team at over 24 people, mixing researchers, engineers and programmers; Wikipedia says the project was given a staff of 15 in 2007.

The match

A practice round was recorded on 13 January 2011 and the official games the next day. They aired over three evenings, 14 to 16 February 2011, as two complete games whose scores were added together.

Watson appeared on stage as a screen showing a glowing globe and spoke with a synthetic voice. It was not connected to the internet. Show staff insisted it press a physical buzzer, like the humans, so it used a mechanical "finger." Viewers could see Watson's top three candidate answers and its confidence in each.

The most famous moment was a mistake. In the first game's Final Jeopardy!, in the category "U.S. Cities," Watson answered "What is Toronto?????" The string of question marks signalled very low confidence. Watson still finished with a commanding lead.

Final totals across both games:

Player	Score	Prize
Watson (IBM)	$77,147	$1,000,000, donated to charity
Ken Jennings	$24,000	$300,000
Brad Rutter	$21,600	$200,000

After the game, Jennings remarked: "'Quiz show contestant' may be the first job made redundant by Watson, but I'm sure it won't be the last."

How Watson worked

Watson ran on IBM software called DeepQA (for "deep question answering"). Despite the name, it did not use deep neural networks. It was a huge pipeline of separate methods working together:

Analyse the clue. Work out what kind of thing is being asked for (a city, a person, a year) and pick out the key terms.
Generate candidates. Search its stored library for many possible answers. Watson held about 200 million pages of material, including encyclopedias, dictionaries, novels, plays and the full text of Wikipedia.
Gather evidence. For each candidate, find passages that support or contradict it.
Score. Hundreds of different algorithms each judged the candidates from a different angle: dates, places, word matches, categories. The more of them that agreed on an answer, the higher Watson's confidence.
Decide. Buzz only if confidence passed a threshold, all within about three seconds.

Separate strategy modules decided which clue to pick next, where Daily Doubles were likely hidden and how much to wager.

The hardware filled a room: 90 IBM Power 750 servers in 10 racks. IBM's history page counts 2,880 processor cores; Wikipedia describes 2,880 processor threads with 16 terabytes of memory.

Try it: Take the Toronto clue above and answer it the way Watson did. Write down three candidate cities, then list one piece of evidence for and against each. Notice how a single misleading hint, like a city with an American League baseball team, can tip the scores.

Why the win mattered

Watson beat top humans at a game decided almost entirely by everyday language. Chess, for all its depth, has fixed rules; Jeopardy! clues can be about anything and are often deliberately tricky. IBM presented the result as a leap in natural language processing, the branch of AI that deals with human language.

After the show, IBM turned Watson into a business, applying the technology to areas such as customer service, weather forecasting and, from 2016, a product for suggesting cancer treatment options.

The match also became a reference point for public conversations about AI and jobs, helped by Jennings's quip.

How is Watson different from a language model like GPT-3?

Watson was a carefully engineered collection of hundreds of separate parts, each written or tuned by experts, that searched a fixed library of documents for evidence. Its strength was knowing how confident it was, and it never wrote its own sentences beyond short answers.

GPT-3, which OpenAI introduced in 2020, takes almost the opposite approach. It is a single very large neural network trained to predict the next word in vast amounts of text. It can write paragraphs, translate and answer questions without a hand-built pipeline. But it does not search for evidence and has no reliable sense of when it is wrong, so it can state false answers with total fluency.

Each approach has something the other lacks. Researchers in 2021 are actively exploring ways to combine them: language models that look up documents before answering, and that can say how sure they are.