# How did Arthur Samuel's checkers program learn, and where does the term "machine learning" come from?

Date: 2019-04-12 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/how-did-arthur-samuels-checkers-program-learn/

**Answer:** In the July 1959 issue of the IBM Journal of Research and Development, IBM engineer Arthur Samuel reported a checkers program for the IBM 704 that improved with experience until it played better than he did, after only 8 to 10 hours of machine play. The paper's title, "Some Studies in Machine Learning Using the Game of Checkers," helped make "machine learning" the name for computers that improve from experience instead of being told every step.

Every programmer knows the frustration of having to spell out each step a computer must take. Arthur Samuel, an engineer at IBM, saw a way around it. If a computer could learn from experience, he wrote, it would one day save people from much of that detailed programming. To prove the point, he taught a computer to play checkers and then let it get better on its own.

The results appeared in July 1959 in the *IBM Journal of Research and Development*, volume 3, number 3, under the title "Some Studies in Machine Learning Using the Game of Checkers." Samuel showed that a program could learn to play a better game than the person who wrote it, and that it could do so in about 8 to 10 hours of playing time.

## Who Arthur Samuel was

Samuel was born in 1901 in Emporia, Kansas, and taught at MIT and the University of Illinois before joining IBM in Poughkeepsie, New York, in 1949. He worked on checkers in his spare time for years. An early version ran on the IBM 701, IBM's first commercial computer, and its public demonstration is said to have pushed IBM's stock up 15 points overnight.

In August 1956 he brought his work to the Dartmouth summer workshop on artificial intelligence. Notes from that summer record a talk on how to score checkers positions and set intermediate goals, and an observer's remark that Samuel cared about the learning problem far more than about checkers itself.

The term "machine learning" is widely credited to Samuel because of this 1959 paper, which uses it in its title and throughout the text.

## Why checkers?

Samuel explained his choice carefully. Games strip away the messy details of real life while keeping what matters for studying learning: a clear goal (winning), definite rules, and no known recipe that guarantees a win. He picked checkers over chess because its simpler rules let him focus on the learning methods instead of the game. He also liked that people could play against the machine, which made a convincing demonstration for anyone who doubted that machines could learn.

Even checkers is too big to solve by brute force. Samuel estimated that exploring every possible game would take an absurd number of centuries, so the program had to be selective.

## How the program played

The program ran on the IBM 704. It stored each board in four of the computer's 36-bit words, using 32 bits for the 32 dark squares of the board. To choose a move, it:

1. Listed every legal move from the current position.
2. Looked ahead several moves for both sides, usually about three at the start, going deeper when captures were possible.
3. Scored the positions at the end of each line with a formula, adding up weighted features such as how many pieces each side had.
4. Worked back up the tree assuming each player picks its best option, a method called *minimax*, and chose the move with the best result.

The learning was in improving that scoring formula and in remembering positions.

## Two ways of learning

**Rote learning.** The program saved board positions it had met, along with their scores. When it saw one again, it could reuse the stored score, which already reflected a look-ahead from the earlier game. In effect, it could see further each time a position recurred. Samuel's memory tape held over 53,000 positions, and rote learning alone made the program a better-than-average novice, strong in openings and endgames but weak in the middle.

**Learning by generalization.** This is the method that looks most like modern machine learning. The program kept a list of possible features and adjusted the weights in its scoring formula after each move. It did this by comparing its quick score of a position with the deeper score it got by looking ahead. If the quick score was wrong, the terms that caused the error were changed.

To give it practice, Samuel had the program play itself, as two players he called Alpha and Beta:

| Player | What it does | What happens after each game |
|---|---|---|
| Alpha | Updates its scoring formula after every move | If Alpha wins, Beta receives Alpha's formula |
| Beta | Keeps the same formula all game | If Beta wins, Alpha gets a black mark |
| Rule | After three black marks, Alpha's formula is shaken up | This keeps it from getting stuck |

In early tests the learning was fast but erratic. Amateur players described the program as tricky but beatable, and after a few dozen games it was approaching the level of a better-than-average player. Samuel then fixed several sources of instability, such as the program being fooled by an opponent's bad play.

> **Try it:** Play tic-tac-toe against a friend ten times. After each game, write one rule of thumb you would add or drop, such as "take the center first." Count how many rules survive by game ten. You are doing what Samuel's Alpha did: adjusting a scoring method from experience.

## Why it mattered

Samuel's work showed, with a working program and careful measurements, that a computer could get better at a task without being reprogrammed. It also introduced ideas that are still central: self-play, learning an evaluation function, and adjusting predictions to match better-informed later predictions. He even described a "neural-net approach" to learning as a separate path, which he judged in 1959 to be a long way from producing practical devices.

He later moved to Stanford, retiring from IBM in 1966, and remained active in computing into his late eighties. He died in 1990.

## How does Samuel's checkers learning compare with how game-playing AI learns today?

The family resemblance is strong. DeepMind's AlphaGo, which beat Lee Sedol at Go in 2016, and its successor AlphaZero, described in the journal *Science* in December 2018, both learned largely by playing against themselves, just as Alpha played Beta. They also combine a look-ahead search with a learned way of judging positions, which is exactly the structure of Samuel's program.

The main difference is what does the judging. Samuel's scoring formula was a weighted sum of features he chose by hand, such as piece counts. Today's systems use deep neural networks that learn their own features from raw board positions, with millions of adjustable weights instead of a few dozen. And they train on enormous amounts of computing power, playing millions of games where Samuel's program played a few dozen in its early tests.

Samuel's core insight still holds: a program that compares its quick guesses with what a deeper look reveals, and corrects itself, can teach itself to play better than its maker.

## Sources

- Some Studies in Machine Learning Using the Game of Checkers: https://people.csail.mit.edu/brooks/idocs/Samuel.pdf
- Arthur Samuel: https://web.archive.org/web/20190225155629/https://en.wikipedia.org/wiki/Arthur_Samuel
- Ray Solomonoff and the Dartmouth Summer Research Project in Artificial Intelligence, 1956: http://raysolomonoff.com/dartmouth/dartray.pdf