How did DeepMind's deep Q-network learn to play Atari games from the pixels alone?

On 19 December 2013, the London startup DeepMind posted a paper showing one neural network that learned to play seven Atari 2600 games using only the screen pixels and the score, with no game-specific rules, beating all earlier methods on six games and an expert human on three. It combined deep learning with reinforcement learning, learning by trial and reward, and launched the field now called deep reinforcement learning.

How did DeepMind's deep Q-network learn to play Atari games from…

Imagine handing someone a game console they have never seen, with no manual, and saying only: "Watch the screen, press buttons, and try to make the number in the corner go up." That is roughly the challenge a small London company called DeepMind set for a computer program in 2013. Its program learned to play Pong, Breakout and other Atari classics that way, and on several games it ended up playing better than a human expert.

The paper, "Playing Atari with Deep Reinforcement Learning," went online on 19 December 2013. Its seven authors, led by Volodymyr Mnih, worked at DeepMind Technologies. The agent they described became known as the deep Q-network, or DQN, the name used in the fuller version that appeared in the journal Nature in February 2015.

What the program did

The team used the Arcade Learning Environment, a research tool that runs games for the Atari 2600 home console, titles from the 1970s and 1980s, inside a computer. They picked seven well-known titles: Pong, Breakout, Space Invaders, Seaquest, Beam Rider, Enduro and Q*bert.

The program saw exactly what a player sees: the picture on the screen. It received the score as its only feedback, and it could press the same buttons a player could. Nobody told it what a paddle, a ball or an enemy was. The same network design and the same learning settings were used for every game.

The results: DQN outperformed every earlier learning method on six of the seven games, and scored higher than an expert human on three: Breakout, Enduro and Pong. In Breakout, its average score was 168 against the human's 31. It still fell far short of the human scores on Q*bert, Seaquest and Space Invaders.

How it worked

DQN joins two ideas.

The first is reinforcement learning: learning by trial and error from rewards. The program tries actions, sees what happens to the score, and gradually prefers the actions that pay off. A classic version, Q-learning, keeps an estimate called a Q-value for each situation and action: "If I am here and do this, how much reward can I expect from now on, counting what comes later?"

The second is deep learning. Atari screens contain far too many possible pictures to keep a table of Q-values for each. So DeepMind used a convolutional neural network, the kind that had just transformed image recognition, to look at the screen and estimate the Q-value of every button at once.

One game step went like this:

Take the last four frames, convert them to grayscale and shrink them to 84 by 84 pixels. Stacking four frames lets the network see motion, such as which way the ball is moving.
Feed them to the network, which outputs a predicted value for each possible action.
Usually choose the highest-value action; sometimes choose a random one, to keep exploring. Early in training it acts randomly all the time, and this falls to 10 percent of the time over the first million frames.
Record what happened: the screen before, the action, the reward, the screen after.
Store that record in a large memory and train the network on a random handful of old records.

That last trick, called experience replay, was key. Learning from consecutive moments is unstable, because each frame looks almost like the previous one. Shuffling a memory of one million recent frames gives the network a varied mix, rather like a student reviewing old flashcards instead of rereading the last page. Each game was trained for 10 million frames.

Try it: Play a simple browser game you have never tried, without reading the instructions. Note how many attempts it takes before you understand what earns points. That gap between pressing buttons and understanding the goal is exactly what DQN had to learn to close.

Why it mattered

Before DQN, reinforcement learning worked mostly when engineers hand-crafted the features a program should notice. DQN showed that a single system could learn its own features straight from pixels, and do so across different games. The 2015 Nature version extended the test to 49 Atari games, using one algorithm and one set of settings, and reached a level comparable to a professional human games tester across them.

The work also changed DeepMind's fortunes. It reportedly helped persuade Google to buy the company in January 2014. The same blend of deep networks and reinforcement learning then powered AlphaGo, which beat Go champion Lee Sedol 4 to 1 in March 2016, and AlphaZero, which taught itself Go, chess and shogi. In 2020 DeepMind's Agent57 surpassed the human benchmark on all 57 games of the Atari suite.

What is reinforcement learning, and where is it used besides games?

Reinforcement learning is learning from consequences. Instead of being shown correct answers, as in most image recognition, an agent acts in an environment, receives rewards or penalties, and adjusts its behavior to collect more reward over time. The idea goes back a long way: Alan Turing proposed teaching machines with "pleasure" and "pain" signals in 1948, and the Nature DQN paper points to parallels with how dopamine signals work in animal brains.

Games are popular testbeds because the score is a clear reward and a simulator can run millions of rounds cheaply. Outside games, the most visible use is robotics. Google researchers, for example, have had robot arms make hundreds of thousands of grasping attempts so they could teach themselves to pick up objects from experience. Any task that is a long chain of decisions with a measurable goal is a candidate.

The hard part is the reward. A game hands you a score; the real world rarely does. Defining a reward that captures what people actually want, without rewarding shortcuts, is one of the main open problems in the field, and it matters more as learning systems take on real tasks.