# What are generative adversarial networks (GANs), and how do two networks learn by competing?

Date: 2021-05-27 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-are-generative-adversarial-networks/

**Answer:** Generative adversarial networks, introduced by Ian Goodfellow and colleagues at the Université de Montréal in a paper posted on 10 June 2014, train two neural networks against each other: a generator that creates fake samples and a discriminator that tries to tell them from real data. As each improves to beat the other, the generator learns to produce new images that look real, which made GANs the leading way to create realistic pictures with AI.

Until 2014, most of deep learning's successes were about recognizing things: is this photo a cat, which word was spoken, what digit is this? Creating new things, such as a realistic photo of a face that never existed, was much harder. On 10 June 2014, a team at the Université de Montréal posted a paper with a new idea for doing it. Instead of one network, use two, and make them rivals.

The paper, "Generative Adversarial Nets," has eight authors, led by Ian Goodfellow, then finishing his PhD, and including Yoshua Bengio, a senior fellow of the Canadian research institute CIFAR. The approach is now known as generative adversarial networks, or GANs. "Generative" means the system generates new data; "adversarial" means it learns through a contest.

## The idea: counterfeiters and police

The authors explained their method with an analogy. Picture a team of counterfeiters trying to print fake banknotes, and the police trying to catch them. At first the fakes are crude and easy to spot. Each time the police catch one, the counterfeiters learn what gave it away and improve. As the fakes improve, the police must become sharper. If the contest runs long enough, the fakes become indistinguishable from real money.

In a GAN, both sides are neural networks:

- The **generator** takes random numbers as input and turns them into a sample, for example an image.
- The **discriminator** looks at an image and outputs a probability that it is real, meaning taken from the training data, rather than made by the generator.

The generator never sees the real images directly. It learns only from the discriminator's verdicts: it adjusts itself to make the discriminator wrong as often as possible. The paper describes this as a two-player "minimax" game, a term from game theory for a contest in which one side tries to maximize what the other tries to minimize. In theory, the game ends when the generator copies the real data's patterns perfectly and the discriminator can do no better than a coin flip, guessing "real" with probability one half.

## A training round, step by step

Here is one round of GAN training, using photos of faces as the example:

| Step | Discriminator | Generator |
|---|---|---|
| 1 | Is shown a batch of real face photos, labeled "real" | Turns random noise into a batch of fake faces |
| 2 | Is shown the fakes, labeled "fake" | Waits |
| 3 | Adjusts itself to tell the two groups apart better | Waits |
| 4 | Judges a fresh batch of fakes | Adjusts itself so those fakes score as more "real" |
| 5 | Repeat thousands of times | Repeat thousands of times |

A practical advantage noted in the paper is that both networks can be ordinary multilayer networks trained with backpropagation, the standard method for adjusting a network's connections. Earlier generative methods often needed slow sampling procedures called Markov chains; GANs did not.

The authors tested the idea on three standard datasets: MNIST, a collection of handwritten digits; the Toronto Face Database; and CIFAR-10, small photos of objects such as cars and animals. The samples were blurry by today's standards, but recognizable, and the team checked that the generator was producing new images rather than copying training examples.

> **Try it:** Play the counterfeiter game with a friend. One of you writes three real facts and one invented "fact" about a topic; the other tries to spot the fake and explains how. Swap roles for five rounds and notice how both the fakes and the detective work get better.

## The weak spots

The paper was frank about the difficulties. The two networks must stay in balance. If the generator is trained too much without the discriminator keeping up, it can find one output that fools the discriminator and produce nearly the same thing for every input. The authors called this "the Helvetica scenario"; researchers often call it mode collapse. GANs also give no direct measure of how well they model the data, which makes them hard to evaluate.

## Why GANs mattered

Despite those problems, GANs caught on quickly, and researchers kept publishing new variants. By 2017 GANs were generating convincing faces, and in 2019 faces made by a system called StyleGAN were being compared with deepfakes. In 2018 a GAN-made portrait, *Edmond de Belamy*, sold at auction for US$432,500. GANs are also used to sharpen low-resolution images, to simulate data for astronomy and physics, and to propose new drug molecules. In May 2020, Nvidia's GameGAN recreated the game Pac-Man simply by watching it played.

The same power raised concerns. GAN-based "deepfakes," realistic fake videos of real people, prompted California in 2019 to pass laws against non-consensual fake pornography and deceptive videos of election candidates.

## How do GANs compare with newer image generators like DALL-E?

DALL-E, which OpenAI announced on 5 January 2021, takes a different route. It does not use a generator and a discriminator. OpenAI describes it as a version of GPT-3 with 12 billion parameters. It reads a caption and an image as one long sequence of tokens, small pieces of text and image, and learns to produce them one after another. Given a prompt such as "an armchair in the shape of an avocado," it writes out the image tokens that should follow. OpenAI then uses CLIP, a model that scores how well an image matches a caption, to rank the results and keep the best.

That difference has practical effects. Most well-known GANs are trained on one kind of image, such as faces, and become very good at that one kind. DALL-E aims for range: almost any scene you can describe in words. GANs were not absent from that goal: OpenAI's own announcement points out that earlier text-to-image systems, such as StackGAN and AttnGAN, were GANs guided by text.

Which approach will lead is an open question. What seems settled is the idea Goodfellow's team introduced: a model can improve by trying to fool a critic, and a critic can improve by refusing to be fooled. CLIP's role in DALL-E, judging which images best fit the words, is a gentler echo of that same partnership.

## Sources

- Generative Adversarial Nets: https://arxiv.org/abs/1406.2661
- Generative adversarial network: https://web.archive.org/web/20210115000000/https://en.wikipedia.org/wiki/Generative_adversarial_network
- DALL·E: Creating Images from Text: https://openai.com/blog/dall-e/