# What is a diffusion model?

Date: 2022-09-08 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/diffusion-model/

**Answer:** A diffusion model is an AI system that learns to make images by learning to remove noise. During training it watches pictures being buried under random static step by step and learns to undo each step, so that it can later start from pure static and clean it up, a little at a time, into a brand-new image.

The image generators everyone is talking about this year, OpenAI's DALL·E 2, Google's Imagen and the newly released Stable Diffusion, all rely on the same core idea. They don't paint a picture from left to right or stamp one out in a single pass. They start with a field of random noise, like the static on an old television, and refine it over many small steps until a picture appears.

That method is called **diffusion**, after the way a drop of ink spreads out in water until it is evenly mixed. The models learn to run that process in reverse: from evenly mixed noise back to a structured image.

## Where the idea comes from

The approach has been around in research for several years, but a 2020 paper made it practical. Jonathan Ho, Ajay Jain and Pieter Abbeel of UC Berkeley, in "Denoising Diffusion Probabilistic Models", showed that diffusion could produce images as good as the best methods of the day on standard test sets.

Their setup has two halves. The **forward process** is fixed and needs no learning: take a real photo and add a little random noise, then a little more, over a long series of steps, until nothing of the photo remains. The **reverse process** is the part a neural network learns. At each step, the network looks at a noisy image and estimates the noise that was added, so that it can be subtracted. Ho's team used 1,000 steps in their experiments. Training is surprisingly simple: show the network a huge number of images at random noise levels and ask it, again and again, to predict the noise.

## Why Stable Diffusion works in a compressed space

Running all those steps on full-size images is expensive. Robin Rombach and colleagues at LMU Munich, Heidelberg University and Runway ML note in their December 2021 paper that training a strong diffusion model directly on pixels can take hundreds of GPU-days.

Their fix is the **latent diffusion model**. First, a separate network called an autoencoder learns to squeeze an image into a much smaller grid of numbers, a "latent" version, and to rebuild the full image from it. The diffusion process then happens entirely in that small latent space. Only at the very end is the result decoded back into pixels. This cuts the computing cost a great deal while keeping fine detail.

Stable Diffusion is a latent diffusion model. According to its project page, it was trained on 512 by 512 images drawn from part of the LAION-5B dataset, has an 860-million-parameter denoising network (a U-Net), and can run on a graphics card with at least 10 GB of memory. That modest size is a big reason it can run on a home computer rather than only in a data center.

## The role of the text encoder

Noise removal alone would produce random pictures. To get the picture you asked for, the model needs to read your prompt. Stable Diffusion uses a **text encoder**, a separate model that turns words into numbers, called CLIP ViT-L/14. It is frozen, meaning it was trained beforehand and is not changed further. Its numerical description of your prompt is fed into the denoising network at every step through a mechanism called cross-attention, which lets each part of the image "look at" the relevant words.

The reference software also uses a technique called classifier-free guidance. At each step the model makes two noise estimates, one with your prompt and one with an empty prompt, and then pushes the result further in the direction of the prompt version. A setting called the guidance scale controls how hard it pushes; the reference script uses 7.5 by default.

> **Try it:** If you have access to a Stable Diffusion demo, generate the same prompt with only the number of steps changed, for example 10, 25 and 50, keeping the seed fixed. Compare how much detail each version has.

## How does noise become a picture?

Picture one run of Stable Diffusion with the prompt "a red bicycle leaning against a stone wall", using the reference script's 50 steps. The table describes, in a simplified and illustrative way, what you would see if you decoded the work in progress at a few points along the way:

| Step | What the in-progress image looks like |
|---|---|
| 0 | Pure static, no structure at all |
| 10 | Soft blobs of color: a warm patch in the middle, grey around it |
| 20 | Rough shapes: something wheel-like, a darker band behind it |
| 35 | A recognizable bicycle against a wall; edges still blurry |
| 50 | Spokes, brick or stone texture, shadows and fine detail |

That order is not a guess. Ho's team observed it directly: when they looked at the model's best estimate of the final image during sampling, large-scale features appeared first and details appeared last. Early steps decide the composition, where things go and what colors dominate. Later steps sharpen edges and add texture.

Each step is a small correction. The network looks at the current noisy latent, reads the prompt through the text encoder, estimates which part of what it sees is noise, and removes a portion of it. Repeat that fifty times and the static has been shaped into an image that matches the words. Because the starting static is random, a different seed, the number that sets that static, gives a different picture from the same prompt.

## Sources

- High-Resolution Image Synthesis with Latent Diffusion Models: https://arxiv.org/abs/2112.10752v1
- Denoising Diffusion Probabilistic Models: https://arxiv.org/abs/2006.11239v1
- CompVis/stable-diffusion (README): https://github.com/CompVis/stable-diffusion