# What is DALL·E 2?

Date: 2022-04-07 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-dall-e-2/

**Answer:** DALL·E 2 is a new OpenAI system, announced on April 6, 2022, that creates realistic images and artwork from a written description and can also edit or remix existing pictures. It is a research preview for a small group of testers, with a waitlist for everyone else.

Yesterday OpenAI introduced DALL·E 2, a system that turns a sentence into a picture. Type a description such as "a painting of a fox sitting in a field at sunrise in the style of Claude Monet," and it produces an original image that matches. OpenAI's first DALL·E, shown in January 2021, already did this in rough form. The new version makes images at four times the resolution and looks far more realistic.

For now very few people can use it. OpenAI is previewing DALL·E 2 with a limited group of trusted users and has opened a waitlist. It is not available through the OpenAI API, and OpenAI describes it as a research project it is studying before any wider release.

## What it can do

DALL·E 2 has three main abilities, according to OpenAI:

1. **Create from text.** It combines concepts, attributes and styles, so you can ask for an unusual subject in a particular look.
2. **Edit with words.** Give it an existing image and a caption, and it can add or remove elements while keeping shadows, reflections and textures consistent.
3. **Make variations.** Show it an image, and it creates new versions inspired by the original.

When human evaluators compared 1,000 images from each system, they preferred DALL·E 2 over the first DALL·E 71.7% of the time for matching the caption and 88.8% of the time for photorealism.

## How it works, in plain terms

DALL·E 2 builds on CLIP, an earlier OpenAI model that learned to connect images with the words used to describe them. The research paper published with DALL·E 2 describes two stages:

| Stage | What happens |
|---|---|
| Prior | Turns your caption into a CLIP image embedding, a list of numbers that captures what the picture should contain |
| Decoder | Turns that embedding into an actual picture |

The decoder uses a technique called diffusion. It starts with random noise, like television static, and refines it step by step until an image appears that fits the description. It first makes a small 64 by 64 pixel image, then enlarges it to 256 by 256 and finally to 1024 by 1024.

## Limits and safeguards

The paper is frank about weak spots. DALL·E 2 sometimes mixes up which property belongs to which object: asked for "a red cube on top of a blue cube," it can swap the colors. It struggles to write readable text inside images, and complex scenes can lack fine detail.

OpenAI also lists safety measures. It removed the most explicit material from the training data, limited the system's ability to produce violent, hateful or adult images, and uses techniques to prevent photorealistic faces of real people, including public figures. Its content policy forbids violent, adult and political images, and filters check both text prompts and uploaded images.

> **Try it:** Look at a photo you like and write a one-sentence description of it. Then rewrite it with four additions: the subject, the setting, the lighting and the style or medium. Read both versions aloud and ask which one a stranger could draw more accurately.

## How do you describe an image so an AI can draw it?

Describe it the way you would brief a human illustrator who cannot ask you questions. The system only knows what your words say, so every important choice should be in the sentence. A helpful order is: **subject**, **action or setting**, **style or medium**, then **details** like lighting, color and viewpoint.

Compare a vague request with a complete one:

```prompt
A dog.
```

```prompt
A golden retriever puppy sitting in a field of tall grass at sunset, warm backlight, shallow depth of field, photograph.
```

The first leaves almost everything to chance. The second fixes the breed, age, pose, place, time of day, lighting and medium. OpenAI's own example works the same way: a fox (subject), sitting in a field at sunrise (setting), as a painting in the style of Monet (style).

A few habits help with DALL·E 2's known weaknesses. When colors or positions matter, say them simply and plainly, and check the result, since the paper shows it can mix up attributes between objects. Avoid relying on written words in the picture; the paper notes it struggles to spell. Keep scenes reasonably simple if detail matters. And treat the first image as a draft: change one part of the description at a time, so you can see what each word does.

## Sources

- DALL·E 2: https://openai.com/dall-e-2/
- Hierarchical Text-Conditional Image Generation with CLIP Latents: https://cdn.openai.com/papers/dall-e-2.pdf