What is DALL·E, and how does it draw pictures from a sentence?

Yesterday, 5 January 2021, OpenAI showed off a neural network that turns short descriptions into pictures. Type "an illustration of a baby daikon radish in a tutu walking a dog," and DALL·E produces a set of cartoon radishes in tutus, each walking a dog. Ask for "an armchair in the shape of an avocado," and you get chairs that look like halved avocados, with the pit as a cushion.

The name blends the painter Salvador Dalí and Pixar's robot WALL·E. Behind the playful examples is a serious idea: the same kind of model that writes text can also learn to draw, if you train it on words and images together.

What OpenAI announced

DALL·E is a version of GPT-3, OpenAI's large language model from last year, with 12 billion parameters, the adjustable numbers a network tunes during training. MIT Technology Review describes it as a smaller version of GPT-3. Instead of only text, it was trained on a dataset of text-image pairs.

OpenAI's blog post, written mostly by project lead Aditya Ramesh, presents dozens of interactive examples. It does not offer the model for download or public use, and it promises a research paper with more technical detail soon. Ilya Sutskever, OpenAI's chief scientist, oversaw the project.

The announcement came together with CLIP, a second model that judges how well a caption matches an image. The two are designed to work as a pair.

How DALL·E turns words into an image

To a model like GPT-3, everything is a sequence of tokens, small units drawn from a fixed vocabulary, like letters from an alphabet. DALL·E puts a caption and an image into one sequence:

Part of the sequence	How it is stored	Size
Caption	Up to 256 text tokens, from a vocabulary of 16,384	Like a short sentence
Image	A 256 × 256 picture compressed into a 32 × 32 grid of image codes	1,024 tokens, from a vocabulary of 8,192
Total	One stream the model reads and writes in order	Up to 1,280 tokens

The compression step uses a separate network, a discrete variational autoencoder, trained to squeeze each picture into those 1,024 codes and rebuild it. You can think of each code as a tile chosen from 8,192 kinds of mosaic tile.

Training then works just like text prediction. DALL·E reads the caption tokens and learns to predict the image tokens that follow, one after another. To create a new image, you give it a caption and it fills in the 1,024 image tokens itself; the decoder turns them back into a picture. Because it fills in tokens in order, it can also finish an image you have started, completing the rest of a photo in a way that fits the text.

DALL·E draws many candidates for each caption. For the examples in its post, OpenAI generated 512 images per caption and had CLIP rank them, then showed the top 32. OpenAI says no images were picked by hand, apart from thumbnails and standalone pictures outside the interactive sections.

Try it: Write three captions that combine two things that do not normally go together, using OpenAI's tip of phrasing like "in the shape of" or "made of," for example "a teapot in the shape of a pumpkin." Sketch, or simply describe in a sentence, what you would expect a sensible drawing to look like. That mental picture is the problem DALL·E must solve: it fills in every detail the words leave out.

What it does well, and where it fails

OpenAI's examples show a model that can:

Combine unrelated ideas, such as a snail made of a harp or a giraffe crossed with a turtle.
Change attributes, such as drawing a clock as a green pentagon, or giving a cube a porcupine's spiky texture.
Control the view and style, including close-ups, aerial views, claymation or x-ray looks.
Write simple text in a scene, such as a shop front with the word "openai" on it.
Transform a photo, for example recoloring a teapot or adding text to it, even though it was never specifically trained to do so.

The failures are just as instructive. DALL·E cannot reliably count past three. When a caption names several objects with different colors, it often mixes them up, and success drops sharply as more objects are added. Some positions, like "sitting on," sometimes work, while "standing behind" or "left of" do not. Swapping a caption's words for synonyms can give completely different results. Longer strings of text are harder for it to write correctly.

MIT Technology Review adds outside views. Mark Riedl, who studies computational creativity at Georgia Tech, called the set of examples impressive but suspects the daikon drawings may partly reflect cartoons the model saw online. Researchers at the Allen Institute for AI said they had not seen earlier text-to-image systems control several objects this well.

Why it matters

Turning text into images is not new. Earlier research used GANs, generative adversarial networks, in which one network makes images and another judges them. DALL·E takes a different route, showing that a GPT-style model can handle pictures and words in the same stream and combine ideas in ways that look creative.

OpenAI describes the result as giving people some of the powers of a 3D rendering program, but through plain language and with the ability to "fill in the blanks" a caption leaves out. It also acknowledges risks: the post says the lab plans to study DALL·E's possible effects on jobs, bias in its outputs and longer-term ethical questions. Its own tests already show one form of bias, with stereotyped answers for prompts about the food or wildlife of a country.

Sutskever told MIT Technology Review that he sees DALL·E and CLIP as steps toward models that understand both text and images.

Can you use DALL·E yourself?

Not yet. The announcement offers no download and no public version you can type into. What you can do today is explore the post's interactive examples, which show DALL·E's results for many variations of each caption.

OpenAI says a paper with details about the architecture and training is coming. That will let researchers study the method closely, even without the model itself.

In the meantime, CLIP's code is available, and it offers a hands-on way to understand half of the system: the part that judges whether a picture matches its words. Many expect wider access to text-to-image tools to come eventually, but OpenAI has given no timeline, and its stated plan to study social effects suggests it will move carefully.