What is OpenAI's CLIP, and how does it recognize images from words?

Yesterday, 5 January 2021, OpenAI introduced two models that connect language and images. One, DALL·E, draws pictures from text. The other, CLIP, does the reverse kind of job: it looks at a picture and judges which piece of text describes it best.

That sounds modest, but it changes how image recognition can work. Most vision systems learn a fixed list of categories from pictures that people labeled by hand. CLIP learned from the captions that already sit next to images across the web. As a result, you can point it at a new task simply by describing the categories in words.

What OpenAI released

CLIP stands for Contrastive Language–Image Pre-training. OpenAI published a blog post, a research paper and code on GitHub. The paper has twelve authors, led by Alec Radford and Jong Wook Kim, and includes OpenAI's chief scientist, Ilya Sutskever.

The headline result concerns ImageNet, the famous benchmark of about 1.28 million labeled photos in 1,000 categories. Without seeing a single one of those labeled training photos, CLIP's best model matches the accuracy of ResNet-50, a standard network trained directly on them. OpenAI calls this zero-shot performance: doing a task with no examples of it, much as GPT-2 and GPT-3 can attempt language tasks they were never trained for.

The team tested CLIP on more than 30 existing image datasets, covering tasks such as reading text in photos, recognizing actions in video frames, guessing where a photo was taken and telling apart closely related objects.

How CLIP learns from captions

CLIP has two parts: an image encoder, which turns a picture into a list of numbers, and a text encoder, which does the same for a sentence. Training pushes the numbers for an image and its true caption close together, and the numbers for mismatched pairs far apart. That is what "contrastive" means: learning by contrasting right pairs with wrong ones.

The training game is a giant multiple-choice quiz. Given one image, CLIP must pick which of 32,768 randomly drawn text snippets really came with it. To win this game across 400 million image-text pairs, it has to learn what a huge variety of objects, scenes and styles look like, and which words go with them.

OpenAI reports two choices that cut the computing cost. Matching whole captions, instead of trying to write them word by word, was 4 to 10 times more efficient. Using a Vision Transformer, an image network built on the same attention design as GPT models, gave roughly another threefold saving. Even so, the largest model trained for two weeks on 256 graphics processors.

A worked example: classifying with words

Here is how CLIP sorts a photo without any task-specific training:

1. Choose your categories. Say you want to know whether pictures show a dog, a cat or a rabbit.
2. Turn each category into a caption. OpenAI's suggested pattern is "a photo of a dog," "a photo of a cat," "a photo of a rabbit."
3. Encode everything. The text encoder turns each caption into numbers; the image encoder does the same for your photo.
4. Compare. CLIP measures how close the photo's numbers are to each caption's numbers.
5. Pick the closest. The caption with the best match becomes the answer.

Want a different task, such as sorting paintings by art style, or telling a sunny street from a snowy one? Change the captions. No new training data is required.

Where it struggles

OpenAI lists clear limits. CLIP does poorly at counting objects and at judging distances, such as the distance to the closest car in a picture; on both, it barely beats random guessing. It is weaker than specialized systems at very fine distinctions, such as car models, aircraft variants or flower species. It reads printed text in images surprisingly well, yet reaches only 88 percent on handwritten digits, a task where people score about 99.75 percent.

Wording matters too. The same task can give better or worse results depending on how the captions are phrased, so users may need some trial and error, which OpenAI calls "prompt engineering."

There are also social risks. Because anyone can define categories, poorly chosen labels can produce harmful results. In one test, when offensive terms were among the options, CLIP placed images of people aged 0 to 20 in those categories about 32 percent of the time; adding "child" as an option cut that to under 9 percent. OpenAI also tested celebrity recognition to study surveillance risks and found CLIP useful but not competitive with commercial systems.

Why it matters

CLIP tackles three problems that have long held back computer vision. Labeled datasets are expensive: ImageNet needed more than 25,000 workers to annotate its images. Standard models are narrow and must be retrained for every new task. And models that ace benchmarks often stumble in the real world. OpenAI argues that because CLIP never trains on a benchmark's own data, its scores there better reflect how it will perform on new kinds of images. On sketches and other hard test sets, it closes much of the gap that trips up ordinary ImageNet models.

CLIP also works hand in hand with DALL·E. OpenAI used it to rank DALL·E's drawings, and MIT Technology Review reports that the showcased results were the 32 images per caption that CLIP judged the best match, not picks made by hand. Sutskever told the magazine he expects future models to understand both text and images, and that seeing what words mean should help AI understand language better.

How can you use CLIP without training your own model?

The simplest route is the code OpenAI has posted on GitHub, linked from the announcement. The idea is the one in the worked example: give CLIP your images and a list of captions, and it scores how well each caption fits. Anyone comfortable with a little Python can use this to sort their own photos into categories written as plain sentences, with no labeled training set.

For good results, follow the lessons in OpenAI's post. Describe each category the way a caption on the web might, starting with "a photo of" or a similar phrase. Try several wordings and keep the one that works best on a handful of test images. Avoid tasks CLIP is known to handle badly, such as counting or very fine-grained identification.

Most of all, check the labels you choose. Since CLIP will sort images into whatever categories you give it, the choice of categories shapes both its accuracy and its biases. Test it on varied examples before trusting it with anything important.