What changed in Stable Diffusion 2.0?

Three months after the first public release of Stable Diffusion, Stability AI published version 2.0 on Thursday. Like the original, it is open: the model files and code are free to download and run on your own computer, and developers can build it into their own tools. But this is not a small update. Several core parts of the model have been replaced, and the first days of reactions show that people who have spent months perfecting their prompts are finding the new model behaves differently.

To understand why, it helps to know that a text-to-image model like Stable Diffusion has two main jobs. First, a text encoder turns your prompt into a list of numbers that captures its meaning. Then the image generator, a diffusion model, starts from random noise and removes it step by step, steering toward an image that matches those numbers. Version 2.0 changes both the encoder and what the generator learned from.

The main changes

A new text encoder. Version 1 used a frozen copy of OpenAI's CLIP model (the ViT-L/14 version) to read prompts. Version 2.0 uses OpenCLIP, an open model trained by the nonprofit LAION with support from Stability AI. Stability says the new encoder greatly improves image quality. It also means the model has a different "vocabulary" of associations between words and visual ideas.

Filtered training data. The 2.0 models learned from an aesthetic subset of LAION-5B, a public collection of billions of image and caption pairs gathered from the web, filtered with LAION's detector for adult content to remove nude and pornographic images. Emad Mostaque, Stability AI's founder, told The Verge that artists' images were not specifically removed.

Bigger images. There are now two text-to-image models, one with a default size of 512x512 pixels and one at 768x768. A separate upscaler model enlarges an image four times, for example from 128x128 to 512x512, and together they can reach 2048x2048 or more.

New tools for editing. A depth-to-image model, called depth2img, estimates which parts of a photo are near and far, then generates a new picture that keeps that structure while following your text. An updated inpainting model replaces a selected part of an image with something you describe.

Stability says it kept the model small enough to run on a single graphics card (GPU), as with version 1.

A worked prompt comparison

Many popular version 1 prompts lean on the names of living artists. The Verge reports that prompts naming the digital painter Greg Rutkowski, whose name many users add as a shortcut to a polished look, no longer produce images that closely resemble his work in 2.0. Mostaque says this comes from changes in how the model encodes and retrieves information, not from filtering out artists.

Here is a typical version 1 style prompt:

a lighthouse on a cliff at sunset, by Greg Rutkowski, trending on artstation, highly detailed

And here is a rewrite for 2.0 that describes the look instead of naming a person:

a lighthouse on a rocky cliff at sunset, digital fantasy painting, dramatic warm backlight, crashing waves, visible brush strokes, rich orange and teal colors, highly detailed, wide view

The table below shows what to expect. It is an illustrative summary of the reported behavior, not measured results.

Why some users are unhappy

The reaction has been split. Some users told The Verge the update "nerfs" the model, meaning it has been weakened, because it is harder to copy specific artists, make photorealistic pictures of celebrities or produce nude images. Others say the new version understands prompts better and produces more convincing photographs. Many artists have objected to their work being used for training without consent, and Mostaque said future versions would use datasets that let artists opt in or opt out. Because the model is open, the earlier versions remain available, and people can fine-tune the new one on additional images.

Do your old prompts still work?

They still run, but many won't give you the same picture. The reason is the new text encoder. Your prompt is only words; what the model draws depends on how the encoder translates those words into numbers. OpenCLIP learned its associations from different training data than OpenAI's CLIP, so a phrase that had a strong, specific meaning for version 1, such as an artist's name or a tag like "trending on artstation," may have a weaker or different effect in 2.0. Prompts that lean on these shortcuts change the most.

Plain descriptive prompts travel best. Words for subject, medium, lighting, color and composition carry meaning in any encoder, so they tend to transfer from one version to the next. A good habit is to treat 2.0 as a new instrument: keep a small test set of five or six prompts you know well, run them on both versions with fixed seeds, and adjust one phrase at a time.

Finally, check your settings, not only your words. If you use the 768x768 model, generate at its default size of 768x768 rather than carrying over your old 512 settings. And if a workflow depends on a specific version 1 look, you can keep using version 1 for that job while you learn the new model.