How do you teach an image model a new face or style?

Text-to-image models such as Stable Diffusion know what "a cat" looks like, but not what your cat looks like. No amount of description will fix that: you can write "orange tabby with a white chin and one torn ear" and still get a stranger's cat. Two research methods published this summer close that gap by teaching the model a specific subject from a few pictures.

Both are now within reach of hobbyists. Hugging Face's free diffusers library added textual inversion and DreamBooth in its October 6 release, each with a ready-made Google Colab notebook, a shared online computer you can run from a browser.

Textual inversion: a new word, same model

Textual inversion comes from a team at Tel Aviv University and NVIDIA, who posted their paper on August 2. Their title sums it up: an image is worth one word.

A text-to-image model first turns each word of your prompt into an embedding, a list of numbers that stands for its meaning. Textual inversion adds one new, made-up word, written S* in the paper, and searches for the embedding that best reproduces your example images. Everything else in the model stays frozen, meaning unchanged. The result is a tiny file you can share and combine with others.

Once learned, the word behaves like any other: "a photograph of S* on the beach," or even "a drawing of S*1 in the style of S*2." In the authors' setup, learning one concept took about two hours on two powerful graphics cards. The paper notes the method captures the general essence of a concept well but can miss precise shapes, which suits styles better than exact likenesses.

DreamBooth: adjusting the model itself

DreamBooth, from Google Research and Boston University, was posted on August 25. Instead of adding a word, it fine-tunes the model: it slightly changes the model's own internal numbers so that it learns the subject.

Each training photo is captioned in a fixed pattern: "a [V] dog," where [V] is a rare token, a scrap of text the model barely knows, and "dog" is the general class. To stop the model from forgetting what ordinary dogs look like, DreamBooth also trains on dog pictures the model generates itself. The authors call this prior preservation. They used Google's Imagen and report about 15 minutes of training on one of Google's TPU chips; the method is not tied to one model, and the diffusers version runs on Stable Diffusion.

In a side-by-side test in the DreamBooth paper, it kept more of a subject's fine details than textual inversion, though the authors point out the two ran on different base models, so the match may not be fully fair. The cost is size: you end up with a whole adjusted model rather than a small word file.

A worked example: your own pet

Here is how a first project with a cat named Miso could go, using either notebook.

1. Pick 3–5 photos. Different angles, rooms and lighting, Miso clearly visible, no other animals. Crop them square.
2. Choose the new word. For DreamBooth, pair a rare made-up token with the class, such as "a zqx cat." For textual inversion, set a placeholder such as <miso> and a starting word like "cat."
3. Train. Follow the notebook's steps and leave the settings at their defaults the first time.
4. Test with new scenes.

a photo of a zqx cat sleeping in a basket of yarn, morning light
a watercolor painting of a zqx cat on a windowsill
a zqx cat wearing a tiny astronaut helmet, studio photo

1. Check for the known failures. The DreamBooth paper lists them: unusual settings may not render, the scene can change the subject's colors (a blue background turning a cat bluish), and prompts close to the training photos may just copy them. Pets are the easy case: the paper notes the model already knows dogs and cats very well.

What are the privacy and consent risks?

The same steps that work for a cat work for a human face, and that is where care is needed. Both research teams raise it. The DreamBooth authors note that bad actors might use such images to mislead viewers. The textual inversion authors warn that personalized models could help someone forge convincing images of private individuals, although they judge their own method does not yet hold a person's identity closely enough for that.

A few rules keep a project on the right side:

• Train only on faces of people who have agreed, and tell them what the images will be used for. For children, ask a parent, and think twice even then.
• Do not train on someone else's photos from social media, even if they are public.
• Label results as AI-made when you share them, and never present a generated image of a real person as a real event.
• Keep the trained file private. A model or word file that has learned a face can make new pictures of that person for anyone who has it.

Styles raise a related question. The textual inversion paper itself notes that learning an artist's style without consent could be used to avoid paying the artist, while suggesting artists could also license their style. Before training on an artist's work, ask whether they would agree.

In Québec, a face is personal information, and a business that uses customers' or employees' photos this way should treat the project like any other use of personal information, with consent for a specific purpose.