What can text-to-video research like Make-A-Video and Imagen Video do?

In less than two weeks, two of the largest AI labs have shown that the text-to-image trick now works for moving pictures. On September 29, Meta announced Make-A-Video. On October 5, Google researchers posted Imagen Video. Type a sentence such as "a red kite rising over a snowy field" and the system produces a short clip in which the scene actually moves.

Text-to-video simply means generating a video from a written description, the way DALL·E 2, Imagen or Stable Diffusion generate a still image. The clips are short, silent and often a little dreamlike, but they show something new: a model that has some sense of how things move, not only how they look.

Two different routes to the same goal

The two projects start from the same problem. There are billions of images with captions on the web, but far fewer videos with good text descriptions. Each team works around that gap in its own way.

Make-A-Video (Meta) splits the learning in two. From image-and-caption pairs, the model learns what the world looks like and what things are called. From ordinary video clips with no captions at all, it learns how the world moves. The paper's title makes the point: no captioned video is needed. The system first makes 16 tiny frames of 64 by 64 pixels, then fills in extra frames between them for smoother motion, then enlarges the picture in two stages, up to 768 by 768 pixels.

Imagen Video (Google) extends Google's Imagen image model into time. It is a cascade, a chain of seven separate diffusion models. A diffusion model creates a picture by starting from random noise and removing it step by step until an image matches the prompt. In Imagen Video, a base model makes a small, short, low-frame-rate clip; the next models each add detail in space (more pixels) or in time (more frames). The final output is 128 frames at 1280 by 768 pixels and 24 frames per second, about 5.3 seconds. Google trained it on an internal collection of video-text and image-text pairs, plus the public LAION-400M image dataset.

What the demos show

Here is how the two compare on the points the papers report:

Both systems carry over a strength of image models: style. Ask Imagen Video for a scene in the style of van Gogh or as a watercolor and the motion follows that look. Google's samples also include readable animated words, a strength carried over from Imagen's skill at spelling text. Meta highlights flexibility: give Make-A-Video a single photo and it can bring it to life, or give it a clip and it can make new versions that resemble it.

The limits the researchers name

The papers are careful about what does not work yet:

• Length. These are clips of a few seconds. Meta lists longer videos with several scenes and real stories as future work.
• Motion that only video can explain. Meta notes its method cannot learn text about things you only see in motion, such as whether a person waves left to right or right to left.
• Consistency. Google says its rotating 3D objects keep their shape only roughly.
• Bias and misuse. Both teams say models trained on web data absorb social biases. Google warns that video generation could be misused for fake, hateful or explicit content, and says filtering cannot yet catch every stereotype.

A worked way to read a demo clip, then, is to ask four questions: How long is it? Does anything move in a specific direction the prompt asked for? Do objects keep their shape from start to finish? And would the same prompt work on a person, a place or a brand, where errors and misuse matter more?

How far is it from a tool you can use?

For now, still some distance. Meta says it is sharing the research and plans to release a demo experience, but there is no public tool to try. Google is firmer: its paper says the team has decided not to release Imagen Video or its code until the safety concerns are reduced. Both models are also large. Imagen Video's cascade holds 11.6 billion parameters, and even with a speed-up technique called progressive distillation, which the team says makes sampling about 18 times faster, it is far from the kind of model you run on a home computer.

There is at least one text-to-video model released publicly: CogVideo, an earlier research system that Meta's paper describes as the only public one of its kind and uses as a comparison. Its results are lower in quality than the new demos. For practical work, creators today get more from still-image tools. Stable Diffusion runs on a home graphics card, and DALL·E 2 is open to everyone, so you can generate storyboards, backgrounds and style frames, then edit them into motion with ordinary video software.

The pace of the past year suggests public tools may arrive, but neither company has given a date. The sensible move for a creator is to learn prompting on still images now: the skills of describing subject, style, lighting and camera angle carry straight over to video once these systems open up.