AI tools

What text-to-video tools can you actually use today?

· By Robert Scales · 4 min read

Short answer

The two text-to-video tools most people can sign up for right now are Runway's Gen-2 and Pika, both of which turn a written description or a still image into a clip a few seconds long. OpenAI's Sora, shown in February, is still limited to safety testers and a small group of artists, so it is not something you can try yet.

Since OpenAI showed its Sora demos in mid-February, many people have asked where they can make a video like that. The honest answer is that you can't, for now. OpenAI has given Sora to red teamers, specialists who look for ways a system can cause harm, and to a small group of artists, designers and filmmakers whose feedback it wants. There is no public sign-up.

What you can use today is a smaller, rougher generation of tools. The two best known are Runway's Gen-2 and Pika. Both produce short clips, both are open for sign-ups, and both reward a carefully written prompt more than a long one.

Runway Gen-2: the first widely available option

Runway's Gen-2 model became open to the public last June after a waitlist, which made it one of the first text-to-video models anyone could pay for. You type a description, or upload an image to animate, and Gen-2 returns a clip.

When TechCrunch tested it at launch, the clips were four seconds long with a visibly low frame rate (the number of still images shown per second, which is what makes motion look smooth). The reviewer also saw grainy footage, bodies and objects that warped as they moved, and camera directions such as "slow zoom" that were ignored. Styles that already look stylized, like anime or claymation, hid those flaws best. Runway has added features since then, so treat those early limits as a guide to what is hard for these models rather than an exact description of today's version.

Under the hood, Gen-2 is a diffusion model: it starts from random visual noise and removes it step by step until frames that match your text appear. Sora uses the same basic idea at a much larger scale.

Pika: a newer challenger

Pika began as a beta on Discord, the messaging app. On November 28 the company launched Pika 1.0, a website for desktop and mobile that generates and edits video from text in styles such as 3D animation, anime and "cinematic." It said more than half a million people were already using it, and it opened access gradually from a waitlist.

Pika's pitch is editing as much as generating. The company describes turning images or existing videos into new clips, widening the frame to a different aspect ratio (the shape of the picture, such as 16:9 for widescreen or 9:16 for phone video), changing an object inside a clip, and extending a video's length.

How the options compare

Runway Gen-2Pika 1.0Sora
Can you use it today?Yes, public sign-upYes, via website or Discord, with gradual accessNo, testers and selected artists only
Starts fromText or an imageText, an image or a videoText, an image or a video (per OpenAI)
Typical clipA few secondsA few secondsUp to a minute (per OpenAI)
Best forShort stylized shotsShort shots plus edits to existing clipsNot available to try

Even OpenAI lists weaknesses for Sora: it can struggle with physics, with cause and effect (a cookie with a bite taken out that shows no bite mark), and with telling left from right. Expect the same problems, more often, from the tools you can use now.

Try it: Pick one tool, write the five-part prompt from the next section, and generate it three times without changes. Compare the three clips and note what stayed the same and what changed. That tells you which parts of your prompt the model is actually following.

How do you write a prompt for a short video clip?

Write for one shot, not a story. Four or five seconds is about the length of a single camera shot in a film, so describe one subject doing one simple thing in one place. A useful order is subject, action, setting, camera and style. Here is a worked example for a five-second clip:

Prompt
A red canoe glides slowly across a calm lake at sunrise,
mist rising from the water, pine forest on the far shore.
Static wide shot from the shore, camera at water level.
Soft golden light, realistic, gentle ripples.

Notice what it leaves out. There is no person paddling, because hands and faces are where these models most often go wrong. There is only one movement, the canoe gliding, and the camera stays still, since camera directions are among the instructions these tools follow least reliably. "Calm," "slowly" and "gentle" ask for little motion, which hides warping.

Then change one thing at a time. If the canoe drifts in the wrong direction, add "moving from left to right." If the water looks like plastic, try a style word such as "35mm film" or "watercolor." If you want a longer scene, generate several short shots with matching style words and join them in any video editor, the usual way to build anything longer than a few seconds with today's tools.

Sources

  1. Runway's Gen-2 shows the limitations of today's text-to-video tech — TechCrunch, June 9, 2023
  2. Pika Labs raises $55M, launches new AI video platform to take on Runway — VentureBeat (Shubham Sharma); read via Wayback Machine capture of 2023-12-22, November 28, 2023
  3. Sora — OpenAI, February 15, 2024

Get the free Guide to Artificial Intelligence

Sign up for the weekly newsletter and receive the free guide: what changed in AI this week, one lesson, one prompt to try. Unsubscribe in one click.

How we handle your address

Back to top

Display

Your choices are saved in this browser only. No data is sent.

Text size
100%
Colours
Reading
More settings
Accessibility statement