Short answer
The two text-to-video tools most people can sign up for right now are Runway's Gen-2 and Pika, both of which turn a written description or a still image into a clip a few seconds long. OpenAI's Sora, shown in February, is still limited to safety testers and a small group of artists, so it is not something you can try yet.
Follow-up question How do you write a prompt for a short video clip?
Since OpenAI showed its Sora demos in mid-February, many people have asked where they can make a video like that. The honest answer is that you can't, for now. OpenAI has given Sora to red teamers, specialists who look for ways a system can cause harm, and to a small group of artists, designers and filmmakers whose feedback it wants. There is no public sign-up.
What you can use today is a smaller, rougher generation of tools. The two best known are Runway's Gen-2 and Pika. Both produce short clips, both are open for sign-ups, and both reward a carefully written prompt more than a long one.
Runway Gen-2: the first widely available option
Runway's Gen-2 model became open to the public last June after a waitlist, which made it one of the first text-to-video models anyone could pay for. You type a description, or upload an image to animate, and Gen-2 returns a clip.
When TechCrunch tested it at launch, the clips were four seconds long with a visibly low frame rate (the number of still images shown per second, which is what makes motion look smooth). The reviewer also saw grainy footage, bodies and objects that warped as they moved, and camera directions such as "slow zoom" that were ignored. Styles that already look stylized, like anime or claymation, hid those flaws best. Runway has added features since then, so treat those early limits as a guide to what is hard for these models rather than an exact description of today's version.
Under the hood, Gen-2 is a diffusion model: it starts from random visual noise and removes it step by step until frames that match your text appear. Sora uses the same basic idea at a much larger scale.
Pika: a newer challenger
Pika began as a beta on Discord, the messaging app. On November 28 the company launched Pika 1.0, a website for desktop and mobile that generates and edits video from text in styles such as 3D animation, anime and "cinematic." It said more than half a million people were already using it, and it opened access gradually from a waitlist.
Pika's pitch is editing as much as generating. The company describes turning images or existing videos into new clips, widening the frame to a different aspect ratio (the shape of the picture, such as 16:9 for widescreen or 9:16 for phone video), changing an object inside a clip, and extending a video's length.
How the options compare
| Runway Gen-2 | Pika 1.0 | Sora | |
|---|---|---|---|
| Can you use it today? | Yes, public sign-up | Yes, via website or Discord, with gradual access | No, testers and selected artists only |
| Starts from | Text or an image | Text, an image or a video | Text, an image or a video (per OpenAI) |
| Typical clip | A few seconds | A few seconds | Up to a minute (per OpenAI) |
| Best for | Short stylized shots | Short shots plus edits to existing clips | Not available to try |
Even OpenAI lists weaknesses for Sora: it can struggle with physics, with cause and effect (a cookie with a bite taken out that shows no bite mark), and with telling left from right. Expect the same problems, more often, from the tools you can use now.
Try it: Pick one tool, write the five-part prompt from the next section, and generate it three times without changes. Compare the three clips and note what stayed the same and what changed. That tells you which parts of your prompt the model is actually following.
How do you write a prompt for a short video clip?
Write for one shot, not a story. Four or five seconds is about the length of a single camera shot in a film, so describe one subject doing one simple thing in one place. A useful order is subject, action, setting, camera and style. Here is a worked example for a five-second clip:
A red canoe glides slowly across a calm lake at sunrise,
mist rising from the water, pine forest on the far shore.
Static wide shot from the shore, camera at water level.
Soft golden light, realistic, gentle ripples.Notice what it leaves out. There is no person paddling, because hands and faces are where these models most often go wrong. There is only one movement, the canoe gliding, and the camera stays still, since camera directions are among the instructions these tools follow least reliably. "Calm," "slowly" and "gentle" ask for little motion, which hides warping.
Then change one thing at a time. If the canoe drifts in the wrong direction, add "moving from left to right." If the water looks like plastic, try a style word such as "35mm film" or "watercolor." If you want a longer scene, generate several short shots with matching style words and join them in any video editor, the usual way to build anything longer than a few seconds with today's tools.
Sources
- Runway's Gen-2 shows the limitations of today's text-to-video tech — TechCrunch, June 9, 2023
- Pika Labs raises $55M, launches new AI video platform to take on Runway — VentureBeat (Shubham Sharma); read via Wayback Machine capture of 2023-12-22, November 28, 2023
- Sora — OpenAI, February 15, 2024