# What is Sora, OpenAI's text-to-video model, and how does it work?

Date: 2024-02-16 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-openai-sora/

**Answer:** Sora is a new OpenAI model, previewed on February 15, 2024, that turns a written description into a realistic or imaginative video up to a minute long. It is not available to the public: for now only red teamers testing it for risks and a group of visual artists, designers and filmmakers can use it.

Yesterday, February 15, 2024, OpenAI showed the world **Sora**, a model that creates video from text. You type a description, for example a woman in a red dress walking down a rain-slicked Tokyo street at night, lit by neon signs, and Sora produces a video clip of that scene, up to one minute long. OpenAI published many sample videos and says they were generated directly by Sora, without editing.

The videos spread quickly online because of their detail: reflections on wet pavement, crowds moving in the background, camera movements that look planned. They are also a preview, not a product. No one outside a small testing group can use Sora yet.

## What OpenAI announced, and who can use it

OpenAI calls Sora a text-to-video model. Here is the announcement in brief:

| Question | Answer from OpenAI's announcement |
|---|---|
| What does it make? | Videos up to 60 seconds long from a text prompt |
| What else can it do? | Animate a still image, extend an existing video, fill in missing frames |
| Who can use it now? | Red teamers, plus some visual artists, designers and filmmakers |
| Is it in ChatGPT or the API? | No; OpenAI says it is still research |
| Who leads the work? | Researchers Bill Peebles and Tim Brooks |

**Red teamers** are outside experts who attack a system on purpose. Here, OpenAI says they work in areas such as misinformation, hateful content and bias, and will try to make Sora produce harmful material so the problems can be found before any public release. The artists and filmmakers are there to tell OpenAI how the model could become useful for creative work.

OpenAI says it is sharing the research early to get outside feedback and to show the public what AI may soon be able to do. Axios reports that an OpenAI spokesperson said the company does not expect to open Sora widely in the near future.

## How Sora works, in plain terms

Sora combines two ideas you may already know from image generators and chatbots.

**1. Diffusion.** Sora is a **diffusion model**. It begins with a video that looks like pure static, the "snow" of an untuned television, and removes that noise a little at a time over many steps, until a clear scene appears. During training, the model learned which removals turn noise into realistic footage that matches a description. A helpful picture is a sculptor working from a rough block: each pass takes away what does not belong.

**2. Transformers and patches.** Like the GPT models behind ChatGPT, Sora uses a **transformer**, the type of neural network that is good at relating many pieces of information to each other. GPT breaks text into **tokens**, small chunks of words. Sora breaks videos and images into **patches**, small chunks of picture across space and time. Because every video becomes the same kind of patch, OpenAI could train Sora on clips of different lengths, resolutions and shapes, from wide screen to vertical.

Two more design choices stand out. First, Sora creates the whole clip at once rather than frame by frame, and can see many frames ahead. That helps with a classic problem: a person who walks behind a tree should come out the other side looking the same. Second, it borrows a trick from DALL·E 3 called **recaptioning**: the training videos were given long, detailed descriptions, which helps the model follow your words more faithfully.

## What it still gets wrong

OpenAI is open about Sora's weaknesses, and its sample page shows failures alongside successes:

- **Physics.** It may not model cause and effect: someone takes a bite of a cookie, and the cookie stays whole.
- **Space.** It can confuse left and right.
- **Time.** It struggles with precise sequences, such as a specific camera path.
- **Objects.** In one example a chair bends like rubber; in others, animals or people appear from nowhere in crowded scenes.

## Safety plans and reactions

Before Sora appears in any OpenAI product, the company lists several steps: the red-team testing, a **detection classifier** meant to recognize Sora-made videos, and plans to add **C2PA** metadata, an industry standard label that records where a piece of media came from, if it ships the model. It will also reuse the filters built for DALL·E 3: one that rejects prompts asking for extreme violence, sexual content, hateful imagery, celebrity likenesses or other people's intellectual property, and image classifiers that check every frame before a video is shown.

Reactions were mixed. Axios describes responses ranging from excitement and awe to fear and revulsion, including filmmakers who object that such systems learn from artists' work without permission. Other companies, including Meta, Google and Runway, have shown or released their own text-to-video systems, so Sora enters a field that is already moving fast.

> **Try it:** Write a one-paragraph video prompt for a scene you know well, such as your street on a winter morning. Include the subject, the setting, the light, the camera movement and the mood. You cannot run it on Sora yet, but it will show you how much detail a video prompt needs.

## How can you prepare to use text-to-video tools like Sora?

Start by practicing descriptive prompts. Video models respond to the same elements a film crew discusses: who or what is in the shot, where it happens, the time of day and lighting, how the camera moves, and how long the moment lasts. Writing these down clearly is a skill you can build now with image generators.

Next, learn to read AI video critically. Look for the weaknesses OpenAI lists: objects that change shape, people who appear from nowhere, actions with no consequence. Those signs will help you judge clips you see online, and the C2PA labels OpenAI mentions may help too, once they are in use.

Finally, think about where video would genuinely help you, such as a short explainer for a class or a mock-up for a project, and where it would raise questions of consent or honesty. When tools like Sora do open to the public, you will be ready to use them well.

## Sources

- Sora: https://openai.com/sora
- OpenAI joins race to make videos from text prompts: https://www.axios.com/2024/02/15/openai-text-to-video-sora