# What did OpenAI announce at DevDay 2024, and what is the Realtime API?

Date: 2024-10-02 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-openai-realtime-api-devday-2024/

**Answer:** At DevDay 2024 in San Francisco on October 1, OpenAI released the Realtime API, which lets developers build apps where people talk with GPT-4o and hear it answer almost instantly, plus three tools to cut costs and customize models: prompt caching, vision fine-tuning and model distillation. There were no new models this year; the focus was on making voice apps possible and existing models cheaper to use well.

Yesterday, Tuesday, October 1, 2024, OpenAI held its second DevDay, its conference for developers, the people who build apps on its models. The event took place in San Francisco. The main announcement was the **Realtime API**, a way for apps to hold spoken conversations with GPT-4o, similar to the Advanced Voice Mode in ChatGPT. Around it came three tools aimed at cost and customization: **prompt caching**, **vision fine-tuning** and **model distillation**.

Unlike last year's DevDay, OpenAI did not announce a new model. It says more than 3 million developers now build with its models, and that the cost of using its API has fallen 99% over the past two years.

## What was announced

| Announcement | What it does | Who can use it |
|---|---|---|
| Realtime API (public beta) | Speech in, speech out, with low delay, through one connection to GPT-4o | All paid developers, rolling out from October 1 |
| Audio in Chat Completions | Send text or audio to GPT-4o, get text, audio or both back | Coming in the next few weeks |
| Prompt caching | Automatically discounts input text the model has seen recently, by 50% | Developers using supported models |
| Vision fine-tuning | Train GPT-4o further on your own images as well as text | Developers |
| Model distillation | Use a large model's answers to train a smaller one, with tools to store examples and test results | Developers |

## How the Realtime API works

Until now, building a voice assistant meant chaining three separate models. One turned speech into text (speech recognition, such as OpenAI's Whisper). A second read the text and wrote a reply. A third turned the reply back into speech. Each handoff added delay, and the middle model never heard your voice, so tone, emphasis and accent were lost.

The Realtime API replaces the chain with one model that hears and speaks. Your app opens a **WebSocket**, a connection that stays open so both sides can send messages at any moment, much like a phone line compared with sending letters. Audio streams in both directions, and the model can handle being interrupted mid-sentence, as people do in conversation.

It also supports **function calling**, where the model asks your app to do something, such as check an order or book a table. TechCrunch describes a demo by Romain Huet, OpenAI's head of developer experience: a travel app where you plan a trip to London by voice, and the assistant marks restaurants on a map as it talks. Another demo had the assistant phone a person to order food for an event, using Twilio, a calling service.

Developers choose from six preset voices. OpenAI says they differ from the ones in ChatGPT, and outside voices are not allowed.

**Prices** are set per token, the small units of text or sound the model reads and writes:

| | Input | Output |
|---|---|---|
| Text | $5 per million tokens | $20 per million tokens |
| Audio | $100 per million tokens (about 6 cents a minute) | $200 per million tokens (about 24 cents a minute) |

Early users include Healthify, a nutrition coaching app, and Speak, a language-learning app that uses it for role-play practice.

## The three money-saving tools

**Prompt caching.** Many apps send the same long instructions or documents with every request. Now, when a request repeats input the model has processed recently, those repeated tokens cost half price, with no extra setup. TechCrunch notes that Anthropic offers a similar feature with a 90% discount.

**Vision fine-tuning.** Fine-tuning means training a model further on your own examples. Developers can now include images, which should help with tasks like reading a particular kind of form or spotting items in product photos. OpenAI's Olivier Godement told TechCrunch that copyrighted images, violent images and others that break its policies are not accepted.

**Model distillation.** A large model is accurate but costly; a small one is cheap but less capable. Distillation uses the large model's answers as lessons for the small one. OpenAI's new tools make this a workflow on its platform: **stored completions** saves your real requests and answers so they can become training data, and a beta **Evals** tool measures whether the smaller model is good enough. At a session, OpenAI staff said distillation suits narrow tasks and works best with thousands of examples rather than millions. For example, you could have GPT-4o or o1-preview answer customer questions for a week, then use those pairs to train GPT-4o mini.

## Why it matters this week

Voice is the headline. Apps for language learning, customer service, accessibility and coaching can now talk with people far more naturally, without stitching together three systems. That also raises questions. OpenAI's policies require developers to tell users they are speaking with AI unless it is obvious, but TechCrunch points out that the models do not announce themselves on calls automatically; it is up to developers.

The other tools are less visible but practical. They help businesses do more with smaller, cheaper models, which is where much of everyday AI use happens.

> **Try it:** Think of one phone call or spoken exchange you repeat often, such as booking an appointment. Write the five pieces of information the conversation must collect. That list is the core of a voice assistant's instructions and the functions it would need.

## How can developers start building with the Realtime API?

OpenAI says developers can try it in the Playground over the coming days, or start from its documentation and a reference client on GitHub. The model is called `gpt-4o-realtime-preview`. Partners LiveKit and Agora offer audio libraries that handle echo cancellation and reconnection, and Twilio connects it to phone calls.

Start small: a single task, a clear system prompt and one or two functions. For example:

```prompt
You are a friendly booking assistant for a hair salon. Ask for the customer's name, preferred day and service, one question at a time. Confirm the details back before calling the booking function. Tell the caller at the start that you are an AI assistant.
```

Keep an eye on cost, since audio is priced well above text, and on limits: during the beta, the top usage tier allows about 100 sessions at once. OpenAI plans to add vision and video, prompt caching and GPT-4o mini support to the Realtime API later.

## Sources

- Introducing the Realtime API: https://openai.com/index/introducing-the-realtime-api/
- OpenAI's DevDay brings Realtime API and other treats for AI app developers: https://techcrunch.com/2024/10/01/openais-devday-brings-realtime-api-and-other-treats-for-ai-app-developers/
- OpenAI DevDay 2024 live blog: https://simonwillison.net/2024/Oct/1/openai-devday-2024-live-blog/