# How do you build a voice agent with GPT-Realtime-2?

Date: 2026-05-20 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/build-voice-agent-gpt-realtime-2/

**Answer:** Design the call flow first (greeting, what the agent may look up, what it must confirm, when it hands over to a person), then connect to OpenAI's Realtime API with the model gpt-realtime-2, describe your tools in a session.update event and answer each function call from your own code. Start with the default low reasoning effort and raise it only for steps that fail in testing.

A **voice agent** is a program you talk to out loud that can also do things: look up an order, book a slot, send a confirmation. OpenAI's **Realtime API** lets one model listen to speech and answer in speech directly, without a separate transcription step and a separate voice step. On May 7, 2026, OpenAI added GPT-Realtime-2 to that API and described it as its first voice model with "GPT‑5‑class reasoning." In plain terms, it can think a little before it speaks and call tools more reliably than the earlier realtime models.

What makes a voice agent pleasant or frustrating is the design around the model: which actions it may take, what it must read back first, and what happens when the caller cuts in or a tool fails.

## What GPT-Realtime-2 changes for builders

OpenAI's announcement lists several features that matter when you plan a call:

| Feature | What it means for your design |
|---|---|
| Reasoning effort: minimal, low, medium, high, xhigh (low by default) | More effort means better answers on hard requests, but slower replies. Keep it low for simple lookups. |
| Preambles | Short spoken phrases such as "one moment, I'm checking" while the model works, so the caller doesn't hear silence. |
| Parallel tool calls | The model can ask your code to run several functions at once, for example checking two dates. |
| 128K-token context (up from 32K) | Longer calls keep more of what was said. A **token** is a small chunk of text or audio, the unit the API counts and bills. |
| Pricing | $32 per million audio input tokens and $64 per million audio output tokens. |

OpenAI's usage policies also require that people know they are talking to an AI unless that is already obvious, so your greeting should say so.

## Step 1: design the call flow on paper

Take a small bakery that wants a phone agent for cake pre-orders. Before writing code, write the flow:

1. **Greet** and say it is an automated assistant.
2. **Collect** the cake type, size, pickup date and the caller's name.
3. **Check** availability with a tool: `check_pickup_slot(date)`.
4. **Read back** every detail and wait for a clear "yes".
5. **Act** with a second tool: `place_preorder(...)`.
6. **Hand over** to staff for allergies, wedding cakes or anything the agent is unsure about.

The rule in step 4 matters most. Reading back names, dates and numbers before any action that changes something is what keeps a misheard "fifteenth" from becoming a "fiftieth".

## Step 2: connect and configure the session

For a server program, OpenAI's documentation recommends a **WebSocket**, a connection that stays open so both sides can send messages at any time. (For browser and mobile apps it recommends WebRTC instead.) Every message is a JSON **event**. Here is a minimal Node.js server using the `ws` package, with the bakery's two tools:

```javascript
import WebSocket from "ws";

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2", {
  headers: { Authorization: "Bearer " + process.env.OPENAI_API_KEY },
});

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      type: "realtime",
      model: "gpt-realtime-2",
      output_modalities: ["audio"],
      audio: {
        input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: { type: "semantic_vad" } },
        output: { voice: "marin" },
      },
      instructions: "You are the automated pre-order assistant for a bakery. Say you are an AI at the start. " +
        "Collect cake type, size, pickup date and name. Before calling place_preorder, read every detail back " +
        "and wait for a clear yes. For allergies or wedding cakes, offer to transfer to staff.",
      tools: [
        { type: "function", name: "check_pickup_slot", description: "Check if a pickup date has space.",
          parameters: { type: "object", properties: { date: { type: "string" } }, required: ["date"] } },
        { type: "function", name: "place_preorder", description: "Save a confirmed pre-order.",
          parameters: { type: "object", properties: {
            cake: { type: "string" }, size: { type: "string" }, date: { type: "string" }, name: { type: "string" } },
            required: ["cake", "size", "date", "name"] } },
      ],
      tool_choice: "auto",
    },
  }));
});
```

`semantic_vad` is the **voice activity detection** setting: the API decides when the caller has finished speaking and when to answer. Audio travels as Base64-encoded chunks in `input_audio_buffer.append` events (to the model) and `response.output_audio.delta` events (from it).

## Step 3: answer tool calls from your code

The model never touches your database. When it wants a tool, the `response.done` event contains an item of type `function_call` with a name, its arguments as a JSON string, and a `call_id`. Your code runs the function, sends back the result with the same `call_id`, then asks the model to continue:

```javascript
ws.on("message", async (raw) => {
  const event = JSON.parse(raw.toString());
  if (event.type === "response.done") {
    for (const item of event.response.output) {
      if (item.type !== "function_call") continue;
      const args = JSON.parse(item.arguments);
      const result = await runTool(item.name, args);   // your own code: calendar, order system
      ws.send(JSON.stringify({ type: "conversation.item.create",
        item: { type: "function_call_output", call_id: item.call_id, output: JSON.stringify(result) } }));
    }
    if (event.response.output.some(i => i.type === "function_call")) {
      ws.send(JSON.stringify({ type: "response.create" }));
    }
  }
});
```

> **Try it:** Open OpenAI's Playground, choose GPT-Realtime-2, paste the bakery instructions above and order a cake out loud. Halfway through, change your pickup date. Note whether the agent reads the new date back before it would place the order, and whether it admits being an AI in its greeting.

## How do you handle interruptions and errors?

Callers interrupt, and a good agent stops talking when they do. With voice activity detection on, the API notices the caller speaking (an `input_audio_buffer.speech_started` event) and cancels its own answer. Over WebSocket, though, your program is the one playing the audio, so it has two jobs: stop playback at once, and send a `conversation.item.truncate` event with the id of the model's last reply and how many milliseconds were actually played. That trims the unheard part from the conversation, so the model doesn't assume the caller heard a sentence that was cut off. With WebRTC or a phone (SIP) connection, the server tracks playback and trims automatically.

For errors, the server sends an `error` event. If you add your own `event_id` to each event you send, the error will carry the same id, so you can tell which request failed. Tool failures are different: they are your code's problem, and the kindest fix is to return a clear result such as `{"ok": false, "reason": "calendar unavailable"}` rather than nothing. The model can then say it is having trouble and offer the staff hand-over; OpenAI lists better recovery among GPT-Realtime-2's improvements. Log every tool call, remember that sessions end at 60 minutes, and test the awkward cases: a mumbled date, a changed mind, a tool that times out.

## Sources

- Advancing voice intelligence with new models in the API: https://openai.com/index/advancing-voice-intelligence-with-new-models-in-the-api/
- Realtime conversations (Managing conversations): https://developers.openai.com/api/docs/guides/realtime-conversations
- Realtime API with WebSocket: https://developers.openai.com/api/docs/guides/realtime-websocket