How do you build a voice agent with GPT-Realtime-2?

A voice agent is a program you talk to out loud that can also do things: look up an order, book a slot, send a confirmation. OpenAI's Realtime API lets one model listen to speech and answer in speech directly, without a separate transcription step and a separate voice step. On May 7, 2026, OpenAI added GPT-Realtime-2 to that API and described it as its first voice model with "GPT‑5‑class reasoning." In plain terms, it can think a little before it speaks and call tools more reliably than the earlier realtime models.

What makes a voice agent pleasant or frustrating is the design around the model: which actions it may take, what it must read back first, and what happens when the caller cuts in or a tool fails.

What GPT-Realtime-2 changes for builders

OpenAI's announcement lists several features that matter when you plan a call:

Feature	What it means for your design
Reasoning effort: minimal, low, medium, high, xhigh (low by default)	More effort means better answers on hard requests, but slower replies. Keep it low for simple lookups.
Preambles	Short spoken phrases such as "one moment, I'm checking" while the model works, so the caller doesn't hear silence.
Parallel tool calls	The model can ask your code to run several functions at once, for example checking two dates.
128K-token context (up from 32K)	Longer calls keep more of what was said. A token is a small chunk of text or audio, the unit the API counts and bills.
Pricing	$32 per million audio input tokens and $64 per million audio output tokens.

OpenAI's usage policies also require that people know they are talking to an AI unless that is already obvious, so your greeting should say so.

Step 1: design the call flow on paper

Take a small bakery that wants a phone agent for cake pre-orders. Before writing code, write the flow:

Greet and say it is an automated assistant.
Collect the cake type, size, pickup date and the caller's name.
Check availability with a tool: check_pickup_slot(date).
Read back every detail and wait for a clear "yes".
Act with a second tool: place_preorder(...).
Hand over to staff for allergies, wedding cakes or anything the agent is unsure about.

The rule in step 4 matters most. Reading back names, dates and numbers before any action that changes something is what keeps a misheard "fifteenth" from becoming a "fiftieth".

Step 2: connect and configure the session

For a server program, OpenAI's documentation recommends a WebSocket, a connection that stays open so both sides can send messages at any time. (For browser and mobile apps it recommends WebRTC instead.) Every message is a JSON event. Here is a minimal Node.js server using the ws package, with the bakery's two tools:

import WebSocket from "ws";

const ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2", {
  headers: { Authorization: "Bearer " + process.env.OPENAI_API_KEY },
});

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      type: "realtime",
      model: "gpt-realtime-2",
      output_modalities: ["audio"],
      audio: {
        input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: { type: "semantic_vad" } },
        output: { voice: "marin" },
      },
      instructions: "You are the automated pre-order assistant for a bakery. Say you are an AI at the start. " +
        "Collect cake type, size, pickup date and name. Before calling place_preorder, read every detail back " +
        "and wait for a clear yes. For allergies or wedding cakes, offer to transfer to staff.",
      tools: [
        { type: "function", name: "check_pickup_slot", description: "Check if a pickup date has space.",
          parameters: { type: "object", properties: { date: { type: "string" } }, required: ["date"] } },
        { type: "function", name: "place_preorder", description: "Save a confirmed pre-order.",
          parameters: { type: "object", properties: {
            cake: { type: "string" }, size: { type: "string" }, date: { type: "string" }, name: { type: "string" } },
            required: ["cake", "size", "date", "name"] } },
      ],
      tool_choice: "auto",
    },
  }));
});

semantic_vad is the voice activity detection setting: the API decides when the caller has finished speaking and when to answer. Audio travels as Base64-encoded chunks in input_audio_buffer.append events (to the model) and response.output_audio.delta events (from it).

Step 3: answer tool calls from your code

The model never touches your database. When it wants a tool, the response.done event contains an item of type function_call with a name, its arguments as a JSON string, and a call_id. Your code runs the function, sends back the result with the same call_id, then asks the model to continue:

ws.on("message", async (raw) => {
  const event = JSON.parse(raw.toString());
  if (event.type === "response.done") {
    for (const item of event.response.output) {
      if (item.type !== "function_call") continue;
      const args = JSON.parse(item.arguments);
      const result = await runTool(item.name, args);   // your own code: calendar, order system
      ws.send(JSON.stringify({ type: "conversation.item.create",
        item: { type: "function_call_output", call_id: item.call_id, output: JSON.stringify(result) } }));
    }
    if (event.response.output.some(i => i.type === "function_call")) {
      ws.send(JSON.stringify({ type: "response.create" }));
    }
  }
});

Try it: Open OpenAI's Playground, choose GPT-Realtime-2, paste the bakery instructions above and order a cake out loud. Halfway through, change your pickup date. Note whether the agent reads the new date back before it would place the order, and whether it admits being an AI in its greeting.

How do you handle interruptions and errors?

Callers interrupt, and a good agent stops talking when they do. With voice activity detection on, the API notices the caller speaking (an input_audio_buffer.speech_started event) and cancels its own answer. Over WebSocket, though, your program is the one playing the audio, so it has two jobs: stop playback at once, and send a conversation.item.truncate event with the id of the model's last reply and how many milliseconds were actually played. That trims the unheard part from the conversation, so the model doesn't assume the caller heard a sentence that was cut off. With WebRTC or a phone (SIP) connection, the server tracks playback and trims automatically.

For errors, the server sends an error event. If you add your own event_id to each event you send, the error will carry the same id, so you can tell which request failed. Tool failures are different: they are your code's problem, and the kindest fix is to return a clear result such as {"ok": false, "reason": "calendar unavailable"} rather than nothing. The model can then say it is having trouble and offer the staff hand-over; OpenAI lists better recovery among GPT-Realtime-2's improvements. Log every tool call, remember that sessions end at 60 minutes, and test the awkward cases: a mumbled date, a changed mind, a tool that times out.