How do you build a voice assistant with the Realtime API?

Until this month, a voice assistant built on OpenAI's models was a relay race between three of them: a speech recognition model such as Whisper turned speech into text, a text model wrote the reply, and a text-to-speech model read it aloud. Each handoff added delay, and the middle model never heard the caller's tone. The Realtime API, in public beta for all paid developers since October 1, replaces that chain with one model, GPT-4o, that listens and speaks directly.

This guide walks through the architecture of a small phone assistant for an imaginary bike repair shop that answers questions about opening hours. The same pattern works for a website or an app; only the audio plumbing changes.

The architecture: two WebSockets and a relay

A WebSocket is a connection that stays open so both sides can send messages at any moment, which is what a live conversation needs. A phone assistant uses two of them, with your server in the middle:

1. The caller dials your phone number. Twilio, a telephony service that partnered with OpenAI for the launch, answers and, following your instructions, opens a "media stream": a WebSocket that carries the call's audio to your server.
2. Your server opens a second WebSocket to OpenAI, at the Realtime endpoint, with your API key and a beta header.
3. Your server relays audio both ways. Caller audio goes to OpenAI as input_audio_buffer.append events. The model's speech comes back in response.audio.delta events, which you forward to Twilio.
4. OpenAI decides when the caller has finished speaking. With "server VAD" (voice activity detection) turned on, the API detects pauses and starts replying by itself.
5. Tools handle facts. When the caller asks something your system knows, such as today's hours, the model requests a function call; your server runs it and returns the result.

Phone networks carry audio in a compressed format called G.711 μ-law. The Realtime API accepts it directly (g711_ulaw), so no conversion is needed.

A worked example in Node.js

This sketch shows the OpenAI side of the relay, with one tool. It uses the ws library and the beta model name from launch, and leaves out the Twilio route for brevity; Twilio's own sample on GitHub shows that part.

import WebSocket from "ws";

const HOURS = { mon: "closed", tue: "9-18", wed: "9-18", thu: "9-20", fri: "9-20", sat: "10-16", sun: "closed" }; // illustrative

export function connectToOpenAI(sendAudioToCaller) {
  const ws = new WebSocket(
    "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01",
    { headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}`, "OpenAI-Beta": "realtime=v1" } }
  );

  ws.on("open", () => {
    ws.send(JSON.stringify({
      type: "session.update",
      session: {
        instructions: "You are the phone assistant for Rue Verte Bikes. Say you are an AI assistant at the start. Keep answers to two sentences. Use get_hours for opening hours; never guess.",
        voice: "alloy",
        modalities: ["text", "audio"],
        input_audio_format: "g711_ulaw",
        output_audio_format: "g711_ulaw",
        turn_detection: { type: "server_vad" },
        tools: [{
          type: "function",
          name: "get_hours",
          description: "Opening hours for one day of the week",
          parameters: {
            type: "object",
            properties: { day: { type: "string", enum: Object.keys(HOURS) } },
            required: ["day"]
          }
        }],
        tool_choice: "auto"
      }
    }));
  });

  ws.on("message", (raw) => {
    const event = JSON.parse(raw);
    if (event.type === "response.audio.delta") {
      sendAudioToCaller(event.delta);                 // base64 audio for Twilio
    }
    if (event.type === "response.output_item.done" && event.item.type === "function_call") {
      const { day } = JSON.parse(event.item.arguments);
      ws.send(JSON.stringify({
        type: "conversation.item.create",
        item: { type: "function_call_output", call_id: event.item.call_id, output: JSON.stringify({ day, hours: HOURS[day] }) }
      }));
      ws.send(JSON.stringify({ type: "response.create" }));   // ask the model to speak the answer
    }
    if (event.type === "error") console.error(event.error);
  });

  return (base64Audio) =>                                   // call this for each audio chunk from Twilio
    ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: base64Audio }));
}

The flow of a call: a caller asks, "Are you open Thursday evening?" The API detects the end of the question and the model requests get_hours with {"day": "thu"}. Your code answers with the stored hours, and the model says something like "Yes, we're open until 8 on Thursday." The wording varies; this reply is illustrative.

Two details deserve attention. First, the instruction to say it is an AI is not just polite: OpenAI's usage policies require developers to make it clear to users that they are talking to AI, unless that is obvious from context. Second, callers interrupt. When the API sends input_audio_buffer.speech_started while the assistant is still talking, Twilio's sample stops playback and sends conversation.item.truncate so the model knows how much of its reply was actually heard.

What does a voice minute cost?

OpenAI prices the Realtime API by tokens, the units its models read and write, with audio tokens costing far more than text. Audio input is $100 per million tokens and audio output $200 per million. OpenAI translates that into roughly $0.06 per minute of audio coming in and $0.24 per minute of audio going out. Text tokens, used for instructions and tool results, cost $5 per million in and $20 per million out.

For a rough feel, take a five-minute call in which the caller speaks for three minutes and the assistant for two. That is about 3 × $0.06 + 2 × $0.24, or around $0.66 in audio, plus a little text. These figures are an illustration from the per-minute estimates, not a quote. The assistant's speaking time drives the bill, which is one more reason to ask for short answers. Earlier turns are also processed again as the conversation grows; OpenAI says it plans to add prompt caching so those repeats cost less.

Your telephony provider charges separately for the phone number and call minutes. Rate limits matter too: at launch, Tier 5 accounts can run about 100 simultaneous sessions, with lower limits for other tiers. Test with a handful of calls, read the usage page in your OpenAI dashboard, and set a monthly spending limit before you publish the number.