OpenAI Realtime API: Voice Agents End-to-End

Most voice AI tutorials teach the same architecture: record audio → transcribe with Whisper → send text to GPT → synthesize with a TTS service → play back audio. The pipeline feels logical because each component is familiar. The problem is latency: three API calls, three network hops, three serialization steps — and a minimum floor of around 1,000 ms even on a fast connection.

The OpenAI Realtime API is a categorical break from that pattern. It runs a single speech-to-speech session where audio goes in, reasoning happens, tools fire, and audio comes out — without ever converting your voice to text and back. The result is a time-to-first-audio of around 500 ms from US datacenters under good conditions. (Latent.Space, "Realtime API: The Missing Manual") (retrieved 2026-06-15)

This chapter maps the architecture difference, introduces the six layers every production voice agent needs, and walks you through building your first working session.

Why the Three-Hop Pipeline Always Loses on Latency

To see the problem concretely, time each stage of a classic pipeline:

(Latent.Space, "Realtime API: The Missing Manual") (retrieved 2026-06-15)

Each stage has its own queue, connection, and tokenization cost. The stages also can't overlap: you can't start TTS until GPT finishes, and GPT can't start until Whisper returns. The pipeline is sequential by design.

The Realtime API collapses all three stages into one stateful connection. The model receives audio tokens directly, reasons over them with full context, and emits audio tokens that begin playing within ~500 ms of your speech ending. (Skywork, "OpenAI Realtime API vs WebRTC 2025") (retrieved 2026-06-15) No re-serialization. No queue handoffs.

The Six Layers of a Production Voice Agent

A "voice agent" is not one component — it is a stack. Understanding all six layers is what separates a demo from a deployment.

┌──────────────────────────────────────┐
│  1. Client                           │  Browser / mobile / SIP phone
│  2. Edge media                       │  WebRTC or WebSocket transport
│  3. Agent runtime                    │  Your server: session mgmt, tools
│  4. Model API                        │  OpenAI Realtime API (gpt-realtime-2)
│  5. Tool plane                       │  Functions, databases, third-party APIs
│  6. Observability                    │  Latency tracing, cost metering, audit logs
└──────────────────────────────────────┘

Layer 1 — Client. The microphone and speaker. In a browser this is the Web Audio API; in a mobile app it is the platform audio stack; in telephony it is a SIP bridge. The client is responsible for audio capture quality (sample rate, noise suppression) and playback buffering. Bad audio in means bad understanding out, regardless of model quality.

Layer 2 — Edge media. How audio travels between the client and your server (and onward to the model API). The two options — WebRTC and WebSocket — are the first major architectural decision you will make, and the next section covers the trade-off.

Layer 3 — Agent runtime. Your server code. It manages the session lifecycle (connect, authenticate, configure), dispatches tool calls without blocking the audio stream, and handles reconnects. This is where most production bugs live.

Layer 4 — Model API. The OpenAI Realtime API. As of June 2026, the recommended model is gpt-realtime-2 — released May 7, 2026 with GPT-5-class reasoning and a 128,000-token context window. (OpenAI, "Advancing voice intelligence with new models in the API") (retrieved 2026-06-15) Its predecessor, gpt-realtime, reached general availability on August 28, 2025. (OpenAI, "Introducing gpt-realtime") (retrieved 2026-06-15) Both support audio input, audio output, text, images, and function calling within a single session. (OpenAI, "Realtime conversations guide") (retrieved 2026-06-15)

Layer 5 — Tool plane. The functions your agent can call: database lookups, CRM reads, calendar writes. Tool latency directly affects perceived response time — the agent cannot speak its answer until the tool returns. For MCP-based tool orchestration patterns that apply here, see Claude MCP Mastery — the dispatch model transfers directly. Chapter 3 covers non-blocking tool dispatch in Realtime API sessions specifically.

Layer 6 — Observability. Timestamp logging at every event boundary, per-session cost metering, and audit logs of all tool calls. Without this layer you cannot diagnose latency regressions or catch runaway costs. Audio tokens are priced at approximately $32 per 1M input tokens and $64 per 1M output tokens for gpt-realtime-2 (OpenAI, "API Pricing") (retrieved 2026-06-15), with audio consuming roughly 800 tokens per minute per channel. A 10-minute session without any caching runs about $0.77 in audio tokens alone — multiply by concurrent sessions and observability becomes financial hygiene. Chapter 5 builds this out.

WebRTC vs WebSocket: One Decision, Many Consequences

The Realtime API supports both WebRTC and WebSocket as first-class transports with the same event schema. Your application-layer event handling code looks nearly identical — but the underlying behavior is very different.

Choose WebRTC when your client is a browser or mobile app on variable network conditions — home WiFi, LTE, 5G. WebRTC's congestion control automatically degrades audio quality to maintain low latency rather than stalling. A dropped audio packet in a conversational stream is less harmful than a 200 ms TCP retransmission freeze. WebRTC also handles firewall traversal through ICE negotiation without you configuring anything.

Choose WebSocket when you are running server-to-server: a telephony bridge (SIP to Realtime API), a backend pipeline that post-processes audio before sending, or any scenario where you want fine-grained control over every frame. WebSocket is also the simpler transport for getting started — no ICE negotiation, no SDP offer/answer, just a connection and a message loop.

Hello World: Your First Voice Session

Before any architecture, you need a working session. The script below connects to the Realtime API over WebSocket, streams a 3-second audio clip, and timestamps the first response.output_audio.delta event. That timestamp is your baseline RTT — you will optimize it in Chapter 4.

Prerequisites: - OpenAI API key with Realtime API access (check platform.openai.com — Realtime is in the API dashboard) - Python 3.10+ with websockets installed (pip install 'websockets>=12.0') — additional_headers requires v12+ - A 3-second mono 24 kHz PCM16 audio clip saved as hello.pcm (raw bytes, no WAV header). Generate with: ffmpeg -i any_audio.wav -f s16le -ar 24000 -ac 1 hello.pcm

API_KEY = os.environ["OPENAI_API_KEY"] # never hardcode keys; add import os at top MODEL = "gpt-realtime-2" WS_URL = f"wss://api.openai.com/v1/realtime?model={MODEL}"

async def main(): headers = { "Authorization": f"Bearer {API_KEY}", }

async with websockets.connect(WS_URL, additional_headers=headers) as ws: # 1. Configure session: audio in + audio out, manual turn detection await ws.send(json.dumps({ "type": "session.update", "session": { "modalities": ["audio", "text"], "voice": "alloy", "turn_detection": None, # we commit manually } }))

for i in range(0, len(pcm), 4800): chunk_b64 = base64.b64encode(pcm[i : i + 4800]).decode() await ws.send(json.dumps({ "type": "input_audio_buffer.append", "audio": chunk_b64, }))

asyncio.run(main()) ```

Expected output (US datacenter, good broadband): `` [1718444400.123] Committed audio, waiting… [1718444400.623] First audio delta received Time to first audio: 500 ms `` </RunPromptCell>

Write down your result. If you see 400–600 ms, your setup is healthy. Above 1,000 ms suggests a regional routing issue or an oversized audio chunk stalling the first parse — reduce your chunk size from 4800 to 2400 bytes and retry.

<Callout type="hot"> Common error: {"type": "error", "error": {"message": "audio buffer empty"}} — this means the base64 payload arrived malformed or the PCM file was a WAV file (with a 44-byte header). Strip the WAV header before encoding: pcm = f.read()[44:]. </Callout>

Session Lifecycle: What Happens Under the Hood

Every Realtime API session goes through the same lifecycle. Knowing this prevents the most common bugs.

1. Connect — WebSocket handshake to wss://api.openai.com/v1/realtime?model=gpt-realtime-2. Authenticate via Authorization: Bearer <key>.
2. `session.created` — Server confirms the session with default settings (voice, VAD threshold, modalities).
3. `session.update` — You configure the session before any audio: voice, VAD sensitivity, system instructions, tool definitions.
4. `input_audio_buffer.append` — Stream audio chunks continuously.
5. Turn commit — Either input_audio_buffer.commit (manual mode) or the server's VAD fires automatically.
6. `response.create` — Model starts generating. First response.output_audio.delta arrives in ~500 ms.
7. `response.output_audio.delta` stream — Base64-encoded PCM chunks. Decode and queue for playback.
8. `response.done` — Model finished its turn. Session stays open — continue the conversation.
9. Close — Either party closes the WebSocket. Maximum session duration: 60 minutes. (OpenAI, "Realtime conversations guide") (retrieved 2026-06-15)

Sessions carry a 128,000-token context window and consume approximately 800 audio tokens per minute per channel of speech. A 10-minute conversation with both parties active occupies roughly 16,000 audio tokens (8,000 input + 8,000 output) — comfortable within the context window, but track it if you are mixing audio with long tool results. (OpenAI, "Advancing voice intelligence with new models in the API") (retrieved 2026-06-15)

Expected event sequence for a single turn: `` session.created session.updated conversation.item.created response.created rate_limits.updated response.output_item.added conversation.item.created response.content_part.added response.audio_transcript.delta response.output_audio.delta ← first audio here, ~500 ms after response.create response.output_audio.delta … response.output_audio.done response.done `` </RunPromptCell>

Hands-On Exercise

Goal: Measure your baseline round-trip time and compare WebSocket vs WebRTC.

Steps: 1. Run hello_realtime.py with a 3-second hello.pcm clip. Record time-to-first-audio. 2. Run it 5 more times and compute the median. Write down your P50 and P95. 3. If you have a browser environment available, connect to the Realtime API via WebRTC using the OpenAI WebRTC quickstart and measure the same TTFA. Compare to your WebSocket baseline.

Success criteria: You have a recorded TTFA for WebSocket, understand which segment of the session lifecycle produces it, and can explain in one sentence why your P95 is higher than your P50.

Next chapter: 02-hello-world-websocket — continuous audio capture, server-side VAD for automatic turn detection, and real-time audio playback.

Chapter Summary

Connect a persistent WebSocket to the OpenAI Realtime API, stream microphone audio in real time, and play back speech deltas as they arrive — with under 700 ms response latency from end of speech to first audio byte. This chapter delivers that session end-to-end: continuous 24 kHz PCM streaming, event-driven response handling, server-side VAD for automatic turn detection, barge-in cancellation via response.cancel, and a PTT fallback you can toggle at runtime without reconnecting.

By the end you have a working Python voice agent you can run against a live microphone, a clear model of the three turn_detection parameters that control how aggressive VAD fires, and the data you need to decide which mode ships as the default in your product.

Connecting a Continuous Voice Session

Chapter 1 showed a one-shot audio send. This chapter replaces that with a proper continuous voice loop: the client captures audio in real time, streams it to the API as fast as it arrives, and handles responses asynchronously on the same WebSocket connection.

The full session script below is the foundation for every chapter that follows. Read through it before running it — the inline comments map directly to the event schema section that follows.

API_KEY = "sk-..." MODEL = "gpt-realtime-2" WS_URL = f"wss://api.openai.com/v1/realtime?model={MODEL}"

pa = pyaudio.PyAudio()

async def capture_and_stream(ws, stop: asyncio.Event): """Reads mic audio and streams base64-encoded chunks to the WebSocket.""" stream = pa.open(format=pyaudio.paInt16, channels=CHANNELS, rate=SAMPLE_RATE, input=True, frames_per_buffer=CHUNK_FRAMES) try: while not stop.is_set(): chunk = stream.read(CHUNK_FRAMES, exception_on_overflow=False) await ws.send(json.dumps({ "type": "input_audio_buffer.append", "audio": base64.b64encode(chunk).decode(), })) await asyncio.sleep(0) # yield to event loop finally: stream.stop_stream(); stream.close()

async def receive_events(ws, stop: asyncio.Event): """Handles all server events and plays back audio deltas in real time.""" play = pa.open(format=pyaudio.paInt16, channels=CHANNELS, rate=SAMPLE_RATE, output=True) try: async for raw in ws: event = json.loads(raw) t = event["type"]

if t == "session.created": print("Session ready:", event["session"]["id"]) elif t == "input_audio_buffer.speech_started": print("[VAD] Speech detected — cancelling any active response") await ws.send(json.dumps({"type": "response.cancel"})) elif t == "input_audio_buffer.speech_stopped": print("[VAD] End of turn — waiting for response") elif t == "response.audio.delta": play.write(base64.b64decode(event["delta"])) # real-time playback elif t == "response.done": print("[Done] Agent finished speaking") elif t == "error": print("Error:", event["error"]["message"]) stop.set(); break finally: play.stop_stream(); play.close()

async def main(): headers = { "Authorization": f"Bearer {API_KEY}", } async with websockets.connect(WS_URL, additional_headers=headers) as ws: await ws.send(json.dumps({ "type": "session.update", "session": { "modalities": ["audio", "text"], "voice": "alloy", "turn_detection": { "type": "server_vad", "threshold": 0.5, "prefix_padding_ms": 300, "silence_duration_ms": 600, }, "max_response_output_tokens": 800, } })) stop = asyncio.Event() await asyncio.gather(capture_and_stream(ws, stop), receive_events(ws, stop))

asyncio.run(main()) ```

To run: pip install websockets pyaudio then python continuous_voice.py. Speak into your microphone — the agent responds in the alloy voice after VAD detects your turn ending. Press Ctrl-C to stop.

gpt-realtime-2 was released May 7, 2026 (OpenAI, "Advancing voice intelligence with new models in the API", openai.com, retrieved 2026-06-15) and is the recommended model identifier for all new Realtime API sessions; its predecessor gpt-realtime reached GA on August 28, 2025 (OpenAI, "Introducing gpt-realtime", openai.com, 2025-08-28). For rate limits and tier details see the model spec (OpenAI, "GPT-Realtime-2 Model", developers.openai.com, retrieved 2026-06-15). (OpenAI, "Realtime Sessions", developers.openai.com, retrieved 2026-06-15) </RunPromptCell>

The Realtime API Event Schema

Every interaction with the Realtime API is a JSON event sent or received over the WebSocket connection. Understanding which event does what separates a 5-minute debug from a 5-hour one. (OpenAI Realtime API, "Client Events", developers.openai.com, retrieved 2026-06-15)

There are three event families in scope for this chapter:

*`input_audio_buffer.` — audio you send and the server's response to it**

*`response.` — the agent's reply stream**

*`conversation.item.` — the persistent conversation history**

The conversation history is stored server-side across the session's 128,000-token context window (OpenAI, "Realtime Sessions", developers.openai.com, retrieved 2026-06-15). Audio consumes approximately 800 tokens per minute (OpenAI, "Realtime Guide", developers.openai.com, retrieved 2026-06-15) — at the June 2026 list price of $32/1M audio-input tokens (OpenAI, "API Pricing", openai.com, retrieved 2026-06-15) that is roughly $0.026 per minute of audio input — so a typical 5-minute conversation uses around 4,000 audio tokens of context — leaving plenty of headroom for tool results and system instructions.

Server-Side VAD and Turn Detection

VAD is what makes a voice agent feel conversational rather than robotic. Without it you need a button press or a fixed silence timer. With it, the model begins responding the moment you stop talking — typically delivering the first audio token within 500–700 ms total. (OpenAI, "Voice Activity Detection (VAD)", developers.openai.com, retrieved 2026-06-15)

The turn_detection object in session.update controls all VAD behavior:

"turn_detection": {
  "type":                "server_vad",
  "threshold":           0.5,
  "prefix_padding_ms":   300,
  "silence_duration_ms": 600
}

• `threshold` (0.0–1.0, default 0.5): The audio energy level above which a frame is classified as speech. Lower values detect whispers but generate more false positives from keyboard noise and background chatter. In a typical call-center or office environment, 0.6–0.7 is a safer starting point than the default.
• `prefix_padding_ms` (default 300): How many milliseconds of audio before the VAD trigger to include in the committed buffer. This prevents clipping the first syllable of an utterance. Setting this above 500 ms wastes context tokens on pre-speech silence.
• `silence_duration_ms` (default 500): How long the audio must stay below threshold before the turn is committed. 500 ms works for deliberate speakers in quiet environments; raise to 700–800 ms if users pause mid-sentence and get interrupted by the agent responding too early.

When VAD fires, the server emits this event sequence automatically — your code does not need to send input_audio_buffer.commit or response.create in VAD mode:

input_audio_buffer.speech_started   ← energy above threshold
input_audio_buffer.speech_stopped   ← sustained silence detected
input_audio_buffer.committed        ← buffer committed by VAD
response.created                    ← model starts processing
response.audio.delta                ← first audio chunk (≈500 ms after commit)

Barge-in handling. When input_audio_buffer.speech_started fires while the agent is still speaking, the correct behavior is to cancel the in-flight response immediately and stop playback. The cancelled response items are not added to the conversation history, so context stays clean.

The key insight: response.cancel is safe to send even when no response is active. It is idempotent. </RunPromptCell>

Managing Session Configuration

The session.update event configures everything about how the session behaves, and you can call it multiple times during a session. This is the mechanism for runtime mode switching — switching VAD to PTT when a user enters a noisy environment, or changing the system prompt when a user authenticates. (OpenAI Realtime API, "session.update", developers.openai.com, retrieved 2026-06-15)

Voice selection. Available voices as of June 2026 include alloy, echo, shimmer, verse, ballad, coral, sage, and ash (OpenAI, "Realtime Sessions", developers.openai.com, retrieved 2026-06-15). Voice cannot be changed mid-session — configure it before the first audio exchange. For enterprise and customer service contexts, alloy and echo test well in user research; shimmer is warmer and more casual.

`max_response_output_tokens`: Caps the agent's response length in tokens (approximately 20 audio tokens per second of speech). Set to 800 for responses under 40 seconds. For conversational agents with long explanations, use 2000 or omit the field entirely. When a response exceeds the cap, response.done arrives with status: "incomplete" — the agent's audio cuts off mid-sentence, which sounds broken to users. Size this conservatively or handle "incomplete" status explicitly.

`input_audio_format` and `output_audio_format`: Default is pcm16 (24 kHz, 16-bit mono). Telephony integrations often require g711_ulaw or g711_alaw (8 kHz, 8-bit) to match PSTN codec standards. Switching to g711 reduces bandwidth by approximately 6× but is audibly lower quality — reserve it for telephony bridges, not browser clients.

`instructions`: The session's system prompt. Set it in session.update, not as a conversation message. You can update it mid-session to implement mode switching — for example, from an "intake" persona to a "billing support" persona after the user authenticates.

Adding Push-to-Talk Mode

PTT is the right choice when VAD cannot be tuned well enough for the deployment environment — high ambient noise, multiple speakers in the room, or accessibility contexts where users prefer explicit control. Switching is a single config change and works seamlessly mid-session.

Set turn_detection to null to disable VAD entirely:

await ws.send(json.dumps({
    "type": "session.update",
    "session": {"turn_detection": None}   # manual mode
}))

In manual mode the server never auto-commits the buffer. Your application controls the turn: send input_audio_buffer.commit followed by response.create when the user releases the PTT trigger.

async def ptt_loop(ws, stop: asyncio.Event): """Polls spacebar state; commits turn on key release.""" was_pressed = False while not stop.is_set(): pressed = keyboard.is_pressed("space") if pressed and not was_pressed: print("[PTT] Recording…") elif not pressed and was_pressed: print("[PTT] Sending turn") await ws.send(json.dumps({"type": "input_audio_buffer.commit"})) await ws.send(json.dumps({"type": "response.create"})) was_pressed = pressed await asyncio.sleep(0.02)

Comparing the two modes. In controlled tests, PTT adds approximately 150 ms to perceived response time (the user must release the key before the turn commits). VAD adds 500–700 ms of silence window but removes the button entirely. Neither is universally better: ship VAD as the default with a clearly labeled PTT fallback toggle in the UI. Without a visible indicator — "Listening…" / "Processing…" — users cannot tell when VAD is waiting for speech versus waiting for the model. That missing indicator is the most common source of user confusion in production voice agents, not VAD threshold tuning.

Hands-on Exercise: Dual-Mode Voice Agent with Runtime Switching

Goal: A single Python script that starts in VAD mode and switches to PTT when the user says "switch to push-to-talk," then back to VAD when they say "switch to voice mode."

Steps:

1. Start from continuous_voice.py from the session connection section above.
2. Add a response.audio_transcript.delta handler that accumulates the agent's spoken text into a string buffer that resets on each response.done event.
3. In the response.done handler, check the accumulated transcript for the phrase "push-to-talk." If found, send session.update with turn_detection: null, print [Mode: PTT], and launch the ptt_loop coroutine.
4. Add a second phrase check for "voice mode" that re-enables VAD with the original threshold and silence_duration_ms values.
5. Ensure the ptt_loop coroutine terminates cleanly when mode switches back to VAD (use a shared asyncio.Event).

Success criteria:

• Agent responds without any button press in VAD mode.
• After saying "switch to push-to-talk," the agent confirms the switch verbally. Subsequent turns require holding spacebar to record and releasing to send.
• After saying "switch to voice mode," VAD resumes with the original silence_duration_ms: 600 configuration.
• No error events or audio glitches occur during either mode transition.
• The full 5-minute session log shows clean session.updated events at each transition.

Next chapter: 03-tool-calling-live-session — registering function tools in a Realtime session, dispatching them non-blockingly, and injecting results back into the audio flow within 500ms.

Register tools in session.update, listen for response.function_call_arguments.done, dispatch your implementation asynchronously so the WebSocket stays alive, inject the result with conversation.item.create + response.create, and wrap everything in error handling so your agent speaks a recovery phrase instead of going silent. That is the complete tool calling loop for a live voice session — and this chapter gives you runnable TypeScript for every step.

Why Voice Tool Calling Is Different From Text

In a Chat Completions tool call, the user sends a message and waits for a response. A two-second tool execution is annoying but tolerable. The user is not listening — they submitted a form and walked away mentally. In a live voice session, the user is still on the line. Their ear is trained on your agent's output stream. The moment the model decides to call a tool, audio stops, and the user hears silence. Every millisecond of tool latency is perceptible dead air.

This is the governing constraint for everything in this chapter: voice tool calls need a latency budget. Target 200ms end-to-end — from the moment the model issues the call to the moment the first audio token of the model's verbal response begins streaming. If your tool cannot hit that SLA, you buffer the gap with a verbal acknowledgment: the model speaks "Let me check that for you" while the real tool runs in the background. You will implement both strategies below.

The other difference is the execution model. The OpenAI Realtime API runs over a persistent WebSocket connection where events arrive continuously. Tool calling is not a single request/response exchange — it is a sequence of events streaming in over time. You accumulate argument fragments, detect completion, fire the tool, and inject a result — all without breaking the event loop that keeps your WebSocket processing new audio from the user.

There is also a model-level difference worth naming explicitly. In Chat Completions, tool calls are part of an atomic response: the model generates a function call in one shot and the API returns it synchronously. In the Realtime API, the model is generating tokens continuously inside a live session. It decides mid-stream to call a tool, emits that decision as a stream of events, and then pauses output until you inject a result. Your server code must observe those events, execute the tool entirely outside the model's processing, and re-enter the session with a result. The model does not pause and call you — you watch its output stream, respond when it acts, and feed it what it needs to continue speaking.

Registering Tools in a Realtime Session

Tools are declared in session.update using the same JSON Schema format as Chat Completions. Add a tools array and set tool_choice: "auto" to let the model decide when to call.

```typescript // session-with-tools.ts import WebSocket from "ws";

const ws = new WebSocket( "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1", { headers: { Authorization: Bearer ${process.env.OPENAI_API_KEY}, }, } );

ws.on("open", () => { ws.send( JSON.stringify({ type: "session.update", session: { modalities: ["text", "audio"], voice: "alloy", turn_detection: { type: "server_vad", silence_duration_ms: 500 }, tool_choice: "auto", tools: [ { type: "function", name: "get_weather", description: "Get current weather for a location. Returns temperature in Celsius and a short condition string like 'partly cloudy'.", parameters: { type: "object", properties: { location: { type: "string", description: "City name, e.g. 'London' or 'New York'", }, }, required: ["location"], }, }, { type: "function", name: "create_reminder", description: "Create a reminder for the user. Returns the reminder ID and confirms the scheduled time.", parameters: { type: "object", properties: { text: { type: "string", description: "Reminder message content" }, time: { type: "string", description: "ISO 8601 datetime string, e.g. '2026-06-15T15:00:00Z'", }, }, required: ["text", "time"], }, }, ], }, }) ); }); ```

Tool descriptions are not documentation — they are instructions the model reads at inference time to decide whether and how to call the tool. A vague description like "Gets weather" leads to missed calls and poor argument population. Specific descriptions that include the return format ("Returns temperature in Celsius and a short condition string") help the model compose natural verbal responses after the tool completes. Write descriptions the way you would brief a human assistant who has never used the tool before.

Handling response.function_call Events

When the model decides to call a tool, it emits a sequence of events you must handle in order. Each tool call starts with response.output_item.added — carrying the tool name and a unique item ID — then streams argument JSON in fragments via response.function_call_arguments.delta, and signals completion with response.function_call_arguments.done. The Realtime API server events reference documents the full shape of each event.

You need a buffer per in-flight call to accumulate argument deltas. Use the item ID as the key, since multiple tools can be called in a single response. The item ID you receive in response.output_item.added is the same ID you will use as call_id when injecting the result — store it immediately when the function call item first appears, not when arguments are complete. If you plan to send a verbal acknowledgment before dispatching (covered in the next section), you need the ID available at response.output_item.added time, well before response.function_call_arguments.done arrives.

```typescript // event-handler.ts const pendingCalls = new Map<string, { name: string; argsBuffer: string }>();

ws.on("message", async (raw) => { const event = JSON.parse(raw.toString());

switch (event.type) { case "response.output_item.added": if (event.item.type === "function_call") { pendingCalls.set(event.item.id, { name: event.item.name, argsBuffer: "", }); } break;

case "response.function_call_arguments.delta": if (pendingCalls.has(event.item_id)) { pendingCalls.get(event.item_id)!.argsBuffer += event.delta; } break;

case "response.function_call_arguments.done": { const call = pendingCalls.get(event.item_id); if (call) { const args = JSON.parse(call.argsBuffer); // Fire and do NOT await — see the next section for why dispatchTool(event.item_id, call.name, args); pendingCalls.delete(event.item_id); } break; } } }); ```

The comment on the dispatch line is the most important line in this file. The next section explains it in full.

Non-Blocking Tool Dispatch: The Critical Constraint

The // do NOT await comment from above is not a stylistic choice — it is a correctness requirement. Node.js runs on a single-threaded event loop. If you await dispatchTool(...) inside the WebSocket message handler, you block the event loop for the duration of the tool call. During that time, the WebSocket cannot process incoming messages — including input_audio_buffer.speech_started events that signal barge-in, and input_audio_buffer.append events carrying new user audio. Your agent goes deaf while the tool runs.

The fix is to fire the async function without awaiting it, letting it resolve in the background while the event loop remains free to process new events.

``typescript // dispatch.ts async function dispatchTool( callId: string, name: string, args: Record<string, unknown> ): Promise<void> { const start = Date.now(); try { const result = await runTool(name, args); console.log([tool] ${name} completed in ${Date.now() - start}ms); injectResult(callId, JSON.stringify(result)); } catch (err) { const errorMsg = err instanceof Error ? err.message : "Tool failed unexpectedly"; console.error([tool] ${name} failed: ${errorMsg}`); injectResult(callId, JSON.stringify({ error: errorMsg })); } }

async function runTool( name: string, args: Record<string, unknown> ): Promise<unknown> { switch (name) { case "get_weather": return fetchWeather(args.location as string); case "create_reminder": return saveReminder(args.text as string, args.time as string); default: throw new Error(Unknown tool: ${name}); } }

async function fetchWeather(location: string) { // Mock: replace with real HTTP call. Target < 150ms. await new Promise((r) => setTimeout(r, 80)); return { temperature: 18, condition: "partly cloudy", location }; }

async function saveReminder(text: string, time: string) { // Mock: replace with your reminder store await new Promise((r) => setTimeout(r, 40)); return { success: true, reminder_id: rem-${Date.now()}, scheduled_at: time }; } ```

<Callout type="warning"> The 200ms SLA — and what to do when you can't hit it

Tool execution plus result injection should complete in under 200ms. For tools that genuinely need longer — external API calls, database queries — send a verbal acknowledgment before dispatching the tool. Inject an assistant text item and call response.create immediately:

ws.send(JSON.stringify({
  type: "conversation.item.create",
  item: {
    type: "message",
    role: "assistant",
    content: [{ type: "text", text: "Let me check that for you." }],
  },
}));
ws.send(JSON.stringify({ type: "response.create" }));

The model speaks "Let me check that for you" while the real tool runs in the background. The user hears natural filler instead of silence. Do this before calling dispatchTool, not after. </Callout>

Injecting Tool Results Back into the Session

After your tool resolves, you inject the result using two back-to-back WebSocket sends: a `conversation.item.create` event with a function_call_output item, then a response.create event to trigger the model's verbal response.

```typescript // inject-result.ts function injectResult(callId: string, resultJson: string): void { // Step 1: add the tool result to the conversation history ws.send( JSON.stringify({ type: "conversation.item.create", item: { type: "function_call_output", call_id: callId, // must match the item ID from response.output_item.added output: resultJson, }, }) );

// Step 2: ask the model to generate its next turn using the result ws.send(JSON.stringify({ type: "response.create" })); } ```

call_id is the load-bearing field. It must exactly match the id from the response.output_item.added event — that is the event.item.id you stored in pendingCalls. A mismatched or absent call_id produces a session error. Track this value from the moment the function call item appears, not from response.function_call_arguments.done, where only item_id is available.

response.create is what triggers the model's next speech turn. Without it, the function_call_output item sits in conversation history but the model generates nothing. The user hears silence. Always send response.create immediately after conversation.item.create.

Handling Tool Errors Gracefully

In a text interface, an unhandled tool error shows an error UI component. Embarrassing, but the user recovers — they click retry or reload. In a live voice session, the same failure produces silence. The user hears nothing, assumes a crash, and hangs up.

The rule is: dispatchTool must never fail silently. Wrap every tool in a try/catch, catch all thrown errors, and inject a result even when that result describes a failure. The output field of function_call_output is a plain string — it can carry any content, including an error description.

// Error path — already shown in dispatchTool above, isolated here for clarity
try {
  const result = await runTool(name, args);
  injectResult(callId, JSON.stringify(result));
} catch (err) {
  // Always inject something — never leave the call_id unresolved
  injectResult(
    callId,
    JSON.stringify({ error: "Weather service unavailable. Try again shortly." })
  );
}

Given an output like { "error": "Weather service unavailable..." }, the model generates a natural verbal recovery: "I'm sorry, I wasn't able to get the weather right now — the service seems to be down. You could try asking me again in a moment." No crash, no silence, conversation continues.

One timing edge case: if the user barge-ins and speaks again before your tool resolves, the conversation moves forward and the model may start a new response turn. When your tool finally completes and you call injectResult, the function_call_output still lands in conversation history and response.create still triggers a new turn. This is correct behavior — the model incorporates the tool result into its understanding of the full updated context. You do not need to cancel in-flight tool calls or suppress injection if the conversation has progressed. Just inject and continue.

There is one additional failure mode to guard against: uncaught promise rejections from dispatchTool itself. Because you fire it without awaiting, an unhandled rejection does not propagate to the message handler. In Node.js, it surfaces as an unhandledRejection event that — in older Node versions or misconfigured environments — crashes the process and terminates every active session. Add a top-level handler during development:

process.on("unhandledRejection", (reason) => {
  console.error("[fatal] unhandled rejection in tool dispatch:", reason);
  // Alert your monitoring system — do not crash in production
});

Hands-On Exercise: Weather + Reminder Tools

Extend your Chapter 2 WebSocket agent with the tools from this chapter.

What to build: a voice session with get_weather(location) and create_reminder(text, time), both fully integrated into the event handling, dispatch, and injection loop above.

Success criteria:

1. Say "What's the weather in Tokyo?" — the agent responds with temperature and condition within 500ms of your end-of-utterance. Check your timestamp logs to verify the tool completed in under 200ms.
2. Say "Remind me to call Alice at 3pm tomorrow" — the agent confirms the reminder was created and reads back the scheduled time in natural language.
3. Simulate a tool failure by throwing inside fetchWeather. Verify the agent responds with a verbal acknowledgment rather than going silent. Measure that the silence gap before verbal recovery is under 300ms.
4. Add a verbal acknowledgment ("Let me check that for you.") before get_weather dispatches. Verify the user hears speech immediately while the 80ms mock delay runs in the background.

Once all four criteria pass, your agent handles the full tool calling loop: registration, event handling, async dispatch, result injection, and graceful failure recovery.

Next chapter: 04-latency-engineering — profiling your voice pipeline end-to-end, implementing barge-in interrupt handling, and applying speculative response patterns to eliminate the silences that make users hang up.

The model's inference speed is not yours to optimize. You cannot make the Realtime API think faster, and chasing that goal wastes engineering time you could spend on levers you actually control. What you can control — entirely — is what happens on every side of that inference window: how fast you commit audio into the session, whether you start audio playback on the first token or the last, how instantly the agent goes silent when the user interrupts, and how much of the tool-call gap you fill with something useful. A 500ms model response that starts playing at token one feels faster than a 300ms response that sits in a client buffer for 250ms before the first speaker sample.

This chapter is about the four levers that move perceived latency without touching inference: a profiling harness that makes the breakdown visible, interrupt handling that stops audio within a single event loop tick, speculative verbal acknowledgments that fill tool-call silence, and WebRTC jitter buffer configuration that keeps audio smooth when the network isn't.

The Latency Budget: Where Every Millisecond Goes

A full voice round-trip has five measurable segments with very different ownership profiles:

The model TTFAT column dominates the total and is completely fixed. That 200–600ms window is model scheduling, KV-cache state, and token sampling — none of which you influence through application code. Focus instead on the two rows you own: VAD sensitivity (reducing false turn-ends that trigger premature responses) and the playback queue startup delay, which many implementations silently leave at 150–200ms by waiting for response.done before starting playback. Start on response.audio.delta, not response.done, and you typically recover 100–180ms for free.

Network latency to the OpenAI API is the one structural lever you partially control: deploying your relay server in the same AWS region as the API endpoint (currently us-east-1 for api.openai.com) cuts your RTT contribution roughly in half compared to a client calling from Europe. See the OpenAI Realtime API guide for transport architecture recommendations.

A practical way to think about the budget: anything above 250ms total perceived latency from end-of-user-speech to first audible agent token registers as a noticeable pause in conversation. Anything above 800ms breaks the conversational flow entirely — users start to wonder if the system heard them and often speak again, triggering a barge-in. Your goal is to stay under 500ms total across all segments you control, leaving the model's TTFAT as the one uncontrolled variable inside that envelope.

Instrumentation First: Seeing the Pipeline

Measure before you optimize. Add a lightweight profiler that records performance.now() at each event boundary; run five back-to-back turns; compare. Any segment averaging more than 30ms beyond its baseline is your first target.

```typescript // latency-profiler.ts export interface Checkpoint { label: string; ts: number; }

export class LatencyProfiler { private checkpoints: Checkpoint[] = []; constructor(private sessionId: string) {}

mark(label: string) { this.checkpoints.push({ label, ts: performance.now() }); }

report(): void { if (this.checkpoints.length < 2) return; console.log(\n[Latency — ${this.sessionId}]); for (let i = 1; i < this.checkpoints.length; i++) { const delta = this.checkpoints[i].ts - this.checkpoints[i - 1].ts; console.log( ${this.checkpoints[i - 1].label} → ${this.checkpoints[i].label}: ${delta.toFixed(1)} ms); } const total = this.checkpoints.at(-1)!.ts - this.checkpoints[0].ts; console.log( TOTAL: ${total.toFixed(1)} ms\n); this.checkpoints = []; } } ```

Wire it into your Chapter 3 WebSocket event loop:

```typescript const profiler = new LatencyProfiler(sessionId); let firstAudioDeltaSeen = false;

ws.on('message', (raw: string) => { const event = JSON.parse(raw); switch (event.type) { case 'input_audio_buffer.speech_started': profiler.mark('VAD_speech_started'); break; case 'input_audio_buffer.speech_stopped': profiler.mark('VAD_speech_stopped'); break; case 'input_audio_buffer.committed': profiler.mark('buffer_committed'); break; case 'response.created': profiler.mark('response_created'); break; case 'response.audio.delta': if (!firstAudioDeltaSeen) { profiler.mark('first_audio_token'); firstAudioDeltaSeen = true; startAudioPlayback(); // ← start here, not on response.done } break; case 'response.done': profiler.mark('response_done'); profiler.report(); firstAudioDeltaSeen = false; break; } }); ```

After instrumentation, most implementations surface the same top two offenders: a 100–200ms first_audio_token → playback start gap caused by buffering on response.done, and a 200ms–2s gap in response_created → response_done whenever a tool call is in flight. The first is fixed in one line. The second requires the speculative pattern in the next section.

Interrupt Handling: Stop on a Dime

When the user talks over the agent, they must hear silence within the same event loop tick — not after the current audio chunk finishes, not after a 200ms drainage cycle. Any perceptible overlap of old agent speech and new user speech signals that the system is not listening, which destroys trust faster than any latency metric.

The Realtime API fires `input_audio_buffer.speech_started` the instant server-side VAD detects voice onset. Your handler has two mandatory obligations: cancel the in-flight model response and flush the local speaker buffer synchronously.

```typescript let responseInFlight = false;

ws.on('message', (raw: string) => { const event = JSON.parse(raw);

switch (event.type) { case 'response.created': responseInFlight = true; break;

case 'response.done': case 'response.cancelled': responseInFlight = false; break;

case 'input_audio_buffer.speech_started': if (responseInFlight) { // Cancel server-side generation immediately ws.send(JSON.stringify({ type: 'response.cancel' }));

// Flush client-side speaker buffer synchronously audioPlayer.flush(); // must be synchronous — schedule = audible bleed } break; } }); ```

Two failure modes to guard against:

Asynchronous flush. If audioPlayer.flush() posts a task to the event queue instead of draining inline, the old audio continues for one or two frames. Use AudioContext.close() followed by constructing a fresh context, or sourceNode.stop(0) with offset zero and a new BufferSourceNode for the next response. The 0 offset is the critical difference: stop() without an offset stops at the next render quantum (one frame delay); stop(0) stops at the earliest safe boundary.

Double-cancel. Short sounds — background noise, a breath, a click — may fire speech_started followed immediately by speech_stopped with no real barge-in intent. The responseInFlight guard prevents cancelling a response that has already finished or was never started.

Speculative Response: Audio Before the Tool Completes

Every tool call introduces a mandatory silence: the model requests a result, your server fetches it, you inject it, then you request a continuation response. At 300ms per tool that gap is noticeable; at 800ms it sounds broken. The fix is to send a verbal acknowledgment to the conversation before the tool finishes, letting the model generate a natural filler phrase while execution runs in parallel.

The Realtime API conversation model accepts conversation.item.create events out of band. Inject a short assistant message immediately on response.function_call_arguments.done, then issue response.create for that message. While the model streams audio for "Let me check that for you…", your tool is executing in parallel.

```typescript ws.on('message', async (raw: string) => { const event = JSON.parse(raw);

if (event.type === 'response.function_call_arguments.done') { const { call_id, name, arguments: argsJson } = event; const args = JSON.parse(argsJson);

// 1. Inject acknowledgment text immediately ws.send(JSON.stringify({ type: 'conversation.item.create', item: { type: 'message', role: 'assistant', content: [{ type: 'text', text: getAcknowledgment(name) }], } })); ws.send(JSON.stringify({ type: 'response.create', response: { modalities: ['audio', 'text'] } }));

// 2. Execute tool in parallel const result = await executeTool(name, args);

// 3. Inject result and request continuation ws.send(JSON.stringify({ type: 'conversation.item.create', item: { type: 'function_call_output', call_id, output: JSON.stringify(result) } })); ws.send(JSON.stringify({ type: 'response.create' })); } });

function getAcknowledgment(toolName: string): string { const map: Record<string, string> = { get_weather: 'Let me check the current conditions…', create_reminder: 'Setting that up now…', lookup_account: 'Pulling up your account…', }; return map[toolName] ?? 'One moment…'; } ```

This trades a single-response turn for a two-turn structure. The cost is minimal — a short text token sequence for the filler. The benefit is that 300–800ms of tool latency becomes inaudible because the agent is speaking through it. For tools that take less than ~150ms (local lookups, in-memory caches), skip the acknowledgment — the added turn introduces more overhead than the silence it would hide.

WebRTC Jitter Buffer Tuning

WebRTC transport (recommended for browser clients in production) introduces one additional latency variable: the jitter buffer, which the browser uses to smooth out packet reordering on congested or mobile networks. The default target delay is conservative — browsers typically buffer 80–120ms of audio to absorb packet jitter — and this default adds constant startup latency to every response.

The W3C WebRTC specification exposes RTCRtpReceiver.jitterBufferDelayHint as a hint to the browser's jitter buffer algorithm. Setting a smaller value tells the browser to trade smoothness for lower startup delay:

```typescript pc.ontrack = (event) => { const receiver = event.receiver;

// Hint a 30ms target (vs browser default ~100ms). Advisory — browser may override on poor links. if ('jitterBufferDelayHint' in receiver) { (receiver as any).jitterBufferDelayHint = 0.03; // seconds }

const stream = new MediaStream([event.track]); audioElement.srcObject = stream; }; ```

For mobile networks where packet loss is the primary problem rather than jitter, the better lever is codec configuration. The Realtime API uses Opus over WebRTC, and you can influence its parameters during SDP negotiation. A lower bitrate with in-band FEC enabled tolerates packet loss better than a higher-bitrate stream:

```typescript async function applyLatencyBiasedSDP( pc: RTCPeerConnection, offer: RTCSessionDescriptionInit ): Promise<RTCSessionDescriptionInit> { await pc.setRemoteDescription(offer); const answer = await pc.createAnswer();

// Set Opus to voice-optimized 24kbps mono with forward error correction answer.sdp = answer.sdp!.replace( /a=fmtp:(\d+) (.+useinbandfec.+)/g, (_m, pt, params) => a=fmtp:${pt} ${params};maxaveragebitrate=24000;stereo=0 );

await pc.setLocalDescription(answer); return answer; } ```

On very flaky mobile connections, also consider setting iceTransportPolicy: 'relay' on your RTCPeerConnection configuration to force TURN relay. Direct ICE paths through mobile NAT often add 30–80ms of unpredictable variance; a good TURN server in the same region as your server is more consistent. See the MDN WebRTC API documentation for full ICE configuration options and when relay is and isn't worth the extra hop.

Optimization Priority Stack

After instrumentation, virtually every Realtime API implementation surfaces the same top three offenders in roughly this order:

First: Playback startup delay (50–200ms). Fixed by calling startAudioPlayback() on the first response.audio.delta event, not on response.done. One line change, immediate improvement.

Second: Tool execution silence (200ms–2s). Fixed with the speculative acknowledgment pattern. The agent speaking through the tool call gap costs one additional conversation turn — a worthwhile trade for any tool taking more than 150ms.

Third: Client jitter buffering (20–100ms). Fixed with jitterBufferDelayHint and Opus SDP tuning. The payoff is largest on mobile connections where the default buffer is deepest.

One pattern that pays dividends across all three: pre-fetch tool data for the queries most likely to arrive in a session before the user asks. If your voice agent handles weather queries and 60% of sessions ask about the same city the user previously queried, pre-fetching that city's conditions during the greeting exchange converts a 400ms tool call into a synchronous map lookup. This is the aggressive end of the "hide latency" philosophy — not faster execution, but execution that already completed before the user formed the intent.

Hands-On Exercise

Goal: Measure your Chapter 3 agent's latency profile and reduce your single largest bottleneck by ≥ 20%.

Setup: Start from your working Chapter 3 agent (WebSocket transport, two tool calls wired). Add the LatencyProfiler class above and instrument every event handler as shown.

Steps:

1. Run five back-to-back voice turns. After each turn, capture the profiler console output. Compute the average delta for each segment across the five runs.

1. Identify the single segment with the highest average. Implement the matching optimization:
2. - first_audio_token → playback start averaging > 100ms → start playback on response.audio.delta instead of response.done.
3. - response_created → response_done averaging > 300ms during tool calls → implement the speculative acknowledgment pattern for your get_weather tool.
4. - Barge-in response delay > 100ms → implement response.cancel + synchronous audio flush on speech_started.

1. Run five more turns after your change. Compare the before and after averages for the targeted segment.

Success criteria: The targeted segment's average drops by ≥ 20%. Paste the before and after profiler output as your submission.

Stretch goal: Throttle your browser to "Slow 3G" in DevTools Network panel, then apply the jitterBufferDelayHint and Opus SDP rewrite. Observe the difference in chrome://webrtc-internals under the inboundRtp jitter metric before and after.

Next chapter: 05-production-deployment-scaling — sticky sessions, horizontal scaling under concurrent voice load, PII redaction, and session reconnect protocols for when the network drops mid-call.

The path to horizontal scaling for WebSocket voice agents is not adding servers — it's routing each session to exactly one server and keeping it there. This chapter gives you a runnable nginx sticky-session configuration, a Postgres-backed reconnect protocol, a PII redaction pipeline for transcripts, an audit log schema that captures every tool call, and a per-session token budget with graceful exhaustion handling. All four production concerns, working together.

Why Stateless Scaling Breaks Voice Sessions

In a stateless HTTP service, any server can handle any request — there is no per-client state. The load balancer can round-robin freely. Voice agents are the opposite. A WebSocket voice session accumulates state on the server that established the connection: the model's conversation history, the VAD buffer state, the pending tool call map from Chapter 3, and the session configuration. None of that lives in the load balancer. It lives in the server process's heap.

When your load balancer sends a reconnect request to a different server — which is the default with round-robin or least-connections balancing — the new server has none of that state. It opens a fresh model session. The user's conversation context is gone. If they asked about their account balance two turns ago, the agent no longer knows. If a tool call was in flight when the reconnect happened, the call_id is orphaned. The session appears to restart from scratch.

Two architectures avoid this failure. Sticky sessions: the load balancer commits all requests from a given client to the same upstream server for the life of the connection. Shared session state: all servers write live state to Redis or Postgres so any server can serve any session. Shared state sounds more resilient, but it introduces a remote store roundtrip on every streaming event — at 10–20 events per second in a live voice session, that's a Redis read on every audio delta. Teams that choose shared state as their first architecture consistently report that the latency cost alone makes it non-viable without a caching layer that reintroduces the consistency problems they were trying to solve. Start with sticky sessions.

Sticky Sessions with nginx

The ip_hash directive in the nginx upstream module routes requests from the same client IP to the same upstream server. For most voice agent deployments — browser clients, mobile apps, dedicated call-center workstations — client IPs are stable within a session, making ip_hash a reliable default.

```nginx # nginx.conf — sticky WebSocket proxy for voice agents upstream voice_agents { ip_hash; # same client IP → same upstream, consistently server 10.0.0.1:3000; server 10.0.0.2:3000; server 10.0.0.3:3000; keepalive 64; # persistent upstream connection pool }

server { listen 443 ssl; server_name api.your-domain.com;

location /voice { proxy_pass http://voice_agents; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 3600s; # hold open for up to 1 hour proxy_send_timeout 3600s; } } ```

The proxy_http_version 1.1 and the Upgrade/Connection headers are required for WebSocket proxying — without them, nginx treats the connection as HTTP/1.0 and the upgrade handshake fails silently.

For deployments where clients share a corporate NAT gateway — a common call-center scenario where hundreds of agents appear to originate from a single IP — ip_hash breaks down: all sessions land on one upstream server. In that case, use AWS ALB's session stickiness by cookie or nginx Plus's sticky cookie directive to route by a session-scoped cookie instead.

Session Lifecycle Management

Production sessions end in three ways: the user disconnects cleanly, the network drops, or the server restarts. Each requires a different code path.

Idle timeouts. Track the timestamp of the last input_audio_buffer.speech_started event. Schedule a teardown if no user speech arrives within your tolerance window. Five minutes is a reasonable default for customer support; 30 seconds for kiosk deployments where idle sessions block hardware resources.

```typescript // session-manager.ts const IDLE_TIMEOUT_MS = 5 60 1000;

class VoiceSession { private idleTimer: NodeJS.Timeout | null = null;

resetIdleTimer(): void { if (this.idleTimer) clearTimeout(this.idleTimer); this.idleTimer = setTimeout(() => this.teardown("idle_timeout"), IDLE_TIMEOUT_MS); }

async teardown(reason: string): Promise<void> { console.log([session:${this.id}] teardown — reason: ${reason}); await this.persistConversationHistory(); await this.flushAuditLog(); this.ws.close(1000, reason); } } ```

Reconnect protocol. The Realtime API does not persist session context across dropped TCP connections. Each reconnect opens a fresh model session with no history. Your server must maintain its own durable history store: after each conversation.item.created server event, write the item to Postgres keyed by session ID. On reconnect, re-inject stored items into the new session via conversation.item.create before the user's first utterance.

Critical: not all stored item types survive `conversation.item.create` replay. The API accepts function_call, function_call_output, and message items whose content consists only of text or input_text parts. It rejects assistant audio items — turns where the model responded with speech, stored as content.type: "audio" — silently or with an error depending on the client. This is the central production footgun for reconnect implementations: teams store every conversation.item.created event faithfully, then replay the full history and discover audio items are dropped or cause 400s, corrupting the context injection order. Filter at replay time. Convert assistant audio turns to text using the transcript captured from the preceding response.audio_transcript.done event if you need to preserve them as context; otherwise skip them.

``typescript // reconnect-handler.ts async function handleReconnect(sessionId: string, ws: WebSocket): Promise<void> { const { rows } = await db.query( SELECT item_payload FROM session_history WHERE session_id = $1 ORDER BY seq ASC`, [sessionId] );

let replayed = 0; for (const row of rows) { const item = JSON.parse(row.item_payload);

// conversation.item.create cannot populate assistant audio items. // Replay only: function_call, function_call_output, and message items // whose content parts are exclusively text/input_text. const replayable = item.type === "function_call" || item.type === "function_call_output" || (item.type === "message" && Array.isArray(item.content) && item.content.every( (c: { type: string }) => c.type === "text" || c.type === "input_text" )); if (!replayable) continue;

ws.send(JSON.stringify({ type: "conversation.item.create", item, })); replayed++; }

ws.send(JSON.stringify({ type: "session.update", session: await loadSessionConfig(sessionId), }));

console.log([session:${sessionId}] reconnected — ${replayed}/${rows.length} history items restored); } ```

Orphan cleanup. When a server crashes, sessions in its memory die without a clean teardown. Their records in the session store remain open. Run a background sweep every five minutes that queries for sessions with a last_heartbeat older than your idle timeout and marks them closed, freeing associated locks and audit log handles.

PII Redaction from Transcripts

The Realtime API emits transcripts via response.audio_transcript.done events. If your agent operates in a regulated domain — healthcare, finance, insurance, customer support — those transcripts contain structured PII: card numbers, SSNs, phone numbers, and email addresses spoken aloud by users. Writing raw transcripts to any persistent store exposes your system to compliance violations.

Redact before you write. Apply the redaction pass to each transcript before it is inserted into your database, not as a post-processing job that runs later — a service outage during that window leaves raw PII in your store.

```typescript // pii-redactor.ts const PII_PATTERNS: Array<{ pattern: RegExp; replacement: string }> = [ { pattern: /\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g, replacement: "[CARD_REDACTED]" }, { pattern: /\b\d{3}-\d{2}-\d{4}\b/g, replacement: "[SSN_REDACTED]" }, { pattern: /\b(\+1\s?)?\(?\d{3}\)?[- .]?\d{3}[- .]?\d{4}\b/g, replacement: "[PHONE_REDACTED]" }, { pattern: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, replacement: "[EMAIL_REDACTED]" }, ];

export function redactPII(text: string): string { return PII_PATTERNS.reduce((t, { pattern, replacement }) => t.replace(pattern, replacement), text); }

// In your event handler: case "response.audio_transcript.done": { const safeTranscript = redactPII(event.transcript); await db.query( "INSERT INTO session_transcripts (session_id, role, content) VALUES ($1, $2, $3)", [sessionId, "assistant", safeTranscript] ); break; } ```

Regex catches structured PII reliably. For unstructured PII — a person's full name spoken mid-sentence — route transcripts through a cloud NLP entity-detection pass before final long-term storage. The regex handles the highest-risk tokens synchronously at write time; NLP handles the longer tail in an async enrichment job. Do not rely on regex alone when asserting compliance to an auditor.

Audit Logging All Tool Calls

Every tool call your agent makes must be traceable for debugging and compliance: which session, which tool, with which arguments, at what time, for how long, and what the result was. This is the minimum audit record for explaining agent behavior to a customer, an auditor, or your own engineering team when a session goes wrong.

```typescript // audit-logger.ts interface ToolCallLog { sessionId: string; callId: string; toolName: string; arguments: Record<string, unknown>; result: unknown; errorMsg: string | null; durationMs: number; timestamp: string; }

async function logToolCall(log: ToolCallLog): Promise<void> { await db.query( INSERT INTO tool_call_audit (session_id, call_id, tool_name, arguments, result, error_msg, duration_ms, called_at) VALUES ($1, $2, $3, $4, $5, $6, $7, $8), [ log.sessionId, log.callId, log.toolName, JSON.stringify(log.arguments), JSON.stringify(log.result), log.errorMsg, log.durationMs, log.timestamp, ] ); }

// Wrap dispatchTool from Chapter 3: async function auditedDispatch( sessionId: string, callId: string, name: string, args: Record<string, unknown> ): Promise<void> { const start = Date.now(); let result: unknown = null; let errorMsg: string | null = null;

try { result = await runTool(name, args); injectResult(callId, JSON.stringify(result)); } catch (err) { errorMsg = err instanceof Error ? err.message : "unknown error"; injectResult(callId, JSON.stringify({ error: errorMsg })); } finally { // always logs — even if runTool throws or injectResult fails await logToolCall({ sessionId, callId, toolName: name, arguments: args, result, errorMsg, durationMs: Date.now() - start, timestamp: new Date().toISOString(), }); } } ```

The finally block guarantees a log entry even on error paths. An audit log with gaps is worse than no audit log — the gaps look like evidence tampering, not system failures.

Set a retention policy when you design the schema. Most compliance frameworks require audit records for 6–12 months; regulated healthcare and financial services contexts can mandate 7 years or more. Keep your audit table in a separate Postgres database from your conversation data — a single database failure should not simultaneously take down your voice service and its compliance evidence. Add an archived_at column and run a weekly job to move records older than your hot window to cold storage, keeping the table fast for recent incident queries.

For incident triage, the audit log supports two essential query patterns: all tool calls for a specific session (for a customer complaint) and all calls to a specific tool across sessions in a given time window (for diagnosing a bug introduced by a tool code change). Index on (session_id, called_at) and (tool_name, called_at) separately — the first query drives customer support workflows; the second drives engineering postmortems.

Per-Session Token Budgets and Model Fallback

A 30-minute customer support call can consume 50,000+ tokens. Without a budget, one runaway session — a confused user who keeps repeating themselves — can consume a disproportionate share of your monthly quota. The response.done event carries a usage object with token counts for each completed model turn.

```typescript // budget-manager.ts const MAX_TOKENS_PER_SESSION = 50_000;

class BudgetManager { private tokenCount = 0;

consumeTokens(usage: { total_tokens: number }): boolean { this.tokenCount += usage.total_tokens; return this.tokenCount < MAX_TOKENS_PER_SESSION; } }

// In your event handler: case "response.done": { const withinBudget = budget.consumeTokens(event.response.usage); if (!withinBudget) { // Inject a closing message and schedule graceful teardown ws.send(JSON.stringify({ type: "conversation.item.create", item: { type: "message", role: "assistant", content: [{ type: "text", text: "I need to wrap up our session — we've reached the session limit. Is there anything final you need before we close?" }], }, })); ws.send(JSON.stringify({ type: "response.create" })); setTimeout(() => session.teardown("budget_exhausted"), 15_000); } break; } ```

For API-level quota exhaustion — OpenAI returns an error event on the WebSocket when you hit a rate limit — implement a fallback that injects a human-readable apology and closes the session cleanly. Do not let the connection hang in an error state; the user should hear a verbal acknowledgment before the call drops.

Distinguish between soft and hard budget limits in your implementation. A soft limit triggers the closing message and starts a graceful teardown countdown, as shown above. A hard limit disconnects immediately when the countdown expires. In practice, giving users a 15–30 second warning before disconnecting is the minimum courteous behavior — the alternative is a voice call that falls silent, which in customer support contexts reads as a system failure rather than a designed boundary. Log all budget exhaustion events with the session ID and final token count: these records reveal which user flows generate the most expensive sessions, directly informing prompt engineering and tool dispatch improvements that reduce cost without degrading call quality.

Hands-On Exercise: Two-Server Deployment with Full Audit Trail

Deploy your Chapter 3 agent to a two-server Node.js cluster behind the nginx sticky-session configuration from this chapter.

What to build: 1. Two Node.js server instances on different ports, both running your voice agent 2. nginx ip_hash routing both instances with proxy_read_timeout 3600s 3. A Postgres table tool_call_audit with the schema above 4. A BudgetManager tracking tokens per session, limit set to 10,000 for the exercise 5. A handleReconnect function that restores conversation history from a session_history table

Success criteria:

1. Sticky routing: Start two sessions from the same machine. Both must route to the same upstream server — verify by checking server process logs for session IDs.
2. Reconnect: Kill one server instance mid-conversation, restart it, reconnect the client. Confirm the model references prior conversation context correctly in the first post-reconnect turn.
3. Audit log: After a 5-minute session with at least 3 tool calls, query SELECT * FROM tool_call_audit WHERE session_id = '<your id>'. Every tool call must appear with correct arguments, result, and a duration under 500ms for mocked tools.
4. Budget cap: Reduce the limit to 5,000 tokens and hold a conversation until it exhausts. Verify the agent delivers the closing message rather than silently dropping the connection.

Once all four criteria pass, your deployment handles the production concerns that separate a demo from a shippable service: sticky scaling, resilient reconnection, complete tool call auditability, and bounded cost per session.

For cost modeling across Realtime API vs Whisper + LLM + Kokoro architectures, see 06-cost-quality-model-tradeoffs — the next chapter compares true per-session costs across all three stacks.

The OpenAI Realtime API is the fastest path from user speech to agent response. It is not always the cheapest. Audio tokens are priced at a significant premium over text tokens, and for high-volume or latency-tolerant use cases — automated notifications, IVR trees, batch audio generation — a Whisper + LLM + self-hosted TTS stack costs 70–90% less per session. This chapter gives you the math to choose correctly, the code to implement the alternatives, and a rubric to evaluate voice quality across all three architectures so you can defend any architecture decision with numbers.

Audio Token Pricing: How the Realtime API Bills

Every second of audio in a Realtime API session is tokenized at approximately 800 tokens per minute of natural speech. Both sides of the conversation are metered independently: audio input tokens (what the user says) and audio output tokens (what the agent says) are billed at different rates, with output priced at 2× input. (OpenAI, "API Pricing")

The 2:1 output-to-input ratio matters in practice because agents often speak more than users. In a customer support session where the agent delivers multi-sentence responses, output tokens frequently outpace input by 1.5–2×, pushing the effective cost per session above the symmetric estimate in the table above.

Text tokens — your system prompt, tool schemas, and tool call results — are priced far below audio. A 1,500-token system prompt and 500 tokens of tool I/O costs about $0.006 per session at gpt-realtime-2 rates, roughly 1.5% of the audio bill. Input caching reduces cached text input to $0.40/1M tokens, which helps if your system prompt is large and stable across sessions, but audio tokens dominate and cannot be cached.

You can measure your actual token consumption in real time by reading the usage field on each response.done event the server emits. The object includes input_token_details and output_token_details, both broken down into audio_tokens and text_tokens sub-fields. Log this per session from the start — it is the only reliable way to know whether your actual costs match the per-session estimate.

Cost-Per-Session Modeling: Three Architectures

To choose an architecture, you need comparable numbers across options. The following models a 5-minute session with roughly equal user and agent speaking time:

Architecture B line-item breakdown: (OpenAI, "Speech to Text")

• Whisper-1 transcription (5 min × $0.006/min): $0.030
• GPT-4o-mini text reasoning (~2,000 input + ~1,000 output tokens): <$0.001
• Cartesia Sonic TTS (~2.5 min of agent speech): ~$0.01–0.03
• Total: ~$0.041–0.061

Architecture C replaces Cartesia with Kokoro running on your own GPU. The only variable API costs are Whisper and the text LLM. At 1,000 sessions per day, a single A10G GPU instance handles the TTS load with throughput to spare, bringing the effective TTS cost to under $0.001 per session when amortized. The total is approximately $0.031 in API fees plus a small infrastructure share — call it $0.031–0.035 all-in at moderate volume.

The Realtime API costs roughly $0.39 for the same session. Architecture B saves approximately 85%; Architecture C saves 90–92% in API fees alone. The spec figure of "70–80% less" is a conservative floor that accounts for the infrastructure costs of self-hosting.

The Realtime API earns its premium in three ways you cannot easily replicate in a pipeline: (1) sub-500ms round-trip latency without complex streaming orchestration across three services, (2) built-in Voice Activity Detection and barge-in handling so the agent stops speaking the instant the user interrupts, and (3) a single stateful audio session that eliminates multi-hop state management between separate STT, LLM, and TTS services. When any of those directly improve your user experience, the premium is justified. When they don't — batch notifications, IVR trees, async pre-rendering — you are paying for latency no one will ever notice.

Kokoro and Chatterbox: Self-Hosted TTS for Async Workloads

Kokoro-82M is an 82-million-parameter open-source TTS model released under the Apache License 2.0. At its size it runs on a single consumer GPU or CPU (more slowly), and on an A10G it generates audio significantly faster than real time — meaning you can batch-render a 30-second audio clip in under 5 seconds. It covers American and British English across multiple voices, and achieves naturalness scores competitive with mid-tier commercial alternatives for standard speech content.

For async workloads — outbound notifications, pre-rendered IVR prompts, batch audio generation — Kokoro is the correct default. The integration is straightforward:

```python # pip install kokoro soundfile numpy from kokoro import KPipeline import soundfile as sf import numpy as np

pipeline = KPipeline(lang_code='a') # 'a' = American English

def synthesize(text: str, output_path: str, voice: str = 'af_heart') -> None: chunks = [] for _, _, audio in pipeline(text, voice=voice, speed=1.0, split_pattern=r'\n+'): chunks.append(audio) sf.write(output_path, np.concatenate(chunks), samplerate=24000)

synthesize( "Your appointment is confirmed for 3 PM tomorrow.", "notification.wav" ) ```

Chatterbox, released by Resemble AI under the MIT License, extends the self-hosted option with explicit controls for emotional expressiveness. The exaggeration parameter lets you tune warmth from flat and neutral to noticeably personal — useful for notifications that should feel human rather than robotic:

```python # pip install chatterbox-tts torchaudio import torchaudio from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")

def synthesize_expressive(text: str, output_path: str, exaggeration: float = 0.5) -> None: wav = model.generate(text, cfg_weight=0.3, exaggeration=exaggeration) torchaudio.save(output_path, wav, model.sr)

synthesize_expressive( "Great news — your order has shipped and arrives tomorrow.", "shipping_alert.wav", exaggeration=0.6 ) ```

Deploy either model on any GPU instance: Vast.ai, Lambda Labs, or HuggingFace Inference Endpoints all work for variable load. For sustained volume above 5,000 sessions per day, provision a dedicated instance and wrap the pipeline function above in a FastAPI handler that accepts text payloads and returns audio bytes. The wrapper adds roughly 100 lines of code and turns your TTS layer into a private HTTP service you can version and monitor independently.

<Callout type="hot"> Kokoro and Chatterbox do not support real-time streaming audio output — they generate a complete audio file before returning. Do not use them in live interactive sessions where the user is waiting for a response. Use them only for batch or async workloads where audio is pre-rendered, or for outbound calls where no user interruption is expected. </Callout>

Cartesia: Managed Streaming TTS for Production Pipelines

Cartesia provides a managed TTS API with low-latency streaming output — audio begins arriving within 50–80ms of the request. This makes it viable for live pipeline architectures where you need streaming TTS but want independent observability on each stage (transcription quality, LLM reasoning accuracy, and TTS naturalness all measured and logged separately).

The streaming integration pattern: feed GPT-4o-mini output to Cartesia's /tts/bytes endpoint sentence by sentence as the LLM generates, and Cartesia returns audio frames you forward to the client as they arrive. This brings total pipeline latency (Whisper → LLM → Cartesia → client) into the 700–1,200ms range under good conditions — slower than the Realtime API's 500ms target but competitive enough for many production use cases that do not require barge-in interruption.

Cartesia also supports voice cloning and style transfer, which matters when your product requires a branded voice persona. Self-hosted Kokoro and Chatterbox have fixed voice libraries and limited fine-tuning support as of mid-2026. If a custom trained voice is a product requirement, Cartesia or a similar managed API is the practical choice today.

Building a Voice Quality Rubric

Cost tells you what you pay; quality tells you what you deliver. A voice agent needs a rubric that makes quality trade-offs visible and measurable across any architecture. Four dimensions cover the essential surface:

Naturalness is the only dimension that resists full automation. Internal MOS panels catch obvious regressions quickly, and session abandonment rate gives you a complementary signal from real traffic: when users hang up early in the first exchange, naturalness is usually the first thing to examine.

Response speed is fully automatable and the easiest to improve. Every Realtime API event carries a server-assigned timestamp. Log input_audio_buffer.speech_stopped and the first response.audio.delta, compute the delta, and track P50 and P95 across sessions. P95 matters more than P50 — a 95th-percentile TTFA above 1,500ms means roughly 1 in 20 users experiences a conversation-breaking pause.

Tool accuracy and error recovery require scripted test harnesses. Maintain a library of test conversations that exercise each tool your agent supports, run them nightly, and track regression. Error recovery requires deliberate fault injection: configure staging tool endpoints to return errors on command and verify the agent's verbal fallback triggers correctly rather than silently failing or repeating the question.

Choosing Your Architecture: A Decision Framework

The right architecture depends on three variables: required latency, daily session volume, and whether the interaction is live or async.

Many production deployments end up hybrid: the Realtime API handles live customer-facing sessions where latency and barge-in detection are visible to the user, while Kokoro handles batch rendering of thousands of personalized audio notifications overnight. The cost of each architecture matches its use case, and the two stacks coexist without conflict. Making the decision explicit — writing the table above for your specific use case — is more durable than defaulting to the most visible option.

Hands-On Exercise: Build a Three-Architecture Cost Comparison

Goal: Calculate the real cost-per-session for all three architectures using your own usage data.

Steps:

1. Instrument a Realtime API session. Run a 5-minute test call and log the usage field on each response.done event. Sum audio input tokens, audio output tokens, and text tokens. Multiply by current rates from openai.com/api/pricing.

1. Model Pipeline B (Whisper + 4o-mini + Cartesia). Estimate Whisper STT cost from your user speech duration. Add GPT-4o-mini text token cost from a comparable Chat Completions session on the same topic. Add Cartesia TTS cost for your agent's speech duration from current rates at docs.cartesia.ai.

1. Model Pipeline C (Whisper + 4o-mini + Kokoro self-hosted). Use the same Whisper and LLM numbers from Pipeline B. Estimate Kokoro GPU cost: pick an instance type, estimate concurrent session throughput, and calculate cost-per-session at your expected daily volume.

1. Build the comparison table. Three rows: architecture, 5-min session cost, monthly cost at your expected volume, and one capability you give up by choosing it.

Success criteria: - You have real token counts from a live Realtime API session, not estimates. - Your Pipeline B and C costs land within 20% of the formulas in this chapter. - You have a written one-sentence justification for which architecture you would choose for your next production voice deployment.

Apply everything built across this course — sessions, tools, latency engineering, production deployment, and the cost framework from this chapter — in the Build SupportVoice Capstone Project (see the course capstone overview in the outline).