Realtime Audio Recipes

Realtime voice has two security boundaries: the standard provider key stays on your server, while the device receives only a short-lived client secret or sends its SDP through your server. It also has three resources to clean up: microphone tracks, data channels, and peer connections.

Mint a Short-Lived Realtime Client Secret on the Server

Authenticate the application user before creating a provider session. Keep the model configurable so a model update does not require a client release.

import { z } from "zod";

const clientSecretSchema = z.object({
  value: z.string().min(1),
  expires_at: z.number().optional(),
});

export async function POST(request: Request): Promise<Response> {
  const user = await requireUser(request);
  if (!user) return Response.json({ code: "UNAUTHORIZED" }, { status: 401 });

  const apiKey = process.env.OPENAI_API_KEY;
  const model = process.env.OPENAI_REALTIME_MODEL;
  if (!apiKey || !model) {
    return Response.json({ code: "REALTIME_NOT_CONFIGURED" }, { status: 503 });
  }

  const response = await fetch("https://api.openai.com/v1/realtime/client_secrets", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "OpenAI-Safety-Identifier": await privacyPreservingUserHash(user.id),
    },
    body: JSON.stringify({
      session: {
        type: "realtime",
        model,
        audio: { output: { voice: "marin" } },
      },
    }),
  });

  if (!response.ok) {
    return Response.json({ code: "REALTIME_UNAVAILABLE" }, { status: 502 });
  }

  const secret = clientSecretSchema.parse(await response.json());
  return Response.json(secret, {
    headers: { "Cache-Control": "no-store" },
  });
}

Rate-limit this route by authenticated user. Never return OPENAI_API_KEY to the app and never put it in an EXPO_PUBLIC_* variable.

Own the Entire Peer Lifecycle

react-native-webrtc requires a development build or native app; it is not a JavaScript-only Expo Go feature. Keep connection setup in one owner and make cleanup safe to call more than once.

import {
  mediaDevices,
  RTCPeerConnection,
  RTCSessionDescription,
  type MediaStream,
} from "react-native-webrtc";
import { z } from "zod";

const secretSchema = z.object({ value: z.string().min(1) });

export async function connectRealtime(accessToken: string) {
  const tokenResponse = await fetch(`${process.env.EXPO_PUBLIC_API_ORIGIN}/realtime/token`, {
    method: "POST",
    headers: { Authorization: `Bearer ${accessToken}` },
  });
  if (!tokenResponse.ok) throw new Error("Could not start voice session");
  const { value: ephemeralKey } = secretSchema.parse(await tokenResponse.json());

  const peer = new RTCPeerConnection();
  let localStream: MediaStream | null = null;

  try {
    localStream = await mediaDevices.getUserMedia({ audio: true, video: false });
    for (const track of localStream.getTracks()) peer.addTrack(track, localStream);

    const events = peer.createDataChannel("oai-events");
    let activeEvents: typeof events | null = events;
    let activePeer: RTCPeerConnection | null = peer;
    let closed = false;
    const offer = await peer.createOffer();
    if (!offer.sdp) throw new Error("Realtime offer had no SDP");
    await peer.setLocalDescription(offer);

    const answerResponse = await fetch("https://api.openai.com/v1/realtime/calls", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${ephemeralKey}`,
        "Content-Type": "application/sdp",
      },
      body: offer.sdp,
    });

    if (!answerResponse.ok) throw new Error("Realtime negotiation failed");
    await peer.setRemoteDescription(
      new RTCSessionDescription({ type: "answer", sdp: await answerResponse.text() }),
    );

    return {
      peer,
      events,
      localStream,
      close() {
        if (closed) return;
        closed = true;
        activeEvents?.close();
        activeEvents = null;
        localStream?.getTracks().forEach((track) => track.stop());
        localStream = null;
        activePeer?.close();
        activePeer = null;
      },
    };
  } catch (error) {
    localStream?.getTracks().forEach((track) => track.stop());
    localStream = null;
    peer.close();
    throw error;
  }
}

Attach remote audio using the rendering mechanism supported by the installed WebRTC adapter. Register connection-state handlers so a failed peer changes the UI from “Connecting” to an actionable retry rather than silently hanging.

Validate Data-Channel Events

Data-channel messages are network data. Parse JSON safely, then narrow the event before updating state.

const realtimeEventSchema = z.object({
  type: z.string().min(1),
  event_id: z.string().optional(),
}).passthrough();

export function parseRealtimeEvent(raw: string) {
  try {
    return realtimeEventSchema.safeParse(JSON.parse(raw));
  } catch {
    return { success: false as const };
  }
}

Define stricter schemas only for events the UI consumes. Ignore unknown event types so new server events do not crash old app versions.

Model Voice State for Interruption

export type VoiceState =
  | { status: "idle" }
  | { status: "requesting-permission" }
  | { status: "connecting" }
  | { status: "connected"; muted: boolean }
  | { status: "failed"; message: string };

The hang-up control must work during connection, not only after success. Keep a visible non-voice path for users who deny microphone access or cannot use audio.

Realtime Checklist

• Ask for microphone access in response to a clear user action.
• Show listening/muted/connecting states with text, not color or animation alone.
• Stop local tracks on hang-up, negotiation failure, and unmount.
• Close stale sessions before starting a new one.
• Bound reconnect attempts and mint a new short-lived secret for a new session.
• Set a stable privacy-preserving safety identifier on the trusted server.

Sources

OpenAI-specific behavior is checked against the official Realtime API with WebRTC guide. The React Native lifecycle wrapper is independently written around the public react-native-webrtc interface and should be checked against the installed adapter version.