Realtime conversations

Hold a live voice (or text) conversation over a WebSocket. The endpoint speaks OpenAI's Realtime API (the GA interface, session type: "realtime"), so OpenAI's SDKs and realtime clients work unchanged; the public model must support audio.converse.realtime. Today that means:

• Gemini's Live models: gemini-3.8-live, gemini-3.8-live-extended-thinking, gemini-3.1-flash-live-preview, gemini-2.5-flash-native-audio-preview-12-2025, and gemini-3.5-live-translate-preview for speech-to-speech translation.
• OpenAI's realtime models: gpt-realtime-2.1, gpt-realtime-2.1-mini, gpt-realtime-2 and gpt-realtime-1.5, through OpenAI's own Realtime API.

Every model's catalog profile says what it accepts, and a session that asks for more is refused with an error naming the field. Most of this page describes Gemini, whose protocol differs from OpenAI's; OpenAI models lists what changes for them. For transcription only, see Realtime transcription.

Connecting

Open a WebSocket to $GATEWAY/v1/realtime?model=<public-model>. Authenticate with your virtual key in Authorization: Bearer, or, from a browser, which cannot set headers, as the subprotocol openai-insecure-api-key.<key> next to realtime, the way OpenAI's browser clients do. A browser should not hold your virtual key: give it a client secret instead.

The gateway answers with session.created. A connection without model, or with an intent other than realtime, is refused with 400 before the upgrade. The model cannot change for the life of the connection.

The provider session opens with the first event that needs it (audio, an item, or response.create), so session.updates sent first configure it instead of reconnecting it.

Client secrets

Your backend mints a short-lived key for the browser with POST /v1/realtime/client_secrets, as with OpenAI:

curl $GATEWAY/v1/realtime/client_secrets \
  -H "Authorization: Bearer $AI_GATEWAY_KEY" -H "Content-Type: application/json" \
  -d '{"expires_after": {"anchor": "created_at", "seconds": 600},
       "session": {"type": "realtime", "model": "voice-agent", "instructions": "Be brief."}}'

The answer carries value (ek_…), expires_at and the rendered session. The browser connects with value in place of the key, and needs no ?model=: the connection starts from the session the secret was minted with (type: "transcription" opens a transcription session). The URL and session.update can still change it.

• expires_after.seconds runs from 10 to 7200 (default 600). The secret stops opening sessions then; a session already open keeps going.
• A secret can open several sessions until it expires. It acts with the minting key's allowed models, budget and rate limits, and it stops working if that key is disabled or expires.
• It opens /v1/realtime and nothing else: any other endpoint answers 401.
• The models in session are checked against the key when minting (403 if not allowed).

Session

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are a concise support agent.",
    "output_modalities": ["audio"],
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "transcription": { "model": "gpt-transcribe", "keywords": ["AI Gateway"] },
        "turn_detection": { "type": "server_vad", "silence_duration_ms": 600 }
      },
      "output": { "format": { "type": "audio/pcm", "rate": 24000 }, "voice": "Kore" }
    },
    "tools": [
      { "type": "function", "name": "order_status", "parameters": { "type": "object", "properties": { "id": { "type": "string" } } } },
      { "type": "google_search" }
    ]
  }
}

Gemini fixes a session's setup when it opens, and a resumed session keeps its old setup. So changing instructions, tools, voice, turn detection or any other provider setting mid-session opens a new provider session with the new setup, seeded with the conversation so far: the transcripts of both sides, the client's text and image items, and function calls with their results. The model keeps the content of the conversation, not the audio itself. The switch waits until neither the model nor the user is speaking.

Turns and responses

• Server VAD. Stream input_audio_buffer.append; Gemini detects the turn, the gateway emits input_audio_buffer.speech_started/speech_stopped/committed and the user item, and the model answers on its own.
• Manual turns (turn_detection: null). Audio is buffered in the gateway until input_audio_buffer.commit (so input_audio_buffer.clear really discards it); response.create asks the model to answer.
• Text and images. conversation.item.create with a user message (input_text, input_image as a base64 PNG/JPEG/WebP data URL, input_audio) or an assistant message adds it to the conversation; response.create asks for an answer. Items can only be appended at the end.
• Function calls. The model's call arrives as a function_call item with response.function_call_arguments.done, and the response ends there. Reply with a function_call_output item, then response.create. A JSON object output reaches Gemini as is; anything else is wrapped as { "output": "…" }.
• Interruptions. Speech during a reply cancels it: response.done arrives with status: "cancelled" and reason: "turn_detected". Stop playback and send conversation.item.truncate as usual; the gateway acknowledges it, but Gemini keeps its own record of what it said.

Responses stream as OpenAI's events: response.created, response.output_item.added, response.content_part.added, response.output_audio.delta with response.output_audio_transcript.delta, then the .done events and response.done with usage. Gemini reports a turn over only once playing it would have finished, and its usage comes with that. The content's .done events arrive when generation ends; response.done waits for the usage, or is sent at once (with usage: null) when the client creates another response or the user speaks.

OpenAI models

OpenAI's realtime models take the session much as OpenAI defines it:

• session.update applies on the same connection, with no new session.
• create_response: false, audio.output.speed and reasoning.effort up to xhigh (2.x models) are accepted.
• audio.input.transcription.model picks OpenAI's transcription model (default gpt-transcribe). OpenAI bills that transcription on its own; it is not in the session's logged usage.
• response.cancel stops the reply upstream.
• Input audio is resampled to 24 kHz, the only rate OpenAI takes.

What the gateway does not support yet is refused for them too: out-of-band responses, per-response overrides, idle_timeout_ms, item deletion, MCP tools and stored prompts.

What Gemini cannot do

These are refused with an error event naming the field, and the session stays open:

• out-of-band responses (response.conversation: "none", response.input) and per-response tools, voice, format, modalities, reasoning or token limits;
• system messages mid-conversation (use instructions), client-created function_call items, and inserting items anywhere but the end (previous_item_id);
• conversation.item.delete: Gemini cannot forget what it heard;
• output_audio_buffer.clear, which is WebRTC-only.

response.create with instructions and nothing else to answer sends them to the model as a user turn, since Gemini has no per-response instructions; this covers the common "greet the user" case. response.cancel ends the response and drops the rest of the model's turn, but Gemini finishes generating it, and it is billed.

Limits and billing

• A connection lasts at most 60 minutes. Gemini ends each provider connection after about ten minutes; the gateway resumes the session on a new one behind the same socket. The translation model has no resumption and reconnects without context.
• REALTIME_WEBSOCKET_MAX_CONNECTIONS and REALTIME_WEBSOCKET_MAX_CONNECTIONS_PER_KEY cap open sessions per process and per key.
• Each connection is logged as one realtime.conversation operation with the usage Gemini reported and its cost. Live models price audio and text separately (audioInputCentsPerMTokens, audioOutputCentsPerMTokens in the catalog). If the provider reports no usage, the gateway estimates it (25 tokens per second of audio, about 4 characters per text token) and marks the operation usageEstimated.