Skip to content

feat(voice): add Gemini Live ephemeral token support - #56888

Draft
kitlangton wants to merge 1 commit into
devfrom
feat/voice-gemini-live
Draft

kitlangton wants to merge 1 commit into
devfrom
feat/voice-gemini-live

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

Why

Clients calling POST /api/experimental/voice/session can mint ephemeral voice sessions for OpenAI (gpt-live, gpt-realtime) and xAI (grok-voice), but requesting a Google Gemini Live model fails with Unknown voice model. Mobile and web clients need short-lived, single-use Google ephemeral tokens so they can connect directly to Gemini Live over WebSocket without holding a raw Google API key.

What Changes

Adds the gemini-live engine (provider: "google") to Realtime and connects Voice.Service to Google API key credentials via Integration.

Model family Engine Provider Transport
gemini-3.8-live (-extended-thinking, etc.) gemini-live google WebSocket
gemini-3.5-live (-translate-preview, etc.) gemini-live google WebSocket
gemini-3.5-transcribe-live gemini-live google WebSocket
gemini-3.1-flash-live (-preview, etc.) gemini-live google WebSocket
gemini-2.5-flash-native-audio (-preview-*, etc.) gemini-live google WebSocket
gemini-live (-2.5-flash-preview, etc.) gemini-live google WebSocket
sequenceDiagram
    participant Client
    participant Server as OpenCode Server (Voice.Service)
    participant Google as Google Auth Tokens (v1beta/auth_tokens)
    participant Live as Gemini Live WS (BidiGenerateContentConstrained)

    Client->>Server: POST /api/experimental/voice/session { model, session, transport: { type: "websocket" } }
    Server->>Google: POST /v1beta/auth_tokens (x-goog-api-key, uses: 1, bidiGenerateContentSetup)
    Google-->>Server: { name: "auth_tokens/..." }
    Server-->>Client: { type: "websocket", url: "wss://...BidiGenerateContentConstrained", token, expiresAt }
    Client->>Live: WebSocket connect (?access_token=token)
Loading

Ephemeral Token Minting

Realtime.connect posts to https://generativelanguage.googleapis.com/v1beta/auth_tokens with the server's Google API key in x-goog-api-key:

  • uses: 1 (single-use token)
  • newSessionExpireTime: 60 seconds from mint time (expiresAt returned as Unix seconds matching this window)
  • expireTime: 30 minutes from mint time for the active session stream
  • bidiGenerateContentSetup: locks { ...session, model: "models/${model}" } into the token as required by BidiGenerateContentConstrained

Requesting webrtc transport for a Gemini Live model fails before network I/O with Voice.InvalidRequestError (HTTP 400). When no stored Google key or GEMINI_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY / GOOGLE_API_KEY environment variable is configured, Voice.Service returns Voice.UnavailableError (service: "google").

Scope

Covers server-side ephemeral token minting and google credential lookup for POST /api/experimental/voice/session. Client WebSocket audio framing and tool bridging live in the client apps.

Verification

bun test packages/ai/test/realtime.test.ts --rerun-each 10
bun --cwd packages/core run script/test.ts test/voice.test.ts --rerun-each 10
bun --cwd packages/server test test/voice.test.ts
bun run check
  • packages/ai/test/realtime.test.ts: 30/30 passed across 10 reruns (engine routing for Gemini Live families and non-Live exclusions, v1beta/auth_tokens request headers/body and response decoding, WebRTC rejection).
  • packages/core/test/voice.test.ts: 10/10 passed across 10 reruns (GEMINI_API_KEY lookup via Integration, missing Google key UnavailableError, WebRTC InvalidRequestError).
  • packages/server/test/voice.test.ts: passed.
  • bun run check (lint + turbo typecheck): 39/39 packages passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant