Repository navigation
feat(voice): add Gemini Live ephemeral token support - #56888
Draft
kitlangton wants to merge 1 commit into
Draft
kitlangton wants to merge 1 commit into
kitlangton wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Clients calling
POST /api/experimental/voice/sessioncan mint ephemeral voice sessions for OpenAI (gpt-live,gpt-realtime) and xAI (grok-voice), but requesting a Google Gemini Live model fails withUnknown voice model. Mobile and web clients need short-lived, single-use Google ephemeral tokens so they can connect directly to Gemini Live over WebSocket without holding a raw Google API key.What Changes
Adds the
gemini-liveengine (provider: "google") toRealtimeand connectsVoice.Serviceto Google API key credentials viaIntegration.gemini-3.8-live(-extended-thinking, etc.)gemini-livegooglegemini-3.5-live(-translate-preview, etc.)gemini-livegooglegemini-3.5-transcribe-livegemini-livegooglegemini-3.1-flash-live(-preview, etc.)gemini-livegooglegemini-2.5-flash-native-audio(-preview-*, etc.)gemini-livegooglegemini-live(-2.5-flash-preview, etc.)gemini-livegooglesequenceDiagram participant Client participant Server as OpenCode Server (Voice.Service) participant Google as Google Auth Tokens (v1beta/auth_tokens) participant Live as Gemini Live WS (BidiGenerateContentConstrained) Client->>Server: POST /api/experimental/voice/session { model, session, transport: { type: "websocket" } } Server->>Google: POST /v1beta/auth_tokens (x-goog-api-key, uses: 1, bidiGenerateContentSetup) Google-->>Server: { name: "auth_tokens/..." } Server-->>Client: { type: "websocket", url: "wss://...BidiGenerateContentConstrained", token, expiresAt } Client->>Live: WebSocket connect (?access_token=token)Ephemeral Token Minting
Realtime.connectposts tohttps://generativelanguage.googleapis.com/v1beta/auth_tokenswith the server's Google API key inx-goog-api-key:uses: 1(single-use token)newSessionExpireTime: 60 seconds from mint time (expiresAtreturned as Unix seconds matching this window)expireTime: 30 minutes from mint time for the active session streambidiGenerateContentSetup: locks{ ...session, model: "models/${model}" }into the token as required byBidiGenerateContentConstrainedRequesting
webrtctransport for a Gemini Live model fails before network I/O withVoice.InvalidRequestError(HTTP 400). When no stored Google key orGEMINI_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY/GOOGLE_API_KEYenvironment variable is configured,Voice.ServicereturnsVoice.UnavailableError(service: "google").Scope
Covers server-side ephemeral token minting and
googlecredential lookup forPOST /api/experimental/voice/session. Client WebSocket audio framing and tool bridging live in the client apps.Verification
packages/ai/test/realtime.test.ts: 30/30 passed across 10 reruns (engine routing for Gemini Live families and non-Live exclusions,v1beta/auth_tokensrequest headers/body and response decoding, WebRTC rejection).packages/core/test/voice.test.ts: 10/10 passed across 10 reruns (GEMINI_API_KEYlookup viaIntegration, missing Google keyUnavailableError, WebRTCInvalidRequestError).packages/server/test/voice.test.ts: passed.bun run check(lint+turbo typecheck): 39/39 packages passed.