Gemini Live API: build an AI that sees and hears in real time

The Gemini Live API is Google's interface for two-way real-time conversation: you stream microphone and camera to the model, and it answers with voice while you are still talking. We run it in production for aifixly's 14 AI experts. This guide is everything we wish we had known before starting.

What is the Gemini Live API — and how is it different from regular Gemini?

Regular Gemini calls are request/response: you send a question over HTTP, wait, get an answer. The Live API is an open WebSocket session instead. Audio, video and text go in continuously, and the model's reply streams back as audio chunks and transcripts while you are still speaking.

The practical difference is interruptibility. In a live session you can say "no, wait" mid-sentence and the model stops immediately. You cannot fake that with turn-based calls, no matter how fast the model is.

• Regular Gemini: HTTP, one answer per call, great for text and single-image analysis.
• Gemini Live API: WebSocket, continuous stream, audio in and audio out, video as a frame stream.
• Audio formats: 16 kHz PCM in, 24 kHz PCM out — resampling is your job in the client.
• Video is sent as individual frames (0.5–2 fps is usually enough), not as a video stream.

Is Google AI Studio free?

Google AI Studio is free as a development environment — sign in with a Google account, try models in the browser and generate an API key without adding a card. The API also has a free tier with daily quotas, which is plenty for prototypes.

Cost appears when you go to production. You pay per token, and for the Live API audio counts as tokens just like text — both what you send in and what the model speaks back. Real-time audio is more expensive per minute than plain text, because the session consumes tokens the whole time it is open.

How do you stream audio and video in real time?

Short answer: the browser opens a WebSocket straight to Google, but it never sees your real API key. Your server issues a short-lived token instead.

• 1. The client calls your own server endpoint, for example /api/gemini-token.
• 2. The server uses the secret GEMINI_API_KEY to mint an ephemeral token — ours lasts 30 minutes and is single-use.
• 3. The client opens the WebSocket session against the Live API with that token.
• 4. The setup message goes first: model name, system instruction, response modality (audio) and voice.
• 5. Then microphone PCM chunks and camera frames stream continuously into the session.
• 6. Replies come back as audio chunks that you queue and play, plus transcripts for the on-screen text.

Add rate limiting and an origin check to the token endpoint on day one. Without them anyone can pull tokens from your domain and run up your bill. We cap it at 5 requests per minute and 30 per hour per IP.

What does it cost in production?

Think per minute, not per call. A live session consumes tokens continuously while it is open — even during silence, because the audio stream keeps flowing. A ten-minute conversation on Gemini Live typically lands around $0.20–$0.50 depending on how much the AI itself talks.

• The biggest cost driver is the AI's own speech — output audio tokens are the priciest.
• The second biggest is idle open sessions. Add an inactivity timeout.
• Video is cheaper than you expect if you send few frames per second.
• Short, precise system instructions save tokens in every session, not just once.

Which pitfalls did we hit?

All four bugs below looked like API problems at first. None of them were.

A race condition on connect

We started the audio stream inside the onopen callback and sent the first frame before the session object had been assigned. The reference was still null and the first seconds vanished. The fix was to initialise the stream after the session is assigned, not inside the open event.

"WebSocket is already in CLOSING or CLOSED state"

The client kept sending audio and video frames after Google closed the session. The error floods the console, but the real damage is that the UI still says "live". We added an explicit flag that is cleared on both close and error, and every send checks it first.

A false live state

An open WebSocket does not mean the model is answering. We showed "live" as soon as the socket connected, and users sat in silence. Now we only show connected once the first response actually arrives, with a 12-second timeout that produces a readable error instead of endless silence.

The wrong audio format

The browser's AudioContext often runs at 48 kHz. The Live API wants 16 kHz in and sends 24 kHz out. Skip the resampling and you get either noise or a chipmunk. This is not an API bug — it is one easy-to-miss line in the docs.

What is it good for — and what is it not?

The Live API shines when the context is visual and the question is immediate: you are standing at a dripping tap, a broken bike chain, an unknown plant, or a job interview in ten minutes. Those are exactly the situations we built aifixly's experts for.

• Good fit: real-time tutoring, language practice, troubleshooting where you need to show something.
• Good fit: coaching where interruptions and follow-up questions are the whole point.
• Poor fit: tasks needing exact fact-checking — text with sources is better there.
• Poor fit: anything requiring a licensed professional, like electrical or gas work. The AI should refer you to a professional, not guess.

Want to see it in practice: pick an expert, allow camera and microphone, and start talking. No account needed to try.