Vision AI that sees, hears and talks back in real time — how it works

Until recently we talked to AI by typing. Now a model can see what you see through the camera, hear what you say, and answer out loud before you've finished the sentence. These are the vision-language models — VLMs — and they change what an "AI assistant" actually means.

What is a vision-language model (VLM)?

A vision-language model is an AI that takes in images, video and text in the same "thought" — not in two separate steps. Earlier language models only read text; if you wanted them to understand a picture you first ran the picture through an image recogniser that translated it into words, and then passed the words on. VLMs do it in a single pass.

That sounds like a technical detail, but it's the difference between "AI that knows there's a frying pan in the picture" and "AI that understands the pan is getting too hot for what you're trying to cook". The latter can answer follow-up questions about what it sees, compare two pictures, read text on a sign in the background, and tie words and sight together the way a person does.

How does it actually "see" and "hear"?

Short answer: the model translates image and audio segments into the same internal representation it uses for words — so-called tokens. To the model, a second of video, an English sentence and a couple of seconds of audio are just different kinds of tokens in the same stream. That's why it can weave all three together in one answer.

For it to feel like real time, being smart isn't enough — the model also has to be fast. Instead of uploading a video and waiting, camera and microphone are streamed continuously in small chunks. The model works with what it has, starts replying before you've finished talking, and can interrupt itself when you say something new.

What does "real time" mean here?

Real time means the flow is bidirectional and continuous. A normal chat is turn-taking: you type, wait, read the reply, type again. A live session works more like a phone call: both sides can talk at the same time, interrupt each other, and refer to things happening right now.

What can it actually do today?

The tech has moved from "impressive demo" to "actually useful" over the past year. Things that work now:

• Help with practical stuff through the camera — point your phone at a broken tap, a dough that won't rise, or a squat that feels off, and get an answer that takes into account exactly what it's looking at.
• Read screens and documents — decode an error message, explain a receipt in a foreign language, or walk you through a form you have in front of you.
• Translate in real time — you speak English, the person on the other end hears Spanish immediately, with no one hitting a "translate" button.
• Coach conversations — language, interview or sales practice where the AI gives feedback on words, tone and timing.
• Describe surroundings for someone with low vision — continuous, live scene description instead of one photo at a time.

What is it not good at (yet)?

It's easy to be impressed and forget the limits. Areas where today's VLMs still stumble:

• Long-term memory between sessions — it forgets most of the last conversation unless you explicitly save it and feed it back in.
• Precise measurements from a picture — it guesses sizes of what it sees, and often gets it wrong. For anything that really has to fit, measure yourself.
• Safety-critical work — electricity, gas, medicine, law. Use it as guidance, not as the final answer. Licensed professionals still apply.
• Bad audio quality — background noise, echo and cheap microphones make it hear the wrong words, which produces the wrong answer.
• Fast motion and poor light — if the camera can't see clearly, neither can the model.

How to try it in 30 seconds

The easiest way to understand what a real-time VLM feels like is to talk to one. On aifixly you do this:

• Go to the home page and pick an expert that fits — physio, cooking, home repair, interview coach, whatever.
• Tap "Start live session".
• Allow camera and microphone when the browser asks.
• Say hi and tell it what you need help with — show the thing you're wondering about through the camera.

What's coming next?

Three things are on the way that will change what you do with VLMs next year:

• Longer memory — models that remember your home, your habits and what you talked about last week, without you having to type it in again.
• Agent behaviour — instead of just answering, the model can do things for you: book, order, fill in, call.
• Local models — VLMs that run entirely on your phone or laptop, faster and without the video stream ever leaving the device.

This is a relatively new technology and it's moving fast. The easiest way to keep up is to actually use it on something you do every week — that's where you'll notice the progress, and get a feel for when it helps and when it's still the wrong tool for the job.