Open Vision Agents by Stream. Build Vision Agents quickly with any model or video provider. Uses Stream's edge network for ultra-low latency.

Vision Agents give you the building blocks to create intelligent, low-latency video experiences powered by your models, your infrastructure, and your use cases.

• Video AI: Built for real-time video AI. Combine YOLO, Roboflow, and others with Gemini/OpenAI in real-time.
• Low Latency: Join quickly (500ms) and maintain audio/video latency under 30ms using Stream's edge network.
• Open: Built by Stream, but works with any video edge network.
• Native APIs: Native SDK methods from OpenAI (create response), Gemini (generate), and Claude ( create message) — always access the latest LLM capabilities.
• SDKs: SDKs for React, Android, iOS, Flutter, React Native, and Unity, powered by Stream's ultra-low-latency network.

This example shows you how to build golf coaching AI with YOLO and Gemini Live. Combining a fast object detection model (like YOLO) with a full realtime AI is useful for many different video AI use cases. For example: Drone fire detection, sports/video game coaching, physical therapy, workout coaching, just dance style games etc.

# partial example, full example: examples/02_golf_coach_example/golf_coach_example.py
agent = Agent(
    edge=getstream.Edge(),
    agent_user=agent_user,
    instructions="Read @golf_coach.md",
    llm=gemini.Realtime(fps=10),
    # llm=openai.Realtime(fps=1), # Careful with FPS can get expensive
    processors=[ultralytics.YOLOPoseProcessor(model_path="yolo11n-pose.pt", device="cuda")],
)

This example shows a security camera system that detects faces, tracks packages and detects when a package is stolen. It automatically generates "WANTED" posters, posting them to X in real-time.

It combines face recognition, YOLOv11 object detection, Nano Banana and Gemini for a complete security workflow with voice interaction.

# partial example, full example: examples/04_security_camera_example/security_camera_example.py
security_processor = SecurityCameraProcessor(
    fps=5,
    model_path="weights_custom.pt",  # YOLOv11 for package detection
    package_conf_threshold=0.7,
)

agent = Agent(
    edge=getstream.Edge(),
    agent_user=User(name="Security AI", id="agent"),
    instructions="Read @instructions.md",
    processors=[security_processor],
    llm=gemini.LLM("gemini-2.5-flash-lite"),
    tts=elevenlabs.TTS(),
    stt=deepgram.STT(),
)

Apps like Cluely offer realtime coaching via an invisible overlay. This example shows you how you can build your own invisible assistant. It combines Gemini realtime (to watch your screen and audio), and doesn't broadcast audio (only text). This approach is quite versatile and can be used for: Sales coaching, job interview cheating, physical world/ on the job coaching with glasses

Demo video

agent = Agent(
    edge=StreamEdge(),  # low latency edge. clients for React, iOS, Android, RN, Flutter etc.
    agent_user=agent_user,  # the user object for the agent (name, image etc)
    instructions="You are silently helping the user pass this interview. See @interview_coach.md",
    # gemini realtime, no need to set tts, or sst (though that's also supported)
    llm=gemini.Realtime()
)

Step 1: Install via uv

uv add vision-agents

Step 2: (Optional) Install with extra integrations

uv add "vision-agents[getstream, openai, elevenlabs, deepgram]"

Step 3: Obtain your Stream API credentials

Get a free API key from Stream. Developers receive 333,000 participant minutes per month, plus extra credits via the Maker Program.

Processors let your agent manage state and handle audio/video in real-time.

They take care of the hard stuff, like:

• Running smaller models
• Making API calls
• Transforming media

… so you can focus on your agent logic.

Check out our getting started guide at VisionAgents.ai.

• Quickstart: Building a Voice AI app
• Quickstart: Building a Video AI app
• Tutorial: Building a real-time meeting assistant
• Tutorial: Building real-time sports coaching

🔮 Demo Applications
Cartesia Using Cartesia's Sonic 3 model to visually look at what's in the frame and tell a story with emotion. • Real-time visual understanding • Emotional storytelling • Frame-by-frame analysis >Source Code and tutorial
Realtime Stable Diffusion Realtime stable diffusion using Vision Agents and Decart's Mirage 2 model to create interactive scenes and stories. • Real-time video restyling • Interactive scene generation • Stable diffusion integration >Source Code and tutorial
Golf Coach Using Gemini Live together with Vision Agents and Ultralytics YOLO, we're able to track the user's pose and provide realtime actionable feedback on their golf game. • Real-time pose tracking • Actionable coaching feedback • YOLO pose detection • Gemini Live integration >Source Code and tutorial
GeoGuesser Together with OpenAI Realtime and Vision Agents, we can take GeoGuesser to the next level by asking it to identify places in our real world surroundings. • Real-world location identification • OpenAI Realtime integration • Visual scene understanding >Source Code and tutorial
Phone and RAG Interact with your Agent over the phone using Twilio. This example demonstrates how to use TurboPuffer for Retrieval Augmented Generation (RAG) to give your agent specialized knowledge. • Inbound/Outbound telephony • Twilio Media Streams integration • Vector search with TurboPuffer • Retrieval Augmented Generation >Source Code and tutorial
Security Camera A security camera with face recognition, package detection and automated theft response. Generates WANTED posters with Nano Banana and posts them to X when packages disappear. • Face detection & named recognition • YOLOv11 package detection • Automated WANTED poster generation • Real-time X posting >Source Code and tutorial

See DEVELOPMENT.md

Want to add your platform or provider? Reach out to nash@getstream.io.

Our favorite people & projects to follow for vision AI

@demishassabis CEO @ Google DeepMind Won a Nobel prize	@OfficialLoganK Product Lead @ Gemini Posts about robotics vision	@ultralytics Various fast vision AI models Pose, detect, segment, classify

@skalskip92 Open Source Lead @ Roboflow Building tools for vision AI	@moondreamai The tiny vision model that could Lightweight, fast, efficient	@kwindla Pipecat / Daily Sharing AI and vision insights

@juberti Head of Realtime AI @ OpenAI Realtime AI systems	@romainhuet Head of DX @ OpenAI Developer tooling & APIs	@thorwebdev Eleven Labs Voice and AI experiments

@mervenoyann Hugging Face Posts extensively about Video AI	@stash_pomichter Spatial memory for robots Robotics & AI navigation	@Mentraglass Open-source smart glasses Open-Source, hackable AR glasses with AI capabilities built in

@vikhyatk AI Engineer Open-source AI projects, Creator of Moondream AI

• Livekit Agents: Great syntax, Livekit only
• Pipecat: Flexible, but more verbose.
• OpenAI Agents: Focused on openAI only

• Working TTS, Gemini & OpenAI

• Simplified the library & improved code quality
• Deepgram Nova 3, Elevenlabs Scribe 2, Fish, Moondream, QWen3, Smart turn, Vogent, Inworld, Heygen, AWS and more
• Improved openAI & Gemini realtime performance
• Audio & Video utilities

• Production-grade HTTP API for agent deployment (uv run <agent.py> serve)
• Metrics & Observability stack
• Phone/voice integration with RAG capabilities
• 10 new LLM plugins (AWS Nova 2, Qwen 3 Realtime, NVIDIA Cosmos 2, Pocket TTS, Deepgram TTS, OpenRouter, HuggingFace Inference, Roboflow, Twilio, Turbopuffer)
• Real-world examples (security camera, phone integration, football commentator, Docker deployment with GPU support, agent server)
• Stability: Fixes for participant sync, video frame handling, agent lifecycle, and screen sharing

• Excellence on documentation/polish
• Better Roboflow annotation docs
• Automated workflows for maintenance
• Local camera/audio support AND/OR WebRTC connection
• Embedded/robotics examples

Video AI is the frontier of AI. The state of the art is changing daily to help models understand live video. While building the integrations, here are the limitations we've noticed (Dec 2025)

• Video AI struggles with small text. If you want the AI to read the score in a game it will often get it wrong and hallucinate
• Longer videos can cause the AI to lose context. For instance if it's watching a soccer match it will get confused after 30 seconds
• Most applications require a combination of small specialized models like Yolo/Roboflow/Moondream, API calls to get more context and larger models like gemini/openAI
• Image size & FPS need to stay relatively low due to performance constraints
• Video doesn’t trigger responses in realtime models. You always need to send audio/text to trigger a response.

Join the team behind this project - we’re hiring a Staff Python Engineer to architect, build, and maintain a powerful toolkit for developers integrating voice and video AI into their products.

Apply here