jarvis-realtime-assistant
Realtime Voice AI & Iron Man HUD with WebSockets and Gemini 2.0 Flash
End-to-end voice-in to voice-out pipeline
Full-duplex WebSockets audio streaming
High-throughput multimodal reasoning
Voice Activity Detection cuts network overhead
Problem Statement & Target Users
The real-world business and technical bottleneck addressed
Standard conversational AI interfaces rely on sluggish REST request-response cycles, leading to 2-4 second dead air pauses that destroy natural conversational cadence. Managing continuous audio streaming, microphone permission states, and WebSocket reconnects across varying network conditions presents significant stability hurdles.
Target User Personas:
- ✓Developers seeking hands-free multimodal voice development assistants
- ✓Engineers exploring low-latency streaming WebSocket architectures with frontier LLMs
- ✓Users who value interactive, futuristic cybernetic interfaces with visual audio feedback
Technology Stack & Architecture Philosophy
Curated tools selected for performance, reliability, and developer experience
A full-duplex WebSocket architecture streaming 16kHz PCM audio chunks from browser Web Audio API to a Python FastAPI engine with Voice Activity Detection (VAD), LLM streaming tokens, and incremental audio playback.
jarvis-realtime-assistant Architecture & Data Flow
Interactive structural nodes & deterministic processing sequence
1. Browser Audio Worklet
Captures microphone input and chunks 16kHz 16-bit PCM audio.
2. WebSocket Gateway
Maintains bi-directional socket connections with heartbeat and reconnection.
3. VAD & Whisper STT
Detects speech boundaries and transcribes audio into text streams.
4. Gemini 2.0 Flash Stream
Generates streaming token responses with contextual memory.
5. Edge-TTS Audio Streamer
Synthesizes spoken sentences into binary audio buffers for immediate client playback.
⚡ Deterministic Execution Pipeline (End-to-End Flow)
- 1Browser microphone captures user voice using Web Audio API AudioWorkletNode.
- 2Continuous PCM audio stream transmitted over secure WebSocket channel to FastAPI backend.
- 3Voice Activity Detection (VAD) monitors speech pauses to trigger prompt finalization without manual clicks.
- 4Transcribed speech passes to Gemini 2.0 Flash API with system instructions tuned for concise conversational pacing.
- 5As Gemini streams response sentences, Edge-TTS synthesizes audio chunks in parallel.
- 6Client receives audio byte stream and plays immediately through Web Audio context while Iron Man HUD animates waveform frequencies.
Technical Tradeoffs & Architecture Decisions
Why specific design decisions were chosen over common alternatives
Engineering Rationale: WebSockets allow full-duplex communication: user can interrupt assistant mid-sentence by speaking, triggering immediate audio cancellation.
Engineering Rationale: Local VAD filtering discards ambient silence before transcription, drastically reducing latency and cloud token costs.
Failure Handling & Edge-Case Resilience
Protecting uptime, data integrity, and degraded operational states
- !Exponential Backoff Reconnect: WebSocket client automatically attempts reconnecting with exponential backoff on network drop.
- !Audio Context Recovery: Gracefully recovers from browser suspended audio policies upon user gesture.
- !LLM Fallback: If Gemini API encounters rate limits or latency spikes, falls back to lightweight cached intent handlers.
Security, Privacy & Data Retention
Ethical data handling and client isolation principles
- 🔒Ephemeral Audio Streams: Microphone audio is processed in-flight in memory and never written to disk or persistent storage.
- 🔒CORS & WebSocket Origin Validation: Rejects unauthorized cross-origin socket initiation attempts.
Results & Measurable Outcomes
Verified performance metrics and business deliverables
- ★Sub-350ms end-to-end voice loop latency achieved.
- ★Interactive visual HUD responding to dynamic audio frequencies in real-time.
- ★Published open-source codebase with setup instructions and architecture documentation.
Known Limitations
- •Requires modern browser supporting Web Audio API and getUserMedia permissions.
- •High ambient background noise can occasionally trigger premature Voice Activity Detection interrupts.
Future Roadmap
- •WebRTC direct data channel support for ultra-low latency mobile streaming.
- •Local on-device Whisper model running via WebAssembly/WebGPU.
Explore More or Review Credentials
Ready to see how jarvis-realtime-assistant fits into real-world production engineering?