Real-Time Voice for an Agentic Assistant

Most "voice AI" demos are push-to-talk: record a clip, transcribe it, send it to an LLM, speak back the response. That works for a toy, but it doesn't feel like a conversation. For Quanta — the agentic assistant I built at Rectify — the goal was full-duplex audio: the user can interrupt mid-sentence, and the assistant reacts the way a person on a call would.

Why the browser's built-in recorder wasn't enough

MediaRecorder is fine for capturing a clip, but it buffers in chunks that are too coarse for a live, interruptible stream. Gemini Live expects raw PCM16 frames on a tight cadence. The fix was a custom AudioWorklet running on the audio rendering thread, downsampling the mic input and posting fixed-size PCM16 frames back to the main thread over a MessagePort. No garbage collection pauses, no dropped frames under load.

Two audio formats, one pipeline

Twilio phone calls don't speak PCM16 — they speak µ-law at 8kHz. Rather than maintaining two separate voice pipelines (one for the web client, one for phone calls), Quanta normalizes both to PCM16 internally and does the µ-law encode/decode at the edges. That meant writing a small codec bridge rather than depending on a heavier media server, which kept latency down for a feature where every extra 100ms is noticeable.

What actually mattered for "feels real-time"

Buffer size tuning made a bigger difference than model choice. Too large, and responses feel laggy; too small, and you get choppy audio under normal network jitter. The setting that worked came from testing on an actual flaky connection, not a wired dev machine — a reminder that voice UX bugs rarely show up on localhost.