On 18 November 2026 AssemblyAI released STT-3.7 as three models. The two streaming ones matter for anyone building live call or interview overlays:
- AssemblyAI-STT-3.7-Realtime for meetings, captions, and ambient transcription ($0.30/hr). Speaker labels are final the moment a turn ends, no revision pass.
- AssemblyAI-STT-3.7-Realtime-Agent for voice agents ($0.45/hr). Tuned for short replies, numbers, and entities, with Voice Focus cutting background speech.
Both support 32 languages in one stream with automatic detection. AssemblyAI’s own measurements put the Realtime model at the lowest live diarization error rate they saw on telephone calls (CallHome) and multi-party meetings recorded on a single distant microphone (NOTSOFAR), ahead of ElevenLabs Scribe v3 Live and Meta Muse Voice Transcribe. The Agent variant is quoted around 86 ms to final on their voice-agent corpus.
Those numbers are real and useful. Final labels at turn end remove the old “wait for a correction” problem that made live diarization annoying for notetakers and coaching overlays. A model that can keep up with short technical answers and names is also practical for suggestion engines that need to see “who just said the API contract detail.”
For a developer shipping a Windows overlay that shows translated suggestions or short replies during a technical interview or sales call in another language, the model is still only half the stack.
The dual-track capture path still dominates
A usable live overlay needs two clean PCM streams that stay separate:
- Local microphone (what you say).
- Loopback of the render device (what the other party says through Teams, Zoom, Meet, or the browser).
Both go through the Windows Audio Engine (audiodg.exe + audioeng.dll) in shared mode so the call client keeps control of the device. Exclusive mode would break the call. Typical shared-mode period is 10 ms. You can request a smaller period with IAudioClient3; the engine frequently ignores it or falls back under load. Communications enhancements, spatial audio, vendor APOs, and Bluetooth HFP/A2DP add another 10–40+ ms and become non-deterministic when the CPU or GPU is busy.
A practical stack looks like this:
- Endpoint device → WASAPI shared-mode client (10–20 ms buffer).
- Audio engine mixing and any enabled enhancements.
- Capture callback (copy PCM, do minimal work, return immediately).
- Framing + optional local VAD or silence detection.
- Streaming STT (AssemblyAI-STT-3.7-Realtime or Agent, or local GGUF).
- Local model for translation polish or short suggestion.
- Overlay paint on a dedicated thread (<16 ms if you stay off the UI thread).
The model inference cost for a short suggestion is now low enough that it no longer dominates. The first 20–50 ms are still spent inside the audio engine before any sample reaches a model. Add network RTT if the STT is remote, framing, and the time for the first usable partial, and the overlay still appears tens to hundreds of milliseconds after the speaker finishes. That is the floor.
Why dual-track matters more than the model name
Single-stream capture (mic only, or loopback only) is useless for a bilingual call. You need both sides so the translator and the suggestion engine see the full conversation. Mixing them in one buffer loses speaker identity and makes barge-in detection harder. Separate tracks let you:
- Run independent VAD or energy gates.
- Apply different gain or noise profiles.
- Feed context only for the remote side.
- Keep the overlay responsive even when one track is quiet.
On Windows the cleanest way is two IAudioClient instances: one for the capture device, one for the loopback of the default render device (or the device the call client is using). Open both in shared mode, event-driven or polling, with a small period if the driver allows it. Copy into lock-free SPSC rings on the callback thread. Do nothing else there—no allocation, no locks, no network.
// Minimal callback sketch — real code needs proper error handling and format negotiation HRESULT CaptureCallback::OnData(const BYTE* data, UINT32 numFrames) { // Copy into pre-allocated ring, return immediately ring_->Write(data, numFrames * frameSize_); SetEvent(consumerEvent_); return S_OK; } The consumer thread pulls, resamples if needed (48 kHz float32 → 16 kHz mono int16 is common), runs a cheap VAD, and ships 20–40 ms chunks over the WebSocket. Speaker labels from the STT arrive later; you can still paint partials without them.
Tradeoffs that actually show up in production
- Shared vs exclusive: Shared is mandatory if the call app must keep working. Exclusive gives lower latency but kills the call.
- Period size: Requesting 5–8 ms sometimes works; more often the engine falls back to 10 ms under load or with Bluetooth.
- Enhancements: Turn off communications enhancements and spatial audio on the endpoints you care about. They add latency and make the path non-deterministic.
- Bluetooth: HFP is especially bad. Prefer a wired headset or a good USB one when latency matters.
- Local vs remote STT: Local GGUF removes RTT but costs CPU/GPU. Remote AssemblyAI or similar is fine if the network is stable; the capture floor is still there either way.
- Diarization timing: Final labels at turn end are better than revision passes, but you still cannot act on “who said it” until the turn closes. Partials are unlabeled or provisional.
The AssemblyAI numbers move the model half of the stack forward. They do not move the Windows audio engine. Until the capture path gets shorter or more predictable, that is the number that decides whether the overlay feels live.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.





