Author: Wang Guo (@wangguode)
Source Lab: https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein
Stack: FreeSWITCH 8kHz Raw RTP / Pipecat Full-Duplex / SenseVoice / SOXR VHQ Resampling
1. The Telephony Gap: Browser Demos vs. PSTN Reality
Most Voice AI tutorials work great over a clean laptop microphone in a quiet room using browser WebSockets. The moment you connect that same pipeline to real enterprise telephony (PSTN / SIP trunks / FreeSWITCH), three brutal physical realities hit at once:
- The 8kHz vs. 16kHz Sample-Rate Wall: Traditional PSTN/SIP networks stream
8kHzG.711 (PCMU/PCMA) orL16audio, while modern acoustic models (SenseVoice, Whisper, Silero VAD) expect16kHzPCM. Naïve linear interpolation introduces aliasing artifacts that tank ASR accuracy and cause false VAD triggers. - TCP Head-of-Line Blocking: Tunneling continuous 20ms telephony audio frames over TCP WebSockets creates packet-queue stalls under cellular jitter—turning a 50ms interruption into a 700ms cross-talk collision.
- Echo & Unflushed Outbound Ring Buffers: Even when your backend decides to stop speaking, an unflushed 20ms–200ms outbound RTP playback queue keeps talking over the caller.
2. The 60ms Enterprise RTP Barge-In Architecture
In our open-source lab (01-voice-agent-rtp-bargein), we bypassed generic JSON-over-WebSocket wrappers and engineered a deterministic telephony loop:
[PSTN / SIP Caller (8kHz RTP, 20ms frames)] │ ▼ (UDP Raw RTP) [FreeSWITCH Media Gateway] │ ├──► [SOXR VHQ Upsampler (8kHz ➔ 16kHz)] ──► [SenseVoice / Acoustic VAD (~18ms)] │ │ │ ▼ (Barge-In Triggered!) └──◄ [256-Frame Outbound Ring Buffer] ◄──── [0.16ms clearAudio() Hard Flush] Key Architectural Primitives
- 20ms Raw RTP Frame Alignment: Audio arrives in strict 20ms packets (160 samples at 8kHz). Processing directly on frame boundaries avoids arbitrary chunk buffering delays.
- SOXR VHQ Zero-Artifact Resampling: Using SoX Resampler (
SOXR_VHQ) for8kHz <-> 16kHzconversion preserves vocal formants and prevents high-frequency aliasing from poisoning the local VAD classifier. - 256-Frame Outbound Ring Buffer + 0.16ms
clearAudio(): Rather than letting TTS audio accumulate in an uncontrolled OS socket buffer, outbound TTS frames sit in a fixed 256-frame circular buffer. The instant acoustic barge-in fires,clearAudio()resets the read/write pointers in0.16ms—producing immediate silence on the caller's phone.
3. Why Decoupling Interruption from Transcription Matters
A common mistake in Voice AI is waiting for the LLM or full-sentence STT transcript to confirm "the user is interrupting". Transcription requires context windows (often 200ms–400ms).
By splitting the pipeline into two asymmetric loops:
- Fast Loop (~50–60ms): Acoustic VAD + energy/formant gate ➔ triggers immediate
clearAudio()buffer flush. - Semantic Loop (~200–400ms): Full ASR transcript ➔ feeds the LLM context & updates turn state (discarding false backchannels like "uh-huh" or "yeah" without permanently derailing the agent's thought).
4. Explore the Open-Source Lab
All reference code, architecture diagrams, and runnable barge-in experiments are open-sourced:
- RTP Barge-In Lab: https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein
- Web & macOS Companion (HardCall 52ms): https://github.com/wangguode/hardcall
