Author: Wang Guo (@wangguode)

Source Lab: https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein

Stack: FreeSWITCH 8kHz Raw RTP / Pipecat Full-Duplex / SenseVoice / SOXR VHQ Resampling

1. The Telephony Gap: Browser Demos vs. PSTN Reality

Most Voice AI tutorials work great over a clean laptop microphone in a quiet room using browser WebSockets. The moment you connect that same pipeline to real enterprise telephony (PSTN / SIP trunks / FreeSWITCH), three brutal physical realities hit at once:

  1. The 8kHz vs. 16kHz Sample-Rate Wall: Traditional PSTN/SIP networks stream 8kHz G.711 (PCMU/PCMA) or L16 audio, while modern acoustic models (SenseVoice, Whisper, Silero VAD) expect 16kHz PCM. Naïve linear interpolation introduces aliasing artifacts that tank ASR accuracy and cause false VAD triggers.
  2. TCP Head-of-Line Blocking: Tunneling continuous 20ms telephony audio frames over TCP WebSockets creates packet-queue stalls under cellular jitter—turning a 50ms interruption into a 700ms cross-talk collision.
  3. Echo & Unflushed Outbound Ring Buffers: Even when your backend decides to stop speaking, an unflushed 20ms–200ms outbound RTP playback queue keeps talking over the caller.

2. The 60ms Enterprise RTP Barge-In Architecture

In our open-source lab (01-voice-agent-rtp-bargein), we bypassed generic JSON-over-WebSocket wrappers and engineered a deterministic telephony loop:

[PSTN / SIP Caller (8kHz RTP, 20ms frames)] │ ▼ (UDP Raw RTP) [FreeSWITCH Media Gateway] │ ├──► [SOXR VHQ Upsampler (8kHz ➔ 16kHz)] ──► [SenseVoice / Acoustic VAD (~18ms)] │ │ │ ▼ (Barge-In Triggered!) └──◄ [256-Frame Outbound Ring Buffer] ◄──── [0.16ms clearAudio() Hard Flush] 

Key Architectural Primitives

  1. 20ms Raw RTP Frame Alignment: Audio arrives in strict 20ms packets (160 samples at 8kHz). Processing directly on frame boundaries avoids arbitrary chunk buffering delays.
  2. SOXR VHQ Zero-Artifact Resampling: Using SoX Resampler (SOXR_VHQ) for 8kHz <-> 16kHz conversion preserves vocal formants and prevents high-frequency aliasing from poisoning the local VAD classifier.
  3. 256-Frame Outbound Ring Buffer + 0.16ms clearAudio(): Rather than letting TTS audio accumulate in an uncontrolled OS socket buffer, outbound TTS frames sit in a fixed 256-frame circular buffer. The instant acoustic barge-in fires, clearAudio() resets the read/write pointers in 0.16ms—producing immediate silence on the caller's phone.

3. Why Decoupling Interruption from Transcription Matters

A common mistake in Voice AI is waiting for the LLM or full-sentence STT transcript to confirm "the user is interrupting". Transcription requires context windows (often 200ms–400ms).

By splitting the pipeline into two asymmetric loops:

  • Fast Loop (~50–60ms): Acoustic VAD + energy/formant gate ➔ triggers immediate clearAudio() buffer flush.
  • Semantic Loop (~200–400ms): Full ASR transcript ➔ feeds the LLM context & updates turn state (discarding false backchannels like "uh-huh" or "yeah" without permanently derailing the agent's thought).

4. Explore the Open-Source Lab

All reference code, architecture diagrams, and runnable barge-in experiments are open-sourced: