When a caller cuts off a voice agent halfway through a sentence, two things have to happen. The audio has to stop right away, and the agent's memory of the conversation has to be fixed so it only contains the words the caller actually heard. Most of us get the first part working in an afternoon. The second part is easy to miss, and when it's missing the agent starts acting on things the caller never heard.

This post walks through both parts on a Twilio phone line, with a short note on how the OpenAI Realtime API handles the same problem.

Why the transcript drifts

LLMs and TTS engines produce audio faster than real time. By the time the caller hears the first word of a reply, your server may have already generated and sent the whole thing. So your conversation history says the assistant spoke three full sentences, while the caller only heard the first five words before saying "no, wait".

Here's a simple example. The agent is booking a dentist appointment:

Agent (generated): "I can book you for Tuesday at 3 PM. I'll also cancel your Friday slot so you don't have two bookings. Is that okay?"

Caller (heard): "I can book you for Tuesday at..." and then says "Actually, can we do Wednesday?"

If the full reply stays in the history, the model now believes it told the caller about cancelling Friday, and that the caller moved on without objecting. A few turns later it might say "as I mentioned, your Friday slot is cancelled." The caller never heard that. Worse, if your code treats "the agent said it and the caller didn't object" as a kind of consent, you've built a real bug.

The fix is to keep a record of what was played, separate from what was generated, and cut the history down to the played part when an interruption happens.

Step 1: stop the audio for real

On a bidirectional Twilio Media Stream (the kind you start with <Connect><Stream>), you send audio to the call as media messages. Twilio buffers that audio and plays it. If you only stop your LLM and TTS when the caller speaks, Twilio keeps playing whatever is already in its buffer, which can be a few seconds of speech.

To flush that buffer, you send a clear message:

function stopPlayback(ws: WebSocket, streamSid: string) { ws.send(JSON.stringify({ event: "clear", streamSid })); } 

This is easy to forget when you use a framework adapter, so it's worth testing on a real phone call and not only in a browser demo. Talk over the agent and listen for how long it keeps going.

Step 2: track what was played with marks

Twilio gives you one tool for knowing what was played: the mark message. After you send some audio, you send a mark with a name. When that audio finishes playing, Twilio sends the mark back with the same name. If you send clear, Twilio empties the buffer and also sends back any marks that were still waiting.

That last detail matters. A returned mark doesn't tell you whether the audio finished or got cleared. The message looks the same in both cases. So your own code has to know which state it's in. A simple rule works: any mark that comes back after you sent clear belongs to audio that was thrown away.

I send audio in sentence-sized chunks and put a mark after each one. Each chunk keeps its text and its length in milliseconds. Twilio expects mu-law audio at 8000 Hz, which is one byte per sample, so 8 bytes of decoded audio equals 1 millisecond. That makes the duration math simple.

type Chunk = { mark: string; text: string; durationMs: number; startedAt?: number; // when we believe it started playing }; class PlaybackTracker { private queue: Chunk[] = []; private spoken: string[] = []; private clearing = false; private seq = 0; send(ws: WebSocket, streamSid: string, text: string, mulaw: Buffer) { const mark = `chunk-${++this.seq}`; const durationMs = mulaw.length / 8; // 8000 bytes per second ws.send(JSON.stringify({ event: "media", streamSid, media: { payload: mulaw.toString("base64") }, })); ws.send(JSON.stringify({ event: "mark", streamSid, mark: { name: mark } })); const chunk: Chunk = { mark, text, durationMs }; if (this.queue.length === 0) chunk.startedAt = Date.now(); this.queue.push(chunk); } onMark(name: string) { const i = this.queue.findIndex((c) => c.mark === name); if (i === -1) return; const [done] = this.queue.splice(i, 1); if (!this.clearing) { this.spoken.push(done.text); if (this.queue[0]) this.queue[0].startedAt = Date.now(); } if (this.queue.length === 0) this.clearing = false; } } 

Step 3: work out the cut point

When the caller barges in, the first chunk in the queue is the one that was playing. Everything before it was heard in full, because its mark already came back. Everything after it was never heard.

For the playing chunk you have to estimate. You know roughly when it started (when the previous mark came back) and how long it is, so the fraction played is the elapsed time divided by its duration. Then you keep that fraction of the words. Add this method to the tracker:

interrupt(ws: WebSocket, streamSid: string): string { this.clearing = true; ws.send(JSON.stringify({ event: "clear", streamSid })); const current = this.queue[0]; if (current?.startedAt) { const elapsed = Date.now() - current.startedAt; const ratio = Math.min(elapsed / current.durationMs, 1); const words = current.text.split(" "); const kept = words.slice(0, Math.floor(words.length * ratio)); if (kept.length) this.spoken.push(kept.join(" ") + "..."); } return this.spoken.join(" "); } 

This is an estimate. Network delay and Twilio's own buffering shift the timing a bit, and words aren't all the same length. A few things make it better:

  • Use word timings if your TTS gives them. Some TTS providers can return a timestamp for each word or character. Then you cut at the last word whose end time is before the elapsed time, instead of guessing from word count.
  • Round down. If you're not sure whether a word was heard, drop it. It's safer for the agent to repeat something than to assume the caller heard it.
  • Keep chunks short. Sentence-sized chunks mean the estimate only has to cover one sentence, not a whole paragraph.

Step 4: rewrite the history

Once you have the spoken text, replace the assistant's last message with it before the next LLM call. I also add a small note so the model knows it was cut off. Without that, the model sometimes picks up as if it had finished its point.

function applyInterruption(history: Message[], spokenText: string) { const last = history[history.length - 1]; if (last?.role !== "assistant") return history; const content = spokenText.trim() ? `${spokenText} [interrupted by caller]` : "[interrupted before speaking]"; return [...history.slice(0, -1), { ...last, content }]; } 

With this, the dentist example becomes:

assistant: I can book you for Tuesday at... [interrupted by caller] user: Actually, can we do Wednesday? 

Now the model knows Friday was never mentioned, so it won't claim it was.

The tool call trap

There's one case where trimming text isn't enough. Sometimes the agent calls a tool and then speaks about it, like "Done, I've cancelled your Friday slot." If the caller interrupts that sentence, the words get cut, but the cancellation already happened.

So keep tool results in the history even when the speech about them was cut. The history should show what actually happened in your systems, plus what the caller actually heard. If those two don't match, the model needs both facts so it can tell the caller about the change on its next turn. For risky actions, the safer design is to confirm before calling the tool at all, but that's a topic on its own.

If you use the OpenAI Realtime API

The Realtime API has this built in. When the caller interrupts, you send a conversation.item.truncate event with the assistant item's item_id, a content_index of 0, and audio_end_ms set to how much audio was actually played. The server cuts the audio at that point and removes the text transcript past it, so the model's context only holds what the user heard. It replies with a conversation.item.truncated event.

realtimeWs.send(JSON.stringify({ type: "conversation.item.truncate", item_id: currentAssistantItemId, content_index: 0, audio_end_ms: playedMs, })); 

You still have to know playedMs yourself, though. The server can't see your playback buffer, and on a phone line that buffer lives at Twilio. So the mark tracking above is still the part you build. One more thing to watch: if audio_end_ms is longer than the real audio, the server returns an error, so clamp it.

Not every sound is a barge-in

One last thing. All of this assumes the interruption was real. A caller saying "mm-hmm" or a cough on the line shouldn't stop the agent and trim its reply. If your barge-in trigger is just "voice activity detected," you'll cut the agent off all the time and fill the history with "[interrupted by caller]" notes that mean nothing. It helps to wait for a short stretch of real speech, or a couple of transcribed words, before you call interrupt().

Checklist

  • Send clear to Twilio on barge-in, not just stop generation.
  • Put a mark after each audio chunk and keep the chunk's text and duration.
  • Treat marks that come back after clear as unplayed audio.
  • Estimate the cut point in the playing chunk, and round down.
  • Replace the last assistant message with the spoken part and mark it as interrupted.
  • Keep tool results even when the speech about them was cut.
  • On the Realtime API, send conversation.item.truncate with the played duration.

The test I use is simple: interrupt the agent mid-sentence on a real call, then ask it "what did you just say?" If it repeats words you never heard, the history is still wrong.