ElevenLabs passed $500M in annual recurring revenue this year, and a big part of what people pay for is simple to describe: take a video, keep the speaker's voice, and make them speak Spanish, Hindi or German. We made a reel about it this week (an uncle whose car starts talking in his own voice, in Spanish), and people asked how the pipeline actually works.

So here it is, end to end, on three open pieces: Whisper for speech to text, Claude for the translation, and Chatterbox Multilingual for the cloned voice. It runs on one rented GPU.

A rule before any code: only clone a voice with the owner's written permission. Store that consent next to the voice sample. Everything below assumes you have it.

The pipeline

  1. Whisper transcribes the video and gives you each line with a start and end time.
  2. Claude translates every line, keeping names, jokes and tone, and keeping each line short enough to say in the same slot.
  3. Chatterbox Multilingual speaks each translated line in the cloned voice, from a 10 to 20 second reference sample.
  4. ffmpeg places each new line at its original timestamp, nudges the speed if a line runs long, and muxes it back onto the video.

Setup 

pip install openai-whisper chatterbox-tts anthropic torchaudio export ANTHROPIC_API_KEY=... # your own key

You need ffmpeg on the PATH and a CUDA GPU for reasonable speed. Chatterbox runs on CPU too, just slowly.

Step 1: transcribe with timestamps 

import json, subprocess, sys import whisper, torchaudio as ta from anthropic import Anthropic from chatterbox.mtl_tts import ChatterboxMultilingualTTS VIDEO, VOICE, LANG = sys.argv[1], sys.argv[2], sys.argv[3] # in.mp4 consented_sample.wav es asr = whisper.load_model("small") segments = asr.transcribe(VIDEO)["segments"] lines = [{"id": i, "start": s["start"], "end": s["end"], "text": s["text"].strip()} for i, s in enumerate(segments)]

Whisper's segments are usually one sentence or a natural phrase, which is exactly the unit you want to dub. If a speaker talks very fast, the "medium" model splits lines more cleanly than "small".

Step 2: translate with Claude, line by line

The trick is to translate for the ear, not the page. A literal translation of a joke is rarely funny, and a long translation will not fit the slot.

client = Anthropic() prompt = f"""You are dubbing a video into the language with code "{LANG}". Translate each line so it sounds natural when spoken. Keep names, jokes and tone. Each translation must be speakable in about the same time as the original line. Return only JSON: a list of objects {{"id": int, "text": str}}, one per input line. Lines: {json.dumps(lines, ensure_ascii=False)}""" msg = client.messages.create(model="claude-sonnet-4-5", max_tokens=4000, messages=[{"role": "user", "content": prompt}]) raw = msg.content[0].text translated = {t["id"]: t["text"] for t in json.loads(raw[raw.find("["): raw.rfind("]") + 1])}

Sending the whole transcript in one call matters. Claude sees the context, so a pun set up in line 3 still lands in line 7, and names stay consistent.

Step 3: speak every line in the cloned voice 

tts = ChatterboxMultilingualTTS.from_pretrained(device="cuda") parts = [] for l in lines: wav = tts.generate(translated[l["id"]], language_id=LANG, audio_prompt_path=VOICE) path = f"line_{l['id']:03d}.wav" ta.save(path, wav, tts.sr) parts.append((l, path, wav.shape[-1] / tts.sr)) # file, length in seconds

audio_prompt_path is the consented reference sample. A clean 10 to 20 second recording with no music under it gives a far better clone than a longer, noisy one.

Step 4: put the lines back on the timeline 

inputs, filters = [], [] for n, (l, path, secs) in enumerate(parts): slot = max(l["end"] - l["start"], 0.3) speed = min(max(secs / slot, 1.0), 1.35) # only speed up, never above 1.35x ms = int(l["start"] * 1000) inputs += ["-i", path] filters.append(f"[{n + 1}:a]atempo={speed:.3f},adelay={ms}:all=1[a{n}]") mix = "".join(f"[a{n}]" for n in range(len(parts))) filters.append(f"{mix}amix=inputs={len(parts)}:normalize=0[dub]") subprocess.run(["ffmpeg", "-y", "-i", VIDEO, *inputs, "-filter_complex", ";".join(filters), "-map", "0:v", "-map", "[dub]", "-c:v", "copy", "-c:a", "aac", f"dubbed_{LANG}.mp4"], check=True)

Run it:

python dub.py talk.mp4 consented_sample.wav es

That is the whole thing, under 70 lines.

What breaks first, and the fixes

Lines that run long. Spanish and German are often 15 to 25 percent longer than English. The 1.35x speed cap keeps voices natural. If a line still overflows, send it back to Claude with "shorter, same meaning" and the target seconds. One retry fixes almost all of them.

Background music disappears. This script replaces the whole audio track. For real videos, split the original into voice and background with a source separation model first, then mix the new voice over the original background.

Numbers and names. TTS models read "$49" and "H3" in odd ways. Ask Claude to write numbers as words in the target language, and keep a small pronunciation list per creator.

Lip sync. This pipeline does not move the lips. For talking-head content, a lip-sync model is the next layer. For courses, screen recordings, podcasts and voiceover-led videos, you do not need it.

From script to product

The script is the easy 20 percent. A product people pay for monthly adds a consent flow and voice vault, a GPU job queue so ten uploads do not fight over one card, a review screen where the creator can fix any line before export, and per-minute metering. That is the part that decides margins.

The rough maths from our reel: 5,000 creators at $49 a month is about $3M a year, and the models cost nothing to license. You pay for GPU time and the Claude calls.

We wrote up the full open-source stack, the build order and the niches worth selling to here: https://axionry.com/open-source-ai-products/ai-voice-cloning-dubbing-app

If you want a product like this built for your own niche without hiring a team, that is what we do at Axionry: https://axionry.com