Short-form captions are not a decoration. On mute, they are the audio. A YouTube Short or TikTok clip that highlights the current word, stays inside the safe area, and does not cover the face will usually outperform a burned-in paragraph that arrives late and sits too high. You do not need a cloud editor for that. A local pipeline with faster-whisper and FFmpeg is enough to caption a talking-head clip on your own machine, including German speech.

This guide walks through a working path: extract audio, get word timestamps, turn them into an ASS file with a live-word highlight, and burn the result into a 9:16 clip. It is the caption stage only. Cutting, silence removal, and face framing are separate problems; I will point at them, not pretend they are solved here.

What “good enough” captions look like on a Short

A useful default for a 9:16 talking-head:

  • Two lines at most, roughly three to five words per line, centered in the lower third.
  • The current word is a different colour or weight. The rest of the line stays readable.
  • Leave a margin above the platform UI (the caption block often sits around 70–80% down the frame, not on the very bottom edge).
  • Do not cover the mouth. If you already reframed the face into the upper two thirds, the lower third is free.
  • Timing follows the word, not the sentence. A word that appears 400 ms late feels sluggish; one that appears early feels like a leak.

SRT can do the job if you only need sentences. Word-level highlight is much easier in ASS.

Step 1: extract a clean mono track

Whisper does not need a 48 kHz stereo mix. A 16 kHz mono WAV is smaller, faster, and honest enough for speech.

ffmpeg -y -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav

If the source is already a clip you will publish, use that same file as input.mp4. If you caption a long podcast first and cut later, word times will still be correct for the long file; you will have to shift them when you cut. It is usually simpler to caption after you have the 20–60 second clip.

Step 2: transcribe with word timestamps

faster-whisper is the practical local choice: same word timings as Whisper, less RAM, good CPU performance with int8. A first script:

from faster_whisper import WhisperModel model = WhisperModel("small", device="auto", compute_type="int8") segments, info = model.transcribe( "audio.wav", language="de", # or "en"; omit to let it detect word_timestamps=True, vad_filter=False, beam_size=5, ) words = [] for seg in segments: for w in seg.words: token = (w.word or "").strip() if not token: continue words.append({"start": w.start, "end": w.end, "text": token}) print(info.language, len(words), "words")

A few honest limits:

  • small is a sensible default on a laptop. medium is cleaner on messy German audio and proper names; it is also slower. tiny will caption a demo, but it will miss names.
  • Set language if you know it. Auto-detect on a short clip can flip to the wrong language after a loanword or a title in English.
  • For files longer than a few minutes, transcribe in chunks of about 60 seconds and cut at a quiet spot, then stitch the word lists. A single pass on a long episode can drift.
  • Do not treat the text as final. Whisper will invent a plausible name. Keep a small replacements file ("Müller", "Kelvaro", product names) and apply it before you render.

Times are in seconds, float, relative to the start of audio.wav.

Step 3: wrap words into lines, then into ASS events

Do not emit one event per word if you can avoid it. Build a line of three to five words, then, for each word in that line, write an ASS event that covers only that word’s time span and highlights just that word. The line stays on screen; the highlight moves.

A compact way to wrap: accumulate words until the next one would exceed about 24 characters or a five-word cap, and break before a new clause if you can (und, aber, weil, and, but). Avoid ending a line on a tiny function word if the next word is the payload.

ASS needs timestamps as H:MM:SS.cc (centiseconds). A highlight can be a colour override:

[Script Info] ScriptType: v4.00+ PlayResX: 1080 PlayResY: 1920 WrapStyle: 0 [V4+ Styles] Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding Style: Word,Poppins ExtraBold,72,&H00FFFFFF,&H00000000,&H00101010,&H80000000,-1,0,0,0,100,100,0,0,1,6,0,2,80,80,280,1 [Events] Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text Dialogue: 0,0:00:01.20,0:00:01.55,Word,,0,0,0,,Clips {\c&H00E6C600&}automatisch{\c&H00FFFFFF&} schneiden

Notes that save pain later:

  • In ASS, colours are &HBBGGRR&, not HTML. The example highlight is a yellow.
  • Alignment=2 is bottom-centre. MarginV=280 on a 1920-tall frame lifts the block above typical UI chrome; tweak it on a real screenshot of YouTube Shorts or TikTok, not by taste alone.
  • Use a font you actually ship (Poppins, Inter, or a bundled OFL font). A missing font silently falls back and changes the layout.
  • Escape ASS special characters in the spoken text: {, }, and backslashes.

Generate the Dialogue lines in Python from your word list. Keep the full line visible in every event of that group; only the active word gets the \c override.

Step 4: burn the captions in with FFmpeg

Once you have captions.ass and a 9:16 clip.mp4:

ffmpeg -y -i clip.mp4 -vf "ass=captions.ass" -c:a copy captioned.mp4

If the clip is not 9:16 yet, do the scale/crop first, then caption, so the PlayResX/PlayResY of the ASS file match the frame you actually burn into. Mixing a 1920×1080 source with a 1080×1920 ASS style is the usual reason text lands in the wrong place.

The subtitles= filter can also read ASS, but it goes through a slightly different font-config path. If a font you installed does not appear, ass= plus an explicit fonts directory (FFmpeg’s fontsdir option on the filter) is the more predictable route.

A short quality checklist

Watch the file once, with sound off, then once with sound on.

  • Names, numbers, and product words: fix them in the word list, do not “paint over” in a timeline.
  • Words that start mid-breath: shift the start forward a little rather than letting a caption flash on a pause.
  • Lines that sit on the face or on a hook title: drop MarginV or reserve the top of the frame for the title (about the first 2.5 seconds).
  • German compounds and capitalization: Whisper sometimes splits or lowercases them. A replacements file is faster than a second model.

And the non-technical constraint: only caption recordings you or your client have the rights to. Running the model locally does not change copyright.

Where this stops

This path captions a clip you already have. It does not pick the six best moments in a 20-minute talk, drop silences, or keep a face in the middle of a 9:16 crop. Those are earlier stages, and they are worth automating only after the captions look right — because a wrong word, highlighted, is more visible than a slightly late cut.

If you want that full local pipeline instead of wiring only the caption stage, I packaged it as a pay-once kit (the Auto-Clipping-System): Python source, a German step-by-step guide, word-level captions (including German), silence removal, face framing, hook titles, and optional zooms or B-roll. It is 29 € once, no subscription. Windows is the tested baseline. Preview every file before you post. The kit is here: https://whop.com/kelvaro-creator-kits/auto-clipping-system

Disclosure: that kit is my own paid product. No guarantee of views or revenue.