How to cut a video by its transcript with ffmpeg

Text-based video editing looks like a word processor, but underneath it is a list of time ranges and one ffmpeg command. The transcript gives each word a start and an end time, the words you delete become cuts, and everything between the cuts is kept. The video editor in the osFoundry desktop app does this in Rust (canvas_transcript.rs). Below is the pipeline written out in Python, run on a real 37.3-second interview clip, and a comparison with the shortcut that looks equivalent but isn't: cutting each piece with -c copy and joining them.

Start from word timings

Any transcriber that returns word timestamps works. Whisper does, and so do most hosted speech-to-text APIs when asked. The clip here is a NASA interview with astronaut Christina Koch, 640×360 at 30 fps, and its transcript is 119 words:

{"words": [ {"index": 0, "text": "Well", "start_ms": 0, "end_ms": 500}, {"index": 1, "text": "I", "start_ms": 740, "end_ms": 840}, {"index": 2, "text": "think", "start_ms": 840, "end_ms": 1060}, ... ], "duration_ms": 37300} 

Words are addressed by index, not by time, because an index stays the same while the user selects and deselects. The edit below deletes words 30 to 81 (a 16.7-second tangent) and 111 to 118 (the sign-off).

The app sends only the audio to the transcriber, and that step is worth copying even for a script. Providers charge by duration, but there is no reason to upload the video track:

ffmpeg -i clip.mp4 -vn -ac 1 -ar 16000 -f wav clip.wav 

Snap every cut to a frame

A word that starts at 9,420 ms starts in the middle of a frame at 30 fps. If the cut time isn't moved to a frame boundary, the number the user sees and the frame ffmpeg actually cuts at differ. The app rounds to the nearest frame rather than truncating, and its comment explains why: truncating pulls every cut earlier, and over a dozen edits that audibly clips the start of words.

def snap(ms, fps): frame = 1000 / fps return round(round(ms / frame) * frame) 

Turn deleted words into kept ranges

The deleted words become cut spans. Overlapping or touching spans merge, and whatever lies between the cuts is kept, including the pauses between kept words. That last rule is what keeps the result from sounding like an automated jump-cut edit: only the time a deleted word occupies goes, not every silence.

def keep_ranges(words, deleted, duration, fps): cuts = sorted((snap(w["start_ms"], fps), snap(w["end_ms"], fps)) for w in words if w["index"] in deleted) merged = [] for s, e in cuts: if merged and s <= merged[-1][1]: merged[-1][1] = max(merged[-1][1], e) else: merged.append([s, e]) keeps, cursor = [], 0 for s, e in merged: if s > cursor: keeps.append((cursor, s)) cursor = max(cursor, e) if cursor < duration: keeps.append((cursor, duration)) return [k for k in keeps if k[1] - k[0] >= 1000 / fps] # a sub-frame keep can't be encoded 

For this edit it returns four ranges, not the two you might expect:

(0, 9433) (16833, 16967) (26133, 34700) (36933, 37300) 

The second one is 134 ms long. Inside the deleted tangent, the speaker pauses for 120 ms between "road" and "but". No deleted word covers that pause, so the rule "keep every gap" keeps it, and the output gets a four-frame flash of the middle of a sentence you deleted. The last range is the same thing at the end: 367 ms of room tone after the deleted sign-off.

The fix is to say what a pause belongs to. A gap between two deleted words, or between a deleted word and the start or end of the clip, goes with them. A gap next to a kept word stays:

def cut_spans(words, deleted, duration, fps): spans = [] for i, w in enumerate(words): if w["index"] not in deleted: continue s, e = w["start_ms"], w["end_ms"] if i == 0 or words[i - 1]["index"] in deleted: s = 0 if i == 0 else words[i - 1]["end_ms"] if i == len(words) - 1 or words[i + 1]["index"] in deleted: e = duration if i == len(words) - 1 else words[i + 1]["start_ms"] spans.append((snap(s, fps), snap(e, fps))) return spans 

With cut_spans in place of the word spans, the same edit gives two ranges, (0, 9433) (26133, 34700), and an 18.0-second result.

Trim video and audio in one graph

Each range becomes a trim for video and an atrim for audio, and one concat joins them all, video and audio together:

def graph(keeps): p = [] for i, (s, e) in enumerate(keeps): p.append(f"[0:v]trim=start={s/1000:.3f}:end={e/1000:.3f},setpts=PTS-STARTPTS[v{i}]") p.append(f"[0:a]atrim=start={s/1000:.3f}:end={e/1000:.3f},asetpts=PTS-STARTPTS[a{i}]") p.append("".join(f"[v{i}][a{i}]" for i in range(len(keeps))) + f"concat=n={len(keeps)}:v=1:a=1[outv][outa]") return ";".join(p) subprocess.run(["ffmpeg", "-y", "-i", "clip.mp4", "-filter_complex", graph(keeps), "-map", "[outv]", "-map", "[outa]", "out.mp4"], check=True) 

This re-encodes the whole output, which is the cost: 4.5 to 7.4 seconds for this clip with ffmpeg 7.1 on an Apple M1, depending on what else was running.

Why not -c copy

The obvious faster route is to cut each range without re-encoding and join the pieces with the concat demuxer:

ffmpeg -ss 26.133 -to 34.700 -i clip.mp4 -c copy seg2.mp4 ffmpeg -f concat -safe 0 -i list.txt -c copy out-copy.mp4 

Both versions were run on the same four ranges and checked against the source. Video frames were matched to the source by hash (the copy keeps the original frames) and by nearest thumbnail, and audio by correlating short windows of the output against the source audio.

The source has a keyframe only every 8.33 seconds, so a copied cut can't start where the transcript says. Each copied piece came out with video and audio of different lengths (8.73 s against 8.58 s for the third range), the join brought back four frames of a deleted word, and ffmpeg warned about timestamps that go backwards while decoding the joined audio. The matches on the copied audio were also weak (correlation 0.2 to 0.8, against 1.00 for the filter graph), so "about one second" is approximate. What is clear is that it was not in sync, and a viewer would see lips that don't match the words.

Stream copy is fine for one cut at a keyframe. For cuts placed by a transcript, the re-encode is the price of the cuts being where the words are.

The whole thing

The pieces above come to under 60 lines of Python, using only the standard library and an ffmpeg binary. The canvas video editor in osFoundry puts this kind of pipeline behind a transcript panel: drag over words, Delete, Apply cuts. The app is free from the osFoundry download page.