How to Crop a Video With FFmpeg (16:9 to 9:16, One Window per Shot)

To crop a video with FFmpeg, use the crop filter with an output width, height and top-left offset, re-encode the video and copy the audio:

ffmpeg -i in.mp4 -vf "crop=606:1080:(iw-ow)/2:0,setsar=1" \ -c:v libx264 -crf 18 -pix_fmt yuv420p -c:a copy out.mp4 

That turns a 1920×1080 clip into a centered 606×1080 vertical (9:16) clip. It is also where most vertical crops go wrong: a single centered window is fine for one static shot, but an edited clip has several shots, and the subject is rarely in the middle of all of them. Below I crop a real four-shot clip, show where the center crop fails, and fix it with one crop window per shot driven by FFmpeg's own scene detection. Everything is checked with 11 pytest tests.

The test clip

A 15.16 s, 1920×1080, 24 fps clip with AAC audio: a four-shot classroom scene with two students. It is one of the text-to-video examples on Voor's HappyHorse 1.1 page, so you can download the same file and reproduce every number here. ffprobe with -count_frames reports 363 frames.

Crop syntax in 30 seconds

crop=w:h:x:y keeps a w×h rectangle whose top-left corner is at (x, y). Inside the expressions, iw/ih are the input size and ow/oh the output size, so (iw-ow)/2 centers the window horizontally. Add setsar=1 so players do not stretch the result.

Cropping changes pixels, so the video has to be re-encoded. If you try -c:v copy, FFmpeg refuses with "Filtering and streamcopy cannot be used together". The audio is untouched, so -c:a copy is safe and keeps it bit-exact.

The width you ask for is not always the width you get

9:16 of 1080 is 607.5 px, and yuv420p needs even dimensions. Here is what FFmpeg 9.0.1 actually did:

crop quietly rounds to the chroma grid, so your script and your file can disagree by two pixels without any error. That matters when you later compute offsets or stack crops side by side. Compute an even size yourself:

def even(n: float) -> int: return int(n) // 2 * 2 def crop_size(src_w, src_h, ratio_w, ratio_h): w, h = src_h * ratio_w / ratio_h, src_h if w > src_w: # source narrower than target: crop height instead  w, h = src_w, src_w * ratio_h / ratio_w return even(w), even(h) crop_size(1920, 1080, 9, 16) # (606, 1080) crop_size(1920, 1080, 1, 1) # (1080, 1080) 

Why the center crop fails on this clip

The top row of the image above is the plain centered crop. Shot 2 cuts straight between the two faces, and in shot 4 the window lands on the blurry background between them. Nothing is technically wrong with the file; the framing is just useless.

Step 1: find the shot boundaries with scdet

FFmpeg's scdet filter scores every frame for a scene change and can print the cut times:

import re, subprocess def detect_shots(path, threshold=10.0): out = subprocess.run( ["ffmpeg", "-hide_banner", "-nostats", "-i", path, "-vf", f"scdet=threshold={threshold},metadata=print:file=-", "-an", "-f", "null", "-"], capture_output=True, text=True, check=True).stdout return [float(t) for t in re.findall(r"lavfi\.scd\.time=([0-9.]+)", out)] detect_shots("happyhorse-t2v.mp4") # [1.083333, 4.25, 9.0] 

Three cuts, four shots. Pull one frame from the middle of each shot (-ss 0.5, 2.6, 6.6, 12) into a contact sheet and pick one x per shot by eye. Here that took four numbers: 980, 1040, 633 and 1150.

Step 2: one crop expression that changes x at each cut

crop evaluates x per frame, and t is the frame time, so a nested if(lt(t,…)) switches windows exactly at the cuts. Commas inside a filter argument must be escaped:

def piecewise_x(cuts, xs): assert len(xs) == len(cuts) + 1, "need one x per shot" expr = str(xs[-1]) for cut, x in reversed(list(zip(cuts, xs[:-1]))): expr = f"if(lt(t\\,{cut:.4f})\\,{x}\\,{expr})" return expr def crop(src, dst, w, h, x, y=0, crf=18): vf = f"crop={w}:{h}:{x}:{y},setsar=1" return subprocess.run( ["ffmpeg", "-v", "error", "-y", "-i", src, "-vf", vf, "-c:v", "libx264", "-crf", str(crf), "-preset", "medium", "-pix_fmt", "yuv420p", "-c:a", "copy", "-movflags", "+faststart", dst], capture_output=True, text=True) w, h = crop_size(1920, 1080, 9, 16) crop("happyhorse-t2v.mp4", "vertical.mp4", w, h, piecewise_x(detect_shots("happyhorse-t2v.mp4"), [980, 1040, 633, 1150])) 

The bottom row of the first image is the result: every shot keeps one readable face. Shot 4 still loses one of the two speakers. At 9:16 there is no single window that keeps both, so either pick the speaker that matters or reframe the shot.

Step 3: prove the output with tests

These are the checks that run against the real clip (abridged):

def test_crop_silently_rounds_odd_width_but_scale_fails(): ... # crop=607:1080 -> 606 wide, rc 0; scale=607:1080 -> "not divisible by 2"  def test_shot_cuts_match_the_example(): assert [round(c, 2) for c in detect_shots(SRC)] == [1.08, 4.25, 9.0] def test_frames_duration_and_audio_survive(per_shot): a, b = probe(SRC), probe(per_shot) assert (b["w"], b["h"], b["sar"]) == (606, 1080, "1:1") assert b["frames"] == a["frames"] == 363 assert abs(b["duration"] - a["duration"]) < 1 / 24 @pytest.mark.parametrize("t,shot", [(0.5, 0), (2.6, 1), (6.6, 2), (12.0, 3)]) def test_each_shot_uses_its_own_window(per_shot, t, shot): # output frame vs the same region cut from the source frame with Pillow  assert psnr(out, right) > 30 assert psnr(out, right) > psnr(out, centre) + 3 def test_cropdetect_finds_letterbox(): ... # 1920x800 padded to 1080 -> crop=1920:800:0:140 def test_stream_copy_cannot_crop(): ... 

11 passed in 5.36s 

The PSNR test is the useful one: it pulls a frame from each shot, cuts the window you meant from the source frame with Pillow, and requires the output to match it better than the centered window. If the cut detection drifts by a few frames or someone edits the offsets, it fails.

Bonus: remove black bars with cropdetect

If the input is letterboxed, let FFmpeg measure the picture area instead of guessing:

ffmpeg -t 2 -i boxed.mp4 -vf cropdetect=limit=24:round=2:reset=0 -f null - 2>&1 | grep -o "crop=[0-9:]*" | sort | uniq -c 

On a 1920×800 picture padded to 1920×1080 it reports crop=1920:800:0:140. Feed that straight into -vf crop=….

When not to crop at all

Cropping 16:9 to 9:16 throws away about 68% of the width (606 of 1920 px). If the vertical version is the one you actually need, it is often better to generate or shoot it vertical. The same HappyHorse 1.1 model has a 9:16 aspect ratio option; on Voor, 9:16 at 720p was quoted at 244 credits for 5 s and 732 credits for 15 s when I checked (Dialogue scene preset, prompt required). You can generate the vertical version directly and skip the crop, but you will still want the test above if you crop anything else.

Limitations

• The per-shot x values were picked by eye from a contact sheet. There is no subject tracking here; a subject that moves within a shot needs keyframed offsets or a tracker.
• scdet thresholds are content-dependent. Fades and dissolves may not register as cuts; check the list against the contact sheet.
• Measurements are from FFmpeg 9.0.1 on macOS (Apple Silicon) with libx264; older builds may round crop sizes differently.