AI video generators are very good at motion and often silent. If you generate a clip of a ball bouncing, a door closing, or a glass being set on a table, you usually get pixels and no audio. Adding the right "thud" by hand in an editor is fine for one clip. It doesn't scale to a batch.
This post builds a small Python + ffmpeg pipeline that does three things:
- Finds the frames where an impact happens.
- Mixes a short sound effect at exactly those times.
- Measures how far the audio actually lands from the true impact time.
That third step is the one most tutorials skip. "It sounds about right" isn't a number. I wanted one.
All code below ran on a Linux box with Python 3.13, NumPy and ffmpeg 7.1. Everything is in three files you can copy.
The test clip is synthetic, on purpose
To measure alignment error you need ground truth: the exact moment each impact really happens. Real AI video doesn't give you that. So the test clip is synthetic: a white ball bouncing on a grey floor, rendered frame by frame with NumPy, piped into ffmpeg.
The bounce period is 1.27 s. At 30 fps that's 38.1 frames, so impacts deliberately fall between frames. A period like 1.3 s (exactly 39 frames) would land every impact on a frame and make any detector look perfect. I tried that first and got 0.0 ms error everywhere, which tells you nothing.
"""Render a synthetic silent clip: a ball bouncing on a floor. Ground-truth impact times are known exactly.""" import json, subprocess, numpy as np W, H, FPS, DUR = 640, 360, 30, 8.0 FLOOR, R, PERIOD = 300, 18, 1.27 # one bounce every 1.27 s (not frame-aligned on purpose) N = int(FPS * DUR) def y_at(t): ph = (t % PERIOD) / PERIOD # 0 -> 1 within a bounce return FLOOR - R - 220 * 4 * ph * (1 - ph) # parabola, touches floor at ph=0 impacts = [round(k * PERIOD, 4) for k in range(1, int(DUR / PERIOD) + 1) if k * PERIOD < DUR] ff = subprocess.Popen(["ffmpeg", "-y", "-loglevel", "error", "-f", "rawvideo", "-pix_fmt", "gray", "-s", f"{W}x{H}", "-r", str(FPS), "-i", "-", "-c:v", "libx264", "-pix_fmt", "yuv420p", "silent.mp4"], stdin=subprocess.PIPE) yy, xx = np.mgrid[0:H, 0:W] for i in range(N): t = i / FPS f = np.full((H, W), 30, np.uint8); f[FLOOR:, :] = 90 cx, cy = 80 + i * 2, y_at(t) f[(xx - cx) ** 2 + (yy - cy) ** 2 <= R * R] = 230 ff.stdin.write(f.tobytes()) ff.stdin.close(); ff.wait() json.dump({"fps": FPS, "impacts_s": impacts}, open("ground_truth.json", "w"), indent=1) print("impacts:", impacts) Output:
impacts: [1.27, 2.54, 3.81, 5.08, 6.35, 7.62] 
Step 1: detect impacts from motion
For this clip, detection is simple: threshold the bright ball, take its vertical centroid per frame, and call an impact wherever downward motion flips to upward. ffmpeg decodes to raw grayscale and Python reads it from a pipe, so there are no temporary image files.
The key detail is sub-frame refinement. The detected frame is only accurate to ±half a frame (±16.7 ms at 30 fps). Fitting a parabola through the three samples around the lowest point gives a fractional frame index. It's a cheap trick, and the measurements below show whether it pays off.
Step 2: mix the effect with adelay + amix
The sound effect here is also synthetic: a 120 ms decaying 70 Hz sine with a short click on top, generated with NumPy and written as a WAV. In a real pipeline you'd swap in a recorded or generated effect.
Each impact gets its own adelay branch (delay in milliseconds), then amix sums them. normalize=0 matters: without it, amix scales each input down by the number of inputs, so six thuds come out much quieter than one. The ffmpeg filters documentation covers both filters.
"""Detect impact frames in a silent video and mix a short 'thud' at each one with ffmpeg.""" import json, subprocess, sys, wave, numpy as np SRC, OUT = sys.argv[1], sys.argv[2] W, H = 640, 360 def probe_fps(path): r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", "stream=r_frame_rate", "-of", "csv=p=0", path], capture_output=True, text=True).stdout.strip() a, b = r.split("/"); return int(a) / int(b) def frames(path): p = subprocess.Popen(["ffmpeg", "-loglevel", "error", "-i", path, "-f", "rawvideo", "-pix_fmt", "gray", "-"], stdout=subprocess.PIPE) while (buf := p.stdout.read(W * H)) and len(buf) == W * H: yield np.frombuffer(buf, np.uint8).reshape(H, W) fps = probe_fps(SRC) # 1) track the brightest blob's vertical centroid per frame ys = [] for f in frames(SRC): m = f > 180 ys.append(np.nonzero(m)[0].mean() if m.any() else np.nan) ys = np.array(ys) # 2) an impact = local maximum of y (lowest on screen) where motion reverses from down to up v = np.diff(ys) hits = [i for i in range(1, len(v)) if v[i - 1] > 0 and v[i] <= 0] # refine to sub-frame with a parabola through the 3 samples around the peak def refine(i): a, b, c = ys[i - 1], ys[i], ys[i + 1] d = a - 2 * b + c return i + (0.5 * (a - c) / d if d else 0.0) REFINE = "--no-refine" not in sys.argv times = [(refine(i) if REFINE else i) / fps for i in hits if 0 < i < len(ys) - 1] # 3) synthesize a 120 ms thud (decaying 70 Hz sine + click) as the effect SR = 48000 t = np.arange(int(0.12 * SR)) / SR thud = (np.sin(2 * np.pi * 70 * t) * np.exp(-t * 35) + 0.3 * np.exp(-t * 900)) * 0.8 with wave.open("thud.wav", "wb") as w: w.setnchannels(1); w.setsampwidth(2); w.setframerate(SR) w.writeframes((thud * 32767).astype(np.int16).tobytes()) # 4) one adelay branch per hit, then amix, then mux onto the original video stream dur = len(ys) / fps inputs = ["-i", SRC] + sum([["-i", "thud.wav"] for _ in times], []) graph = ";".join(f"[{k+1}:a]adelay={int(round(tt*1000))}:all=1[a{k}]" for k, tt in enumerate(times)) graph += ";" + "".join(f"[a{k}]" for k in range(len(times))) + \ f"amix=inputs={len(times)}:normalize=0,apad,atrim=0:{dur:.3f}[mix]" subprocess.run(["ffmpeg", "-y", "-loglevel", "error", *inputs, "-filter_complex", graph, "-map", "0:v", "-map", "[mix]", "-c:v", "copy", "-c:a", "aac", "-b:a", "128k", OUT], check=True) json.dump({"fps": fps, "detected_s": [round(x, 4) for x in times]}, open("detected.json", "w"), indent=1) print(f"fps={fps} hits={len(times)}", [round(x, 3) for x in times]) Note -c:v copy: the video stream isn't re-encoded, so frames stay identical to the input.
Step 3: measure where the audio really lands
Here's the part I cared about. measure.py decodes the final MP4's audio track (after AAC encoding, not the intermediate WAV), finds each thud's onset with a simple amplitude threshold, and compares it with the ground truth.
"""Decode the output audio and measure where each thud actually starts vs ground truth.""" import json, subprocess, sys, numpy as np OUT = sys.argv[1]; SR = 48000 raw = subprocess.run(["ffmpeg", "-loglevel", "error", "-i", OUT, "-f", "f32le", "-ac", "1", "-ar", str(SR), "-"], capture_output=True).stdout x = np.abs(np.frombuffer(raw, np.float32)) thr = 0.1 * x.max(); onsets = []; i = 0 while i < len(x): if x[i] > thr: onsets.append(i / SR); i += int(0.3 * SR) else: i += 1 gt = json.load(open("ground_truth.json"))["impacts_s"] det = json.load(open("detected.json"))["detected_s"] rows = [] for g in gt: o = min(onsets, key=lambda t: abs(t - g)) if onsets else float("nan") d = min(det, key=lambda t: abs(t - g)) if det else float("nan") rows.append((g, d, o, (d - g) * 1000, (o - g) * 1000)) print("truth_s detected_s audio_onset_s detect_err_ms audio_err_ms") for r in rows: print(f"{r[0]:7.3f} {r[1]:10.3f} {r[2]:13.3f} {r[3]:13.1f} {r[4]:12.1f}") e = np.array([r[4] for r in rows]) print(f"onsets found={len(onsets)} truth={len(gt)} | audio err mean={e.mean():.1f} ms, " f"max |err|={np.abs(e).max():.1f} ms, frame={1000/json.load(open('ground_truth.json'))['fps']:.1f} ms") json.dump({"rows": rows, "onsets": onsets}, open("measure.json", "w")) Results
I ran the pipeline twice on the same clip: once snapping to the nearest frame (--no-refine), once with sub-frame refinement. These are the actual outputs:
== mode --no-refine truth_s detected_s audio_onset_s detect_err_ms audio_err_ms 1.270 1.267 1.267 -3.3 -3.0 2.540 2.533 2.533 -6.7 -7.0 3.810 3.800 3.800 -10.0 -10.0 5.080 5.067 5.067 -13.3 -13.0 6.350 6.333 6.333 -16.7 -17.0 7.620 7.633 7.633 13.3 13.0 onsets found=6 truth=6 | audio err mean=-6.2 ms, max |err|=17.0 ms, frame=33.3 ms == mode refine truth_s detected_s audio_onset_s detect_err_ms audio_err_ms 1.270 1.268 1.268 -1.5 -2.0 2.540 2.537 2.537 -2.6 -3.0 3.810 3.807 3.807 -2.9 -3.0 5.080 5.078 5.078 -2.2 -2.0 6.350 6.350 6.350 0.0 0.0 7.620 7.622 7.622 2.2 2.0 onsets found=6 truth=6 | audio err mean=-1.3 ms, max |err|=3.0 ms, frame=33.3 ms 
What the numbers say:
- Nearest-frame snapping: mean error −6.2 ms, worst case 17.0 ms. The error drifts because the 1.27 s period slides against the 33.3 ms frame grid, then wraps when the nearest frame flips to the other side.
- Sub-frame parabola: mean error −1.3 ms, worst case 3.0 ms. Roughly a 5–6× smaller worst case for a three-line function.
- The audio errors are whole milliseconds because
adelaytakes integer ms here. Detection error and final audio error match to within 1 ms, so the AAC encode and mux added no measurable offset in this run.
Both modes stay under one frame (33.3 ms) here, but the refined version leaves far more headroom for the extra error a real detector will add.
