Six still PNGs, six narration clips, and the ffmpeg pass that keeps them in sync

A vertical video with narration usually means a timeline editor and someone dragging clips until the words land on the right frame. I had an SSH session on a two-core ARM machine, six PNGs exported at 1080x1920, and ffmpeg.

The result is a 47.9 second reel: six typographic frames, each with one sentence of Portuguese narration. The pipeline is 103 lines of Python calling ffmpeg as a subprocess, with no dependencies beyond ffmpeg, ffprobe and a TTS CLI. Here is what it does, and which decisions the machine forced.

The frames were designed as still images

Every scene is a static frame rendered elsewhere into a PNG at exactly 1080x1920, 8-bit RGB. No animation, no zoom. On a phone held at arm's length, text that moves has to be chased, and each of these frames holds a sentence that the narration finishes in about seven seconds.

That removes most of the usual complexity: no timeline, no transitions, no partial renders. A scene is one image that stays for a fixed number of seconds, and the only open question is how many.

The hardware mattered. Two cores, aarch64, no GPU. libx264 at veryfast is fine for a slideshow and would not be fine for anything with motion. I measured one segment while writing this: 8.44 seconds of video, 211 frames, 5.8 seconds of wall clock, 38 KB on disk and 32 kb/s, because a still frame has nothing to spend bits on.

The narration sets the timing

The obvious approach is to fix five seconds per scene and squeeze the audio to fit, which makes speech sound rushed in one place and padded in another. Doing it the other way is easier: synthesize the narration first, one MP3 per scene, then ask ffprobe how long each file is. That number is the timeline.

BASE = Path("/home/ubuntu/hermes_tools/pecas_plano_fde") FPS = 25 VOZ = "pt-BR-AntonioNeural" EDGE = "/home/ubuntu/hermes_tools/voz/venv/bin/edge-tts" def dur(caminho): saida = subprocess.run( ["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "default=noprint_wrappers=1:nokey=1", str(caminho)], capture_output=True, text=True, check=True).stdout.strip() return float(saida) 

The six clips measured 7.944, 7.080, 6.888, 7.392, 7.128 and 8.472 seconds: 98 words in 44.904 seconds of speech, about 131 words per minute in Portuguese at the default rate. Nothing there was estimated. If a sentence grows, its scene grows, and no other scene compensates.

The narration comes from the edge-tts CLI in a virtualenv, voice pt-BR-AntonioNeural, rate +0%. Separate files instead of one long track is what keeps the rest simple: each one can be measured, replaced and reused on its own.

Each frame becomes a clip of its own length

for i, d in enumerate(duracoes, start=1): png = BASE / f"reel-cena{i}-1080x1920.png" seg = destino / f"seg{i}.mp4" seg_dur = round(d + 0.5, 2) subprocess.run([ "ffmpeg", "-y", "-loop", "1", "-t", str(seg_dur), "-i", str(png), "-vf", "scale=1080:1920:flags=lanczos,format=yuv420p", "-r", str(FPS), "-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p", str(seg) ], check=True, capture_output=True) 

scale with lanczos does nothing on an input already at 1080x1920, and stays in the chain so it survives a frame exported at another size. format=yuv420p is the pixel format phones decode without argument, and it has to reach the final file. -crf 20 with veryfast is a compromise for two cores; on a machine with headroom, medium would shave bytes nobody would notice.

The + 0.5 is breathing room. Without it the next sentence lands on the exact frame where the previous one ended, and the cut feels clipped even with correct timings.

Padding the audio so each line stays with its frame

Concatenating the six MP3s gives 44.9 seconds while the video is 47.9. Muxing those directly means the audio falls behind, and by the fifth scene the voice describes something that left the screen seconds earlier. Padding each narration to the duration of its own segment removes the accumulation.

entradas, filtros = [], [] for i, (_, seg_dur) in enumerate(segmentos, start=1): entradas += ["-i", str(destino / f"narracao{i}.mp3")] filtros.append(f"[{i - 1}:a]adelay=0|0,apad=whole_dur={seg_dur}[a{i - 1}]") concat_in = "".join(f"[a{i}]" for i in range(len(segmentos))) filtros.append(f"{concat_in}concat=n={len(segmentos)}:v=0:a=1[aout]") 

apad=whole_dur=N extends each stream with silence to N seconds, the number computed in the earlier loop. The adelay filter at the front of the chain carries a zero delay and changes nothing audible; it stays there for scenes that need a lead-in before the voice starts. For padding alone, apad is enough.

subprocess.run(["ffmpeg", "-y", "-i", str(video), "-i", str(audio), "-map", "0:v:0", "-map", "1:a:0", "-c:v", "copy", "-c:a", "aac", "-shortest", "-movflags", "+faststart", str(final)], check=True) 

-c:v copy is why this step costs almost nothing: the segments were already encoded and joined by the concat demuxer, so the muxer only interleaves streams. +faststart moves the index to the front of the file, which matters when it is played from a phone over a network.

What the container reports

• video: h264, 1080x1920, 25/1 fps, 1195 frames
• audio: aac, 24 kHz, mono
• duration: 47.9 seconds, reported the same for both streams

The arithmetic matches the file: 44.904 seconds of speech plus six half-second pads is 47.904. The audio stays at 24 kHz mono because that is what the TTS produced; resampling to 44.1 kHz would add bytes and no information, and Instagram re-encodes it anyway.

What that probe answers is whether video and audio agree on duration. When they differ, either a segment was written at a different length than the one used for its pad, or the concat dropped frames at a join. Both show up before anything is uploaded, and neither is visible while watching the file on a laptop at half size.

The silence you hear is longer than the padding you set

I ran a two-scene version of the pipeline while writing and measured the join with silencedetect at -45 dB and 0.25 seconds. The padding was 0.5 seconds and the measured silence at the cut was 1.52 seconds.

The extra comes from the narration files. TTS output carries a little silence at its own start and end, and apad adds on top of it. The same run flagged pauses of about 0.9 seconds inside the speech, so the clips have internal gaps of their own. What you set is a constant; what a listener perceives is that constant plus the edges of two files.

Padding therefore has to be tuned by measuring the join. Half a second worked for six scenes of about seven seconds, where the sentence still holds attention. For a fast cut it would be too much.

What I would change

The docstring of the script says 0.7 seconds of padding. The loop uses 0.5. Nothing broke, but the comment is already wrong, and comments drift faster than the code they describe.

I would also write the durations to a JSON file before encoding instead of recomputing them in the same pass. The script is one long function, so an encode that fails halfway re-synthesizes every narration clip when it restarts, and synthesis is the slow step that touches the network. Splitting them turns a failed render into a five-second retry.

The rest holds. Six frames, six clips, one filtergraph and a format check at the end, with no GUI, no GPU and no cloud render service.

Se esse tipo de detalhe ajuda no seu projeto, deixa um comentário contando o que você já quebrou tentando sincronizar áudio com imagem.