A silent video loses people. But stock sound effects come with licence terms you have to track, and music libraries are worse. For short data videos we render from code, we synthesise every sound with numpy instead: it takes a few lines per effect, it is ours, and because the animation is code too, every sound can land on the exact frame where something happens.
This post continues the canvas-to-MP4 tutorial: same 4-second bar chart, now with sound, normalised to a loudness that phones and platforms handle well.
Disclosure: I build data tools for AI agents as Fast Data on Apify. This post needs none of them; the script is in our public data-recipes repo.
Three sounds from scratch
Every sound is a short numpy array at 48 kHz. An envelope (fast attack, exponential decay) is what turns a raw sine into something that sounds like an event:
import numpy as np SR = 48_000 def env(n, attack=0.003, decay=0.2): t = np.arange(n) / SR return np.clip(t / attack, 0, 1) * np.exp(-t / decay) def kick(f0=150, f1=45, dur=0.35): t = np.arange(int(dur * SR)) / SR freq = f1 + (f0 - f1) * np.exp(-t * 30) # pitch drops fast: the "thump" return np.sin(2 * np.pi * np.cumsum(freq) / SR) * env(len(t), 0.002, 0.12) def pluck(freq, dur=0.5): t = np.arange(int(dur * SR)) / SR tone = sum(np.sin(2 * np.pi * freq * k * t) / k ** 1.5 for k in range(1, 6)) return tone * env(len(t), 0.002, 0.15) def whoosh(dur=0.6): n = int(dur * SR) noise = np.random.default_rng(7).standard_normal(n) smooth = np.convolve(noise, np.ones(40) / 40, mode="same") # crude low-pass return smooth * np.sin(np.pi * np.linspace(0, 1, n)) ** 2 # swell in and out | Sound | Trick | Used for |
|---|---|---|
| Kick | sine whose pitch falls from 150 Hz to 45 Hz in ~0.1 s | the final "hit" when the last bar lands |
| Pluck | 5 harmonics, each quieter, with a short decay | one note per bar, rising in pitch |
| Whoosh | smoothed noise with a swell envelope | frame 0, so the video is not silent at the start |
Note np.cumsum(freq) / SR in the kick: when frequency changes over time, you integrate it into a phase. Writing sin(2π·freq·t) with a changing freq produces a nasty sweep instead.
Put each sound on its timestamp
The animation already knows when things happen; in the demo, bar i starts growing at 0.3 + i * 0.25 seconds. So the soundtrack is a list of (time, sound, gain, pan):
def mix(events, duration): out = np.zeros((int(duration * SR), 2)) for t, sound, gain, pan in events: i = int(t * SR) s = sound[: len(out) - i] * gain out[i:i + len(s), 0] += s * (1 - pan) / 2 * 2 ** 0.5 # constant-power-ish panning out[i:i + len(s), 1] += s * (1 + pan) / 2 * 2 ** 0.5 return out / max(1e-9, np.abs(out).max()) * 0.9 notes = [440.0, 493.9, 587.3, 659.3, 784.0] events = [(0.0, whoosh(), 0.6, 0.0)] events += [(0.3 + i * 0.25, pluck(f), 0.5, -0.4 + 0.2 * i) for i, f in enumerate(notes)] events.append((2.5, kick(), 1.0, 0.0)) Spreading the plucks from left to right (pan from -0.4 to 0.4) follows the bars down the screen. Small thing, noticeable on headphones.
In bigger projects we let the page itself push {t, type} events into an array while it renders each frame, read that array with Playwright at the end, and build this list from it. Then sound and picture can never drift apart.
Loudness: why -14 LUFS, and why two passes
Peak-normalising is not enough: two clips with the same peak can sound very different in loudness. The usual target for online video is -14 LUFS integrated with true peaks below about -1 dBTP; Spotify documents -14 LUFS as its playback level, and YouTube is widely measured to normalise to about the same. Louder files get turned down, quieter ones may stay quiet.
ffmpeg's loudnorm filter does this properly in two passes: measure first, then apply a linear gain with the measured values.
import json, re, subprocess def loudnorm_mux(video, audio, out, target=-14.0): base = f"loudnorm=I={target}:TP=-1.5:LRA=11" p1 = subprocess.run(["ffmpeg", "-hide_banner", "-i", audio, "-af", base + ":print_format=json", "-f", "null", "-"], capture_output=True, text=True) m = json.loads(re.search(r"\{[^{}]*\"input_i\"[^{}]*\}", p1.stderr, re.S).group(0)) second = (f"{base}:measured_I={m['input_i']}:measured_TP={m['input_tp']}:measured_LRA={m['input_lra']}" f":measured_thresh={m['input_thresh']}:offset={m['target_offset']}:linear=true") subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", video, "-i", audio, "-map", "0:v:0", "-map", "1:a:0", "-c:v", "copy", "-af", second + ",aresample=48000", "-c:a", "aac", "-b:a", "192k", "-shortest", "-movflags", "+faststart", out], check=True) Real result on the demo: the raw mix measured -12.3 LUFS; after the second pass, the final MP4 measures -14.05 LUFS integrated with a true peak of -2.6 dBTP. The video stream is copied, not re-encoded.
| Detail | Why |
|---|---|
print_format=json in pass 1 | the measurements are printed to stderr as JSON; the regex picks them up |
linear=true in pass 2 | applies one gain to the whole file instead of dynamic compression; keeps hits punchy |
aresample=48000 after loudnorm | loudnorm works internally at 192 kHz; without resampling you get odd sample rates |
-c:v copy | no second video encode, no quality loss, takes a second |
-shortest | the file ends with the video even if the audio is a few samples longer |
What it does not do
- It is not music. Plucks and kicks make a video feel alive; a melodic bed is a bigger job (we layer a slow pad and a pulsing bass, built the same way).
- No reverb here. A short convolution with exponentially decaying noise as the impulse response adds a lot of polish; it is about 15 more lines.
- Phones still vary. Many people watch muted. Sound supports the video; the first frame and the text still have to work without it.
Try it
soundtrack.py plus the animation and capture script: snippets/canvas-to-mp4. Run node capture.mjs index.html demo.mp4 and then python soundtrack.py demo.mp4 final.mp4.
Question: for data videos, do you prefer sound effects on every event, or one music track and nothing else? Our own numbers are too small to tell, and I am curious what others have measured.
