Re-Encoding Before Upscaling Is Usually a Waste of a Generation (Measured)

Every upscaling tutorial says the same thing: "transcode to ProRes first, so the upscaler isn't working on compressed data." It sounds like hygiene. In most pipelines it is a net loss, not because the advice is wrong about compression, but because it is applied at the wrong stage, with the wrong bit budget, in an order no codec was designed for.

I ran the experiment instead of repeating the advice.

The setup, so you can break it

A 540×810 frame stands in for "the source you were given" (the demo material downscaled with Lanczos, which is what a capture pipeline hands you); the 1080×1620 original is ground truth. Every path is scored against that with PSNR and SSIM from ffmpeg, plus a blockiness index defined below. No neural upscaler is involved, because the question is codec order, not which model is best — including a model would confound the two.

Paths: source straight to 2× Lanczos (oracle); H.264 at bitrate B, decode, then upscale; the same wrapped in FFV1 lossless first (the "transcode to ProRes" advice, tested literally); two lossy encodes then upscale; and upscale first, encode the big frame.

Result 1: the lossless transcode changes literally nothing

At 700 kbps, encode-then-upscale scored PSNR 32.590, SSIM 0.94663. Inserting a full FFV1 round trip between decode and upscale produced PSNR 32.590, SSIM 0.94663. Identical to three decimals, at 300k, 700k and 1500k.

Identical because that is what lossless means: decode(H.264) and decode(FFV1(encode(H.264))) are the same pixel array. The transcode either reproduces the decoded frame exactly — you added a step and changed nothing — or it does not, and you added a generation. There is no third outcome. When a tutorial says transcoding "cleans up" artifacts, it is describing a filter doing the work, not a container.

The last row is why this article exists.

Result 2: at equal bits-per-pixel, compressing after the upscale wins

That last row looks terrible. The comparison is dishonest, and it is the mistake almost everyone makes. 700 kbps on a 437,400-pixel frame is 1.600 bits per pixel; the same 700 kbps on the 1,749,600-pixel upscaled frame is 0.400 — a quarter of the budget per pixel. Of course it loses. You asked a codec to describe four times as many pixels with the same bits.

Give the big frame four times the bitrate and the ordering reverses:

At matched bits-per-pixel, encoding after the resize wins at every rung, by 0.6–1.4 dB. The mechanism is not mysterious. Encoding the small frame spends budget reproducing codec-shaped detail — the DCT basis functions that best approximate the texture — and then the upscaler reconciles those 8×8 patches with its own smoothness prior, amplifying what the codec got wrong, because a resize preserves high-frequency error. Encoding the big frame spends the same budget on the high-frequency content the upscaler actually produced, with no later step to amplify mistakes.

In one sentence: pre-encoding pays for compression twice and lets the more artifact-prone pass be the one the upscaler magnifies.

Result 3: the block grid does not vanish when you upscale — it moves

A codec leaves a discontinuity on an 8-pixel grid, the transform size. Scale by f and the grid lands at period 8*f*. I measured differential gradient energy exactly on the expected period, normalised against off-period energy: 1.0 means no seam, above 1.0 means a periodic seam exists there.

The seam energy is still there after 4× — slightly stronger, not weaker, because the resize stretches each seam over more pixels and raises contrast across it. It has simply relocated to period 32. Every deblocker in every editor assumes an 8-pixel grid and will find nothing there, leaving the real artifact untouched.

I checked. Deblocking the compressed small frame before the upscale dropped the index 1.260 → 1.117 and cost 0.03 dB. Deblocking after, at the wrong period, dropped it to 0.974 but smeared recovered detail — PSNR flat at 30.78, SSIM +0.0003. Neither is a win. The fix is to not create the grid: encode once, at the end, with enough bits.

Result 4: 4:2:0 chroma is gone and upscaling will not bring it back

Against a 4:4:4 encode at 1500 kbps, mean absolute Cb/Cr error was 1.108 levels at source resolution, 1.122 after a 2× upscale, and 1.122 after upscaling and re-subsampling. Interpolation cannot invent bandwidth that subsampling destroyed. This is the one part of "transcode first" that is genuinely right — but the reason is chroma headroom, not artifact removal. An intermediate in 4:2:0 caps final colour detail no matter what the last encode gets.

Result 5: for stills, a second save is nearly free — pre-sharpening is not

Re-saving a JPEG at fixed quality, scored against the original: at q≈70, 41.32 dB for generation one, 41.30 for two, 41.30 for three. At q≈85, 43.41 → 43.40 → 43.40. At q≈92, 44.66 → 44.66 → 44.66. The second generation costs about 0.02 dB, because quantisation is deterministic: the first save already rounded every coefficient to the grid, and saving again lands on the same grid. Generational loss is real but front-loaded. Re-saving at higher quality recovers nothing — 42.217 → 42.183 dB, SSIM unchanged to four decimals; a finer quantiser cannot un-round a coefficient.

Pre-sharpening before the lossy save is where people lose money: sharpen then save gave 36.92 dB / 0.9791, versus 42.22 dB / 0.9839 without. Five dB worse, because sharpening adds high-frequency energy the codec must spend bits on or discard, and discarding it rings exactly where the sharpen halo already is. Sharpen after the last lossy step — with one exception: sharpening before a generative upscaler changes what the model thinks the source distribution is, which is a different question with no general answer.

The decision table

That last row costs people the most. "We have a 10 Mbps budget" is a statement about a delivery target, not a pixel count. Change the pixel count 4× and you silently changed quality 4×. Comparing orderings at equal total bitrate and concluding "compressing after upscale is bad" is measuring your own budget error.

When you should not follow any of this

When the source is already multi-generation. After three encodes the artifacts are no longer on a clean 8×8 grid; they are smeared across scales and no ordering argument applies. The damage is upstream of every choice here.

When the upscaler is generative and expects a specific input distribution. Some models were trained on mildly compressed web images; feeding them pristine lossless frames can push them off-distribution and produce worse output. That inverts the advice — but it is a property of one model, so verify per model rather than generalising.

When the intermediate is doing real work. Deinterlacing, denoising, deflickering, colour management are filters. They belong in the pipeline and should be measured on the final output. My result is about the transcode step, not preprocessing in general.

When you cannot measure. Everything here is PSNR and SSIM against synthetic ground truth, which rewards fidelity and correlates poorly with perceived quality on generative output. If you cannot look at the result at 100%, do not let a metric decide for you.

The two-minute version

Encode once, at the end, at a bitrate chosen for the output pixel count rather than inherited from the input. Insert a lossless intermediate only for chroma headroom or repeated edit cycles. Never insert a lossy intermediate to clean up a lossy source. And whenever you compare two orderings, normalise to bits per pixel first — the most common error in this discussion is comparing a well-funded small encode against a starved large one.

The preset set and pipeline order I use are documented here