Why Video Upscaling Is Not Image Upscaling, Repeated 300 Times

A video upscaler that works frame by frame is not a video upscaler. It is an image upscaler called in a loop, and the failure it produces is invisible on any single frame by construction.

14 November 2026 · 40-frame sub-pixel pan, ffmpeg + numpy

The pitch is always the same: "upscale every frame, get a 4K video." Technically true. Practically, what you get is a sequence of 4K frames that do not agree with each other about what the scene contains, and your visual system is extremely good at noticing disagreement even when it cannot name it.

I built a test to isolate this from every other variable: a static, high-detail scene, panned by 0.2 pixels per frame. Nothing in the scene moves. Nothing appears or disappears. The only thing changing is the sub-pixel phase of the source.

The mechanism, before the numbers

When you shrink an image and then enlarge it, each output pixel is a weighted average of source pixels. The weights depend on where the output pixel falls relative to the source grid. Pan by a fraction of a pixel and those weights change. On a hard edge, a small weight change flips a pixel from mostly-dark to mostly-light.

In a still image that is invisible — you have one phase, you see one result. In a video you have a different phase every frame, so the same patch of wall gets a slightly different edge on every frame. Played at 24 fps, edges shimmer. This is the artifact people describe as "boiling" or "crawling," and it is a property of the resampling kernel, not of the content.

What I measured, and what surprised me

Metric: high-frequency energy per frame, computed as the ratio of 2-D FFT magnitude outside a radius of 0.18 × width to the magnitude inside it. For a smooth pan, this should be nearly constant. Its coefficient of variation (CV) across 40 frames is the instability.

The surprise is in the last two columns of the middle row. The per-frame upscale is more temporally stable than the ground truth — CV 2.87% versus 5.99%. That looks like it contradicts everything I just said, so it is worth being precise about why it does not.

So the measurement refutes the naive version of the claim. Per-frame interpolation does not flicker more than reality does — it flickers less, by flattening the scene until there is nothing left to flicker. The flicker problem is real, but it belongs to the methods that actually try to reconstruct detail, not to the ones that blur.

Where the instability actually comes from

Three sources, in descending order of how much damage they do:

1. Reconstruction models. A generative upscaler samples from a distribution. Run it twice on nearly-identical inputs and it can pick different plausible details — a different hair strand, a different texture direction. At 24 fps that is a scene that reinvents itself every frame. This is the dominant cause of the artifact people complain about, and it is invisible in a single-frame comparison.
2. Sharpening steps. Unsharp mask is nonlinear near edges. Small changes in input produce discontinuous changes in output right at the edge, which is exactly where the eye is looking.
3. Per-frame noise reduction. Any denoiser with a threshold behaves differently on either side of that threshold. Frames straddling it get visibly different treatment.

Plain interpolation — the thing in the table above — is the one case that is stable, because it is a fixed linear filter. It is also the case that adds nothing. That trade is the whole subject.

What to do about it

Not "use a different kernel." The kernel is not the problem. Practical options, in order of how much they help:

• Give the model temporal context. A model that sees frames n−1 and n+1 when producing frame n can hold detail consistent across the shot. This is the only real fix and it is why video-specific models beat image models run in a loop.
• Reduce the scale factor. At 2×, Lanczos keeps 21.5% of high-frequency energy; at 4× it keeps 1.3%. Less aggressive scaling means less to invent means less to disagree about.
• Test on motion, not on a still. Extract three consecutive frames around a hard edge and compare them. If the edge sits in a different place in each, the method will shimmer, no matter how good frame 1 looks.
• Accept the trade explicitly. A softer, temporally stable result is often better viewing than a sharp still that boils in motion. That is a real engineering choice, not a compromise.

Reproducing it

# 40-frame sub-pixel pan (0.2 px/frame) over a static detail scene
# shrink 2x, upscale each frame independently, then compare HF energy per frame
for i in range(40):
    gt = scene.transform((320,320), AFFINE, (1,0,i*0.20, 0,1,0), BICUBIC)
    lo = gt.resize((160,160), LANCZOS)
    up = lo.resize((320,320), LANCZOS)   # per-frame, no temporal context

# instability = coefficient of variation of hf_ratio() across the 40 frames

The full script is temporal2.py in the measurement set. It writes temporal_hf.json with the numbers above.

Check it on your own clip

The fastest way to see whether a source is worth a full pass is to run one short segment and watch the result in motion at full size, not paused. A browser-based upscaler handles both photos and short video without a GPU install: UpgoLAB upscales in the browser to 2K and 4K, which is convenient for exactly this test — a 15-second segment, played back, tells you in under a minute whether the flicker is acceptable.

Pause any frame and it will look fine. That is the trap, and it is the reason this paragraph exists.