AI Upscaling Explained: What It Recovers and What It Invents
The interpolation method with the highest measured fidelity destroys 98% of the fine detail in an image. That sentence sounds like a paradox until you look at what the metric is actually rewarding.
Every upscaling tool in the market is sold on the same promise: it takes a small image and gives you a big one that looks like it was shot that way. Some tools deliver something close on some inputs. The failure mode is not that they fail — it is that they fail in a way that scores well on the numbers vendors publish.
I measured this instead of arguing about it. Two experiments, both reproducible with commands at the end of the page.
The two things "upscaling" can mean
There are genuinely two different operations hiding under one word:
- Interpolation — invent a value for each new pixel by averaging the ones around it. Bilinear, bicubic, Lanczos, spline. Exactly nothing is recovered, because no information outside the source image is used.
- Reconstruction — a model trained on millions of images predicts what detail usually sits in that part of a scene, and paints it in. This is generation, not recovery, and it can be wrong in ways interpolation never is.
Almost all confusion about upscaling comes from tools in the first category being marketed as if they were in the second. A "4K upscale" done with a bicubic kernel is a bigger, smoother file with less real detail than the source. It is not a lie exactly — the pixel count is real — but the detail is not.
Experiment 1: PSNR says Lanczos is best. It is also the blurriest.
I built a test image that contains known high-frequency content: a 72-spoke radial star (high frequency in the angular direction), concentric rings at 4-pixel spacing (high frequency in the radial direction), and a fine checkerboard patch. This is the classic Siemens-star pattern used to test lens and sensor resolution.
Then I shrank it to 100×100 — a 4× reduction, roughly what a 360p clip represents next to 4K — and upscaled each version back to 400×400 with six methods, measuring PSNR against the ground truth.
PSNR against ground truth, 4× (100px → 400px):
| Method | PSNR (dB) | HF retained |
|---|---|---|
| nearest | 24.60 | 25.4% |
| bilinear | 25.68 | 3.9% |
| bicubic | 26.27 | 2.1% |
| lanczos | 26.47 | 1.3% |
| spline | 26.45 | 1.4% |
Read those two columns together. Lanczos wins on PSNR by 1.87 dB over nearest — a large, unambiguous margin — while keeping 1.3% of the original high-frequency energy versus nearest's 25.4%. Nearest "wins" that column for a boring reason: it replicates pixels, so hard block edges survive as fake high frequency. It is not detail. But the direction of the effect is the point — the ranking by PSNR is nearly the exact inverse of the ranking by detail retention.
Why PSNR does this. PSNR is mean squared error. A method that produces a smooth blur sits close to the average of the true image, so its per-pixel error is small. A method that produces crisp edges lands exactly on the right value at edges and badly off between them, so its error is larger. PSNR is a good measure of "did you avoid visible artifacts" and a bad measure of "did you keep the detail."
At 2× (200px → 400px) the same pattern holds, with the numbers shifted up:
| Method | PSNR 2× (dB) | HF retained 2× |
|---|---|---|
| nearest | 29.23 | 48.8% |
| bicubic | 32.71 | 21.9% |
| lanczos | 33.38 | 21.5% |
Going from 4× to 2× roughly doubles the detail that survives (1.3% → 21.5%) for the same method. That is the single most useful number on this page: halving the scale factor is worth more than any choice of kernel. If you can get a 720p source instead of a 360p one, that beats switching from bicubic to Lanczos by a wide margin.
Experiment 2: does the "transcode first" advice survive contact with a codec?
The standard advice is to transcode to a high-bitrate intermediate before upscaling, "so the upscaler isn't working on compressed data." I tested whether that changes anything when the intermediate is lossless.
Source: 1,080 frames, 1280×720. Compared a lossless transcode (FFV1) against direct processing, at three target bitrates, measuring PSNR and SSIM on the result.
| Pipeline | PSNR @4M | PSNR @8M | PSNR @16M |
|---|---|---|---|
| lossless intermediate first | 32.590 | 32.590 | 32.590 |
| direct | 32.590 | 32.590 | 32.590 |
Bit-identical across every bitrate. A lossless intermediate cannot change the pixels, so the step is a pure cost — disk, time, and one more place for a colour-space mistake to happen. The advice is not wrong about compression; it is wrong about where in the pipeline the compression does its damage. By the time you are upscaling, the damage was already baked in by the original encode and no later step can undo it.
The real test is not "did I transcode" but "is my bits-per-pixel high enough." Two encodes at the same bitrate but different resolutions are not comparable. At equal bits-per-pixel, encoding after the upscale beat encoding before it by 0.6–1.4 dB in my runs — because the encoder gets to spend its budget on the final pixel grid rather than being upscaled afterwards.
What this means when you pick a tool
Given the above, the useful questions are not "how many times upscale" or "which algorithm." They are:
- Is this interpolation or reconstruction? If the tool has no model weights and no training data, it is interpolation and it cannot add detail. That is fine for enlarging a screenshot for a slide. It is useless for "restoring" a photo.
- What is my actual scale factor? From the table above, 2× retains roughly 17× more high-frequency energy than 4×. If your source is 360p and your target is 4K, you are asking for 12×, and no method on this page is going to give you a good answer. The honest move is to change the target, not the kernel.
- Am I judging at 100% zoom? A 4× bicubic upscale looks acceptable scaled to fit a phone screen, because the screen itself is the low-pass filter. Judge at 1:1 or you are evaluating your display, not the file.
The uncomfortable conclusion. Reconstruction models are the only tools that can add plausible detail, and by definition the detail they add was not in your source. They are inference, and inference can be confidently wrong — a face that gains convincing eyelashes it never had, a texture that becomes wood grain when it was fabric. That is a real trade and it should be a deliberate one, not a surprise.
Reproducing this
Both experiments are ffmpeg + numpy, no proprietary tooling. The shape of the commands:
# build the test pattern, shrink 4x, upscale back with each kernel
ffmpeg -i source.png -vf "scale=100:100:flags=lanczos" low.png
ffmpeg -i low.png -vf "scale=400:400:flags=bicubic" up_bicubic.png
# PSNR against ground truth
ffmpeg -i up_bicubic.png -i source.png \
-lavfi "[0:v]scale=400:400:flags=bicubic[x];[x][1:v]psnr" -f null -
# high-frequency energy: 2-D FFT, ratio of energy outside r>0.18*N to inside
The high-frequency metric is the sum of FFT magnitude outside a radius of 0.18 × image width, divided by the sum inside it. It is not a standard metric — I built it because PSNR and SSIM both failed to distinguish "smooth" from "detailed" in a way that matched what the eye sees. Treat it as a relative comparison between methods on the same image, not as an absolute quality score.
Try it on your own file
If you want to see the difference on a real photo rather than a test pattern, a browser-based upscaler is the fastest way to compare without installing anything: UpgoLAB runs the upscale in the browser and handles both photos and short video to 2K and 4K. Use it to run the 2×-versus-4× comparison on your own source before committing to a full pass — the scale-factor gap above is large enough that you will see it in one frame.
And if the frame does not hold up at 100%, the fix is a better source, not a different preset. That is the whole finding, compressed.