Output Format After Upscaling: What JPEG, WebP and AVIF Do to the Detail You Just Bought
The last step of every upscaling job is the one nobody measures. You spent the compute to reconstruct detail, then hand the result to a lossy encoder and let it decide which of those details were worth keeping. I took one upscaled frame, encoded it three ways at a matched file size, and measured what came back. AVIF bought 3.28 dB over JPEG at the same 400 KB. But the more useful finding is that all three encoders added high-frequency energy rather than removing it, which means the intuitive metric — "how much detail survived" — is measuring the wrong sign.
The setup: matched size, not matched quality
Comparing encoders at "quality 80" is meaningless, because the three quality scales are not the
same scale. Pillow's quality=80 for JPEG, WebP and AVIF produce files that differ by a
factor of two in size on identical pixels. What a person actually chooses is a file size
budget — a hero image that must stay under half a megabyte, a thumbnail under 60 KB — and
then asks which encoder uses that budget best.
So the comparison below searches each encoder's quality scale for the setting that lands closest to 400,000 bytes, then measures PSNR against the pre-encode array. The three files that competed are within 6% of each other in size, which is as matched as an integer quality knob allows.
| Encoder | Setting | Bytes | PSNR vs source | HF energy vs source |
|---|---|---|---|---|
| JPEG | q68 | 404,460 | 43.153 dB | 116.4% |
| WebP | q90 | 381,016 | 45.176 dB | 106.8% |
| AVIF | q86 | 403,527 | 46.437 dB | 104.4% |
AVIF is 3.28 dB ahead of JPEG and 1.26 dB ahead of WebP at the same budget. That is a real and reproducible margin — but 3.28 dB on a frame that already sits above 43 dB is not a visible difference at normal viewing distance. It matters when the file will be re-encoded downstream, which is the case this whole article keeps returning to.
A negative result: every encoder added detail instead of losing it
The column that surprised me is the last one. I expected all three to fall below 100% — lossy encoding discards, so high-frequency energy should drop. It did not. JPEG came back at 116% of the source's high-frequency energy, WebP at 107%, AVIF at 104%.
The measurement is mean absolute Laplacian on luma. A value above 100% means the decoded image has steeper local transitions than the array that was handed to the encoder. Nothing was added to the picture; the encoder manufactured edges. JPEG does it most because 8×8 block quantisation creates discontinuities at block borders, and a discontinuity is by definition high-frequency — it shows up as a spike in any Laplacian-type measure regardless of whether a human can see it.
Consequence for measurement: "detail retained" cannot be computed as a ratio of high-frequency energy after a lossy step. The ratio exceeds one, so it cannot be interpreted as a retention figure at all. Blocking artifacts are indistinguishable from detail under this metric. This is the same trap I hit in the JPEG-blocking piece: the block edge keeps its contrast and gains area, so a gradient-energy measure reports an increase where the eye reports a defect.
Where the invented detail lives
If the extra high-frequency energy were spread evenly, it would be a mild curiosity. It is not spread evenly. I split the frame into the flattest quartile of gradient magnitude (24.9% of pixels) and the top decile (sharp edges), then measured the same Laplacian statistic in each region against the same regions of the source:
| Encoder | Flat-quartile HF vs source | Edge-decile HF vs source |
|---|---|---|
| JPEG q70 | 276.6% | 106.4% |
| JPEG q95 | 183.5% | 101.8% |
| WebP q80 | 232.0% | 102.9% |
| AVIF q80 | 232.1% | 102.1% |
In the flat regions the encoders roughly double or triple the local variation. On real edges they are within a few percent of the source. That is exactly backwards from what a person wants: the places that were smooth and should stay smooth are where the noise lands, and the places that already had contrast are preserved almost perfectly.
It also explains why raising JPEG quality helps so much in the flat column — 276.6% at q70 falls to 183.5% at q95 — while the edge column barely moves. Blocking is a flat-region phenomenon. If your frame is mostly sky, gradient, or studio backdrop, JPEG quality is the parameter that matters, and it matters for a reason that has nothing to do with detail retention.
Chroma subsampling: what 18.6% of the file actually costs
JPEG at q90 with 4:4:4 chroma, then the same q90 in 4:2:2 and 4:2:0. Everything else held fixed:
| Subsampling | Bytes | PSNR | Chroma RMSE (Cb/Cr only) |
|---|---|---|---|
| 4:4:4 (s=0) | 899,642 | 49.030 dB | 0.566 |
| 4:2:2 (s=1) | 795,568 | 48.245 dB | 0.645 |
| 4:2:0 (s=2) | 732,069 | 47.600 dB | 0.711 |
4:2:0 removes 18.6% of the bytes and raises chroma error by 25.8%. The PSNR cost looks small — 1.43 dB — but PSNR is dominated by luma, which subsampling does not touch. The isolated chroma RMSE is the honest number, and it rises by a quarter. On a saturated colour boundary — red signage, coloured gel lighting, a bright product on a coloured backdrop — that quarter is where the coloured fringe along the edge comes from.
This is worth connecting to the upscale itself: subsampling throws away chroma resolution after you paid to reconstruct it. If the source was already 4:2:0, upscaling reconstructed chroma detail that never existed and 4:2:0 output discards it again for an 18.6% saving. If the source was 4:4:4 or synthetic, that saving is a real loss.
What a second 4:2:0 pass costs when the source was already 4:2:0
Everything above assumed a full-resolution chroma source. Most photos that reach an upscaler
do not have one: they were JPEGs at 4:2:0, or frames pulled out of video at 4:2:0. So the common
pipeline is not 4:4:4 → upscale → encode. It is
4:2:0 → upscale → 4:2:0, and the chroma gets thrown away twice. I measured both passes
separately instead of assuming the second one is free.
Step one: the pure arithmetic, no codec involved
Take the 2× lanczos reference, subsample Cb the way 4:2:0 does (keep one sample in four, box filter down, bilinear back up), then do it again to the result. Two passes, no JPEG anywhere, so nothing but the subsampling can move the number:
| Image | 1st pass RMSE | 2nd pass RMSE | Total vs original | 2nd ÷ 1st |
|---|---|---|---|---|
| clear-reality | 0.532 | 0.323 | 0.752 | 0.61 |
| cinema-vivid | 0.667 | 0.462 | 0.994 | 0.69 |
| natural-hd | 0.534 | 0.347 | 0.778 | 0.65 |
| true-texture | 0.481 | 0.281 | 0.678 | 0.58 |
| ultra-crisp | 0.547 | 0.361 | 0.804 | 0.66 |
| median | 0.534 | 0.347 | 0.778 | 0.65 |
The hypothesis I started with was that the second pass would cost the same as the first — that subsampling is a per-generation tax. It does not. The second pass costs a median 0.65× the first, in all five images, because the first pass already removed the chroma detail the second pass would have destroyed. What is left to lose is a smooth plane, and smoothing a smooth plane is cheap.
That is not a reprieve, only a smaller bill. The two errors are correlated (r = 0.50 on Cb, not independent), which is why they do not add in quadrature: 0.534² + 0.347² would predict 0.637, and the measured total is 0.778 — 46% worse than a single pass, and 22% worse than independence would suggest. The second pass damages the same neighbourhoods the first one already damaged, and the damage compounds rather than accumulating independently.
Step two: the same thing with a JPEG at the end
Now the real question — is any of that visible once the file is actually encoded? Four chains, all encoded at q85 with 4:4:4 forced so the encoder contributes the same amount to every row and the only variable is where the chroma was lost. Reference is the 2× 4:4:4 array; 5 images; medians:
| Chain | Bytes | Luma PSNR | Chroma RMSE | vs 4:4:4 |
|---|---|---|---|---|
| 4:4:4 all the way | 711,558 | 48.263 dB | 0.571 | — |
| 4:2:0 at encode only | 702,844 | 47.594 dB | 0.614 | +7.5% |
| 4:2:0 at source only | 693,958 | 47.583 dB | 0.785 | +37.5% |
| 4:2:0 at both ends | 694,498 | 47.576 dB | 0.978 | +71.3% |
Two things worth flagging. The chroma damage compounds roughly in proportion to how many times the resolution is cut — 0.571 → 0.614 → 0.785 → 0.978 is +7.5%, +37.5%, +71.3% — but the source pass costs five times what the encode pass costs, which is the opposite of what a pipeline diagram suggests. Losing chroma before the upscale is expensive because the upscale then has to invent chroma from a quarter-resolution plane and smooths it across edges; losing it at the end is cheap because the upscaler had real chroma to work with and only the final delivery throws it away.
The second thing is the luma column, and it is a negative result about my own method. Luma PSNR falls ~0.67 dB on all three subsampled chains, including the one where chroma was only touched at the last step. Luma is untouched by 4:2:0 by definition, so that drop cannot be a chroma effect. Isolating it: chroma-subsample → RGB → YCbCr before any encode costs a luma RMSE of 0.086, against a JPEG q85 floor of 0.987 — so 92% of the luma error is the encoder's own quantisation and the remaining 8% is the chain. The honest reading is that these 0.67 dB are not attributable to the chroma pass at all; the small residual comes from the encoder re-allocating bits around the extra clipping (3.83% of pixels clipped, against 2.84% on the clean chain). Reporting the luma column as if subsampling caused it would be wrong.
The penalty grows with the scale factor
If the mechanism is "the upscale has to invent chroma it never had", the damage should scale with how much inventing is required. It does, slowly. Chroma error of the double-subsampled chain divided by the single-pass chain, median of five images:
| Scale factor | One pass | Two passes | Ratio |
|---|---|---|---|
| 2× | 0.614 | 0.978 | 1.59× |
| 3× | 0.517 | 0.843 | 1.63× |
| 4× | 0.452 | 0.773 | 1.72× |
Both absolute numbers fall with the factor — more upscale means more smoothing, and a smoother image is easier to subsample — but the ratio climbs from 1.59× to 1.72×. The gap widens at exactly the scales where people upscale furthest. It is not a dramatic effect, and at 4× the difference is 0.32 RMSE units on a 0–255 scale, which nobody will see on a photograph. Where it will show is a saturated edge, and that is measurable: the 99th-percentile chroma error on the top-5%-gradient mask went from a median 4.0 (4:4:4) to 6.0 (encode only), 11.3 (source only) and 13.5 (both) — 3.4× the clean chain at the pixels where chroma error is most visible.
A fourth negative result: the 20% saving does not depend on chroma detail
The reason 4:2:0 is offered as an export option is that it is ~20% smaller. I expected that saving to shrink on an already-subsampled source, since there is nothing left in the chroma to drop. It does not shrink at all:
| Source | 4:4:4 bytes | 4:2:0 bytes | Saving |
|---|---|---|---|
| full-resolution chroma | 745,940 | 599,683 | 19.6% |
| already 4:2:0 | 731,212 | 594,507 | 18.7% |
| median over 5 images | 21.27% → 20.18% | ||
The saving is 20.18% instead of 21.27% — a one-point difference across five images and four quality levels. So the 20% is not buying the removal of chroma detail, which is already gone; it is buying the removal of chroma resolution, which is a fixed 3/4 of the chroma samples no matter how smooth they are. The practical consequence: you cannot argue "my source was already 4:2:0, so 4:2:0 output costs me nothing" and you also cannot argue "so it saves me less". It saves the same and costs more, because it is the second dose.
And if the byte budget is what is actually fixed, the choice is not binary. Encoding the same 2× frame as 4:4:4 at a lower quality — q76–77, chosen by bisection to match the 4:2:0 file's byte count — beat the 4:2:0 export on chroma error in 5 of 5 images: median chroma RMSE 0.656 against 1.129, a 41.9% reduction, for a luma cost of 0.625 dB (47.553 → 46.928 dB). Quality knobs spend bytes where the encoder thinks they matter; subsampling spends them on a fixed geometric assumption instead. On a photographic frame that is not upscaled — 4:2:0 is still the better trade at these numbers, since 0.6 dB of luma is worth more than 0.47 RMSE of chroma. It is specifically on the second pass, where chroma has already been halved once, that paying in quality rather than in chroma resolution starts to win.
Reproducing it
The whole measurement is one script, and it hides one trap that produced a wrong number on the first
run. Pillow's YCbCr mode keeps Cb and Cr as signed values around zero — on this frame Cb
runs −75.1 to 49.2 and 63.7% of its samples are negative. Clip a signed plane to 0 before sampling it
and every one of those samples becomes a flat zero: the box filter then averages a plane with its
entire lower half deleted, and the measured chroma RMSE comes out 8.38. Cast the same negative floats
straight to uint8 and they wrap to 255, which is worse — RMSE 198.34, a number so absurd
it is the only reason the bug was caught at all. Chroma has to be offset by 128 before it goes near an
8-bit buffer, exactly the way JPEG stores it, which brings the same measurement to 1.07:
import io, numpy as np
from PIL import Image
M = np.array([[ .299, .587, .114], # BT.601 forward
[-.168736, -.331264, .5],
[ .5, -.418688, -.081312]])
Mi = np.linalg.inv(M)
rgb2ycc = lambda a: a @ M.T # float RGB 0-255 -> Y,Cb,Cr (Cb/Cr signed)
ycc2rgb = lambda p: np.clip(p @ Mi.T, 0, 255)
_u = lambda p: np.clip(np.rint(p + 128), 0, 255).astype(np.uint8) # chroma, JPEG-style
_b = lambda u: u.astype(np.float64) - 128
def sub420(p): # keep 1 of every 4 chroma samples
h, w = p.shape
return _b(np.asarray(Image.fromarray(_u(p), 'L')
.resize((w//2, h//2), Image.BOX)).astype(np.float64))
def back(p, w, h): # bilinear back to full size
return _b(np.asarray(Image.fromarray(_u(p), 'L')
.resize((w, h), Image.BILINEAR)).astype(np.float64))
def upscale(a8, f=2):
return np.asarray(Image.fromarray(a8, 'RGB')
.resize((a8.shape[1]*f, a8.shape[0]*f), Image.LANCZOS))
C = rgb2ycc(upscale(src).astype(np.float64)) # 2x 4:4:4 reference
S = rgb2ycc(src.astype(np.float64)) # the 4:2:0 original
h, w = C.shape[:2]
# chroma lost before the upscale, then lost again at the end
cb = back(sub420(back(sub420(S[...,1]), w, h)), w, h)
cr = back(sub420(back(sub420(S[...,2]), w, h)), w, h)
pipeline = ycc2rgb(np.dstack([C[...,0], cb, cr])).astype(np.uint8)
buf = io.BytesIO()
Image.fromarray(pipeline, 'RGB').save(buf, 'JPEG', quality=85, subsampling=2)
decoded = np.asarray(Image.open(io.BytesIO(buf.getvalue())).convert('RGB'))
Output on this machine, medians over five photographs at 1080×1620 upscaled 2×, all encodes forced
to 4:4:4 so the encoder is not a variable: chroma RMSE 0.571 → 0.614 → 0.785 → 0.978 for
444 / 420-encode / 420-source / 420-both, luma 48.263 → 47.594 → 47.583 → 47.576 dB.
The gap between 0.614 and 0.978 is the price of the first subsampling pass, and it is paid before the
upscale, where it is five times more expensive than at the end. Which is the one ordering rule this
experiment supports: if a 4:2:0 source has to pass through an upscale, the subsampling has already
happened — do not do it again on the way out if you can afford 4:4:4, and if you cannot, spend the
saving on quality instead.
Generation loss: JPEG has a fixed point, the modern codecs do not
Files get re-encoded. A CMS resizes, a CDN recompresses, a client opens and saves. So I re-encoded the same frame repeatedly at a fixed setting, feeding each output back in, and watched for two things: how fast quality fell, and whether the process ever stabilised.
| Encoder | PSNR gen 1 | Stabilised? | Generations to fixed point | Bytes gen 1 → last | PSNR at last |
|---|---|---|---|---|---|
| JPEG q85 | 46.379 | yes | 29 | 591,176 → 591,595 | 42.657 dB |
| WebP q85 | 43.768 | no (40 gen) | never | 280,894 → 220,036 | 32.880 dB |
| AVIF q80 | 45.949 | no (20 gen) | never | 339,423 → 347,689 | 32.278 dB |
JPEG reaches a byte-identical output at generation 29 and then stops changing forever. That is a property of DCT quantisation with a fixed table: once the coefficients are already snapped to the quantisation grid, re-quantising them is the identity. The cost is 3.72 dB from generation 1 to the fixed point, and after that the file is free to copy around.
WebP and AVIF show no such floor. Over 40 and 20 generations respectively they lost 10.9 dB and 13.7 dB with no sign of converging, and WebP's file got smaller every generation — 280,894 down to 220,036 bytes — while AVIF's got slightly larger. Neither behaviour is a fixed point; both are still drifting when the loop ends.
Practical reading: "JPEG is bad because it is generational" and "modern codecs are better" are both partly wrong. JPEG degrades faster per generation but terminates. The modern codecs degrade more slowly at first and then never stop. For a file that will be re-encoded a handful of times, AVIF wins. For a file that will be re-encoded an unknown number of times, the guarantee matters more than the first-generation score.
The format gap is not constant — it grows with pixel count
Same photograph, encoded at its native 1080×1620 and at the 2× upscale, same settings per encoder:
| Encoder | 1× (1080×1620) | 2× (2160×3240) | Byte ratio |
|---|---|---|---|
| JPEG q85 | 230,956 | 591,176 | 2.56× |
| WebP q85 | 131,668 | 280,894 | 2.13× |
| AVIF q80 | 160,909 | 339,423 | 2.11× |
Pixel count went up 4×. JPEG's file grew 2.56×, WebP 2.13×, AVIF 2.11×. JPEG is the least efficient at absorbing extra pixels, so the AVIF-over-JPEG size advantage widens from 1.44× at native resolution to 1.74× at the upscale. This is the argument for choosing the output format after deciding to upscale, not before: the bigger the output, the more the format choice is worth.
It also means a size budget set at 1× does not survive the upscale. A 231 KB JPEG at native size becomes 591 KB at 2× — 2.56 times the budget for 4 times the pixels. Switching that output to AVIF recovers 251,753 bytes, 42.6% of the 2× file, without touching the encoder quality knob.
A second negative result: measured after a downscale, the penalty halves
Most content is not viewed at full resolution. A 4K master becomes a 1200-pixel-wide web image. So I re-ran the comparison at matched size and then downscaled both the reference and the decoded file by 2× and 4× before computing PSNR:
| Encoder | Full frame | After 2× downscale | After 4× downscale |
|---|---|---|---|
| JPEG q68 | 43.153 dB | 44.368 dB | 45.924 dB |
| WebP q90 | 45.176 dB | 46.177 dB | 47.978 dB |
| AVIF q86 | 46.437 dB | 47.128 dB | 48.371 dB |
| AVIF − JPEG | 3.284 | 2.760 | 2.447 |
Two things happen at once, and they point in opposite directions. Every encoder's score rises when measured after a downscale — JPEG gains 1.22 dB at 2× and 2.77 dB at 4× — because downscaling averages away exactly the high-frequency error the encoder introduced in flat regions. But the gap between AVIF and JPEG narrows, from 3.28 dB to 2.45 dB, because AVIF's advantage was concentrated in that same high-frequency band.
So benchmarking encoders on full-resolution PSNR overstates their visible difference for downscaled delivery, and benchmarking them after a downscale understates it for full-resolution delivery. Which one is right depends entirely on where the file ends up, and any single number presented as "the" encoder ranking is answering a question that was not asked.
When to skip the choice entirely: lossless
If the file still has an edit ahead of it — a crop, a composite, another resize — every lossy number above is the wrong tool. On this frame:
| Format | Bytes | PSNR |
|---|---|---|
| PNG (optimize=True) | 4,555,198 | lossless |
| WebP lossless (method 6) | 3,276,760 | lossless |
| JPEG q85 (for scale) | 591,176 | 46.379 dB |
WebP lossless is 71.9% of the PNG size on a photographic frame, which is a legitimate saving with zero quality argument attached. But both are about 7.7× the JPEG size. Lossless is for intermediates, not for delivery, and the only reason to mention it in a page about output format is that the intermediate choice is where generation loss is either avoided or invited in.
The decision, reduced to what was measured
- Fixed byte budget, one encode, viewed at full resolution: AVIF, by 3.28 dB over JPEG. If AVIF is not an option in the pipeline, WebP q90 recovers 2.02 dB of that gap.
- Mostly flat frame (sky, backdrop, gradient): the flat-region artifact column is the one to watch; JPEG q70 puts 2.77× the source's local variation into smooth areas. Raise quality before changing format, because that column responds to quality and the edge column does not.
- Will be re-encoded an unknown number of times: prefer the codec with a fixed point. JPEG terminates at generation 29; WebP and AVIF were still drifting at 40 and 20.
- Delivered downscaled: the size advantage of the modern codecs survives, the quality advantage halves. Choose on bytes, not decibels.
- Another edit ahead: WebP lossless at 71.9% of PNG, and neither one is the delivery file.
None of this changes the fact that the frame above 43 dB is already past the point where a viewer at normal distance can see the difference. The format choice is a size decision with a small quality side effect, and treating it as a quality decision with a size side effect is how people end up spending 591 KB on a file that would have looked identical at 403 KB.
Reproducing it
import io, numpy as np
from PIL import Image
SRC = Image.open('photo.jpg').convert('RGB')
UP = SRC.resize((SRC.width*2, SRC.height*2), Image.LANCZOS)
A = np.asarray(UP) # reference: pre-encode array
def rt(fmt, **kw): # round-trip through one encoder
b = io.BytesIO(); Image.fromarray(A, 'RGB').save(b, fmt, **kw)
return len(b.getvalue()), np.asarray(Image.open(io.BytesIO(b.getvalue())).convert('RGB'))
def psnr(a, b):
a = a.astype(np.float64); b = b.astype(np.float64)
m = ((a-b)**2).mean()
return 99.0 if m == 0 else 10*np.log10(255.0**2/m)
def hf(a): # mean |Laplacian| on luma
y = 0.2126*a[...,0] + 0.7152*a[...,1] + 0.0722*a[...,2]
return float(np.abs(4*y[1:-1,1:-1] - y[:-2,1:-1] - y[2:,1:-1]
- y[1:-1,:-2] - y[1:-1,2:]).mean())
# 1. matched-size search
for fmt, key, rng in [('JPEG','quality',range(50,100,2)),
('WEBP','quality',range(50,100,2)),
('AVIF','quality',range(20,95,2))]:
best = min(rng, key=lambda v: abs(rt(fmt, **{key: v})[0] - 400_000))
n, rec = rt(fmt, **{key: best})
print(fmt, best, n, round(psnr(A, rec), 3), round(100*hf(rec)/hf(A), 1))
# 2. generation loss -> does it reach a byte-identical fixed point?
for fmt, kw, limit in [('JPEG', {'quality': 85}, 40),
('WEBP', {'quality': 85}, 40),
('AVIF', {'quality': 80}, 20)]:
cur = A.copy(); prev = None
for g in range(1, limit+1):
b = io.BytesIO(); Image.fromarray(cur, 'RGB').save(b, fmt, **kw)
data = b.getvalue()
cur = np.asarray(Image.open(io.BytesIO(data)).convert('RGB'))
if data == prev:
print(fmt, 'fixed point at gen', g, len(data), round(psnr(A, cur), 3)); break
prev = data
else:
print(fmt, 'no fixed point in', limit, 'generations', round(psnr(A, cur), 3))
Output on this machine, in order: JPEG 68 404460 43.153 116.4 · WEBP 90 381016 45.176
106.8 · AVIF 86 403527 46.437 104.4, then JPEG fixed point at gen 29 591595 42.657,
WEBP no fixed point in 40 generations 32.88, AVIF no fixed point in 20 generations
32.278. One caveat worth stating: the fixed-point generation count depends on the quality
setting and the frame — q85 on this source took 29 — so treat "29" as an order of magnitude, not a
constant. The existence of the fixed point is the reusable finding, and that one is a property
of DCT quantisation rather than of this photograph.