Output Format After Upscaling: What JPEG, WebP and AVIF Do to the Detail You Just Bought

The last step of every upscaling job is the one nobody measures. You spent the compute to reconstruct detail, then hand the result to a lossy encoder and let it decide which of those details were worth keeping. I took one upscaled frame, encoded it three ways at a matched file size, and measured what came back. AVIF bought 3.28 dB over JPEG at the same 400 KB. But the more useful finding is that all three encoders added high-frequency energy rather than removing it, which means the intuitive metric — "how much detail survived" — is measuring the wrong sign.

Measured with numpy 1.26.4 and Pillow 12.2.0 · source is one real photograph at 1080×1620, upscaled 2× with Lanczos to 2160×3240 · the reference for every error figure is the upscaled RGB array itself, so all numbers isolate encoder loss and exclude resampler loss · all statistics computed on uint8-decoded arrays, never on screenshots · every figure reproducible from the snippet at the end

The setup: matched size, not matched quality

Comparing encoders at "quality 80" is meaningless, because the three quality scales are not the same scale. Pillow's quality=80 for JPEG, WebP and AVIF produce files that differ by a factor of two in size on identical pixels. What a person actually chooses is a file size budget — a hero image that must stay under half a megabyte, a thumbnail under 60 KB — and then asks which encoder uses that budget best.

So the comparison below searches each encoder's quality scale for the setting that lands closest to 400,000 bytes, then measures PSNR against the pre-encode array. The three files that competed are within 6% of each other in size, which is as matched as an integer quality knob allows.

EncoderSettingBytesPSNR vs sourceHF energy vs source
JPEGq68404,46043.153 dB116.4%
WebPq90381,01645.176 dB106.8%
AVIFq86403,52746.437 dB104.4%

AVIF is 3.28 dB ahead of JPEG and 1.26 dB ahead of WebP at the same budget. That is a real and reproducible margin — but 3.28 dB on a frame that already sits above 43 dB is not a visible difference at normal viewing distance. It matters when the file will be re-encoded downstream, which is the case this whole article keeps returning to.

A negative result: every encoder added detail instead of losing it

The column that surprised me is the last one. I expected all three to fall below 100% — lossy encoding discards, so high-frequency energy should drop. It did not. JPEG came back at 116% of the source's high-frequency energy, WebP at 107%, AVIF at 104%.

The measurement is mean absolute Laplacian on luma. A value above 100% means the decoded image has steeper local transitions than the array that was handed to the encoder. Nothing was added to the picture; the encoder manufactured edges. JPEG does it most because 8×8 block quantisation creates discontinuities at block borders, and a discontinuity is by definition high-frequency — it shows up as a spike in any Laplacian-type measure regardless of whether a human can see it.

Consequence for measurement: "detail retained" cannot be computed as a ratio of high-frequency energy after a lossy step. The ratio exceeds one, so it cannot be interpreted as a retention figure at all. Blocking artifacts are indistinguishable from detail under this metric. This is the same trap I hit in the JPEG-blocking piece: the block edge keeps its contrast and gains area, so a gradient-energy measure reports an increase where the eye reports a defect.

Where the invented detail lives

If the extra high-frequency energy were spread evenly, it would be a mild curiosity. It is not spread evenly. I split the frame into the flattest quartile of gradient magnitude (24.9% of pixels) and the top decile (sharp edges), then measured the same Laplacian statistic in each region against the same regions of the source:

EncoderFlat-quartile HF vs sourceEdge-decile HF vs source
JPEG q70276.6%106.4%
JPEG q95183.5%101.8%
WebP q80232.0%102.9%
AVIF q80232.1%102.1%

In the flat regions the encoders roughly double or triple the local variation. On real edges they are within a few percent of the source. That is exactly backwards from what a person wants: the places that were smooth and should stay smooth are where the noise lands, and the places that already had contrast are preserved almost perfectly.

It also explains why raising JPEG quality helps so much in the flat column — 276.6% at q70 falls to 183.5% at q95 — while the edge column barely moves. Blocking is a flat-region phenomenon. If your frame is mostly sky, gradient, or studio backdrop, JPEG quality is the parameter that matters, and it matters for a reason that has nothing to do with detail retention.

Chroma subsampling: what 18.6% of the file actually costs

JPEG at q90 with 4:4:4 chroma, then the same q90 in 4:2:2 and 4:2:0. Everything else held fixed:

SubsamplingBytesPSNRChroma RMSE (Cb/Cr only)
4:4:4 (s=0)899,64249.030 dB0.566
4:2:2 (s=1)795,56848.245 dB0.645
4:2:0 (s=2)732,06947.600 dB0.711

4:2:0 removes 18.6% of the bytes and raises chroma error by 25.8%. The PSNR cost looks small — 1.43 dB — but PSNR is dominated by luma, which subsampling does not touch. The isolated chroma RMSE is the honest number, and it rises by a quarter. On a saturated colour boundary — red signage, coloured gel lighting, a bright product on a coloured backdrop — that quarter is where the coloured fringe along the edge comes from.

This is worth connecting to the upscale itself: subsampling throws away chroma resolution after you paid to reconstruct it. If the source was already 4:2:0, upscaling reconstructed chroma detail that never existed and 4:2:0 output discards it again for an 18.6% saving. If the source was 4:4:4 or synthetic, that saving is a real loss.

What a second 4:2:0 pass costs when the source was already 4:2:0

Everything above assumed a full-resolution chroma source. Most photos that reach an upscaler do not have one: they were JPEGs at 4:2:0, or frames pulled out of video at 4:2:0. So the common pipeline is not 4:4:4 → upscale → encode. It is 4:2:0 → upscale → 4:2:0, and the chroma gets thrown away twice. I measured both passes separately instead of assuming the second one is free.

Step one: the pure arithmetic, no codec involved

Take the 2× lanczos reference, subsample Cb the way 4:2:0 does (keep one sample in four, box filter down, bilinear back up), then do it again to the result. Two passes, no JPEG anywhere, so nothing but the subsampling can move the number:

Image1st pass RMSE2nd pass RMSETotal vs original2nd ÷ 1st
clear-reality0.5320.3230.7520.61
cinema-vivid0.6670.4620.9940.69
natural-hd0.5340.3470.7780.65
true-texture0.4810.2810.6780.58
ultra-crisp0.5470.3610.8040.66
median0.5340.3470.7780.65

The hypothesis I started with was that the second pass would cost the same as the first — that subsampling is a per-generation tax. It does not. The second pass costs a median 0.65× the first, in all five images, because the first pass already removed the chroma detail the second pass would have destroyed. What is left to lose is a smooth plane, and smoothing a smooth plane is cheap.

That is not a reprieve, only a smaller bill. The two errors are correlated (r = 0.50 on Cb, not independent), which is why they do not add in quadrature: 0.534² + 0.347² would predict 0.637, and the measured total is 0.778 — 46% worse than a single pass, and 22% worse than independence would suggest. The second pass damages the same neighbourhoods the first one already damaged, and the damage compounds rather than accumulating independently.

Step two: the same thing with a JPEG at the end

Now the real question — is any of that visible once the file is actually encoded? Four chains, all encoded at q85 with 4:4:4 forced so the encoder contributes the same amount to every row and the only variable is where the chroma was lost. Reference is the 2× 4:4:4 array; 5 images; medians:

ChainBytesLuma PSNRChroma RMSEvs 4:4:4
4:4:4 all the way711,55848.263 dB0.571—
4:2:0 at encode only702,84447.594 dB0.614+7.5%
4:2:0 at source only693,95847.583 dB0.785+37.5%
4:2:0 at both ends694,49847.576 dB0.978+71.3%

Two things worth flagging. The chroma damage compounds roughly in proportion to how many times the resolution is cut — 0.571 → 0.614 → 0.785 → 0.978 is +7.5%, +37.5%, +71.3% — but the source pass costs five times what the encode pass costs, which is the opposite of what a pipeline diagram suggests. Losing chroma before the upscale is expensive because the upscale then has to invent chroma from a quarter-resolution plane and smooths it across edges; losing it at the end is cheap because the upscaler had real chroma to work with and only the final delivery throws it away.

The second thing is the luma column, and it is a negative result about my own method. Luma PSNR falls ~0.67 dB on all three subsampled chains, including the one where chroma was only touched at the last step. Luma is untouched by 4:2:0 by definition, so that drop cannot be a chroma effect. Isolating it: chroma-subsample → RGB → YCbCr before any encode costs a luma RMSE of 0.086, against a JPEG q85 floor of 0.987 — so 92% of the luma error is the encoder's own quantisation and the remaining 8% is the chain. The honest reading is that these 0.67 dB are not attributable to the chroma pass at all; the small residual comes from the encoder re-allocating bits around the extra clipping (3.83% of pixels clipped, against 2.84% on the clean chain). Reporting the luma column as if subsampling caused it would be wrong.

The penalty grows with the scale factor

If the mechanism is "the upscale has to invent chroma it never had", the damage should scale with how much inventing is required. It does, slowly. Chroma error of the double-subsampled chain divided by the single-pass chain, median of five images:

Scale factorOne passTwo passesRatio
2×0.6140.9781.59×
3×0.5170.8431.63×
4×0.4520.7731.72×

Both absolute numbers fall with the factor — more upscale means more smoothing, and a smoother image is easier to subsample — but the ratio climbs from 1.59× to 1.72×. The gap widens at exactly the scales where people upscale furthest. It is not a dramatic effect, and at 4× the difference is 0.32 RMSE units on a 0–255 scale, which nobody will see on a photograph. Where it will show is a saturated edge, and that is measurable: the 99th-percentile chroma error on the top-5%-gradient mask went from a median 4.0 (4:4:4) to 6.0 (encode only), 11.3 (source only) and 13.5 (both) — 3.4× the clean chain at the pixels where chroma error is most visible.

A fourth negative result: the 20% saving does not depend on chroma detail

The reason 4:2:0 is offered as an export option is that it is ~20% smaller. I expected that saving to shrink on an already-subsampled source, since there is nothing left in the chroma to drop. It does not shrink at all:

Source4:4:4 bytes4:2:0 bytesSaving
full-resolution chroma745,940599,68319.6%
already 4:2:0731,212594,50718.7%
median over 5 images21.27% → 20.18%

The saving is 20.18% instead of 21.27% — a one-point difference across five images and four quality levels. So the 20% is not buying the removal of chroma detail, which is already gone; it is buying the removal of chroma resolution, which is a fixed 3/4 of the chroma samples no matter how smooth they are. The practical consequence: you cannot argue "my source was already 4:2:0, so 4:2:0 output costs me nothing" and you also cannot argue "so it saves me less". It saves the same and costs more, because it is the second dose.

And if the byte budget is what is actually fixed, the choice is not binary. Encoding the same 2× frame as 4:4:4 at a lower quality — q76–77, chosen by bisection to match the 4:2:0 file's byte count — beat the 4:2:0 export on chroma error in 5 of 5 images: median chroma RMSE 0.656 against 1.129, a 41.9% reduction, for a luma cost of 0.625 dB (47.553 → 46.928 dB). Quality knobs spend bytes where the encoder thinks they matter; subsampling spends them on a fixed geometric assumption instead. On a photographic frame that is not upscaled — 4:2:0 is still the better trade at these numbers, since 0.6 dB of luma is worth more than 0.47 RMSE of chroma. It is specifically on the second pass, where chroma has already been halved once, that paying in quality rather than in chroma resolution starts to win.

Reproducing it

The whole measurement is one script, and it hides one trap that produced a wrong number on the first run. Pillow's YCbCr mode keeps Cb and Cr as signed values around zero — on this frame Cb runs −75.1 to 49.2 and 63.7% of its samples are negative. Clip a signed plane to 0 before sampling it and every one of those samples becomes a flat zero: the box filter then averages a plane with its entire lower half deleted, and the measured chroma RMSE comes out 8.38. Cast the same negative floats straight to uint8 and they wrap to 255, which is worse — RMSE 198.34, a number so absurd it is the only reason the bug was caught at all. Chroma has to be offset by 128 before it goes near an 8-bit buffer, exactly the way JPEG stores it, which brings the same measurement to 1.07:

import io, numpy as np
from PIL import Image

M  = np.array([[ .299, .587, .114],          # BT.601 forward
               [-.168736, -.331264, .5],
               [ .5, -.418688, -.081312]])
Mi = np.linalg.inv(M)

rgb2ycc = lambda a: a @ M.T                   # float RGB 0-255 -> Y,Cb,Cr (Cb/Cr signed)
ycc2rgb = lambda p: np.clip(p @ Mi.T, 0, 255)

_u = lambda p: np.clip(np.rint(p + 128), 0, 255).astype(np.uint8)   # chroma, JPEG-style
_b = lambda u: u.astype(np.float64) - 128

def sub420(p):                                # keep 1 of every 4 chroma samples
    h, w = p.shape
    return _b(np.asarray(Image.fromarray(_u(p), 'L')
                         .resize((w//2, h//2), Image.BOX)).astype(np.float64))

def back(p, w, h):                            # bilinear back to full size
    return _b(np.asarray(Image.fromarray(_u(p), 'L')
                         .resize((w, h), Image.BILINEAR)).astype(np.float64))

def upscale(a8, f=2):
    return np.asarray(Image.fromarray(a8, 'RGB')
                      .resize((a8.shape[1]*f, a8.shape[0]*f), Image.LANCZOS))

C  = rgb2ycc(upscale(src).astype(np.float64))        # 2x 4:4:4 reference
S  = rgb2ycc(src.astype(np.float64))                 # the 4:2:0 original
h, w = C.shape[:2]

# chroma lost before the upscale, then lost again at the end
cb = back(sub420(back(sub420(S[...,1]), w, h)), w, h)
cr = back(sub420(back(sub420(S[...,2]), w, h)), w, h)
pipeline = ycc2rgb(np.dstack([C[...,0], cb, cr])).astype(np.uint8)

buf = io.BytesIO()
Image.fromarray(pipeline, 'RGB').save(buf, 'JPEG', quality=85, subsampling=2)
decoded = np.asarray(Image.open(io.BytesIO(buf.getvalue())).convert('RGB'))

Output on this machine, medians over five photographs at 1080×1620 upscaled 2×, all encodes forced to 4:4:4 so the encoder is not a variable: chroma RMSE 0.571 → 0.614 → 0.785 → 0.978 for 444 / 420-encode / 420-source / 420-both, luma 48.263 → 47.594 → 47.583 → 47.576 dB. The gap between 0.614 and 0.978 is the price of the first subsampling pass, and it is paid before the upscale, where it is five times more expensive than at the end. Which is the one ordering rule this experiment supports: if a 4:2:0 source has to pass through an upscale, the subsampling has already happened — do not do it again on the way out if you can afford 4:4:4, and if you cannot, spend the saving on quality instead.

Generation loss: JPEG has a fixed point, the modern codecs do not

Files get re-encoded. A CMS resizes, a CDN recompresses, a client opens and saves. So I re-encoded the same frame repeatedly at a fixed setting, feeding each output back in, and watched for two things: how fast quality fell, and whether the process ever stabilised.

EncoderPSNR gen 1Stabilised?Generations to fixed pointBytes gen 1 → lastPSNR at last
JPEG q8546.379yes29591,176 → 591,59542.657 dB
WebP q8543.768no (40 gen)never280,894 → 220,03632.880 dB
AVIF q8045.949no (20 gen)never339,423 → 347,68932.278 dB

JPEG reaches a byte-identical output at generation 29 and then stops changing forever. That is a property of DCT quantisation with a fixed table: once the coefficients are already snapped to the quantisation grid, re-quantising them is the identity. The cost is 3.72 dB from generation 1 to the fixed point, and after that the file is free to copy around.

WebP and AVIF show no such floor. Over 40 and 20 generations respectively they lost 10.9 dB and 13.7 dB with no sign of converging, and WebP's file got smaller every generation — 280,894 down to 220,036 bytes — while AVIF's got slightly larger. Neither behaviour is a fixed point; both are still drifting when the loop ends.

Practical reading: "JPEG is bad because it is generational" and "modern codecs are better" are both partly wrong. JPEG degrades faster per generation but terminates. The modern codecs degrade more slowly at first and then never stop. For a file that will be re-encoded a handful of times, AVIF wins. For a file that will be re-encoded an unknown number of times, the guarantee matters more than the first-generation score.

The format gap is not constant — it grows with pixel count

Same photograph, encoded at its native 1080×1620 and at the 2× upscale, same settings per encoder:

Encoder1× (1080×1620)2× (2160×3240)Byte ratio
JPEG q85230,956591,1762.56×
WebP q85131,668280,8942.13×
AVIF q80160,909339,4232.11×

Pixel count went up 4×. JPEG's file grew 2.56×, WebP 2.13×, AVIF 2.11×. JPEG is the least efficient at absorbing extra pixels, so the AVIF-over-JPEG size advantage widens from 1.44× at native resolution to 1.74× at the upscale. This is the argument for choosing the output format after deciding to upscale, not before: the bigger the output, the more the format choice is worth.

It also means a size budget set at 1× does not survive the upscale. A 231 KB JPEG at native size becomes 591 KB at 2× — 2.56 times the budget for 4 times the pixels. Switching that output to AVIF recovers 251,753 bytes, 42.6% of the 2× file, without touching the encoder quality knob.

A second negative result: measured after a downscale, the penalty halves

Most content is not viewed at full resolution. A 4K master becomes a 1200-pixel-wide web image. So I re-ran the comparison at matched size and then downscaled both the reference and the decoded file by 2× and 4× before computing PSNR:

EncoderFull frameAfter 2× downscaleAfter 4× downscale
JPEG q6843.153 dB44.368 dB45.924 dB
WebP q9045.176 dB46.177 dB47.978 dB
AVIF q8646.437 dB47.128 dB48.371 dB
AVIF − JPEG3.2842.7602.447

Two things happen at once, and they point in opposite directions. Every encoder's score rises when measured after a downscale — JPEG gains 1.22 dB at 2× and 2.77 dB at 4× — because downscaling averages away exactly the high-frequency error the encoder introduced in flat regions. But the gap between AVIF and JPEG narrows, from 3.28 dB to 2.45 dB, because AVIF's advantage was concentrated in that same high-frequency band.

So benchmarking encoders on full-resolution PSNR overstates their visible difference for downscaled delivery, and benchmarking them after a downscale understates it for full-resolution delivery. Which one is right depends entirely on where the file ends up, and any single number presented as "the" encoder ranking is answering a question that was not asked.

When to skip the choice entirely: lossless

If the file still has an edit ahead of it — a crop, a composite, another resize — every lossy number above is the wrong tool. On this frame:

FormatBytesPSNR
PNG (optimize=True)4,555,198lossless
WebP lossless (method 6)3,276,760lossless
JPEG q85 (for scale)591,17646.379 dB

WebP lossless is 71.9% of the PNG size on a photographic frame, which is a legitimate saving with zero quality argument attached. But both are about 7.7× the JPEG size. Lossless is for intermediates, not for delivery, and the only reason to mention it in a page about output format is that the intermediate choice is where generation loss is either avoided or invited in.

The decision, reduced to what was measured

None of this changes the fact that the frame above 43 dB is already past the point where a viewer at normal distance can see the difference. The format choice is a size decision with a small quality side effect, and treating it as a quality decision with a size side effect is how people end up spending 591 KB on a file that would have looked identical at 403 KB.

Reproducing it

import io, numpy as np
from PIL import Image

SRC = Image.open('photo.jpg').convert('RGB')
UP  = SRC.resize((SRC.width*2, SRC.height*2), Image.LANCZOS)
A   = np.asarray(UP)                       # reference: pre-encode array

def rt(fmt, **kw):                         # round-trip through one encoder
    b = io.BytesIO(); Image.fromarray(A, 'RGB').save(b, fmt, **kw)
    return len(b.getvalue()), np.asarray(Image.open(io.BytesIO(b.getvalue())).convert('RGB'))

def psnr(a, b):
    a = a.astype(np.float64); b = b.astype(np.float64)
    m = ((a-b)**2).mean()
    return 99.0 if m == 0 else 10*np.log10(255.0**2/m)

def hf(a):                                 # mean |Laplacian| on luma
    y = 0.2126*a[...,0] + 0.7152*a[...,1] + 0.0722*a[...,2]
    return float(np.abs(4*y[1:-1,1:-1] - y[:-2,1:-1] - y[2:,1:-1]
                        - y[1:-1,:-2] - y[1:-1,2:]).mean())

# 1. matched-size search
for fmt, key, rng in [('JPEG','quality',range(50,100,2)),
                      ('WEBP','quality',range(50,100,2)),
                      ('AVIF','quality',range(20,95,2))]:
    best = min(rng, key=lambda v: abs(rt(fmt, **{key: v})[0] - 400_000))
    n, rec = rt(fmt, **{key: best})
    print(fmt, best, n, round(psnr(A, rec), 3), round(100*hf(rec)/hf(A), 1))

# 2. generation loss -> does it reach a byte-identical fixed point?
for fmt, kw, limit in [('JPEG', {'quality': 85}, 40),
                       ('WEBP', {'quality': 85}, 40),
                       ('AVIF', {'quality': 80}, 20)]:
    cur = A.copy(); prev = None
    for g in range(1, limit+1):
        b = io.BytesIO(); Image.fromarray(cur, 'RGB').save(b, fmt, **kw)
        data = b.getvalue()
        cur = np.asarray(Image.open(io.BytesIO(data)).convert('RGB'))
        if data == prev:
            print(fmt, 'fixed point at gen', g, len(data), round(psnr(A, cur), 3)); break
        prev = data
    else:
        print(fmt, 'no fixed point in', limit, 'generations', round(psnr(A, cur), 3))

Output on this machine, in order: JPEG 68 404460 43.153 116.4 · WEBP 90 381016 45.176 106.8 · AVIF 86 403527 46.437 104.4, then JPEG fixed point at gen 29 591595 42.657, WEBP no fixed point in 40 generations 32.88, AVIF no fixed point in 20 generations 32.278. One caveat worth stating: the fixed-point generation count depends on the quality setting and the frame — q85 on this source took 29 — so treat "29" as an order of magnitude, not a constant. The existence of the fixed point is the reusable finding, and that one is a property of DCT quantisation rather than of this photograph.