Colour Matrix and Range Before Upscaling: Two Bugs That Look Identical, Measured

Your upscale comes out with a green cast and crushed shadows. You check the model, the sharpen amount, the encode settings, and nothing explains it. The cause is almost certainly one of two metadata mistakes that happened before the upscaler ever ran — and they are not the same bug, do not cost the same, and cannot be fixed the same way. This is a measurement of both, against a reference we hold, with the numbers that distinguish them.

17 November 2026 · synthetic colour chart + real ffmpeg/x264 round-trips, measured with numpy, swscale and CIEDE2000

1. The two bugs, stated precisely

Video is not stored as RGB. It is stored as a luma plane and two chroma planes, and converting between the two requires two pieces of information that live in metadata rather than in pixels:

Both are metadata. A file can be encoded with one and decoded with the other, and nothing in the pixel data objects. Both produce a visible colour shift. That is why they get confused. Everything below is about pulling them apart.

Why this belongs in an upscaling guide. An upscaler operates on whatever numbers it is handed. If the numbers were decoded with the wrong matrix, the upscaler faithfully enlarges the wrong colour. It cannot detect the mistake, and — as §6 measures — it does not dilute it either. The error goes in and the same error comes out, larger.

2. The reference, and why the chart is built the way it is

A natural photograph is the wrong instrument here, because you cannot separate "the colour is wrong" from "the content is just like that". The reference is a 512×512 chart with four regions, each chosen to exercise a different part of the conversion:

The pipeline under test is the honest one: RGB → YCbCr 4:2:0 limited → back to RGB. That round-trip including chroma subsampling is the noise floor, and every cost below is quoted against it rather than against a perfect RGB copy, because subsampling is a cost you have already accepted.

The conversion maths is written out rather than imported, so it can be audited:

M709 = [[0.2126, 0.7152, 0.0722],
        [-0.1146, -0.3854, 0.5000],
        [ 0.5000, -0.4542, -0.0458]]

def rgb2yuv(rgb, M, limited=True):
    Y, Cb, Cr = rgb @ M[0], rgb @ M[1], rgb @ M[2]
    if limited:                      # the whole difference, right here
        Y  = Y  * 219.0/255.0 + 16.0
        Cb = Cb * 224.0/255.0 + 128.0
        Cr = Cr * 224.0/255.0 + 128.0
    return np.stack([Y, Cb, Cr], -1)

3. What each mistake costs

Four variants, one variable at a time. PSNR is reported alongside CIEDE2000, because PSNR in RGB underweights exactly the kind of error a colour bug produces — it measures per-channel squared error while the eye judges hue and lightness, which is what ΔE measures.

PipelinePSNRCostΔE meanΔE p95
correct — 709, limited, both ways28,64 dB—1,112,68
encoded 709, decoded as 60126,11 dB−2,53 dB2,785,72
encoded 601, decoded as 70925,92 dB−2,72 dB3,178,73
encoded limited, decoded as full24,40 dB−4,24 dB3,554,56
encoded full, decoded as limited25,32 dB−3,32 dB2,865,41
both wrong at once23,29 dB−5,35 dB4,307,94

The ΔE figures are the useful column. ΔE above roughly 2 is the threshold where a trained eye picks up a difference on a side-by-side; the 709-as-601 case sits at a mean of 2,78 with a 95th percentile of 5,72, which is comfortably visible on saturated content and invisible on grey. Note the ordering: a range mistake costs more than a matrix mistake (4,24 dB vs 2,53 dB) on this chart, and the two together cost 5,35 dB — less than the sum, because they partially cancel on some patches.

That ordering is not a law. §4 shows it depends entirely on content, and on mid-tone material both mistakes cost almost nothing at all.

Because table 1 is computed with our own arithmetic, it needs an independent engine. The same two mistakes were re-run through a real libx264 round-trip at CRF 18 with swscale doing the conversion, comparing against a correct decode of the same file:

Mistakeffmpeg costnumpy cost (§3)Channel bias, R/G/B
709 encoded, 601 decoded−2,04 dB−2,53 dB−0,79 / −0,58 / +0,44
limited encoded, full decoded−3,59 dB−4,24 dB−2,59 / −1,68 / −0,57

Two different implementations, 8-bit versus float arithmetic, a real codec in the loop: the magnitudes agree to within half a decibel and the ordering agrees exactly — range costs more than matrix on this content. The channel biases are a useful independent fingerprint too. The matrix mistake is not directionally uniform (−0,79 red, +0,44 blue); the range mistake is monotone-negative on all three (−2,59, −1,68, −0,57), which is what a range compression looks like. If you are staring at a bad output and wondering which bug you have, the sign pattern across channels is the first tell.

4. The headline result: the same bug is 0,02 dB or 16 dB, depending on content

Three content classes, each encoded at CRF 12 so the codec is not the variable, each decoded three ways. The "clean" column is the correct decode and serves as the reference for the two cost columns.

ContentClean601 matrixCostFull rangeCost
mid-tone texture only (60–196)22,79 dB22,77 dB−0,0222,95 dB+0,16
saturated patches44,38 dB28,30 dB−16,0823,62 dB−20,76
smooth gradient, full legal range44,53 dB29,76 dB−14,7729,96 dB−14,57

Read the first row against the second. On texture confined to the middle of the range, a completely wrong colour matrix costs 0,02 dB. That is not a small effect, it is no effect — smaller than the codec's own rounding. The full-range mistake on the same row scores 0,16 dB better than the correct decode, which is measurement noise, not a benefit.

On saturated patches the identical matrix bug costs 16,08 dB. Eight hundred times larger, from the same one-line metadata error.

The mechanism is arithmetic. A neutral pixel has R = G = B, so its Cb and Cr are zero under any matrix — 0.2126+0.7152+0.0722 = 1, and so does 0.299+0.587+0.114. The matrix only affects pixels that carry chroma. Mid-tone grey noise carries none, so the matrix cannot touch it. Reproduce that directly:

>>> M709 @ [1,1,1], M601 @ [1,1,1]
(1.0, 1.0)        # both sum to exactly 1 -- neutrals are invariant

The practical consequence is uncomfortable. If you validate a colour pipeline on a grey ramp, a test chart of flat mid-greys, or any scene that is mostly neutral — and a great many test suites are exactly that — you cannot detect a matrix error at all. The measurement will report a pass. The bug is only visible on saturated colour, which is precisely the content people build colour pipelines to handle.

5. Chroma is the plane that dies quietly

The reason a matrix error is concentrated in chroma is worth quantifying, because it explains why 4:2:0 footage is so much more fragile to it than RGB footage. Pushing a set of horizontal gratings through the 4:2:0 box filter, one frequency at a time, measuring how much amplitude survives:

Period (px)cycles/pxChroma keptdBLuma kept
480,0210,998−0,021,000
240,0420,991−0,071,000
120,0830,966−0,301,000
80,1250,924−0,691,000
60,1670,866−1,251,000
40,2500,707−3,011,000
30,3330,501−6,011,000
2 (phase π/2)0,5000,000null1,000

The last row is exact, and it is worth stating plainly: a 2-pixel chroma grating sampled at phase π/2 survives 4:2:0 as a constant. Input standard deviation 90,0, output standard deviation 0,0. The chroma channel is not blurred there, it is erased.

The luma column is the control and it is a hard 1,000 at every frequency, because 4:2:0 never touches luma. Which produces the asymmetry that governs everything in this article: measured on white noise, luma detail survives 4:2:0 at 27,97 → 27,97 (0,00 dB, unchanged to two decimals) while chroma detail falls 44,58 → 22,29, a loss of 6,02 dB.

So chroma arrives at your upscaler already down 6 dB, carrying at most half its original information. Every bit of matrix error you feed in is applied to that depleted plane. This is why colour mistakes on 4:2:0 sources look worse than the same mistake on an RGB still: the plane being damaged is the one with the least headroom left.

6. Upscaling does not dilute it — and that is provable, not empirical

The intuitive hope is that interpolation averages neighbouring pixels, and averaging might soften a systematic error. It does not, and the reason is that a matrix mistake is not noise — it is an affine transform, and interpolation is linear, so the two commute. A constant per-channel offset passes through any linear resampler completely unchanged.

Demonstrated with a 4× lanczos upscale of random content, then the same content with a three-channel bias added first, and the two results subtracted:

bias in        : [12.0, -9.0, 5.0]
bias out (mean): [12.0, -9.0, 5.0]
bias out per channel (min, max):
  R: (12.0, 12.0)    G: (-9.0, -9.0)    B: (5.0, 5.0)

Not approximately. The minimum and maximum of the difference are both exactly the input bias, on every channel, at every pixel. The upscale changed nothing about the error's magnitude. A 4× upscale turns a small wrong colour into a four-times-larger wrong colour that occupies the same fraction of the frame — every metric that operates on the whole image will report it identically.

The only way this ever changes is at the clamp. Measured on RGB content pushed near the rails, a bias that would otherwise be harmless can drag samples out of gamut and clip them:

ConditionOut-of-range samples% of pixels
matrix mismatch, 709 read as 60160 9937,76%
range mismatch, limited read as full3890,05%
correct decode, then unsharp mask ×1,0113 46914,43%
correct decode, sharpen chroma planes only76 4629,72%

Two things here that are easy to get wrong. First, the matrix bug clips 157 times more samples than the range bug on saturated content, even though §3 found the range bug costs more in dB. Sample count is not damage. A range error is a gentle uniform compression that shifts everything slightly; a matrix error is a large excursion that only affects chroma-carrying pixels but pushes them hard. Judge by ΔE, not by clip count.

Second, sharpening before the clamp produces more out-of-range samples than either metadata bug — 14,43% — with a 95th-percentile overshoot of 67,11 code values. If you sharpen in floating point and then clamp, you are clipping detail you just created. Sharpen, then clamp to a legal range you chose deliberately, and treat the clamp as part of the grade rather than an afterthought.

7. Does upscaling in RGB beat upscaling in YUV?

This is the question people actually ask, and the answer here is a negative result worth recording: it makes almost no difference, and the chroma kernel matters less than expected. Same 4× source, same luma kernel (lanczos), only the chroma path varies.

PathPSNRΔE mean|dCb||dCr|
upscale in RGB15,09 dB8,40——
YUV, chroma lanczos15,05 dB8,529,4610,91
YUV, chroma bilinear14,92 dB8,048,359,52

Converting to RGB, upscaling, and converting back costs 0,04 dB versus staying in YUV with a lanczos chroma kernel. Choosing lanczos over bilinear for chroma is worth 0,13 dB. Both are far below the 2,53 dB a matrix mistake costs, which puts the whole debate in perspective: the RGB-vs-YUV question is a rounding error next to getting the metadata right.

And there is a genuine conflict in the metrics. Lanczos chroma wins on PSNR (15,05 vs 14,92) but loses on ΔE2000 (8,52 vs 8,04). The explanation is ringing: a sharpening chroma kernel creates overshoot at colour boundaries, and overshoot that surrounds a saturated edge shifts hue more than a soft transition does, even while it lowers squared error overall. If your target is perceptual colour accuracy near hard edges — logos, graphics, text overlays, anime line art — a plain bilinear chroma upscale is the better choice despite the worse PSNR. Do not let one number decide this.

8. The three-line diagnostic

Everything above reduces to one decisive test, and it follows from the invariant in §4: neutrals have zero chroma, so they are unaffected by any matrix change. Which means the two bugs leave completely different fingerprints on a neutral ramp.

Render a grey ramp, push it through the suspect pipeline, and look at the output:

ffmpeg -y -i neutral_ramp.png -vf "scale=out_color_matrix=bt709:out_range=limited" \
       -pix_fmt yuv420p -c:v libx264 -qp 0 suspect.mp4

# now decode it the way your pipeline decodes it, and measure
ffmpeg -y -i suspect.mp4 -vf "scale=in_color_matrix=bt601:in_range=full,format=rgb24" out.png

Measured on that pair:

What you decode wrongNeutral ramp max deltaNeutral ramp mean deltaColour bars max delta
matrix only (709 → 601)0,00,00033,0
range only (limited → full)20,014,55316,0

The matrix row is not "close to zero", it is exactly zero — PSNR of the neutral ramp against the correct decode is infinite. Thirty-three code values of error on the colour bars, and literally nothing on the ramp. The range row is the opposite shape: the ramp is wrong by a mean of 14,553, the colour bars by 11,542. Both bugs damage the bars; only one damages the ramp.

The test, in one sentence: if a neutral ramp comes out wrong, your bug is range. If the neutral ramp is perfect but colour is wrong, your bug is matrix. There is no ambiguous case, because the matrix provably cannot touch a neutral.

The remaining confirmation, if you want one, is that a range error is a near-uniform shift while a matrix error is directional in hue. Measured on saturated patches, the range bug shifts lightness by 4,56 and hue by 19,78° — clipping distorts hue badly — while the matrix bug shifts hue by only 0,71° but lightness by 1,42. On a smooth gradient the pattern inverts: the matrix bug shows 4,56° of hue shift, the range bug 1,40°. Content-specific, which is why the neutral-ramp test is the reliable one and these numbers are only corroboration.

9. Fixing it, in ffmpeg, explicitly

The failure mode to avoid is letting swscale guess. Left to itself it reads the container tags; if the tags are missing or wrong, it guesses from resolution, and its guess for anything below 576 lines is bt601. That single heuristic explains a large fraction of "my HD clip looks orange".

# decode, stating the truth about the source rather than trusting tags
ffmpeg -i in.mp4 \
  -vf "scale=in_color_matrix=bt709:in_range=limited:flags=lanczos+accurate_rnd+full_chroma_int,format=rgb24" \
  out.png

# encode, writing the tags so the next tool does not have to guess
ffmpeg -i in.mov \
  -vf "scale=out_color_matrix=bt709:out_range=limited" \
  -pix_fmt yuv420p -colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv \
  out.mp4

Three details that matter more than they look. full_chroma_int keeps chroma interpolation at full precision instead of rounding to 8-bit intermediates. accurate_rnd does the same for the RGB conversion. And -color_range tv on the output is what stops the next tool from repeating the guess; note it is tv for limited and pc for full, which is charmingly unintuitive.

Verify rather than assume:

$ ffprobe -v error -select_streams v:0 \
    -show_entries stream=color_space,color_range,pix_fmt -of json out.mp4

{"streams":[{"pix_fmt":"yuv420p","color_range":"tv","color_space":"bt709"}]}

If color_space is absent from that output, the tags are missing and every downstream tool is guessing. That is the condition to fix before you touch anything else — a file with no matrix tag is not "probably fine", it is undefined, and different tools will resolve it differently, which is how a single file ends up looking correct in one player and wrong in another.

10. What this actually means for an upscale

  1. Fix metadata before the upscaler, not after. The upscaler cannot detect the error and provably does not dilute it (§6). Anything you do downstream is colour correction applied on top of a wrong decode, which is guesswork.
  2. Validate on saturated content. A grey-ramp test passes a wrong matrix at 0,02 dB. If your QA material is neutral, your QA cannot see this class of bug.
  3. Judge with ΔE, not PSNR or clip count. §7 has a case where PSNR and ΔE2000 rank two options in opposite orders, and §6 has a case where the bug with 157× the clipping costs less.
  4. Expect more damage on 4:2:0. Chroma is already down 6,02 dB before the matrix error is applied to it.
  5. Write the tags on output. The bug is nearly always inherited, not invented — someone upstream omitted -colorspace and the next tool guessed bt601 from a resolution heuristic.

The general shape of this result is the same one that recurs across this measurement series: the upscaler is the last and least powerful stage in the chain. A matrix mistake is a two-byte metadata error that costs 16 dB on saturated content, survives every resampler unchanged, and is invisible to grey-ramp testing. No amount of model quality recovers it, because there is nothing to recover — the numbers were already wrong when they arrived. Getting the colour pipeline right before you upscale is worth more than any choice of kernel, and it costs one flag.

Reproducing this

No model weights, no vendor SDK. The chart generator, both colour matrices, the 4:2:0 subsampler, the lanczos resampler and a CIEDE2000 implementation are all plain numpy; the cross-check uses ffmpeg 6.1 with libx264 at -crf 12 or -qp 0 so the codec is not the variable. Total runtime under two minutes on one core. The only non-obvious requirement is that the reference chart must contain saturated patches — a mid-tone chart will report that your broken pipeline is fine.