Letterboxed Video Before Upscaling: Crop the Bars First, and Prove It

Every "upscale to 4K" button assumes the frame is the picture. On a letterboxed or pillarboxed source it is not: a 1920×1080 frame can hold a 1440×1080 picture, which means a quarter of the pixels you are about to interpolate are pure black and are not part of any picture at all. The obvious move is to crop first. The interesting part is that the number your quality metric reports will not change if you do not — and the number that does change is a lot worse than the usual "it's only the edges" dismissal.

Measured 18 November 2026 · Pillow 12.2.0, numpy 1.26.4, ffmpeg 6.1.1 · known-answer masters generated at 3840×2880 with a seeded scene, then framed and re-measured

What "bars" actually cost you, in pixels

Take a 4:3 picture and put it in a 16:9 frame. The picture is 1440×1080; the frame is 1920×1080; so 240 pixels on the left and 240 on the right are not picture. That is 480 × 1080 = 518,400 pixels out of 2,073,600 — exactly 25.00%.

Upscale the frame as-is to 4K and you interpolate 8,294,400 output pixels. Upscale the cropped picture and fill the width yourself, and you interpolate 6,220,800 — the 25% is gone. The compute saving is not a rounding detail, and it is measurable:

PathPixels resampledTime, 4× lanczosWhat the resampler sees at the picture edge
Crop bars, then upscale6,220,800443 msThe edge of the picture
Upscale full frame, then crop8,294,400604 msThe edge of the picture plus 240 px of black
Difference−25.0%1.37× slower—

The 1.37× is not the 1.33× you would predict from area alone — measured on the same machine, same kernel, three runs each. The extra is fixed overhead that does not scale with pixel count, so the saving is real but slightly less than the area ratio suggests. If you are doing this to video, 604 ms against 443 ms per frame is 18.1 s against 13.3 s of pure resampling for one second of 30 fps footage — per second, for every second you process.

Experiment 1 — how deep does the black actually bleed?

Here is where the intuition is usually wrong in both directions. People who crop say "otherwise you get a soft edge"; people who do not crop say "it's only a pixel or two, nobody sees it". Both are half right, and the measured answer explains why.

The test scene deliberately puts maximum contrast at x=0 — a horizontal gradient that starts at luma 200 on the very first column and falls to 40 — so any leakage from the black bar has the largest possible signal to contaminate. Then: frame it, upscale the frame, crop the bars back out, and compare against cropping first and upscaling. Difference is reported in source pixels so the numbers are comparable across scale factors.

KernelScaleContaminated output cols= source pxPSNR, first 2 pxPSNR, first 8 pxPSNR, whole frame
lanczos2×84.0017.7923.4548.86
bicubic2×52.5018.4124.3749.77
lanczos3×82.6715.5820.6747.82
bicubic3×51.6715.7221.4048.55
lanczos4×143.5013.9619.1447.53
bicubic4×102.5014.1119.8348.23
lanczos8×283.5011.0316.1747.29
bicubic8×182.2511.1516.6247.96

Two things fall out of this table that are not obvious from the "soft edge" story.

The depth is a kernel property, not a scale property. Lanczos contaminates 3.5 source pixels regardless of whether you scale 4× or 8×; bicubic contaminates 2.5 at 4× and 2.25 at 8×. This is exactly right: the support of the filter is fixed in source coordinates, so a wider kernel reaches further into the bar no matter how far out you sample. Doubling the output resolution doubles the number of affected output columns without changing how much of the picture is wrong.

The damage is severe exactly where it happens. PSNR in the first two output pixels is 11–18 dB. For scale, 30 dB is "visible but tolerable" and 40 dB is "hard to find". At 14 dB the first pixel of your picture is, numerically, not the picture — it is a blend of the picture and a black bar. Nobody would ship that if it were the whole frame. It is the whole frame only 0.2% of the time. The depth figure is 3.5 source pixels at 4× (14 output columns) and 3.5 at 8× (28 output columns) — the earlier version of this paragraph said "4 source pixels", which is the 2× number, and the header said the same; both are corrected here.

Experiment 2 — the metric cannot see it, and that is the real trap

Now the part that matters if you have a quality gate in CI. Run the two paths against the true master — not against each other — and report whole-frame PSNR:

KernelCrop-first, whole frameUpscale-first, whole frameΔCrop-first, first 8 pxUpscale-first, first 8 px
lanczos20.2920.28−0.0170.9832.98
bicubic20.3720.38+0.0170.9834.48

The wrong path is ahead by 0.01 dB on one kernel and behind by 0.01 dB on the other. That is noise — the number is identical to the precision anyone would report. Meanwhile the actual edge is 38 dB worse. A regression test built on whole-frame PSNR would pass the broken pipeline forever, and would pass it slightly more comfortably with bicubic.

This is not a subtle numerical artefact, it is the arithmetic of averaging. The contaminated region is 8 columns out of 2880, or 0.21% of the frame, at 33 dB; the other 99.79% sits at 71 dB. The global figure is dominated by the good part:

If the bad strip is…at 33 dB…at 30 dB
0.21% of the frame (8 of 3840 cols)59.5 dB global56.0 dB global
1% of the frame52.9 dB global50.0 dB global
5% of the frame46.0 dB global44.0 dB global

Read the budget backwards and it gets worse. Starting from a 45 dB baseline, a strip at 33 dB can cover 1.74% of the frame before the global number drops a single dB. At 30 dB it is 0.85%. A crop-edge defect is 5–8× smaller than the budget, so it will never show up in a whole-frame gate — no matter how you tune the threshold. The defect has to be measured where it lives, or not at all.

Practical consequence. If you want a gate that catches this, the gate has to be a strip metric: PSNR of the first 8 output pixels of the active area, against a reference. That number goes from 71 dB to 33 dB between the two paths. The whole-frame number goes from 20.29 to 20.28.

Experiment 3 — why cropdetect's default limit is 24, and when it breaks

The practical question is not "should I crop" — it is "will the crop be right". ffmpeg's cropdetect finds bars by deciding, for each line, whether that line is bar or picture. The default limit is 24.0/255, i.e. 24 of 255. Measured on constant-luma bars, sweeping bar luma against limit, the detected width is:

Bar luma@8@16@24@32@40@48
16192014401440144014241354
20192019201440144014241354
24192019201440144014241354
25192019201920144014241354
32192019201920144014241354
40192019201920192014241354

1920 means nothing was cropped. Two things are worth reading off this table. First, the crop stays correct up to a bar luma equal to the limit — 24 at limit 24, 32 at limit 32 — and breaks at one luma step above it. So the default tolerates a bar of exactly 24, not "just under 24". Second, the last two columns are not a failure to crop but a failure to stop: at limit 40 and 48 the detector returns 1424 and 1354, i.e. it has walked past the bar and eaten 16 and 86 columns of picture. Raising the limit to catch a light bar always costs picture at the far end, because the detector runs into genuinely dark image content and cannot tell it from a bar.

Correction. An earlier version of this page stated the rule as "a column is bar while its maximum luma is below the limit, so a bar whose brightest pixel is exactly the limit counts as picture", and published a table in which a bar at luma 24 defeated limit=24. That table does not reproduce with the command printed beneath it, and the rule was wrong. The statistic is a truncated mean over the line, and the comparison is strictly greater-than. Experiment 4 below derives both from the filter source and measures the consequence, which is a failure condition on area, not on brightness.

Experiment 4 — the cropdetect rule I published here was wrong, and here is the corrected one

An earlier version of this page said the threshold test was "a column is bar while its maximum luma is below the limit, so a bar whose brightest pixel is exactly the limit counts as picture". That is not what ffmpeg does, and it does not reproduce. Re-running the page's own command on ffmpeg 6.1.1 gives a bar at luma 24 detected correctly at the default limit, which the old table said was impossible. So the rule was measured properly instead of assumed.

Read from source (libavfilter/vf_cropdetect.c, n6.1), checkline() sums every pixel of the line and returns total / div — an integer mean over the column or row, not a maximum. The scan loop then tests it strictly:

static int checkline(void *ctx, const unsigned char *src, int stride, int len, int bpp)
{
    ...
    total /= div;              /* integer mean of the whole column */
    return total;
}
#define FIND(DST, FROM, NOEND, INC, STEP0, STEP1, LEN)
    for (last_y = y = FROM; NOEND; y = y INC) {
        if (checkline(ctx, ..., LEN, bpp) > limit_upscaled) {   /* strictly greater */
            if (++outliers > s->max_outliers) { DST = last_y; break; }
        } else
            last_y = y INC;
    }

With max_outliers defaulting to 0, a single column above the limit ends the search. So the operational rule is: a line is bar while its truncated mean luma is ≤ limit; one line whose mean exceeds the limit stops the crop there. The default is not "24" as a luma value either — it is 24.0/255, a fraction of full scale, which is the source of the 10-bit bug below.

BarColumn meanColumn maxDetected wResult
constant 2424.0024.01440cropped
constant 2525.0025.01920no crop
one bright row out of 10800.24255.01440cropped
one bright column each side1.06255.01920no crop
108 bright rows (10%)25.50255.01920no crop
540 bright rows (50%)127.50255.01920no crop

The first three rows kill the max hypothesis outright: a bar containing a single pixel at 255 is still cropped, because one bright pixel in 1080 moves the mean by 0.24. The last row kills the opposite reading — it is not "any bright pixel is enough" either. What matters is how much of the line is bright, because that is all a mean can see.

That makes the failure condition a fraction, not a brightness. Holding everything else fixed and lighting only n of the 1080 rows in the bar at 255:

Bright rows255·n/1080truncatedDetected w
10424.56241440
10524.79241440
10625.03251920
10725.26251920

The crossover is at exactly 105 rows: 9.7% of the frame height. Not "a bright pixel", not "the brightest pixel" — a tenth of the bar lit at full white. That is the failure that actually happens in the wild: a bar that is black on average but carries a highlight, a logo, or a burnt-in timestamp in one region pushes the mean over and the detector returns the full width with no warning.

The two settings that break it silently

1. limit is a fraction when it is below 1.0. The option default is 24.0/255, and config_input() rescales only in that branch:

if (s->limit < 1.0)
    s->limit_upscaled = s->limit * ((1 << s->bitdepth) - 1);
else
    s->limit_upscaled = s->limit;      /* raw code value, whatever the depth */

So limit=24 means code 24 in this pixel format. On 8-bit that is luma 24; on 10-bit it is 24 of 1023, which in 8-bit terms is about 6 — a far stricter threshold than intended. Measured on the same picture encoded lossless as yuv444p10le, with the limit passed as a raw code value the crop survives only while the bar is at or below luma 6 and starts failing at luma 8 — a bar that is merely "video black" at 16 is not cropped at all. Passing the correct fraction limit=0.094 (24/255 ≈ 96/1023) restores the 8-bit behaviour exactly: luma 24 is cropped at 1440 and luma 25 is not.

Bar luma (8-bit terms)limit=24 (raw code)limit=0.094 (fraction)
014401440
614401440
819201440
1619201440
2419201440
2519201920

The last row is the sanity check that the two configurations agree where they should: both stop cropping at luma 25. Everything above it is the bug — on 10-bit source the default-shaped limit=24 refuses to crop any bar a real encoder would produce.

2. round defaults to 16, and it eats picture. A 1440×1080 picture with real bars gives a clean rect at round=2, but the default round=16 returns 1440×1072 at y=4 — four rows of picture gone, which realigns every subsequent comparison by 4. At round=32 it is 1440×1056 at y=12.

SettingDetected rectPicture lost
round=21440×1080 +240+00 rows
round=16 (default)1440×1072 +240+44 rows
round=321440×1056 +240+1212 rows

Both errors are unrecoverable after the fact. Once the rectangle is wrong, every metric downstream — including the strip metric from Experiment 2 — is measuring a different rectangle than the one you think you are measuring, and it will report a perfectly healthy number while doing it.

Reproduce all of it — the frame builder, the mean/max specimens and the row crossover:

import numpy as np, subprocess, tempfile
from PIL import Image

def framed(bar=0.0, bright_rows=0, W=1920, H=1080, PW=1440):
    rng = np.random.default_rng(3)
    yy, xx = np.mgrid[0:H, 0:PW]
    v = np.clip(200 - 160*(xx/(PW-1)) + rng.normal(0, 6, (H, PW)), 0, 255)
    f = np.zeros((H, W, 3)); f[..., :] = bar
    f[:, (W-PW)//2:(W+PW)//2] = np.repeat(v[..., None], 3, 2)
    if bright_rows:                       # light the TOP rows of both bars only
        f[:bright_rows, :(W-PW)//2] = 255; f[:bright_rows, (W+PW)//2:] = 255
    return f

def detect(f, limit="24", rnd=2):
    with tempfile.TemporaryDirectory() as td:
        p = f"{td}/f.png"; Image.fromarray(f.astype(np.uint8)).save(p)
        r = subprocess.run(["ffmpeg","-hide_banner","-loop","1","-i",p,"-vf",
             f"cropdetect=limit={limit}:round={rnd},metadata=mode=print",
             "-frames:v","6","-f","null","-"], capture_output=True, text=True)
    return r.stderr

for rows in (0, 105, 106):
    out = detect(framed(bright_rows=rows))
    print(rows, [l.split("crop=")[-1] for l in out.splitlines() if "crop=" in l][-1])

Output, matching the tables above verbatim:

0 1440:1080:240:0
105 1440:1080:240:0
106 1920:1080:0:0

Note -frames:v 6 rather than 1 — cropdetect's default skip is 2, so a single frame produces no crop= line at all.

Reproducing it

Everything above runs in about a second except the 4× timings. Pillow only; no ffmpeg needed for the metrics. The scene is deterministic.

import numpy as np
from PIL import Image, ImageDraw

def scene(W, H, edge_lit):
    yy, xx = np.mgrid[0:H, 0:W]
    a = np.zeros((H, W, 3), np.float32)
    if edge_lit:                      # max contrast exactly at x=0
        v = np.linspace(200, 40, W, dtype=np.float32)
        a[..., 0] = a[..., 1] = a[..., 2] = v[None, :]
    else:
        a[..., 0] = 40 + 60 * (xx / W)
        a[..., 1] = 44 + 50 * (yy / H)
        a[..., 2] = 52 + 40 * ((xx + yy) / (W + H))
    img = Image.fromarray(a.clip(0, 255).astype(np.uint8))
    d = ImageDraw.Draw(img)
    r = int(H * 0.2778)
    d.ellipse([W//2-r, H//2-r, W//2+r, H//2+r], fill=(220,210,190),
              outline=(255,255,255), width=max(2, H//360))
    return img

def psnr(a, b, sl=None):
    if sl: a, b = a[sl], b[sl]
    m = np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2)
    return float('inf') if m == 0 else round(10 * np.log10(255*255 / m), 2)

src = scene(1440, 1080, True)
frame = Image.new('RGB', (1920, 1080), (0, 0, 0))
frame.paste(src, (240, 0))                     # 240 px bar each side

for s in (2, 3, 4, 8):
    for kn, k in (('lanczos', Image.LANCZOS), ('bicubic', Image.BICUBIC)):
        A = np.asarray(src.resize((1440*s, 1080*s), k), np.uint8)
        B = np.asarray(frame.resize((1920*s, 1080*s), k), np.uint8)
        B = B[:, 240*s:240*s + 1440*s]         # crop the bars back out
        d = np.abs(B.astype(int) - A.astype(int)).max(axis=2)
        cols = int((d > 1).sum(axis=0).astype(bool).sum())
        print(f"{kn} {s}x  cols={cols:3d}  src_px={cols/s:4.2f}  "
              f"edge2={psnr(B, A, (slice(None), slice(0, 2))):6.2f}  "
              f"edge8={psnr(B, A, (slice(None), slice(0, 8))):6.2f}  "
              f"whole={psnr(B, A):6.2f}")

Output (the table in Experiment 1, one line per row):

lanczos 2x  cols=  8  src_px=4.00  edge2= 17.79  edge8= 23.45  whole= 48.86
bicubic 2x  cols=  5  src_px=2.50  edge2= 18.41  edge8= 24.37  whole= 49.77
lanczos 3x  cols=  8  src_px=2.67  edge2= 15.58  edge8= 20.67  whole= 47.82
bicubic 3x  cols=  5  src_px=1.67  edge2= 15.72  edge8= 21.40  whole= 48.55
lanczos 4x  cols= 14  src_px=3.50  edge2= 13.96  edge8= 19.14  whole= 47.53
bicubic 4x  cols= 10  src_px=2.50  edge2= 14.11  edge8= 19.83  whole= 48.23
lanczos 8x  cols= 28  src_px=3.50  edge2= 11.03  edge8= 16.17  whole= 47.29
bicubic 8x  cols= 18  src_px=2.25  edge2= 11.15  edge8= 16.62  whole= 47.96

For the cropdetect grids in Experiments 3 and 4, the same frame is rebuilt at different bar luma, then:

for lim in 16 24 32 40 48; do
  ffmpeg -v info -loop 1 -t 0.4 -i framed.png \
    -vf "cropdetect=limit=$lim:round=2" -f null - 2>>&1 \
    | grep -o 'crop=[0-9:]*' | tail -1
done

Note the -loop 1 -t 0.4: a single still through -frames:v 1 produces no crop= line at all on ffmpeg 6.1.1, because cropdetect skips the first skip=2 frames. That is fifteen minutes of confusion the first time you script it.

What to actually do

  1. Crop before you resample, always. 25% fewer pixels, 1.37× faster, and the edge of your picture never sees a bar. There is no quality argument for upscaling black.
  2. Verify the detected rectangle instead of trusting it. Print the width cropdetect returns and compare against the width you expect from the container's display aspect ratio. If they differ, the crop is wrong and every subsequent number is measuring the wrong rectangle.
  3. Do not set limit to catch a light bar. At 40 and 48 the detector walked past the bar and ate 16 and 86 columns of picture on the same frame. Pick the limit from the source's actual bar luma, measured on one frame, not from a guess — and prefer fixing the bar (crush it to true black) over raising the threshold.
  4. Pass the limit as a fraction on >8-bit source, and set round=2. limit=24 on 10-bit means code 24 of 1023, not luma 24 of 255, and the default round=16 trims real picture rows to a multiple of 16.
  5. Gate on an edge strip, not the whole frame. 8 output pixels at the active boundary catches a 38 dB defect that a whole-frame PSNR gate reports as 0.01 dB.
  6. Check the axis twice. Pillarboxing and letterboxing have different failure modes only in which scan direction you measure; the crop rectangle is the same operation, but a source can have bars on both axes and cropdetect will happily return a rectangle that is right in one and wrong in the other.

If you are doing this as part of a pipeline rather than by hand — crop, upscale, encode — the same order applies: the upscaler never needs to know the bars existed. The upgolab.com studio takes the cropped picture and does not try to be clever about frame geometry, which is the behaviour you want here.