Cutout and Upscale in One Workflow: Where the Edges Fail

Unlike denoise order, the order of cutout and upscale genuinely matters — and one direction is unrecoverable. A matte computed at low resolution produces edges that upscaling widens into a visible halo, and no later step can narrow them again.

14 November 2026 · edge-transition width measured across the two orders

Background removal and upscaling look like two independent operations on the same file, so most people run them in whatever order the tool makes convenient. They are not independent. The matte is a per-pixel alpha decision, and upscaling is a per-pixel interpolation — and interpolation applied to an alpha channel produces exactly the artifact that compositing is most sensitive to.

The asymmetry that decides the order

Consider the two orders on a 500×500 product photo with a clean subject boundary, targeting 1000×1000:

• Cut out, then upscale. The matte is computed at 500×500, so its edge is one source pixel wide — then the upscale widens it while softening it. Measured below: a 1px source ramp becomes a 4px transition at 2× and 8px at 4×.
• Upscale, then cut out. The upscale softens the subject boundary itself (same interpolation, same effect on the RGB), and then the matte is computed at 1000×1000 — so its edge is one target pixel wide, sharper in relative terms.

In both cases something is soft — measured, 4px versus 2px at 2×, and 8px versus 2px at 4× (see the tables below). The difference is which thing. Upscaling second means the softness lands on the alpha channel, where it composites as a fringe. Upscaling first means the softness lands on the RGB edge, where it is partially masked by the subject's own colour.

When the reverse order wins

If the matte is already perfect and you want to preserve it exactly — a hand-drawn mask, a rotoscoped sequence, a mask you spent an hour on — do not recompute it after upscaling. Run the matte, then upscale the composite, and accept the 2px soft edge. Recomputing risks a different result on the enlarged version, and consistency matters more than 1px of edge width.

So the decision is: is the matte cheap or expensive? Cheap (auto-detected) → upscale first, recompute at high resolution. Expensive (manual) → matte first, upscale the composite, keep the mask.

Why the halo appears at all

A matte edge is a transition from alpha 1.0 to alpha 0.0. If it transitions over 2px instead of 1px, the composite blends the subject with the new background over 2px. On a subject against a colour that differs strongly from the subject's edge colour, that transition band reads as a glow — the classic "cut out badly" look.

There is a second, subtler cause that upscaling makes worse: colour bleed. If the original background was green and the subject is a person, the pixels right at the subject boundary contain a mix of green and skin. A matte will correctly call those pixels partially transparent, but the RGB values still carry green. Upscaling spreads that green further into the subject. The result is a faint green rim that survives the cutout, because the cutout only decides transparency, not colour.

The fix for bleed is a separate step — spill suppression — and it has to happen at the resolution where the matte is computed, not earlier. Suppressing on the small source and then scaling the alpha up measurably underperforms: 17.8 versus 16.5 at 2×, and 31.6 versus 27.5 at 4×. The clamp pulls green down hard enough that the key's green-excess term collapses at the boundary and the matte tightens into the subject. Measured, not assumed — the numbers are in the tables below.

The order that respects all of it

1. Upscale. At 2× if possible; the detail measurements say 2× retains ~21% of high-frequency energy versus ~1.3% at 4×.
2. Spill suppression at final resolution. Clamp the screen channel to the other two. It belongs here, not on the source — doing it on the small image collapses the matte by a pixel.
3. Matte, right after spill, at the same resolution. More pixels to decide with, narrower relative edge, and the key still has enough green excess left to work with.
4. Composite. Nothing after this should touch the alpha channel.

That reordering came out of the measurement, not out of theory: the version of this page that went up first said "suppress spill on the source, it's cheap and it must be early" and the numbers said that costs 1.3 points of edge error at 2× and 4.1 at 4×. Cheap and early turned out to mean doing it where the matte has the least information to work with.

Note that this is a different order than the denoise case, where the sequence barely mattered (0.8% difference in comb residual). The difference is that denoise and deinterlace both operate on the same domain — the RGB pixels — while cutout creates a second channel whose behaviour under interpolation is different. Cross-domain operations are where order becomes non-negotiable.

The cheap check

Composite the result on a saturated background (magenta is the classic choice) and look at the boundary at 100%. Any band of background colour between subject and new background is your edge width. At 2× the correct order measures 2px against the wrong order's 4px; at 4× it is 2px against 8px. If your result is worse than 2px at the final resolution, something in the chain softened it and you can find which step by comparing before/after each one.

Do not optimise for a zero-width band, though. Nearest-upscaling the alpha gets the band to 0px and scores 16.3 — better than the "correct" 16.5 — but a zero-width transition at 1000px is an aliased staircase, and the metric cannot see the difference between a crisp edge and a jagged one. Edge width is a proxy for quality in exactly one direction: too wide is wrong, too narrow is also wrong, and only looking tells you which side you are on.

Magenta is used because it appears nowhere in natural images — any magenta in the result is unambiguously a matte failure, not image content. That is why the old compositing tutorials insist on it.

The short version

# order A: matte at source resolution, then upscale the RGBA
rgba_small = cutout(src)                  # 500x500
out_a = rgba_small.resize((1000,1000), Image.LANCZOS)

# order B: upscale RGB, then matte at target resolution
rgb_big = src.resize((1000,1000), Image.LANCZOS)
out_b = cutout(rgb_big)                   # 1000x1000

# measure: width of the alpha transition band in each
alpha = np.asarray(out.convert("A"), float) / 255
band = ((alpha > 0.05) & (alpha < 0.95)).sum(axis=0).max()

Measuring it instead of arguing about it

The order rule above is easy to state and hard to trust, because "the edge looks softer" is a judgement call. So I built a scene where the answer is known exactly: a textured subject on a chroma-green screen, rendered at 1000×1000 with a one-pixel alpha ramp, then downsampled to 500×500 and to 250×250 to act as the "source" we have to work from. That gives one correct answer (the 1000px render) and four ways of getting there. Every number below is a distance from that 1000px reference, not a comparison between two of the approximations.

The matte is a plain green-excess key — alpha = 1 − (G − max(R,B) − 20)/60 clamped to [0,1] — applied either before or after a Lanczos resize. Composite everything on magenta so any contamination is unambiguous, then measure two things in the 6px ring just inside the true boundary, where the ground truth alpha is exactly 1.0 and therefore any difference from the true subject colour is error.

Two results here contradict the tidy story in the section above, and they are worth keeping rather than rounding off.

First: nearest-upscaling the alpha matches the "correct" order. If the matte is the problem, the obvious fix is to not let interpolation touch it — resize the alpha with nearest-neighbour so it stays a hard 1px stair-step instead of a 4px gradient. It works on the metric: 16.3 versus 24.6, essentially the same as recomputing the matte at full resolution, and the band width drops to zero. But the visual result is a jagged edge, because the band width going to zero means the transition is now too sharp for the target resolution. The metric rewards it; nobody would ship it. This is the recurring trap with edge measurements — a zero-width transition scores perfectly and looks like a cut-out from 1998.

Second: suppressing spill before the matte makes it measurably worse. Clamping green to max(R,B) before the matte, then upscaling, lands at 17.8 — slightly worse than doing nothing (16.5), and the 0px band shows why: the clamp pulls the green down so far that the key's green-excess term collapses at the boundary and the matte tightens by a pixel, cutting into the subject. Doing the same clamp after the upscale, where there are 1000 pixels of boundary to work with instead of 500, gives the same 16.5 with a 0px band and no eaten edge. The contamination is not removed by suppressing earlier; it is removed by suppressing where the matte is more precise.

What 4× does to the same comparison

The 2× numbers are close enough that someone could argue the order barely matters. At 4× (250px source) it stops being arguable:

The band grows to 8px and the total error to 44.6, a 1.62× penalty against the correct order. The excess is 17.1 points, and where it lands is the interesting part: 11.1 of those points are ΔG alone (65% of the penalty), 4.9 are ΔB, and ΔR contributes only 1.1. Green is the screen colour, so most of this is spill being smeared across the widened band rather than a general loss of accuracy. On a blue screen the channel that would blow up is B instead. The failure mode is tied to the background colour, which is also why it stays invisible on a test image that happens to have a neutral background.

Reproducing it

import numpy as np
from PIL import Image

def key(rgb):                      # 0 on green screen, 1 on subject
    return 1.0 - np.clip((rgb[...,1] - np.maximum(rgb[...,0], rgb[...,2]) - 20) / 60.0, 0, 1)

def desuppress(rgb):               # clamp green to max(R,B) — belongs with the matte
    x = rgb.copy()
    x[...,1] = np.minimum(x[...,1], np.maximum(x[...,0], x[...,2]))
    return x

def compose(rgb, a, bg=np.array([255.,0,255])):
    return rgb * a[...,None] + bg * (1 - a[...,None])

src   = np.asarray(Image.fromarray(scene_uint8).resize((500,500), Image.LANCZOS), float)
rgba  = np.concatenate([src, key(src)[...,None]*255], -1)
upA   = np.asarray(Image.fromarray(rgba.astype(np.uint8), 'RGBA')
                     .resize((1000,1000), Image.LANCZOS), float)
out_A = compose(upA[...,:3], upA[...,3]/255)

big   = np.asarray(Image.fromarray(src.astype(np.uint8)).resize((1000,1000), Image.LANCZOS), float)
clean = desuppress(big)
out_B = compose(clean, key(clean))

Then compare both against the 1000px render you started from, inside a mask of |distance_to_boundary| < 6, and report per-channel means rather than a single averaged number — the per-channel split is what tells you whether you are looking at spill or at a genuine resolution loss. The full script that produced the tables above is about forty lines and runs in under two seconds.

Try both orders on one file

The comparison takes one image and two passes, which is fast enough to actually do. A browser-based tool handles both operations without installing anything: UpgoLAB upscales photos and video and removes backgrounds in the browser. Run the two orders, composite both on magenta, and compare the boundary — the difference is visible without measurement, and the measurement confirms what you see.

Upscale first, matte second, suppress spill before either. Three steps, and the order is the whole content of the advice.