Tiled Upscaling Seams: How Much Context a Tile Actually Needs

Every upscaler that cannot fit a frame in memory cuts it into tiles. Every seam artifact blamed on tiling is then described as a memory limitation, or as the price of a big inference batch. I tiled a real photograph deliberately, at six tile sizes, and measured the difference against a single-shot upscale of the whole frame. The answer is smaller than the folklore: a tile needs as much context as the resampler has support, and three source pixels was enough to make 54, 64, 108, 180, 256 and 270-pixel tiles all bit-identical to the full-frame result. What does produce visible seams is not tiling at all — it is any per-tile adaptive step, and I have the numbers for how badly.

Measured with numpy 1.26.4 and Pillow 12.2.0 · source is one real photograph downscaled to 540×810 by area-average (BOX), then upscaled 2× to 1080×1620 · reference is the same resampler applied to the whole frame in one call · all statistics computed on raw uint8-decoded arrays, never from a screenshot · every number reproducible from the snippet at the end

The experiment that decides everything: context, not size

A separable Lanczos-3 resampler reaches three pixels in each direction. If a tile is cut exactly at the boundary it will be asked to produce output pixels whose interpolation window extends past its own edge, and the implementation has to invent something — edge replication, reflection, or zero padding. Whichever it picks, it is not the value the full-frame resampler would have used, because that value lived in the neighbouring tile.

So the question "how much overlap do I need?" has a sharp answer rather than a heuristic answer. I gave each tile ov extra source pixels of context on every side, resampled the padded tile, then cropped the context back off before writing — no blending, no feathering, exact coverage. Increasing ov from zero upward:

Three source pixels — six output pixels at 2× — is the entire requirement. Everything past that is wasted arithmetic. This is not an approximation or a tolerance: np.array_equal returns true, and the maximum absolute difference is exactly 0.0, not 0.004 that rounds to zero. The transition is also not gradual in a way that suggests a tuning knob. Two pixels of context still leaves a 3-level error on 124 pixels; three leaves nothing.

Why one pixel is not almost enough

The first pixel of context recovers the bulk of it — max error drops from 51 to 14 levels and PSNR jumps 7.2 dB — and then the last two pixels recover the tail. That shape is the signature of a filter weight table, not of a statistical effect: each extra pixel of context brings in one more non-zero tap. Lanczos-3 has three taps per side, so the error goes to zero exactly when the tap count is satisfied and stays there. A bicubic kernel with two taps per side needs two; a box filter needs none.

Tile size alone is not monotonic, which is the trap

Before adding any context, the naive tiler — cut clean, no padding — gives a picture that is tempting to summarise as "bigger tiles are better." It is not that clean:

270-pixel tiles have half the seam area of 180-pixel tiles, a lower maximum error, and a worse seam PSNR by 8.65 dB. The reason is that the error is not a property of the grid, it is a property of what happened to be under the cut. A boundary through a flat gradient costs almost nothing; a boundary through the high-contrast edge in this frame costs 20 to 50 levels on the two output columns that straddle it. Tile size changes how many boundaries you get, and only incidentally where they fall.

Blending with overlap is worse than cropping — measured

The common recommendation is to overlap tiles and feather across the seam, on the theory that a gradual transition is less visible than a hard one. When the samples on both sides are correct, feathering is not merely unnecessary, it is detectably destructive, because averaging two identical fields through floating-point cosine weights does not return the original:

Sub-level, invisible, and the wrong direction. If you are going to blend you must be blending because the tiles disagree on purpose — never because you think the seam needs smoothing. On correctly-contexted tiles the blend is buying you rounding noise.

Where blending does help, and how much it does not

Add one per-tile adaptive step — auto-levels, the mildest thing a pipeline does, one percentile stretch per tile — and the picture changes. Now adjacent tiles really do disagree about the mapping from input to output, and a hard cut turns that disagreement into a line:

Max error in these tables is the largest single-channel difference; the largest mean-of-three-channels difference is 137.95, 105.91, 72.84 and 50.71 for the four rows in order.

Blending cuts the visible seam step by a factor of 27 and buys 4.2 dB. It also leaves a 52-level maximum error and a picture 34 dB worse than the un-tiled result. Feathering hides the discontinuity perfectly and cannot touch the gain error, because a blend of a correctly-exposed region and an over-stretched region is simply a wide, soft, wrong region instead of a thin, hard, wrong region.

The seam generator is per-tile state, not tiles

Per-tile auto-levels on a clean cut cost 34 dB — 21.47 against the 55.49 that the same tiling scores with no per-tile step. Each boundary gets its own error step, and the step's visibility depends on the local image gradient, which is exactly the ratio the eye responds to:

Same operator, same grid, and the ratio swings 7× depending on what the cut lands on. Some seams would be invisible even with a 12-level step because the neighbourhood already has 6 levels of gradient there. This is the general rule for every artifact I have measured in this series: absolute error predicts almost nothing, and error relative to local content predicts most of it.

A per-tile sharpen policy: better PSNR, worse seam

I tried the adaptive version deliberately — one threshold on per-tile gradient energy deciding between 0.35 and 1.1 of unsharp amount. Against a single global sharpen at 0.7:

The adaptive policy wins on PSNR by 3.15 dB and loses on seam step by 46%. Both of those are true at the same time, and the reading is obvious once stated: an adaptive policy is optimising the average while creating local disagreements at every change of decision. If the deliverable is a score, adapt per tile. If the deliverable is an image, do not.

Clipping is the operator's fault, not the tiling's

Strong sharpening at 1.4 amount, applied per tile, drives 0.554% of output pixels to 255. The same 1.4 applied once to the whole frame drives 0.555%. The reference has 0.168%. So tiling changed the clipped fraction by 0.001 percentage points while the operator changed it by 3.3×. Pixels at zero: 3.011% tiled against 3.013% global. If a tiled pipeline appears to clip more than a whole-frame one, the cause is upstream in the operator or its state, not in the seams.

What I got wrong first, and how it was caught

My first grid-phase sweep showed PSNR falling from 41.4 to 25.5 dB as the tile grid shifted by one pixel, with maximum errors of 252 levels. That looked like a real and useful finding about lattice alignment — and it was a bug. Shifting the grid origin left an uncovered strip on the left and top edges, which stayed at zero, and a zero-filled strip against real content is a 252-level error. Fixed by always inserting a leading tile at coordinate 0:

1.9 dB of total spread across six phases, no trend, seam step flat at 0.365–0.418. Grid phase does not matter. The hypothesis is dead and I am reporting it dead. The tell was in the diagnostic, not in the metric: a 252-level error is impossible from a resampler disagreement, so the number was describing my compositor, not the algorithm.

Where the error actually is

On a clean 108-pixel cut with no adaptive step, 0.454% of pixels are more than 2 levels off — 7,947 of 1,749,600. Of those, 67.6% sit within two output pixels of a tile boundary, while the seam band is only 3.18% of the frame. That is a 21× concentration of error into a thin line, which is precisely why it reads as a line to a viewer even at a PSNR of 55 dB:

Two pixels of context leaves four pixels in the entire frame above 2 levels. Three leaves none. The distribution goes from a long tail to a point mass at zero in one step, which is the same signature as the PSNR table and the same explanation: filter support.

Cost, measured rather than assumed

Context is not free, but it is cheaper than the seam it removes. Work ratio is the total source area resampled, including padding, divided by the frame area:

Six pixels of context on 108-pixel tiles costs 20.2% more resampling work and produces an exact result. Thirty-two pixels of context, the sort of number that gets recommended as "safe," costs 128.9% more and produces exactly the same exact result. The recommended value is the wrong one by a factor of six in cost terms, and it is the wrong one in the same direction every time: people overlap by feel, and feel is calibrated to hiding a seam rather than to satisfying a kernel.

If your pipeline tiles because the model or the browser tab cannot hold the frame, the tiling itself is not what you are seeing in the output. Feed a clean master, give each tile its kernel radius of context, keep adaptive steps global, and the last browser upscaler at upgolab.com in the chain will see the same image whether it processed it in one call or forty — which is the only condition under which a tiling parameter can be called correct.

Reproducing it

import numpy as np
from PIL import Image

GT  = np.asarray(Image.open('photo.jpg').convert('RGB')).astype(np.float32)
SRC = np.asarray(Image.fromarray(GT.astype(np.uint8))
        .resize((GT.shape[1]//2, GT.shape[0]//2), Image.BOX)).astype(np.float32)
K = 2; H, W = SRC.shape[:2]; OH, OW = H*K, W*K

def up(a, size):
    return np.asarray(Image.fromarray(np.clip(a,0,255).astype(np.uint8))
                      .resize(size, Image.LANCZOS)).astype(np.float32)

REF = up(SRC, (OW, OH))                       # single-shot reference

def tiled(T, ov):
    """ov = extra CONTEXT source px per side, cropped back before writing."""
    out = np.zeros((OH, OW, 3), np.float32)
    xs = list(range(0, W, T)); ys = list(range(0, H, T))
    for y0 in ys:
        for x0 in xs:
            x1, y1 = min(x0+T, W), min(y0+T, H)
            cx0, cy0 = max(0, x0-ov), max(0, y0-ov)
            cx1, cy1 = min(W, x1+ov), min(H, y1+ov)
            t = up(SRC[cy0:cy1, cx0:cx1], ((cx1-cx0)*K, (cy1-cy0)*K))
            oy, ox = (y0-cy0)*K, (x0-cx0)*K
            out[y0*K:y1*K, x0*K:x1*K] = t[oy:oy+(y1-y0)*K, ox:ox+(x1-x0)*K]
    return out

# ov=0,1,2 differ; ov=3 is bit-identical for a 3-tap kernel
for ov in (0, 1, 2, 3):
    o = tiled(108, ov)
    d = np.abs(o-REF).max(2)
    print(ov, np.array_equal(o, REF), round(float(d.max()), 1), int((d > 2).sum()))
# -> 0 False 51.0 7947 / 1 False 14.0 2586 / 2 False 3.0 4 / 3 True 0.0 0

Output on this machine, in that order: 0 False 51.0 7947 · 1 False 14.0 2586 · 2 False 3.0 4 · 3 True 0.0 0. The Image.BOX downscale matters — using NEAREST to build the source would hand the upscaler a pre-decimated image with no intermediate values, and the seam errors would be smaller for a reason that has nothing to do with the algorithm. If your numbers disagree at ov=3, check that the resize target is ((cx1-cx0)*K, (cy1-cy0)*K) and not the nominal tile size; getting that wrong is the most common way to report a seam that is really a half-pixel crop.