GIF and Palette Limits Before an Upscale: 256 Colours, Measured at 1× to 8×
You have a 256-colour image, you upscale it 4×, and the result is worse than the original looked. The instinct is that the upscaler is at fault. It is not. The upscaler did exactly what you asked and produced a correct 24-bit image — which your format then threw away, because the format still only allows 256 colours and now there are four times as many pixels competing for them.
This is the fourth post in a series about things that go wrong before the model sees the pixels. The earlier ones covered chroma subsampling, bit depth and banding and colour matrix and range. This one covers the case where the bottleneck is not a rounding error or a metadata flag but a hard ceiling of 256 entries.
1. The mechanism, stated precisely
A palette image stores two things: a table of at most 256 RGB triples, and one index per pixel. GIF allows one table per frame, HTML/CSS-rendered PNG-8 allows one table per image, and animated GIF usually allocates a fresh 256-entry table for each frame. There is no way to store 257 distinct colours. That is the whole constraint.
An upscaler, in contrast, is a continuous-domain operation. Bilinear, bicubic and Lanczos all compute weighted averages of neighbouring pixels. Averaging two palette entries that differ produces a value that is in neither of them. So the moment you resample a palette image in truecolour — which is what every tool does internally — you leave the 256-entry world and enter a 16-million-entry one. The question is only what you do on the way back out.
The asymmetry that matters. Downscaling a palette image also averages, but it averages towards values that are already present in dense regions, so the output tends to stay near the table. Upscaling does the opposite: it manufactures intermediate values in the gaps between entries, and the gaps are exactly where the eye looks, because that is where the edges are.
2. How many colours does an upscale actually invent?
Take three images, quantise each to a palette, then resample in truecolour and count distinct output colours. If the "upscaling doesn't add information" intuition were the whole story, the count should stay flat. It does not.
| Source | Base colours | Factor | nearest | bilinear | lanczos |
|---|---|---|---|---|---|
| flat graphic (5 colours, hard edges) | 5 | 2× | 5 | 27 | 199 |
| 3× | 5 | 27 | 216 | ||
| 4× | 5 | 73 | 458 | ||
| 8× | 5 | 215 | 1,035 | ||
| anime-style cell (30 colours, 1px outlines) | 30 | 2× | 30 | 210 | 1,499 |
| 3× | 30 | 231 | 1,883 | ||
| 4× | 30 | 498 | 3,570 | ||
| 8× | 30 | 1,204 | 7,673 | ||
| photo-like (256 colours, smooth gradient) | 256 | 2× | 256 | 14,285 | 25,889 |
| 4× | 256 | 26,862 | 46,456 | ||
| 8× | 256 | 40,849 | 67,330 |
Nearest-neighbour is the only resampler that returns the palette unchanged, and that is a tautology — it copies integers, so it cannot create a value. Every interpolating kernel multiplies the colour count by 40 to 250 even on flat artwork. A 30-colour cell animation upscaled 8× with Lanczos holds 7,673 distinct values. You can display 256 of them.
The flat-graphic row is the most instructive: five source colours, 458 distinct values at 4×. All 453 extra values sit on the boundaries between the five regions. There is no true detail there at all; that is pure kernel ringing on a step edge. It is information in the Shannon sense — the output is not predictable from the input without knowing the kernel — but it is not information about the scene.
3. The test that decides everything: does the palette survive the round trip?
Colour count tells you how much the kernel invented. It does not tell you whether those invented values are recoverable. Take a 6-colour sprite — the archetypal case, pixel art from a 40×40 source — upscale it, and measure what fraction of output pixels land exactly on one of the six original palette entries.
| Kernel (8×) | Colours introduced | % pixels on original 6 | PSNR vs nearest 8× |
|---|---|---|---|
| nearest | 6 | 100.00% | ∞ |
| bilinear | 382 | 81.31% | 20.85 dB |
| bicubic | 1,417 | 52.88% | 21.35 dB |
| lanczos | 3,020 | 29.04% | 20.90 dB |
Lanczos leaves 71% of the output on colours the sprite never contained. Bilinear is the mildest at 19%, which is a genuinely useful and slightly surprising result — for pixel art specifically, the "worst" kernel by reputation is the best-behaved by this metric, because it only ever averages two neighbours and so cannot overshoot into the empty space beyond a colour.
Note the PSNR column: bicubic scores higher than lanczos while keeping fewer exact palette pixels. If you are choosing a kernel for pixel art by PSNR against a nearest-neighbour reference, you will pick the one that smears most. This is the same failure mode the kernel-selection guide measured on a Siemens star: PSNR rewards smoothness, and smoothness is the opposite of a hard edge.
4. Band width is the number that actually predicts what people complain about
Nobody complains about "colour count". They complain about stepping — wide flat terraces in what should be a smooth gradient. That is measurable directly: walk the centre row of the output, count how many consecutive pixels share a value, and take the mean run length. A perfect gradient has a run length of 1. A terraced image has long runs.
| Factor | Pixel count | band_src | band, dither | band, no dither | ratio |
|---|---|---|---|---|---|
| 1× | 240×240 | 1.58 | 2.05 | 3.20 | 1.56× |
| 2× | 480×480 | 2.31 | 2.31 | 7.62 | 3.30× |
| 4× | 960×960 | 3.56 | 2.19 | 16.27 | 7.43× |
| 8× | 1920×1920 | 6.34 | 2.23 | 24.94 | 11.18× |
Without dither, band width tracks the scale factor almost exactly: 3.20 → 7.62 → 16.27 → 24.94. That is the terraces growing in proportion to the pixels you added, which is what the mechanism predicts. At 8× a single band is 25 pixels wide; at typical viewing size that is a visible stripe on a face or a sky.
With dither, band width stays flat at 2.0–2.3 regardless of scale. Dither breaks up the terraces
by alternating between adjacent palette entries on a fixed pattern, so the run length never grows.
It looks noisier but it never steps. This is the reason dither exists and the reason the ffmpeg
default is sierra2_4a rather than none.
The cost of that flatness
Dither is not free, and the cost scales worse than the benefit. Same source, same 256-colour
palette, only the dither= flag differs:
| Factor | dither bytes | no-dither bytes | dither overhead | PSNR dither | PSNR none |
|---|---|---|---|---|---|
| 1× | 29,345 | 24,088 | 1.22× | 39.44 dB | 40.71 dB |
| 2× | 85,834 | 57,413 | 1.50× | 39.21 dB | 40.83 dB |
| 4× | 273,293 | 140,789 | 1.94× | 38.87 dB | 40.81 dB |
| 8× | 976,018 | 351,625 | 2.78× | 38.68 dB | 40.84 dB |
Dither lowers PSNR at every scale — by 0.93 dB at 8× — because PSNR measures per-pixel error and dither deliberately introduces per-pixel error. If you accept the dither, you have to accept that your objective metric gets worse while the picture gets better. Every third-party comparison table you have read that ranks GIF encoders by PSNR is measuring the wrong thing for exactly this reason.
5. The counter-intuitive result: quantise first and the picture is sharper
The "obvious" pipeline is: upscale in truecolour, then compress to the delivery format. The "wrong" pipeline is: compress first, then upscale. Everyone assumes the second one loses quality because you throw away information before you need it. Measured against a truecolour 4× reference, it loses on one axis and wins on the other.
| Pipeline | PSNR vs ref | Acutance | Bytes | Distinct colours |
|---|---|---|---|---|
| reference (truecolour 4×) | — | 0.322 | 999,916 | — |
| upscale → GIF (correct order) | 38.87 dB | 1.678 | 273,293 | 255 |
| GIF → upscale (quantise first) | 40.64 dB | 0.549 | 937,227 | 30,629 |
| GIF → upscale → GIF | 38.78 dB | 0.686 | 225,222 | 254 |
Quantise-first scores +1.77 dB — it is closer to the reference — and yet its acutance is 0.549 against 1.678, roughly a third as sharp. Both numbers are correct. Quantising first destroys the fine gradient detail, and a smooth image is easy to predict, so PSNR likes it. The upscale-then-quantise path keeps dither structure that PSNR counts as error but that a human reads as texture.
Record the negative result. This is the same pattern as the temporal page: per-frame reference-comparison metrics favour the pipeline that has already destroyed the detail being measured. If you tune a GIF pipeline against PSNR you will land on quantise-first, and you will ship a soft, banded result that scores well. The metric and the goal point in opposite directions.
The third row is the important one operationally: if you requantise at the end — which any GIF output does — the higher PSNR of quantise-first is gone (38.78 vs 38.87, a 0.09 dB difference, noise) but you keep the softer edge. You paid the loss and kept none of the score.
6. Animated GIF: 256 colours per frame, and the tables drift
An animated GIF is a stack of palette images. If each frame gets its own adaptive palette — which is what both PIL and ffmpeg do by default — the palette changes as the content changes. Static regions can therefore shift colour frame to frame even though nothing in the scene moved, because the table re-centres itself on the new content. That is flicker without motion.
A 24-frame 128×128 animation (smooth sky, one moving disc) written with a per-frame palette:
| Measure | Value |
|---|---|
| frames / file size | 24 / 124,569 B |
| colours per frame (min / max) | 254 / 256 |
| distinct colours across whole animation | 1,794 |
| distinct colours in the truecolour source | 13,746 |
| mean |frame delta| truecolour source | 1.489 |
| mean |frame delta| decoded GIF | 1.988 |
| temporal noise added by quantising | +33.5% |
| colours needed by frame 0 upscaled 4× | 9,328 |
The GIF's frame-to-frame difference is a third larger than the source's own motion. Some of that is quantisation noise appearing and disappearing between frames, which the eye sees as grain crawling over still areas. The last row is the punchline: a single frame of this animation needs 9,328 colours after a 4× upscale. The format offers 256.
With ffmpeg the fix is stats_mode=diff plus diff_mode=rectangle, which
weights the palette towards the pixels that actually change and only re-encodes changed rectangles.
On a 20-frame 160×160 test at 4× that combination holds the GIF at 265,458 B with dither and 130,935 B
without — a 2.03× swing that is entirely the dither= flag, with the same 102-colour union
either way. Turning dither off is always the cheapest lever and it is always the wrong one: it trades
band width for bytes at exactly the ratio table 4 shows.
7. Where GIF is simply the wrong format
Measured on the same content, same frame count:
| Format | 1× (160×160) | 4× (640×640) | vs GIF at 4× |
|---|---|---|---|
| GIF, dither on | 26,529 B | 265,458 B | — |
| GIF, dither off | 17,677 B | 130,935 B | 0.49× |
| animated WebP q75 | 6,362 B | 19,062 B | 0.07× |
| animated WebP lossless | 20,286 B | 171,304 B | 0.65× |
Lossy animated WebP at 4× is 13.9× smaller than the dithered GIF and has no palette ceiling at all. Even lossless WebP beats dithered GIF by 1.55×. The gap widens with scale because GIF's LZW compresses runs of identical indices and dither is specifically designed to eliminate those runs — the format's compression and its only banding remedy are in direct opposition.
Static images show the same thing. A 768×768 Lanczos upscale of a photo-like source:
| Format | Bytes | bpp | Palette ceiling |
|---|---|---|---|
| PNG (truecolour) | 394,892 | 5.36 | none |
| WebP q90 | 24,764 | 0.34 | none |
| JPEG q90 | 49,099 | 0.67 | none |
| GIF | 183,443 | 2.49 | 256 |
GIF is 7.4× larger than WebP q90 for strictly less colour. On the flat-graphic test — the content GIF was designed for — it is 7.1× larger than WebP and still 1.86× larger than PNG, because the interpolated edge colours defeat LZW. The only remaining argument for GIF is support, and animated WebP has been supported in every current browser since 2020.
8. A silent no-op that will waste an afternoon
While building these tests I hit a real one, and it is worth recording because it produces no error at all.
# PIL 12.2.0 — this does nothing
im.quantize(colors=16, method=Image.Quantize.MEDIANCUT,
dither=Image.Dither.FLOYDSTEINBERG)
im.quantize(colors=16, method=Image.Quantize.MEDIANCUT,
dither=Image.Dither.NONE)
# → byte-identical images. ImageChops.difference().getbbox() is None.
# also identical through the palette path:
im.convert('P', palette=Image.Palette.ADAPTIVE, colors=16,
dither=Image.Dither.FLOYDSTEINBERG)
# 1-bit conversion DOES honour the flag:
im.convert('1', dither=Image.Dither.FLOYDSTEINBERG) # != dither=NONE
The dither keyword is accepted, silently ignored, and the call returns P
mode with no warning on this path. It is honoured for 1-bit conversion, and — measured in
section 10 — it is also honoured for palette quantisation when you pass a pre-built palette image
(im.quantize(palette=pal, dither=...) changes 3,204 of 16,384 pixels). The flag is
dropped only when quantize() is also the call that builds the palette. Two
of my first-pass experiments reported identical dither and no-dither figures across three content
types and four scale factors before the duplication made me check — which is the general lesson: if
two settings produce identical numbers everywhere, suspect the setting, not the content.
The reliable path is ffmpeg's palettegen/paletteuse pair, which does
honour dither= and lets you choose the algorithm:
# upscale in truecolour, then quantise once, at the delivery size
ffmpeg -i in.png -vf "scale=iw*4:ih*4:flags=lanczos,\
split[a][b];\
[a]palettegen=max_colors=256:stats_mode=full[p];\
[b][p]paletteuse=dither=sierra2_4a" out.gif
# animation: weight the palette towards changing pixels, re-encode rectangles
ffmpeg -framerate 25 -i up_%03d.png -vf "\
split[a][b];\
[a]palettegen=max_colors=256:stats_mode=diff[p];\
[b][p]paletteuse=dither=sierra2_4a:diff_mode=rectangle" -loop 0 out.gif
# comparison set
ffmpeg -i up.png -vf "\
split[a][b];\
[a]palettegen=max_colors=256[p];\
[b][p]paletteuse=dither=none" out_nodither.gif
# dither algorithm costs on this content (4x, 256 colours, 960x960):
# none 140,789 B band 16.27 px PSNR 40.81
# sierra2_4a 273,293 B band 2.19 px PSNR 38.87
# bayer scale=3 253,367 B band 1.03 px PSNR 37.90
# bayer scale=5 178,807 B band 5.55 px PSNR 40.45
The Bayer results are worth one line of interpretation: bayer:bayer_scale=3 produced
the narrowest bands of any setting I tried (1.03 px mean run, against 2.19 for sierra2_4a) at a PSNR
cost of 0.97 dB, and it was cheaper in bytes (253,367 against 273,293) — the opposite of what
you would guess. Ordered dither is a regular grid, so it never accumulates into runs, but the grid is
visible as a crosshatch on flat areas. bayer:bayer_scale=5 is a useful middle point:
178,807 B, band 5.55 px, PSNR 40.45 — 35% of sierra2_4a's size for a band 2.5× wider. Error-diffusion
hides better on photographs; ordered dither hides better on graphics. Neither is universally right,
and scale 5 is a cheaper compromise than scale 3 on photo content.
9. What this means for an upscale specifically
Four rules, each one directly implied by a table above.
Decide the delivery format before you upscale, not after. If the answer is GIF, the upscale is buying you pixels the format cannot describe. The 8× column in table 4 shows what you get: 25-pixel-wide bands and a 2.78× dither surcharge. Upscaling 8× into a GIF is a net loss against upscaling 4× and letting the viewer's GPU do the rest.
Never quantise before the upscale unless you also stop caring about edges. The +1.77 dB that quantise-first scores is a metric artefact of destroyed detail; acutance drops to a third. It also costs 3.43× the bytes of the correct order because you end up storing 30,629 colours instead of 255.
For pixel art, nearest is not a compromise — it is the only correct kernel. 100% palette retention against 29% for Lanczos, with no invented values to compress. The line-art guide found the same thing from the other direction: hard-edged content fails when you average it.
If any interpolating kernel is involved, dither is mandatory and you should stop reading PSNR. Turn it on, accept the 0.9 dB objective loss, and judge the result by band runs or by eye. An undithered 8× GIF is 11× worse on band width than its own truecolour source, and no subsequent step repairs that.
The broader pattern across all four posts in this series is the same. The expensive part of an upscale is not the model — it is the format contract you signed before the model ran. A 256-colour ceiling, a 4:2:0 subsample, an 8-bit container, a mislabelled matrix: each one caps what the upscaler can ever deliver, and each one is invisible in the output until you measure. Getting the delivery format and colour pipeline right before you upscale costs one flag and saves the entire run.
10. Generation loss: what eight successive upscale/quantise cycles actually do
Everything above measures one round trip. Real work is not one round trip. A designer upscales, saves a GIF, re-opens it next week because the client wants it larger, upscales again, saves again. Each of those is a full upscale → quantise → downscale cycle against a 256-colour ceiling, and the question worth answering is not "does it lose detail" (obviously it does) but which generation loses it, and does the loss compound or saturate?
The answer decides something practical. If loss compounds linearly, the number of times a file has passed through a GIF is the thing to audit. If it front-loads, then one round trip is the only one that matters and re-saving is close to free.
Method
64×64 sprite, 5 actual colours, hard quadrant edges and two diagonals. One generation is: lanczos 2× up, quantise against a palette built once from the first upscale and then held fixed, lanczos back to 64×64. Holding the palette fixed matters — rebuild it each generation and you measure palette adaptation instead of loss. Eight generations, measured against generation 0.
palette = SRC.resize((128,128), Image.LANCZOS).quantize(colors=256, method=Image.MEDIANCUT)
def cycle(im, palette, dither):
big = im.resize((im.width*2, im.height*2), Image.LANCZOS)
q = big.quantize(palette=palette, dither=dither).convert("RGB")
return q.resize((64,64), Image.LANCZOS) # one generation
Note the palette= argument. That detail is the whole difference between this
experiment working and producing garbage — see the trap at the end.
Result: the loss front-loads, hard
| Gen | PSNR (dB) | Δ vs prev | Colours | On palette | Band (src px) |
|---|---|---|---|---|---|
| 1 | 29.548 | — | 256 | 38.26% | 3.839 |
| 2 | 27.579 | −1.969 | 474 | 34.81% | 3.032 |
| 3 | 26.685 | −0.894 | 762 | 30.27% | 2.957 |
| 4 | 26.065 | −0.620 | 872 | 27.83% | 2.872 |
| 5 | 25.651 | −0.414 | 968 | 26.64% | 2.709 |
| 6 | 25.363 | −0.288 | 1.041 | 25.63% | 2.592 |
| 7 | 25.141 | −0.222 | 1.114 | 24.90% | 2.484 |
| 8 | 24.930 | −0.211 | 1.168 | 23.75% | 2.444 |
From generation 1 to generation 8 the image loses 4.618 dB, and that total decomposes unevenly. The single step gen 1 → gen 2 costs 1.969 dB, or 42.6% of the whole loss. The first two steps together cost 2.863 dB — 62%. The final step costs 0.211 dB, about 4.6%.
That is a usable rule rather than a curiosity: the first pass through the palette does the damage and every later pass converges toward a slow floor of roughly 0.2 dB. If you must re-export an upgraded GIF a third or fourth time, the fact that it has been through the format before is not a reason to stop — the expensive trip already happened. The corollary matters more: a file that has been through one quantisation is not "slightly damaged", it is most of the way to the asymptote, and re-doing the upscale from that file will not recover the difference.
The counter-intuitive part: dither makes PSNR worse and bands better
At generation 1 the two paths are level (+0.046 dB, inside measurement noise). By generation 8 dithered output is 0.604 dB worse on PSNR — and simultaneously has narrower bands (2.444 vs 2.543 source pixels) and 1.79× more distinct colours (1,168 vs 654).
| Metric | Floyd–Steinberg | None | What it means |
|---|---|---|---|
| PSNR | 24.930 dB | 25.534 dB | undithered "wins" by 0.604 dB |
| Distinct colours | 1,168 | 654 | dither keeps inventing intermediates |
| Band width (src px) | 2.444 | 2.543 | dither's bands are narrower |
| Total PSNR loss, gen 1→8 | −4.618 dB | −3.968 dB | undithered degrades slower |
This is the same trap as section 3, one layer down. PSNR rewards output that stays close to the average, and dithering deliberately moves pixels away from the local average to break up contour bands. So a PSNR-driven choice picks the undithered path and gets wider bands — the exact artefact people complain about. The metric and the defect point in opposite directions, consistently, and the gap grows with generation count: +0.046 dB at gen 1, −0.279 dB at gen 4, −0.604 dB at gen 8. The longer your pipeline, the more wrong PSNR-guided format choices become.
The growing workflow: detail per source pixel collapses 81%
The other real pattern is not downscale-then-upscale, it is upscale-again-and-again without ever coming back down. Same palette, five generations, 64 → 2048 px. Band width here is normalised to source pixels, so it does not simply track the rising resolution:
| Gen | Scale | Pixels | Colours | Band (src px) |
|---|---|---|---|---|
| 1 | 2× | 16,384 | 184 | 1.605 |
| 2 | 4× | 65,536 | 207 | 1.006 |
| 3 | 8× | 262,144 | 214 | 0.636 |
| 4 | 16× | 1,048,576 | 214 | 0.434 |
| 5 | 32× | 4,194,304 | 215 | 0.307 |
Between generation 1 and generation 5 the flat-run length per source pixel falls from 1.605 to 0.307 — 80.9% of the measurable band structure is gone — while the file has grown 256× in pixel count. The picture gets bigger and flatter at the same time. Colour count also pins at 215 of the 256 available and stops moving: the format ceiling binds long before the resolution does, so every extra generation after the second is pure file size.
The trap that produced two identical columns of "results"
The first version of this experiment reported dither and no-dither as byte-identical across eight generations. That is not a finding, it is a broken flag — and it is worth writing down because the mistake is invisible without an assertion.
# SILENTLY IGNORED: quantize() builds the palette AND drops the dither argument
im.quantize(colors=256, dither=Image.Dither.NONE)
im.quantize(colors=256, dither=Image.Dither.FLOYDSTEINBERG)
# -> np.array_equal(...) is True. 0 differing pixels of 65,536.
# HONOURED: pass a pre-built palette image
pal = im.resize((w*2,h*2), Image.LANCZOS).quantize(colors=256, method=Image.MEDIANCUT)
big.quantize(palette=pal, dither=Image.Dither.FLOYDSTEINBERG) # differs in 3,204 px
Measured on Pillow 12.2.0: with colors=, the dither flag has zero
effect and the call returns P mode with no warning. With palette=, it changes
3,204 of 16,384 pixels on a 16-colour gradient. The same silent drop applies to
convert('P', palette=Image.ADAPTIVE). Only 1-bit conversion honours the flag on the
convenience path.
The assertion that catches it costs one line and should be in every palette experiment:
assert not np.array_equal(dithered, undithered), \
"dither flag did not reach the quantiser - suspect the call signature"
Which generalises past this page: when two settings produce identical numbers everywhere, suspect the setting before you believe the content. Section 8 records the same failure one level up, at the format level. The useful habit is not remembering this particular Pillow quirk — it is refusing to publish a comparison whose two columns you have not proved can differ.
Reproducing this
Every number above comes from generated images, no external assets. The whole set runs in under two minutes.
import numpy as np
from PIL import Image
# colour-count growth (table 2)
n = 192
sp = np.zeros((40,40,3), np.uint8) # 6-colour sprite
PAL = [(20,20,24),(240,240,235),(200,55,50),(45,110,180),(240,200,60),(90,170,90)]
sp[:,:] = PAL[1]
sp[12:20,10:30] = PAL[2]; sp[8:12,18:22] = PAL[3]; sp[28:34,12:28] = PAL[4]
sprite = Image.fromarray(sp)
for f in (2,3,4,8):
for meth in (Image.NEAREST, Image.BILINEAR, Image.LANCZOS):
u = sprite.resize((40*f, 40*f), meth)
print(f, meth, len(np.unique(np.asarray(u).reshape(-1,3), axis=0)))
# band width (table 4)
def band(path):
L = np.asarray(Image.open(path).convert('L')); r = L[L.shape[0]//2]
runs, c = [], 1
for i in range(1, len(r)):
if r[i] == r[i-1]: c += 1
else: runs.append(c); c = 1
runs.append(c); return round(float(np.mean(runs)), 2)
Band width is the metric I would keep from this whole exercise. It is one function, it needs no reference image, and it moves in the direction a human viewer moves — 3.20 at 1× to 24.94 at 8× is the terraces appearing, and 2.05 flat across all four scales is dither doing its job.