DEV Community: hao jia

Counting LZW codes to see why dithered GIFs come out bigger

hao jia — Mon, 02 Nov 2026 08:04:02 +0000

When someone says "turn dithering off and the GIF gets smaller", I want a number I can reproduce before I repeat it in a code review. File size alone does not say why, so I wrote a script that counts how many codes a GIF-style LZW encoder emits for the same frames with dithering off and on. It needs Pillow and numpy and runs locally, so nothing leaves the machine, which matters when the animation is a pre-release product screen.

The sample is MORPH.gif from Wikimedia Commons, by Kowloonese, CC BY-SA 3.0: 675×599, 10 frames, 100 ms each, 3,003,893 bytes. It is a photo-like fade with text, and nearly every pixel changes between frames. I blurred each frame slightly (Gaussian radius 0.8) to flatten noise already baked into the original. That blur is my own processing, so every number below belongs to the blurred copy. One sample, one laptop, one run each, sizes in plain bytes.

The script

It builds one median-cut palette for all frames, maps each frame onto it with dithering off and then Floyd–Steinberg on, and runs a plain LZW over each frame's index stream. The coder follows the GIF idea: 8-bit minimum code size, table reset at 4096 entries. It counts codes instead of writing bits, so it models the encoder and is not a GIF writer.

import numpy as np
from PIL import Image, ImageFilter, ImageSequence

def codes(data):
    d, nxt, cur, n = {}, 258, b"", 0
    for ch in data:
        t = cur + bytes([ch])
        if len(t) == 1 or t in d:
            cur = t
            continue
        n, d[t], nxt, cur = n + 1, nxt, nxt + 1, bytes([ch])
        if nxt >= 4096:
            n, d, nxt = n + 1, {}, 258
    return n + 1

N = 32
fr = [f.convert("RGB").filter(ImageFilter.GaussianBlur(0.8)) for f in ImageSequence.Iterator(Image.open('MORPH.gif'))]
pal = Image.fromarray(np.vstack(fr)).quantize(N, Image.MEDIANCUT, dither=Image.NONE)
for name, mode in (("off", Image.NONE), ("on", Image.FLOYDSTEINBERG)):
    print(name, sum(codes(np.array(f.quantize(palette=pal, dither=mode)).tobytes()) for f in fr))

With N = 32 it printed:

off 418303
on 628637

What the counts say

Dithering took the code count from 418,303 to 628,637 at 32 colours, which is +50.3%. Setting N = 256 gave 964,706 and 1,273,047, which is +32.0%. The sample has 4,043,250 pixels, so each code covers 9.67 pixels without dithering and 6.43 with it at 32 colours, and 4.19 against 3.18 at 256. A side check on the share of pixels equal to their left neighbour (not in the listing, to keep it short) fell from 85.8% to 55.8% at 32 colours.

Real files tell the same story. I saved the same indexed frames as GIFs and ran gifsicle -O3, which barely helped because every frame changes. At 32 colours they came to 572,768 bytes without dithering and 861,321 with it, +50.4%. At 256 colours: 1,364,178 and 1,800,055, +32.0%. Code count and byte count grow almost in step. Bytes × 8 ÷ codes works out near 10.95 bits per code at 32 colours and 11.31 at 256, inside the 9 to 12 bit range a GIF code can use.

Why not just compare file sizes? Because they can differ for reasons unrelated to dithering, such as a local palette or a transparency trick. The code count depends only on the index stream, so it isolates what the dithering switch touches. When the two numbers move together, as they do here, I trust the explanation a bit more. If they had diverged, I would have gone looking for a second effect before writing anything down.

Why I think it happens

This part is an inference the script does not prove. LZW builds its dictionary from strings that repeat. Without dithering, a flat region is a long run of one index, so the dictionary learns long strings and each code swallows many pixels. Error diffusion scatters neighbouring indices to fake in-between colours, which breaks those runs. The left-neighbour numbers fit that story, but I did not test it by dithering only part of a frame.

Two limits on the counter: the files use gifsicle's optimized frames while my counter reads full-frame streams, so the two are close but not the same quantity. And "dithering" here means Floyd–Steinberg only, not ordered patterns.

Trying the same switch in a tool

If you want to see the effect without a script, ImgIng ( https://imging.ai/ ) has an animation workshop in the desktop browser where colour count and dithering can be set by hand for GIF output. On the same MORPH.gif, loaded as the original and not my blurred copy, its export was 811,310 bytes at 32 colours with dithering off and 1,110,431 with it on, +36.9%. The gap differs from the script's +50.3% because the source, palette and encoder differ. At 256 colours the gap shrank to +0.7%, 1,525,126 against 1,535,115. My guess is that the source already behaves like a 256-colour image, so dithering has little error to spread, but I did not verify that. In my network capture, compression sent no request carrying the file.

What this does not cover

This is one photographic fade, 10 frames, about one second long. Screen recordings with flat UI colours, long clips, ordered dithering and frame dropping were not tested, and a flat-colour animation may show a much smaller gap. Dithering exists to hide banding in gradients, and I did not measure that, so whether the smaller file looks acceptable is a call to make by eye on your own content. If you copy anything, copy the script, and run it on the file and palette size you actually ship. If the gap on your file is a few percent, dithering is not your problem, and frame count or dimensions deserve the time instead.

pdf.js render() resolves fine on a huge page, but the canvas reads back transparent

hao jia — Sun, 01 Nov 2026 08:02:52 +0000

A failed render ought to look like a failure: a rejected promise, an exception, something in the console. With pdf.js on an oversized page it does not. The promise resolves, the code after await runs, and the canvas you hand to the next step is empty. All numbers below come from one machine (Apple M4, desktop Chromium 149.0.7827.55, pdfjs-dist 4.4.168), single runs, on a blank vector page I generated myself. Other browsers and phones have different limits, and I did not test them.

What the render call reported

The page was 14400 × 14400 pt. I rendered it at four scales and read pixels back from the canvas afterwards. At scale 0.5 (7200 × 7200) and scale 1 (14400 × 14400) the top-middle pixel was opaque white, as expected. At scale 1.1 the canvas was 15840 × 15840 and still fine, with the page's border colour showing up at the edge.

At scale 1.3333333 the canvas would be 19199 × 19199, an area of 368,601,601 pixels. render().promise resolved after about 1 ms, no error was thrown, and the pixels I read back were 0,0,0,0. The earlier scales took 54, 167 and 195 ms. A 1 ms render of a page that large should have looked suspicious, but nothing in the API told me so.

The canvas itself behaves the same way

To see whether pdf.js was involved at all, I took it out. A bare canvas was filled with a small red square and one pixel was read back. Then I asked for toDataURL() on the ones that failed. This is the complete output of that probe:

Chromium 149.0.7827.55
16384x16384 thrown=no readback=221,0,0,255 dataURL=skipped
16385x16384 thrown=no readback=0,0,0,0 dataURL=data:,
20000x13421 thrown=no readback=221,0,0,255 dataURL=skipped
19199x19199 thrown=no readback=0,0,0,0 dataURL=data:,
65536x1 thrown=no readback=0,0,0,0 dataURL=data:,

So the line sits between 16384 × 16384 (area 268,435,456, works) and 16385 × 16384 (fails). It is not a per-side limit: 20000 × 13421 has an area of 268,420,000 and worked. 65536 × 1 failed on its own, so a width limit exists too, and I only know it is somewhere above 32768, because 32768 × 1 worked in an earlier run. In every failing case getContext('2d') still returned a context and fillRect threw nothing.

My reading, which is an inference and not something I verified in the Chromium source: the browser cannot allocate the backing bitmap, so every drawing call goes to a surface that does not exist. That would explain why pdf.js has nothing to complain about. From its side, it issued all its draw calls and none of them raised.

The check I added

Because the failure is silent, the promise cannot be the success signal. pdf.js paints a white page background by default, so an alpha of 0 in the corner can only mean nothing was drawn. A page that is meant to have a transparent background would break this check, so it only fits the default case.

async function renderChecked(page, viewport, canvas) {
  await page.render({ canvasContext: canvas.getContext('2d'), viewport }).promise;
  const alpha = canvas.getContext('2d').getImageData(0, 0, 1, 1).data[3];
  if (alpha === 0) throw new Error(`blank canvas ${canvas.width}x${canvas.height}`);
}

Reading one pixel is a small call (I did not time it), and toDataURL() returning data:, works as a second signal if you are about to encode the canvas anyway. Neither replaces a size check before you create the canvas. The readback only tells you that it already went wrong, which is enough to avoid uploading or caching an empty image.

Where I would not rely on this

I tested a single blank vector page and a bare canvas. I did not test real complex pages, other desktop browsers, or any phone. If your pipeline sends rendered pages to a server, a silent failure also matters for compliance: an empty image that looks like a success could end up archived as if it were complete.

For PDF to image on the desktop side without writing any of this, ImgIng https://imging.ai/ has a PDF to image page that runs in a desktop browser with 144, 216 and 300 DPI presets. Across 18 runs I captured, I saw only GET requests and no request carrying the file. Nothing above says anything about how it handles oversized pages.

Why getImageData returns different colors than putImageData wrote, at low alpha

hao jia — Sat, 31 Oct 2026 08:02:56 +0000

If you write a pixel with a low alpha into a canvas and read it back, the color is not the one you wrote. In Chromium 149, putImageData with (200, 100, 50) at alpha 2 came back from getImageData as (255, 128, 0). Alpha stays 2, the RGB is simply different. At alpha 0 the color is gone entirely.

I hit this while checking why cutout PNGs go grey or bright at the edge after a resize. The samples were three Wikimedia Commons CC photos (a portrait, a product, a hair close-up) that I cut out with ImgIng, a browser tool at https://imging.ai/ , on a desktop. The readback test below is only a pixel I made up and one browser version, a single run.

The readback test is twelve lines

The loop writes one pixel with RGB fixed at (200, 100, 50) for each alpha from 0 to 22 in steps of 2, then reads it back with getImageData. This is the complete output:

in  (200,100,50, 0) -> out (0,0,0,0)
in  (200,100,50, 2) -> out (255,128,0,2)
in  (200,100,50, 4) -> out (191,128,64,4)
in  (200,100,50, 6) -> out (212,85,42,6)
in  (200,100,50, 8) -> out (191,96,64,8)
in  (200,100,50,10) -> out (204,102,51,10)
in  (200,100,50,12) -> out (191,106,42,12)
in  (200,100,50,14) -> out (200,91,55,14)
in  (200,100,50,16) -> out (207,96,48,16)
in  (200,100,50,18) -> out (198,99,57,18)
in  (200,100,50,20) -> out (204,102,51,20)
in  (200,100,50,22) -> out (197,104,46,22)

Alpha comes back exact every time. The RGB wanders, and it does not settle smoothly as alpha grows. Alpha 10 lands within 4 on every channel, but alpha 12 is already 9 off on red (191 against 200), and at alpha 18 blue is still 7 off (57 against 50).

The numbers match a premultiplied store

My explanation is an inference, not something I checked in the browser source. If the canvas keeps each pixel premultiplied (RGB multiplied by alpha, rounded to a byte) and getImageData divides alpha back out, the pattern falls out of the arithmetic. At alpha 2, red is 200 × 2 / 255 = 1.57, which rounds to 2, and 2 × 255 / 2 gives 255. Green is 0.78, rounds to 1, comes back as 127.5, shown as 128. Blue is 0.39, rounds to 0. That is (255, 128, 0), the exact row above.

Alpha 4 works the same way: 3.14 rounds to 3 and returns 191, 1.57 rounds to 2 and returns 128, 0.78 rounds to 1 and returns 64. I did not run the arithmetic for every row, only these two plus alpha 10 (7.84 rounds to 8, returns 204). At alpha 2 each channel has about three possible stored values, so there is nothing to recover. At alpha 0 every channel is zero, which is why the color vanishes completely.

Cutout PNGs hit this at the edge

In my three exports there was a ring of pixels at alpha 1 to a few dozen, and the color under the lowest of them was the old background: average RGB for alpha 1–5 was about (221, 218, 219) on the portrait, (230, 224, 218) on the product and (30, 31, 32) on the hair photo. Where alpha was 0, the portrait and the hair photo stored (0, 0, 0); I did not check the product one. A PNG file keeps those values as they are. A canvas does not.

So the color under low alpha is not reliable once an image has passed through a canvas. I would guess that is one source of the grey or bright fringe after resizing, though I did not isolate it. A second source I measured separately: scaling the four channels independently, without premultiplying, pulls that background color into the edge. On the portrait the edge band was off from a premultiplied baseline by 2.991 levels on average, and 13.83% of its pixels were off by more than 8 levels (this portrait, a 4× shrink and grow, one run).

Rules I would follow now

Do not use getImageData to learn what color a faint pixel "really" had. Keep the original file buffer if you need it, and treat the RGB under alpha below roughly 10 as noise. If a pipeline round-trips cutouts through a canvas, compare a few edge rows before and after, as in the table above, before trusting the output.

createImageBitmap with premultiplyAlpha: 'none' is not a clean fix either. On a tiny 8-pixel test it let the blue from a fully transparent area mix into the transition pixel, giving (128, 0, 128, 128). Safari and Firefox I did not test, so the whole table is Chromium 149 only.

A bash loop that logs bytes and seconds for each PNG optimization step

hao jia — Fri, 30 Oct 2026 08:03:26 +0000

When you chain two PNG tools, the usual question is which one saves more. The question I care about more is whether I can see, per step, how many bytes and how many seconds each step cost, so a build review has something to argue with. A shell loop that writes one line per file is enough for that, and it reruns the same way on any machine that has bash, perl, pngquant and oxipng.

One loop records bytes and seconds per step

The script takes a folder of PNGs. For each file it runs pngquant with a quality range of 65-80 (speed 4 is its default), copies the result, runs oxipng on the copy, and prints the sizes and three timestamps. awk then groups the lines by the filename prefix, so icon- and photo- become two classes. The timestamps come from perl's Time::HiRes, so the format is the same wherever perl is installed.

now() { perl -MTime::HiRes=time -e 'printf "%.3f", time'; }
tmp=$(mktemp -d)
for f in "$1"/*.png; do
  c=$(basename "$f" | cut -d- -f1)
  t0=$(now); pngquant --quality=65-80 --force --output "$tmp/q.png" "$f"
  t1=$(now); cp "$tmp/q.png" "$tmp/o.png"; oxipng -o 4 --strip safe -q "$tmp/o.png"
  echo "$c $(wc -c <"$f") $(wc -c <"$tmp/q.png") $(wc -c <"$tmp/o.png") $t0 $t1 $(now)"
done | awk '{o[$1]+=$2; q[$1]+=$3; x[$1]+=$4; tq[$1]+=$6-$5; tx[$1]+=$7-$6}
  END {for (c in o) printf "%-13s %8d %8d %8d %5.2fs %5.2fs\n", c, o[c], q[c], x[c], tq[c], tx[c]}' | sort

Drop the awk stage and the per-file lines show which single image moved a class total. --strip safe makes oxipng remove non-critical chunks but keep the color-profile ones (iCCP, sRGB, cICP) and pHYs, according to its own help text, which matters if your pipeline checks color profiles.

I ran it on 12 PNGs from Wikimedia Commons (CC licensed, noted at the end): three screenshots, three icons, three illustrations, three photos stored as PNG. Tools were pngquant 3.0.3 and oxipng 10.2.1 with -o 4 --strip safe, on an Apple M4. Sizes below are bytes, and the two time columns are pngquant and oxipng, summed over the three files of each class, one run.

icon            184137    37173    31496  0.19s  0.95s
illustration    316993    78209    73211  0.20s  1.43s
photo          2063192   694735   612968  0.42s  0.90s
screenshot     1890089   601760   565455  0.44s  1.67s

The columns are class, original, after pngquant, after oxipng, pngquant time, oxipng time. I ran the loop three times and the byte counts never changed. Only the decimals of the times moved. The loop never touches the input files, so rerunning it is safe.

The second step still takes bytes off

Summed over all 12 files, 4,454,411 bytes became 1,411,877 after pngquant, which is 68.3% smaller. After oxipng the total was 1,283,130, which is 71.2% smaller. The second step removed another 128,747 bytes, or 2.9 percentage points of the original total.

Per class the extra points, measured against that class's original size, were 4.0 for photos, 3.1 for icons, 1.9 for screenshots and 1.6 for illustrations. I did not look into why photos gained the most, so I would not turn that order into a rule.

The cost is a few seconds. Across the whole table pngquant took about 1.25 seconds and oxipng about 4.95 seconds, so the second step is the slower of the two here. In a separate run, zopflipng added at most 0.5 points on top of pngquant and took 3.5 to 19.5 seconds per class, so I did not put it in the loop. That run used zopflipng's quick mode (-q), so it says nothing about its full mode, which I did not time here.

The order of the two steps changes the result

Swapping the two commands is a quick test. In an earlier run with the same tools, when oxipng ran first and pngquant second, the photo class ended at 694,253 bytes, against 694,735 for pngquant alone. That is 482 bytes of difference. My reading, which I have not verified, is that pngquant writes a fresh file and the earlier deflate work is thrown away. The numbers do say the same thing: on these files the lossless pass belongs after the palette step.

A tool's own palette output can go through the same loop

I also ran the same 12 files through the batch compression in ImgIng (https://imging.ai/) on desktop Chromium 149, with the output format set to PNG. Seven of the 12 came out larger than the original, and one was kept as the original, so I left those eight out. The other four were palette PNGs, and I fed them to the same oxipng command.

Those four shrank by another 28.6% to 45.5% relative to the tool's output, for example 43,216 bytes down to 23,571 for the blue star icon. Four files is a small sample and I did not compare image quality, so the only conclusion is that palette PNGs like these four are worth a second look in the loop.

What this loop does not prove

Twelve files, one machine, one run per step is the whole evidence. The photos are PNGs of photographs, not camera JPEGs. I did not compare image quality between the settings, so a smaller file here only means smaller. And oxipng at -o 4 is a middle setting, not the slowest one.

To reuse it, point the script at your own folder, keep the output as a CSV if you want a history, and diff the numbers when a tool version changes. If the byte counts differ between two runs on the same input, something in the chain is not deterministic, and that is worth knowing before the step goes into CI.

Sample images are from Wikimedia Commons under CC licenses (CC0, CC BY, CC BY-SA); author credits are on each file's Commons page.

Chromium turned my CFF OpenType font into Type 3 at every fsType value I tried

hao jia — Thu, 29 Oct 2026 08:03:13 +0000

Which fonts survive "print to PDF" as real fonts, and which ones turn into drawing instructions? That decides what a PDF-to-HTML converter can later hand back to a browser as a web font. The fsType flag in the font's OS/2 table is the obvious suspect. In the one test below, it explained two of three fonts and nothing for the third.

Three fonts, six fsType values each

The test is small. I took Pacifico and Liberation Sans (both TrueType outlines) and Noto Serif SC (CFF outlines in an .otf), all open-licensed. For each one I made six local copies with fsType set to 0, 2, 4, 8, 256 and 512, and printed one page of mixed English and Chinese text to PDF with Chromium 149. Then I read the font objects out of each PDF. The modified copies never left my machine.

For the TrueType fonts the flag mattered. Values 4 and 8 gave a normal embedded subset. Values 2 and 512 gave no embedded font at all, and the glyphs were written as Type 3 outlines instead. Value 256, the "no subsetting" bit, made Chromium embed the whole font: the Liberation Sans PDF went from 85,024 to 502,702 bytes.

Noto Serif SC ignored all of that. At 0, 2, 4, 8, 256 and 512 the PDF contained Type 3 fonts only. Value 0 is "installable embedding", the most permissive setting there is, so in this test the flag was not what pushed this font out of the embedded path.

python3 - NotoSerifSC-fsType0.pdf NotoSerifSC-fsType512.pdf Pacifico-fsType0.pdf Pacifico-fsType512.pdf <<'P'
import sys, fitz
from collections import Counter
for p in sys.argv[1:]:
    d = fitz.open(p)
    ids = {f[0]: f[2] for pg in d for f in pg.get_fonts(full=True)}
    print(p, dict(Counter(ids.values())))
P

NotoSerifSC-fsType0.pdf {'Type3': 29}
NotoSerifSC-fsType512.pdf {'Type3': 29}
Pacifico-fsType0.pdf {'Type3': 24, 'Type0': 2}
Pacifico-fsType512.pdf {'Type3': 28}

The counts are PyMuPDF's view of one page. I do not know why it is 29 and not 1, and I did not dig into it. The Pacifico row is a useful contrast: at fsType 0 there are two Type0 objects, which are the embedded TrueType subset of Pacifico, and at 512 there are none. The Type 3 objects next to them come from the Chinese line, which fell back to a system font I did not identify.

What I know and what I am guessing

What I know is narrow. In this one test, with one CFF font, on Chromium 149 on one Mac, every fsType value ended in Type 3, and the two TrueType fonts did not behave that way.

The explanation I find most plausible is that Chromium's PDF writer has an embedding path for TrueType outlines and no equivalent for CFF outlines inside an OpenType wrapper, so those fonts get drawn glyph by glyph as Type 3. That is a guess. I did not read the Chromium or Skia source, I did not test a second CFF font, and I did not try another Chromium version. If Noto Serif SC is odd for some other reason, such as its size or its CJK coverage, my test cannot tell. So the "always" I was tempted to put in the title is really "every time in this test".

What it did to the PDF-to-HTML step

I ran these PDFs through ImgIng (https://imging.ai/), a browser-based PDF-to-HTML tool that is in Beta, runs in desktop browsers and processes the file locally. The outputs matched the font types in the PDF. The TrueType subsets became inline woff fonts holding only the glyphs the page used. The six Noto Serif SC PDFs got no inline font at all: every glyph became a path, with a transparent text layer on top set in Arial, sans-serif so the text can still be selected. In the fsType 2 case, the Pacifico text still looked like Pacifico, because paths do not need the font.

Paths keep the shape, but they give you no font file to reuse. A subset is not a reusable font either way, since it holds only the used characters. The takeaway from this one test is small: for the CFF font I tried, the outline flavor of the file mattered, and the license flag did not.

If you hit Type 3 where you expected a real font, it may be worth checking whether the file is TrueType or CFF before you look at fsType; that is my suggestion from one font, not something I verified more widely. On the PDFs you already have, the snippet above shows how many font objects are Type 3 and how many are embedded.

Alpha is just a plane in the pixel format, and FFmpeg drops it in three quiet places

hao jia — Wed, 28 Oct 2026 08:04:52 +0000

"Transparent video" sounds like a property of a file, but inside FFmpeg it is a property of a frame in memory: either the frame has a fourth plane, or it does not. FFmpeg names that difference yuva420p versus yuv420p. Every step on the path keeps the plane, drops it, or makes up a new one. In the cases I ran, none of the steps printed an error when it dropped the plane.

The clip is one 8-second, 540×960 WebM with VP9 alpha, exported from ImgIng (https://imging.ai/), a browser tool for video background removal. It is Beta, it is built for desktop Chrome and Edge, it processes the video locally, and it does not run on phones. Everything below is that one clip, FFmpeg 7.1 on one machine, one run each. I did not try other FFmpeg builds or any editing software.

The decoder can hand over a frame with no alpha plane

The first drop is at the decoder. With the default VP9 decoder, ffmpeg -i prints the stream as yuv420p. Name libvpx-vp9 as the decoder and the same stream is printed as yuva420p. The extra a is the whole difference, and it is the only hint you get.

On frame 60, the default decoder gives an alpha minimum, average and maximum of 255, so every pixel is opaque. With libvpx-vp9 the average is 46.7618 and the minimum is 0. I read a raw frame from the correct path as well: 1,296,000 bytes, which is 518,400 for luma, two chroma planes of 129,600 each, and 518,400 for alpha at full luma size. So the alpha plane is a real, full-size plane, and its absence on the other path is not a rounding problem. Why the default decoder skips it, I did not look into.

A filter chain can throw the plane away for good

The second drop happens even when the decoder is already right. Frame 60 again, decoded with libvpx-vp9, then two filter chains that differ only in a detour.

$ ffmpeg -hide_banner -c:v libvpx-vp9 -i input.webm -vf "select=eq(n\,60),format=yuva420p,alphaextract,signalstats,metadata=print" -an -f null -
lavfi.signalstats.YMIN=0
lavfi.signalstats.YAVG=46.7618
lavfi.signalstats.YMAX=255

$ ffmpeg -hide_banner -c:v libvpx-vp9 -i input.webm -vf "select=eq(n\,60),format=yuv420p,format=yuva420p,alphaextract,signalstats,metadata=print" -an -f null -
lavfi.signalstats.YMIN=255
lavfi.signalstats.YAVG=255
lavfi.signalstats.YMAX=255

The second chain leaves the plane and comes back, and what comes back is a plane filled with 255. Stats trimmed to the three lines; the rest of the log is omitted. I tried two more detours, through rgb24 and through gray, and both gave 255 as well. Staying inside formats that have alpha was fine: yuva420p, rgba and yuva444p all kept the 46.7618 average.

This matters because a pipeline rarely has one filter. A chain can pick up format=yuv420p early to keep a scaler happy and format=yuva420p later "to be safe". The second one looks like it restores the transparency. It does not. It only makes the stream claim to have alpha. The same thing happens on the default-decoder path: forcing format=yuva420p after it still reads 255 on frame 60.

The encoder's pixel format gets negotiated for you

The third drop is in the encoder. When the output codec does not take the incoming pixel format, FFmpeg inserts a conversion on its own. Running with -v verbose, decoding with libvpx-vp9 and encoding with libx264 without any -pix_fmt, the log has a line saying it is auto-inserting auto_scale_0, and another showing the conversion fmt:yuva420p -> fmt:yuv420p. The command succeeded and the output stream is yuv420p. H.264 has no alpha, so this is correct behavior. I only ran this with -v verbose, so what the default log level prints for the same conversion is untested.

The same conversion helps explain the worst case in these runs. Default decoder into ProRes 4444 with -pix_fmt yuva444p10le: the log shows auto_scale_0 converting fmt:yuv420p -> fmt:yuva444p10le, and the output stream line says yuva444p10le. The file looks like an alpha file by its metadata. Reading the alpha back gives 255 everywhere. That the converter supplies an opaque plane because the input had none is my inference from the readback, not something I inspected in the filter.

Read the alpha back after the step you suspect

The stream line is a first gate, not the last one. If the line right after the decoder does not say yuva, nothing downstream can fix it, and a later format=yuva... only paints a plane of 255. For anything that has to cross a team boundary, such as handing a clip to an editor, run the finished file back through the alphaextract,signalstats pair above and read the minimum. A minimum of 255 means the file is opaque, whatever its stream line says.

For this clip the number to compare against is 0 / 46.7618 / 255 on frame 60.

Why 1-pass ABR landed under my target size: 60-second clip, x264, 25 and 10 MB

hao jia — Tue, 27 Oct 2026 08:33:20 +0000

Say a ticket tool has a hard upload cap, so the browser has to squeeze screen recordings under it before they leave the laptop. The formula that comes to mind first is video bitrate = target × 8 / duration with single-pass ABR. On a 60-second 1080p clip that came out above the cap. After I subtracted the audio track it came out below the target instead, at both targets I tried, and I wanted to know why before trusting that direction.

All numbers below come from two clips, x264 preset medium, AAC 128 kbps, FFmpeg 7.1 on an M4 Mac, one run each. Clip A is a 60-second 1080p action clip (Isak Zvegelj, CC BY 3.0, Wikimedia Commons). Clip B is a synthetic 60-second screen recording, generated frame by frame, so it is quieter than a real one. "MB" means 1,000,000 bytes here.

Subtract the audio first

Audio at 128 kbps over 60 seconds is nominally 960,000 bytes, and the muxed track measured 971,835. That is about 3.9% of a 25 MB target and 9.7% of a 10 MB target. This is the whole budget calculation:

for target in (25_000_000, 10_000_000):
    print(target, (target * 8 / 60 - 128_000) / 1000)

25000000 3205.3333333333335
10000000 1205.3333333333333

On clip A the naive bitrate (no audio subtracted) overshot by 3.14% at 25 MB and 9.17% at 10 MB. With the audio subtracted, single-pass ABR (-b:v 3205k and -b:v 1205k) gave 24,835,380 bytes (−0.66%) and 9,965,325 bytes (−0.35%).

Where the missing 1% went

I split the naive single-pass overshoot at 25 MB into three parts: audio +971,835 bytes, container +66,834 bytes (almost all of it the moov box), and the video stream −253,166 bytes. That adds up to the +785,503 bytes the file was over by. So the video stream came in about 1.0% under the budget I had handed to the rate control, and nothing in the file filled that gap. The 10 MB run went the same direction.

Why? I did not look inside x264, so this part is inference. My guess is that single-pass ABR steers by what it has seen so far and stays slightly on the cautious side of its average. Per-second video bitrate on clip A swung between 790 and 1,950 kbps around a 1,205 kbps average at 10 MB, so there was plenty to steer. That fits the guess but does not prove it.

Two-pass did not fix it here

The textbook answer is two-pass, so I ran it with the same bitrate. On clip A it overshot: +0.28% at 25 MB (69,942 bytes over) and +1.74% at 10 MB. A CRF search that re-encodes the whole file each round landed at −0.86% and −1.45%, and took 157 to 217 seconds against about 23 seconds for single-pass. Under a hard cap, undershooting by a percent is the failure I can live with, and overshooting is the one that gets the upload rejected.

The clip decides how big the miss is

On clip A every method stayed within ±1.8% once the audio was subtracted. Clip B was different, and not in one direction. At 10 MB its single-pass run came in 8.65% under and its two-pass run 12.0% over. One synthetic clip says nothing about real recordings, so treat B as a warning about variance, not as a measurement.

What I would do with it

Keep the formula with the audio subtracted, plus about 70 KB for the container, and on footage like clip A expect single-pass to land up to a percent low. That is only what these two clips showed. If the file ends up far below the cap, check whether the clip is simply easy before blaming the encoder. I would not raise the bitrate automatically to use the slack, because the cap matters more than the last percent.

If you want to see what a ready-made tool does with the same clip, ImgIng (https://imging.ai/) has a video compression page that runs in a desktop browser, and the page says it processes locally. It has no compress-to-target-size mode, so the only lever is its presets: the balanced tier with a 1280 longest side gave 25,938,497 bytes against my 25,000,000 target, so even there I overshot. It does not work on phones. Free tiers and limits of other online tools change, so check their own pages.

One unit trap: ImgIng's interface counts MB as 1,048,576 bytes, 4.86% more than the 1,000,000 I used above. If a limit is written in MB, find out which one before you budget.

Estimating skew from OCR word boxes before row clustering: one of my three estimators broke at 5 degrees

hao jia — Mon, 26 Oct 2026 08:07:57 +0000

Tilt a table image by 2 degrees and the two ends of a row end up about 43 px apart vertically (1,228 px of text width times tan 2 degrees), while the rows on my test sheet are 56 px apart. A fixed y tolerance gives up somewhere in that range. The obvious repair is to estimate one global slope and subtract slope times x before clustering. I assumed that would take ten lines, so I wrote three versions and compared them.

The sheet is a quote list I generated: 7 columns, a header and 8 data rows, 63 cells, no ruling lines, English and digits only. I rotated it by 0.5 to 5 degrees on white and ran tesseract 5.5.3 (English pack, --psm 6) on an Apple M4, about 70 word boxes per image. Afterwards I cluster with half the median word height and count rows; 9 is right. It is one sheet, so nothing here is an accuracy figure.

Three ways to read the slope

Neighbours: for each box take the nearest box to its right within 1.5 word heights vertically, then the median of dy/dx. All pairs: the median of dy/dx over every pair more than 50 px apart, skipping pairs steeper than 0.12. Profile: try slopes from -0.15 to 0.15 and keep the one that piles up y minus slope times x most tightly, which is what flat rows do. I fixed those numbers before the first run and never touched them; only the profile score changed, below.

Tilt	True slope	Neighbours	All pairs	Profile	Rows, no fix
0.5 deg	-0.0087	-0.0097	-0.0087	-0.0086	9
1 deg	-0.0175	-0.0176	-0.0172	-0.0174	10
2 deg	-0.0349	-0.0352	-0.0336	-0.0350	18
3 deg	-0.0524	-0.0511	-0.0475	-0.0528	19
5 deg	-0.0875	-0.0848	-0.0352	-0.0880	24

All three gave 9 rows up to 3 degrees. At 5 degrees neighbours and profile still gave 9 and all pairs gave 19.

All pairs broke at 5 degrees

A pair from two different rows passes the steepness cap once the page leans far enough. Among the pairs that passed at 5 degrees, 253 came from the same row and 263 from other rows. At 2 degrees it was 238 and 226 and the estimate was fine; my guess is that the strays scatter to both sides there and cancel in the median, which I did not plot. A lower cap is one more number to tune, so I dropped it.

My first profile score was jumpy

I first counted boxes in horizontal bins a quarter of a word height wide and scored a slope by the sum of squared counts. On the straight bordered score sheet it returned -0.0022 instead of 0: the score was 534 at a slope of exactly 0.0 and 440 at 1.4e-16. I read that as centres hopping across 5.5 px bin edges, checked only through those two scores. A Gaussian kernel over all pairs of y values has no edges:

def tilt_slope(cx, cy, sigma):
    def tightness(s):
        y = cy - s * cx
        gap = y[:, None] - y[None, :]
        return np.exp(-0.5 * (gap / sigma) ** 2).sum()
    grid = np.arange(-0.15, 0.1501, 0.002)
    best = grid[np.argmax([tightness(s) for s in grid])]
    fine = np.arange(best - 0.002, best + 0.0021, 0.0002)
    return fine[np.argmax([tightness(s) for s in fine])]

With sigma at a quarter of the median word height this is the Profile column. It also needs fewer boxes: over 200 random subsets of 10 boxes from the 2 degree sheet, the kernel landed within 0.003 of the true slope 157 times and the bins 59.

A straight page should not move

A correction that moves flat tables is a regression, so I checked. On three straight sheets (quote, bordered score and merged-header sheets, the last two with lines painted out) the profile estimate stayed between -0.0004 and +0.0016 and row counts did not change: 9, 9 and 10. The merged-header sheet gives 10 either way, for a reason I haven't found.

A simulated phone photo

The same sheet as a simulated phone photo (rotated 4.5 degrees, trapezoid warp, glare patch, JPEG; made with Pillow and OpenCV, not a real photo) gave 17 rows with no fix. Neighbours estimated 4.83 degrees and profile 5.07, both gave 9 rows, all pairs gave 15. Per-row slopes after the fix varied by only about 15 px across the page, against a word height of 39, so one slope was enough here; I have not tried a stronger warp.

For comparison I tried the OCR page of ImgIng (https://imging.ai/) in a desktop browser with its default model. On clean synthetic tables its copied text has tabs between only some cells, so the columns are incomplete. On the four simulated photos it found almost every word and wrote each as one scrambled line; it only offers 90 degree turns, and it does not run on phones.

What I have not settled

Rows were right at 3 and 5 degrees, the text was not: 6 of 9 rows matched the source at 3 degrees and 3 of 9 at 5, since tesseract starts misreading digits (196.10 came back as 496.10). Everything here is one English sheet rotated by a script, so no real scan went through it. Run the estimator on a straight page first: if the slope is not near 0.000 there, the estimator is the problem.

Reading an image's matrix and clip from pdf.js getOperatorList: the last cm was enough until I nested one

hao jia — Sun, 25 Oct 2026 08:03:26 +0000

On a hand-built PDF, pdfimages, PyMuPDF and a pdf.js bitmap all returned the whole stored picture, even where the page showed half of it or showed it turned or mirrored. What the page shows has to come from somewhere else. pdf.js has no extract call, but getOperatorList() returns a page's drawing instructions, so I read each image's matrix and clip out of it.

The setup was pdfjs-dist 4.10.38 (legacy build) under Node 24.14.1 on an Apple M4. The files were a hand-built page with four lossless images (one clipped, one rotated, one mirrored, and a palette image I leave out because it has nothing to do with placement), the 2-page Creative Commons Indonesia brochure (CC BY 4.0) and a 10-page Creative Commons newsletter (CC BY 3.0), both from Wikimedia Commons. Everything below holds for these three files only. Lengths are in pt (1/72 inch), y up.

An image paint carries no position

Page 1 of the hand-built file comes back as 52 operations. For the clipped image they read save, constructPath, clip, endPath, transform, paintImageXObject, restore. The path args are [[19], [60,540,90,120], [60,540,150,660]]: 19 is OPS.rectangle, then x, y, width, height and a bounding box. The paint op is only ["img_p0_1", 240, 160], an id and a pixel size. The position comes entirely from the transform before it.

That transform is [a b c d e f], applied to a unit square. A is [180,0,0,120,60,540]: 180×120 pt with its corner at (60, 540). B is [0,180,-120,0,450,540]: a is 0 and b is positive, a quarter turn counter-clockwise, ending up 120 wide and 180 tall. C is [-180,0,0,120,240,300], with a negative a and a determinant a·d − b·c of −21,600, so it is mirrored. A negative determinant is my mirror test; for C, b and c are 0, so it is left-right.

The last cm is not the matrix

A transform multiplies into whatever matrix is current, and the paint op carries none. The real matrix is every cm since the page started, minus the ones a restore popped. This is the core of my walk, trimmed. The full script also handles form XObjects and groups, and the newsletter has three of each.

let ctm = [1, 0, 0, 1, 0, 0], clips = [], stack = [], path;
for (let i = 0; i < ol.fnArray.length; i++) {
  const f = ol.fnArray[i], a = ol.argsArray[i];
  if (f === OPS.save) stack.push([ctm, clips.length]);
  else if (f === OPS.restore) [ctm, clips.length] = stack.pop();
  else if (f === OPS.transform) ctm = mul(a, ctm);
  else if (f === OPS.constructPath) path = { xy: a[1], ctm };
  else if (f === OPS.clip) clips.push(path);
  else if (f === OPS.paintImageXObject) console.log(a[0], ctm, clips.length);
}

On the brochure (8 image paints) and the newsletter (36), the last cm before each paint was already the full matrix, 44 times out of 44, with save depth up to 4. I don't know why. Then I edited a copy of the hand-built file, wrapping A's drawing in q 0.5 0 0 0.5 100 100 cm … Q. Its last cm stayed [180,0,0,120,60,540] while the accumulated matrix was [90,0,0,60,130,370]. The other three images still matched. I accumulate.

Clips share those coordinates

A clip arrives as a constructPath followed by clip. The rectangle is in the coordinates in force when the path was built, so I keep that matrix, push the corners through it and take the bounding box. A restore drops the clips added since its save. The visible share is the image box intersected with every clip box, over the image box.

File	Visible share	Checked against
Hand-built, A	0.500	a 96 dpi render, cropped to the left 120 columns, differed by 0.00 on a 0–255 scale
Second hand-built file, rotated 90° and clipped	0.500	the render matched the stored image's right half, turned 90°, with error 0.00
Third hand-built file, two nested clips	0.333	a render showed 33.8% visible
Brochure, 8 images	1.000 for all 5 under a clip	arithmetic only
Newsletter, 36 images	14 under a clip, 7 lose 0.4% to 2.3%	arithmetic only, pages not rendered

What I couldn't read

Page 9 of the newsletter has three images tilted by about 7°, 5° and 14°, each under a clip that is not a rectangle. My rectangle math doesn't apply, so I have no visible share for them. W* and text clips are untested.

For the stored side I used ImgIng (https://imging.ai/), a browser tool, on the hand-built file in a desktop browser. It takes the images actually painted on a page, not a screenshot, and A, B and C came back whole, unrotated and unmirrored, the same pixels as pdfimages. It takes one PDF per run, returns 0 results for a page with no images, and names ZIP entries by serial number, with page numbers only on its cards. On the newsletter it listed 32 results where the operator list has 36 paints, since a banner reused on several pages is exported once. The op list keeps every placement with its own matrix, and a deduplicating extractor keeps one file.

To get the part the page shows, crop the stored pixels by the visible share when the matrix is axis-aligned and the clip a rectangle; otherwise render the page and crop, which I haven't compared across tools.

Where did the 254s go? A heatmap of non-255 pixels in a white JPEG

hao jia — Sat, 24 Oct 2026 08:03:15 +0000

A white sneaker photo, cut out and flattened onto white, is exactly 255 in the PNG. Save it as a JPEG at quality 85 and it isn't any more. A percentage tells me how much drifted, but when a rule says "pure white" I want to know where, so I built a heatmap. On this one photo, 10,403 pixels that were exactly 255 in the PNG no longer were after the JPEG, and the map puts them along the outline of the shoes.

The photo is a Wikimedia Commons shot of white sneakers on beige tile (JustinLiang522, CC BY-SA 4.0). I made the white PNG in ImgIng (https://imging.ai/), a browser-based image tool, in desktop Chromium: cutout, background set to white in the refine step, exported at 1,280×1,707. Its export row lists PNG, WebP, AVIF, TIFF and GIF but no JPEG, so the JPEG comes from Pillow 11.3.0 and numpy 2.0.2 on an Apple M4, quality 85 with Pillow's 4:2:0 chroma. Amazon's US seller help page (viewed 2026-10-23) asks main images for RGB 255, 255, 255 and recommends JPEG, which is why I care about the last digit.

A percentage says how much but not where

My first count used the cutout's alpha channel: background is wherever alpha is 0. That gave 7,916 drifted pixels, 0.66% of the background. It needs the cutout file, and I'd guess an image reaches an upload step with the alpha long gone, though I haven't checked any pipeline. So I switched to what the finished files still tell me: which pixels were exactly 255 in the PNG master?

Use the master as the mask

Everything below compares two arrays, the PNG before saving and the JPEG decoded again.

png = Image.open("master.png").convert("RGB")
stream = io.BytesIO()
png.save(stream, "JPEG", quality=85, subsampling=2)
before = np.asarray(png)
after = np.asarray(Image.open(stream).convert("RGB"))

white = (before == 255).all(axis=2)
drift = white & (after < 255).any(axis=2)
print(f"{white.mean():.2%} white, {drift.sum():,} drifted")
# 55.59% white, 10,403 drifted

I expected 7,916 again and got 10,403. The extra 2,487 sit on the shoe itself: 7,891 pixels of the shoe are pure 255 in the PNG, and some of them drift too. That's the price of not having a mask, and I'm keeping this version because it works on any pair of files.

The red hugs the outline

To see it, I widened each drifted pixel with a 5×5 maximum filter, faded the photo to a quarter of its strength and painted the dots red.

dots = Image.fromarray((drift * 255).astype("uint8")).filter(ImageFilter.MaxFilter(5))
canvas = (before * 0.25 + 191).astype("uint8")
canvas[np.asarray(dots) > 0] = (230, 40, 40)
Image.fromarray(canvas).save("drift.png")

I'm not attaching the render. In mine the red is a thin line along the outline of both shoes and almost nothing across the open white. A few isolated specks sit away from the shoes, and I haven't traced them. A count by distance agrees: 7,731 of the 7,916 drifted pixels in the cutout's empty area are within 8 pixels of the subject, so a check that samples only far from the shoe would have seen none of it.

Every threshold shrinks the red

A heatmap with a threshold on it is the useful version. I counted a pixel as still red when any channel fell below the threshold, for five thresholds.

for t in (255, 254, 252, 250, 245):
    print(t, (white & (after < t).any(axis=2)).sum())
# 255 10403 | 254 5129 | 252 1091 | 250 293 | 245 32

Half the red disappears at 254, and by 250 only 293 pixels are left, under 3% of the 10,403. So most of the drift is one or two levels off white. The darkest spots go further than that. A count on the same photo and quality found a few background pixels as low as 239.

The right panel is the part to read: background-only share on the sneakers climbs from 99.34% at 255 to 99.98% at 250 and is at or above 99.99% by 245. The left panel is the trap. Over the whole image the sneaker share moves only from 55.25% to 56.39% between 255 and 250, a 1.14-point shift that looks negligible even though the red count dropped by about 97% over the same range. They are different measures of the same JPEG, and the whole-image one also counts the shoe's own bright pixels: 2.66% of the shoe body is at or above 250. Photos by Dan Bollinger, Kritzolina and JustinLiang522, Wikimedia Commons, CC BY-SA 4.0; the chart is an adaptation under the same license.

If I had to put a line in a check today I'd start at all three channels at or above 250. That is my guess from one photo, not a validated value, and I don't know what any marketplace uses.

What I still can't explain

The specks away from the outline are the open one. The same photo's background count at quality 85 goes from 0.66% to 0.29% with 4:4:4 chroma and never reaches zero, so my next run is the same two blocks with subsampling=0, to see whether the specks go along with the rest of the red.

For your own export, run the first block before you read any percentage and look at where the red sits. Three photos were measured in the background figures above, only the sneakers went through the heatmap, and it was one encoder on one machine.

Decode, draw, close, encode, save: one browser image loop, timed step by step

hao jia — Fri, 23 Oct 2026 08:04:02 +0000

A teammate is building a small internal page where our ops people pick a folder of product photos and get JPEGs back for an upload form, all of it inside the browser. Before reviewing his loop I wanted every line of it backed by output I had run myself, with a stopwatch between the steps. The test photos are my own synthetic 3024×4032 JPEGs, the browser is desktop Chromium 149 on a 16 GB Mac, and MB means 1024×1024 bytes.

Hold one photo at a time

A decoded 3024×4032 photo is about 46.5 MB of RGBA, so the loop's whole memory strategy is to never hold two. In our earlier 300-photo runs, total browser memory for the one-at-a-time version with results thrown away stayed between 111 and 192 MB and the run took 24.6 seconds. Decoding everything first had reached 5,192 MB by photo 100.

Close the bitmap as soon as it is drawn

My rewrite, with a timer between steps. persist is whatever saves the result.

async function convertQueue(queue, persist) {
  for (const item of queue) {
    const label = item.webkitRelativePath || item.name, t = [performance.now()];
    const lap = () => t.push(performance.now());
    let photoBmp = null, paper = null;
    try {
      photoBmp = await createImageBitmap(item); lap();
      paper = new OffscreenCanvas(photoBmp.width, photoBmp.height);
      paper.getContext('2d').drawImage(photoBmp, 0, 0); lap();
      photoBmp.close();
      const jpeg = await paper.convertToBlob({ type: 'image/jpeg', quality: 0.8 }); lap();
      await persist(label, jpeg); lap();
      console.log(label, t.slice(1).map((x, i) => Math.round(x - t[i])).join('/'), 'ms', jpeg.size, 'B');
    } catch (e) {
      console.log(label, 'skipped:', e.name, e.message);
    } finally {
      photoBmp?.close();
      if (paper) paper.width = paper.height = 0;
    }
  }
}

Once drawImage returns, the canvas holds its own copy of the pixels, so the bitmap has no reason to live through the encode. The awaited persist keeps the next decode from starting while the last JPEG is still being written. I haven't measured a version without that await, so this part is reasoning only. The finally covers files that fail halfway, and a second close() on an already closed bitmap threw nothing in my runs.

Two bad files were skipped and the rest finished

The test folder had six valid photos in three nested folders plus two traps: notes.jpg, a 65-byte text file, and IMG_0007.jpg, the first 64 KB of a JPEG. For the test, persist wrote each result into the browser's private file system (OPFS) through createWritable(), the same call a page uses on a folder the user picked. The four numbers are decode/draw/encode/save in milliseconds. Here are four of the eight lines from the first run:

photos/album-a/IMG_0001.jpg 28/3/50/2 ms 1530557 B
photos/album-a/notes.jpg skipped: InvalidStateError The source image could not be decoded.
photos/album-b/IMG_0007.jpg skipped: InvalidStateError The source image could not be decoded.
photos/album-b/IMG_0005.jpg 29/3/51/2 ms 1623854 B

The other four valid photos printed the same shape, with encode between 46 and 49 ms.

Both bad files reported image/jpeg as their File.type, because the browser takes the type from the extension, and only the decode step caught them. Without the try, that rejection would have ended the function at notes.jpg, the third file, and the ops person would have had two JPEGs out of six. The order isn't alphabetical either (IMG_0007 before IMG_0005), because that's how input.files handed them over, so the progress list should show each label instead of assuming a sorted queue.

Decode and encode take most of each photo

Across two runs, decode took 24–29 ms per photo and encode 46–52 ms, while draw took 3–4 ms and the OPFS write 2 ms, so about 75 of every 80 ms went to decode and encode. I don't trust the draw number much. My guess is that Chromium defers the real drawing work to convertToBlob, but I haven't verified it. In the earlier 300-photo runs, moving the loop into a Worker barely changed the total (23.5 s against 24.6 s) but cut the main thread's worst timer delay from 8.5 ms to 2.6 ms. Memory during the Worker loop sat at 160–229 MB, the same range as the main-thread version. For a page people keep clicking around in while it works, that's the version I'd ship.

Zeroing the canvas early did not break the JPEG

I also tried zeroing too early on purpose: call convertToBlob, set paper.width = 0 on the next line, then await. The JPEG was still 1,530,557 bytes and decoded back to 3024×4032, identical to the normal order. That suggests Chromium copies the canvas when the call starts. I only checked Chromium 149, so in our code the zeroing stays after the await.

Where persist writes to

In our page persist will write into a folder from showDirectoryPicker({ mode: 'readwrite' }). According to MDN's compatibility data (2026-10-15) Firefox and Safari don't have that call, so on those browsers persist has to become a download instead, and the loop itself doesn't change. For one-off folders outside our tool I use ImgIng (https://imging.ai/) on the same Mac. Its batch panel says each photo is saved and released before the next one starts, which is the same order as this loop.

Before this goes into review I'm checking four things: every file sits inside its own try/finally, the save is awaited before the next decode, memory is read from the task manager rather than the heap panel, and the test folder includes at least one file that isn't really an image. How the loop behaves in Firefox and Safari I still don't know.

Computing a GIF's per-frame change box: one stray pixel keeps it at full canvas

hao jia — Thu, 22 Oct 2026 08:04:36 +0000

A teammate wanted to add a frame-differencing step to the job that turns our onboarding walkthroughs into GIFs, on the usual promise that storing only what changed roughly halves the file. Before agreeing to another build step I wanted a cheap per-frame number saying whether that step would have anything to remove: the bounding box of the changed pixels, and how much of that box didn't actually change. Getting it to measure the right thing took longer than writing it, and on noisy footage the box turned out to be set by a few stray pixels.

The samples are public, not customer screens: a line-art running horse from Wikimedia Commons (Janke, CC BY-SA 2.5, 307×230, 8 frames), a dim phone clip of a cat wagging its tail (Mateus2019, CC BY-SA 4.0, 320×240, 208 frames), and a camera pan I cut from the cat's first frame, a 240×180 window sliding 2 px right and 1 px down for 40 frames. Pillow 11.3.0 and numpy 2.0.2 on an M4 Mac.

Compare the frames a viewer sees

Seven of the horse's eight frames are stored as partial rectangles, so diffing raw frame data compares patches, not pictures. Pillow gives you the composited canvas when you seek and convert. I compare decoded RGB rather than palette indexes, because with per-frame local palettes one index can mean two colours. A pixel counts as changed when any channel moves by more than tol.

def change_boxes(path, tol=0):
    gif = Image.open(path)
    prev_rgb = None
    for k in range(gif.n_frames):
        gif.seek(k)
        stored = tuple(gif.tile[0].extents)       # read before convert(): loading empties .tile
        rgb = np.asarray(gif.convert("RGB"), dtype=np.int16)  # frame 0 is mode P, later RGB/RGBA
        if prev_rgb is not None:
            changed = (np.abs(rgb - prev_rgb) > tol).any(axis=2)
            if not changed.any():
                yield k, None, stored, 0.0, 0.0, 1.0
            else:
                rows, cols = np.nonzero(changed)
                box = (int(cols.min()), int(rows.min()), int(cols.max()) + 1, int(rows.max()) + 1)
                inside = changed[box[1]:box[3], box[0]:box[2]]
                yield k, box, stored, inside.size / changed.size, changed.mean(), 1 - inside.mean()
        prev_rgb = rgb

Two Pillow details broke my first version

Frame 0 decodes as mode P, a 2-D array of indexes, while later frames come back as RGB, or RGBA for the cat, so skipping the conversion died on frame 1 with a broadcast ValueError. The other trap: gif.tile holds the rectangle the file stores for the current frame, but convert() loads the frame and empties it, and reading it afterwards gives an IndexError that looks unrelated. The not changed.any() branch never fired here, but a recording paused on one page will hit it.

The original encoders already cropped as tight as possible

The first horse frame and the summaries at tol=0 (horse, cat, pan), trimmed from the real output:

f1   box=(5, 39, 282, 225) stored=(5, 39, 282, 225) area=73.0% moved=11.2% still_in_box=84.6%
7 diffs | area 73.6% | moved 12.8% | still_in_box 82.6% | full-canvas boxes 0 | no-change frames 0 | box==stored 7
207 diffs | area 100.0% | moved 91.4% | still_in_box 8.6% | full-canvas boxes 207 | no-change frames 0 | box==stored 207
39 diffs | area 100.0% | moved 82.0% | still_in_box 18.0% | full-canvas boxes 39 | no-change frames 0 | box==stored 39

The last column is the one I cared about: my box matched the rectangle stored in the file on every frame of all three samples. Whatever encoded them already cropped to the minimum, so a rectangle-only pass has nothing left to take.

One stray pixel keeps the box at full canvas

I raised the tolerance to get past the cat's sensor noise. At 16 only 11.3% of pixels differ from the previous frame, yet the box is the whole canvas in 177 of 207 comparisons, and the other 30 only shrank to 96.7%–99.6% of the frame. The pan, with 5.7% of pixels left, kept a full-canvas box every time. A bounding box stays maximal while anything changes near each edge. The horse's numbers didn't move between 0, 8 and 16; my guess is that 16 flat colours either match exactly or differ a lot. These tolerances compare against the real previous frame; an encoder reusing near-matches compares against what is already on screen, so its counts will differ.

The gate I kept

The box became a cheap skip check rather than a forecast. If it equals the stored rectangle on every frame or covers about the whole canvas, the lossless pass is skipped. Anything in between, like the horse's 73.6%, only earns a real encode, because the share of still pixels inside the box said nothing about bytes. 82.6% of the horse's box sat still, yet with one shared palette and my own encoder (weaker than whatever made the originals) the best lossless layout came to 97.5% of full frames. For files I'm not scripting, I use ImgIng (https://imging.ai/) in Chromium 149 on the same Mac; its animation tool runs only in desktop browsers, and Safari can't use it to compress an existing animated GIF. It switches between difference frames and full frames, and I don't know how it decides per frame, so I try both on the actual file. What I haven't settled is a tolerance that keeps our walkthroughs readable, since small UI text may break up well before 16.