DEV Community: Bowen Rui

I held a seance for ten dead PyPI packages

Bowen Rui — Tue, 27 Oct 2026 15:42:11 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

zipfile36 hasn't had a release since October 2016. Last month it was downloaded 1,178,018 times.

It isn't alone. I took the 15,000 most-downloaded projects on PyPI and asked PyPI when each of them
last uploaded a file. 427 haven't uploaded anything since before 2018, and people still
install them every day. Nobody is maintaining them, and Python keeps moving: distutils is gone, collections.MutableMapping is gone, imp is gone,
nntplib is gone.

So I went looking for the ones that died of it. A package counts as a ghost if its own test
suite, as shipped in its last sdist, passes on Python 3.8 and fails on Python 3.13. Of the 427, 45
still passed their own tests on 3.8 (at least three tests each). On 3.13, 35 of those still pass.
Ten don't.

pip-seance holds a seance for each ghost. A medium, which is a Mastra
agent running an open-weight model, gets a copy of the package and six small tools. It may read
anything and edit the library code. It may not touch the tests, which act as the judge. The
whole seance runs as a Temporal workflow, so if the machine dies mid-seance,
it continues somewhere else. At the end, the original tests run on Python 3.13 and on 3.8
(raising the dead shouldn't kill the living), and the package gets an epitaph.

Six of the ten rose. DeepSeek V4 Flash raised five in its first pass, for 27 cents across all ten
attempts, and
a sixth after I fixed a mistake in my own tooling. But the more interesting results are the ones
in between: a revival that wasn't one, a medium that found the cure and couldn't apply it, and a
kill switch that didn't kill.

It's Día de los Muertos this weekend, the time for remembering the dead and the work they left.
This felt like the right way to mark it as a programmer: these packages were written by people a
decade ago, and a million installs a month still lean on that work.

Demo

The graveyard: bowen1314.github.io/pip-seance

One headstone per ghost, generated from the results. Each badge is one seance and links to its
epitaph: before and after test counts, whether Python 3.8 still passes, the diff, the cost, and
the medium's last words. Here is
kafka's.

Code

Bowen1314 / pip-seance

Raising abandoned PyPI packages from the dead with open-weight models: Mastra agent + Temporal workflow, judged by their own tests

pip-seance

Raising abandoned PyPI packages from the dead with open-weight models.

Hundreds of packages on PyPI have not had a release in eight years and are still downloaded hundreds of thousands of times a month. Some of them quietly died when Python moved on: their own test suite passed on Python 3.8 and fails on Python 3.13. pip-seance finds those ghosts, then holds a seance for each one: a Mastra agent running an open-weight model edits the library code (never the tests) inside a Temporal workflow, and the package's untouched original tests decide whether it rose.

The graveyard (live page): https://bowen1314.github.io/pip-seance/

Results

Run on 27 October 2026. Full table with every run: results/SUMMARY.md.

The census. Of the 15,000 most-downloaded PyPI projects (hugovk/top-pypi-packages, 30-day window), 427 have not uploaded a file since before 2018. 245 of those don't ship their tests in the sdist, 109 couldn't pass at…

View on GitHub

Everything behind this post is in the repo: the census, every seance's transcript, diff and
epitaph, my review of each revival, the Temporal history of the chaos test, and a ledger with
every model call priced. seance/replay.py re-applies every recorded diff to a fresh copy of the
package and re-runs the judge, so you can check every number here without calling a model.

How I Built It

Finding the dead. The 15,000 most-downloaded projects come from
hugovk/top-pypi-packages, which is built from
PyPI's public download data. For each project, PyPI's JSON API gives the newest upload time across
all releases. Anything before 2018 is "undead": 427 packages. For each of those,
seance/probe.py downloads the latest sdist, checks its sha256, and runs the test files it
contains, unchanged, on CPython 3.8 and 3.13, with the same pytest 8.3.5 and nothing else besides
mock and six.

Most of the 427 dropped out for dull reasons. In 245 I found no test files in the sdist, 109
couldn't pass three of their own tests on 3.8 (77 of those stopped at an import error, usually a
test dependency like nose or Django that I hadn't installed), 16 have C code, and 12 have no sdist
at all. That left 45 that
were provably alive on 3.8, and 10 of them that are dead on 3.13. The tests that passed on 3.8 are
the target tests. A seance succeeds when every target test passes on 3.13.

One judgment call: Python 3.12 also deleted old unittest spellings like assertEquals. That breaks
the test file, not the library. Since nobody may edit tests, a small pytest plugin puts those
aliases back in every run, before and after, old Python and new. It can't make a broken library
pass, it just stops a test's spelling from counting as a death.

The medium: Mastra over open weights. Mastra already knows Nebius Token Factory as a
provider, so the model is a string and the key comes from NEBIUS_API_KEY:

const agent = new Agent({
  id: 'medium',
  name: 'medium',
  instructions: INSTRUCTIONS,            // ~20 lines: fix the cause, never the tests, keep 3.8 working
  model: `nebius/${modelId}`,            // e.g. nebius/deepseek-ai/DeepSeek-V4-Flash-0731
  tools,                                 // list_files, read_file, search, edit_file, write_file, run_tests
});

The six tools are built with createTool and jailed to the package directory. edit_file and
write_file refuse anything under test/ or tests/, any test file, conftest.py, and the
pytest/tox/setup config files. The verify step also hashes all of those against the pristine
copy afterwards. run_tests runs the original suite on 3.13 and tells the agent how many target
tests pass, which ones still fail, and the end of pytest's output. There's no shell and no network
tool, and the test subprocess gets an environment with the inference key and other API variables
stripped out. It isn't a sandbox, though: code the medium writes does run when the tests run,
which is one reason all of this ran on a batch node rather than my laptop.

Every model step lands in onStepFinish, where its token usage is priced from Nebius's live
/v1/models prices and appended to a ledger file before the next step starts. A run is capped at
4 cents and the whole project at $1.00.

The ritual: Temporal. Each seance is a workflow with four activities:

export async function seanceWorkflow(a: { project: string; model: string; run: string }) {
  const prep = await quick.prepare({ project: a.project, run: a.run });
  const result = await medium.seance({ project: a.project, run: a.run, model: a.model, root: prep.root, brief: prep.brief });
  const verdict = await quick.verify({ project: a.project, run: a.run, root: prep.root, pristine: prep.pristine });
  const attempts = await quick.countAttempts({ run: a.run });
  return await quick.epitaph({ project: a.project, run: a.run, model: a.model, seance: result, verdict, attempts });
}

The agent loop is one activity with a 90-second heartbeat timeout and three attempts. Workflow
ids are deterministic (bench-<model>-<package>), so if the client crashes, re-running it attaches
to the seances already in flight instead of paying for them twice. The dev server keeps its state
in a SQLite file, and everything ran as Slurm jobs on a university cluster, because a seance
executes a decade-old stranger's code and I wanted that off my laptop.

Then I killed it. scripts/chaos_job.sh starts a worker and a seance for money, waits until
the medium has made four model calls, and sends the worker SIGKILL. No cleanup, no warning. Ten
seconds later it starts a second worker. Here's what happened:

15:17:04  attempt 1 started (worker A)
15:17:11  agent finished step 4; SIGKILL worker A
15:17:28  worker B up
15:18:40  attempt 2 started (worker B)      <- Temporal: no heartbeat for 90 s, retry
15:18:55  money: REVIVED 20/20 on 3.13, 20/20 on 3.8, attempts 2

The ledger has both attempts' calls, 0.84 cents in total, because each step was written down
before the kill. (The epitaph's own cost row counts only attempt 2; the ledger is the full bill.) Attempt 2 was told that an earlier attempt had been interrupted and to start with
run_tests. The full event history is
in the repo.

The Seances

Medium (open weights, on Nebius)	Ghosts tried	Raised	Sound on my review	Left 3.8 broken	Cost
DeepSeek V4 Flash	10	5	3	2	$0.27
Qwen3-30B-A3B-Instruct-2507	10	1	1	1	$0.39
GLM-5.3-Flash	4 (the hardest)	0	0	0	$0.07
DeepSeek V4 Flash, round 2	2	1	0	1	$0.06

DeepSeek raised zipfile36, kafka, python3-ldap, null and money. Qwen raised null. Every raised
package still passes its tests on 3.8. Here's what the epitaphs say, beyond the table.

Green isn't the same as alive. I read every diff that turned the tests green. Three I would
merge as they are. kafka's vendored copy of six registers an import hook that only implements
find_module, which Python 3.12 stopped calling. DeepSeek added find_spec, create_module and
exec_module, which is the same shape as the fix six itself shipped in 1.16.0. python3-ldap and
null just needed collections.abc. The other three work, with a caveat. money's fallback for
distutils.version leans on packaging, which happens to be importable only because pytest
depends on it. And logutils' fix re-creates two private functions on the standard library's
logging module rather than in its own code.

zipfile36 was never dead. Its sdist carries its own copy of CPython's test.support, which
does import nntplib, and Python 3.13 removed nntplib. The library was fine; its test kit had
died. The medium first checked that the fence was real ("Let me check whether the test/support file
is truly read-only by attempting a small edit"), then wrote a 27-line nntplib.py stub so the old
test helper could import it. 168 out of 168, for one cent. That counts as raised by the rules, and
it's also the most honest result in the table: the most downloaded ghost wasn't a ghost.

The cure and the clumsy hands. cookies died because one of its regexes has (?ix), a global
flag group, in the middle of a pattern, which has been an error since Python 3.11. DeepSeek
diagnosed it correctly and turned it into a scoped group, (?ix:. Then it had to close the
parenthesis at the other end of a multi-line pattern, and all 8 of its next edits were rejected
because the text it quoted wasn't in the file. The indentation was off every time, and the last
five were each exactly two spaces too deep: remove two spaces per line and every one of them
matches. My guess is that it was counting the two-space gutter after my read_file line numbers. The budget
ran out with the parenthesis still open, and now the package didn't even import on 3.8. A failed
seance can leave the body worse than it found it. That's why each one works on a copy and checks
3.8 at the end.

The medium is only as good as what it can see. logutils died because Python 3.13 removed
logging._acquireLock. In the first bench no model found that, and none of them edited a single
file. The reason was my tool: run_tests led with No module named 'redis', from a test that had
never been a target, and the real AttributeError had been cut from the tail I passed back. After
I changed run_tests to list each failing target with its own error message, DeepSeek raised
logutils in 10 steps for 1.6 cents. Because that's a different tool, I report it as round 2, not
mixed into the bench.

The kill switch didn't kill. The per-run cap originally called abort() from inside
onStepFinish. The bench showed that this doesn't end Mastra's loop: one Qwen run kept going to
$0.098 against a $0.04 cap. The cap is now a stopWhen condition, and every tool refuses work once
the run is over budget. I re-ran the same seance with the cap at $0.004 and it stopped at $0.0045,
one step over. The project-wide $1.00 cap held throughout, because it's checked before each seance
starts.

GLM never got to speak. GLM-5.3-Flash is a reasoning model. In three of its four seances it
spent the whole 4,096-token output budget thinking and ended with an empty reply. That row says
more about my maxOutputTokens than about GLM.

Small isn't free. Qwen3-30B-A3B has only 3B active parameters, small enough to run on a good
desktop GPU. It ran 40-step loops of read_file and search, and once it wrote its tool call as plain
text instead of calling the tool. It cost more than DeepSeek and raised one package.

Why Does Open Innovation Matter?

This project only works because of open source, starting with the dead.

The ghosts are open. Every one of these packages can still be raised because its source
and its tests are still sitting on PyPI, eight to thirteen years after the last release, and the
license lets anyone fix it. A closed SDK whose vendor walked away in 2017 is just gone. The
nntplib stub, the find_spec fix and the collections.abc imports are all possible because
someone published the code.

The mediums are open, and they'll still exist. I compared three open-weight models from three
labs by changing one string, nebius/<model id>. The weights for every model in my table are
published, so this bench can be re-run against the same models in five years, or on my own
hardware if the provider disappears. That's the point of an experiment about software that
outlived its maintainers: I didn't want the medium to be the next thing that vanishes.

Cheap enough to try on everything. DeepSeek V4 Flash attempted all ten ghosts for 27 cents,
about 5 cents per package raised. The whole project, including smoke tests, the cap test, the
chaos test and round 2, cost $0.83 across 523 model calls. At that price you can hold a seance
for every dead dependency in your lockfile and only read the diffs that come back green.

Open frameworks let me see inside. Mastra ships its documentation and provider registry
inside the npm package. That's how I found that Nebius was already a provider and which model ids
it knew. Temporal's dev server is a single open-source binary with no account, so the durability
test ran inside a batch job on a shared cluster. When something misbehaved, like the cap that
didn't stop the loop, I could read the types and fix my side instead of filing a ticket and
waiting.

What I'm not claiming: this isn't offline or private. The code went to a hosted inference
endpoint, as it would with a closed API. I also didn't run a closed model, so I can't tell you
open models are better at this. My claim is narrower: for this job, a model that costs about a
cent per attempt raised more than half the dead, and every part of the pipeline, from the ghosts
to the medium to the ritual, is something I could read, run and fix myself.

My Agent Session

Instead of DevRelay, every seance's full session is in the repo as transcript.jsonl: each step's
tool calls, tool results and tokens. Three worth reading:

kafka, DeepSeek: 41 of 41 passing at step 8 for one cent. It then spent three more cents reading test files to make sure a leftover SyntaxError in a non-target test wasn't its fault, until the cap stopped it. Conscientious, and four times the price.
zipfile36, DeepSeek: tries the read-only fence once, accepts it, writes the stub.
cookies, DeepSeek: right diagnosis, eight rejected edits, one open parenthesis.

The agent that built pip-seance itself was Claude Code, working autonomously from a task brief.

The Spirit of It

The moment it landed for me was kafka. 41 tests, zero passing on Python 3.13, the whole suite
failing at import with No module named 'kafka.vendor.six.moves'. Eight model calls later, all 41
passed on 3.13 and on 3.8, with the same shape of fix the six maintainers shipped for their own
library in 2021. The original authors moved on years ago, and the package is still installed
239,184 times a month. A one-cent seance gave it back to the people still using it.

None of these revivals are going upstream automatically, and they shouldn't. The epitaphs show
why: a green run can be a stub, a leaky fallback, or a fix in the wrong place. pip-seance doesn't
take over the job of maintaining these packages. It finds the dead, says what killed them, and
hands you a candidate fix with the evidence attached. If you depend on one of these ten, the
epitaph is a good place to start.

Prize Categories

Best Use of Mastra. The medium is a Mastra Agent with six createTool tools, running
open-weight models through Mastra's model router (nebius/<model id>, with the key read from
NEBIUS_API_KEY). onStepFinish drives the live cost ledger, and stopWhen enforces the
per-seance budget. The same agent ran DeepSeek V4 Flash, Qwen3-30B-A3B and GLM-5.3-Flash with no
code changes. Notes from the build that might help the Mastra team:

Calling abort() on the abortSignal from inside onStepFinish didn't end a generate() loop for me (one run went 2.4x over its cap). A stopWhen predicate did.
With a reasoning model and maxOutputTokens: 4096, three runs ended on a step that used the full 4,096 output tokens and returned no text, so the loop simply stopped. A warning when a step's output is all reasoning would make that easier to spot.
The bundled provider registry lists Nebius models but not every current one (no MiniMax-M3.1-Flash, for example), so I picked models that were listed.

Best Use of Temporal. Each seance is a Temporal workflow: a heartbeating agent activity with
retries, then verify and epitaph activities. Deterministic workflow ids keep a crashed client from
paying twice. In the chaos test, the worker was SIGKILLed mid-seance, and Temporal handed the
activity to a new worker after the heartbeat timeout. That worker finished the seance in the same
workspace and raised money 20/20. The event history and the exact cost of both attempts are in
the repo.

Built in one day, 27 October 2026, inside the challenge window. Numbers are from
results/SUMMARY.md and
ledger/spend.jsonl.
Package list: hugovk/top-pypi-packages.

I hired a sloth for the night shift

Bowen Rui — Sat, 24 Oct 2026 15:57:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Somewhere in your repo there is a comment like this one:

# TODO: blows up with a traceback on a fresh install (empty ledger).

It has been there for months. Fixing it would take ten minutes. You will not fix it today, because
today there is always something more interesting. Same goes for the deprecation warning that
prints on every run, the README that drifted three releases ago, and the function with no tests
"because it's simple".

nightshift-sloth is built to do one of those chores every night while I sleep. It picks the next open
item from a CHORES.md checklist and hands it to Backboard R-CLI,
the open-source terminal coding agent, running headless on a cheap open-weight model. The agent
works on a throwaway branch with a strict time budget, then the harness runs the tests and
writes a morning report. It never merges and never pushes. In the morning I read one page, and
either keep the branch or delete it.

To find out whether that holds up, I gave four open-weight models the same six chores on a small
demo app, one run each. That is 24 nights' worth of work, which I ran back to back in one morning.
All 24 together cost 39 cents. MiniMax M3.1 Flash got
all six right. DeepSeek V4 Flash got five right for about a quarter of a cent each. And the most
useful thing I learned was not about models at all: green tests are not the same as done.

Who gets to be lazier: anyone who maintains a small project and has a list of chores that are
too boring to start and too small to delegate. The chore still gets a human review, but reviewing
a 6-line diff with green tests takes a minute. Doing it takes a context switch I never make.

Demo

One take, nothing mocked. Bedtime is one command. The sloth takes the first open chore, DeepSeek V4
Flash fixes it in 9 seconds for $0.0016, the tests go from 12 to 14, and the work sits on its own
branch while main stays where it was. Here is a
full morning report
from the bench.

Code

Bowen1314 / nightshift-sloth

One chore a night: Backboard R-CLI + open-weight models on a throwaway branch, with a morning report. Never merges, never pushes.

nightshift-sloth

One chore a night. While you sleep, an open-weight model picks the next item off your CHORES.md, does it on a throwaway branch with a strict time budget, runs your tests, and leaves a morning report. It never merges, never pushes, never touches main. You read the report over coffee and decide: keep it or bin it.

Built on Backboard R-CLI (MIT), the open-source terminal coding agent, running headless against open-weight models on Nebius Token Factory. No Backboard account needed.

Results: four open-weight models, six chores

Each model took each of the six chores in the demo repo once: 24 nights of work, run back to back on 24 October 2026. A chore counts as done only if the repo's tests pass and a hidden acceptance check, which the agent never sees, passes on the sloth branch.

Model (on Nebius Token Factory)	Done	Tests green	Median

…

View on GitHub

Everything the sloth did for this post is in the repo: each run's morning report, its diff, the raw
R-CLI event stream, a readable transcript, and a spend ledger with every model call priced.

How I Built It

The agent: Backboard R-CLI, headless. R-CLI is an MIT-licensed terminal coding agent. Its
headless mode is what a night shift needs:

backboard --cwd <clone> --model nebius/deepseek-ai/DeepSeek-V4-Flash-0731 \
  --print "<the chore>" --timer 600 --format json --permission-mode bypass \
  --memory off --no-lsp --excluded-tools WebSearch,FetchUrl

--format json streams every event (tool calls, results, token usage) as JSON lines. --timer
tells the model how long it has and reminds it as the clock runs down.

The models: open weights on Nebius Token Factory. R-CLI can talk to any OpenAI-compatible
endpoint you define in ~/.backboard/config.json. The key stays in an environment variable:

{ "providers": [{
    "id": "nebius", "protocol": "openai-chat",
    "baseUrl": "https://api.tokenfactory.nebius.com/v1",
    "auth": { "type": "env", "variable": "NEBIUS_API_KEY" },
    "extraArgs": { "stream_options": { "include_usage": true } },
    "models": [{ "id": "deepseek-ai/DeepSeek-V4-Flash-0731", "contextLimit": 131072 }]
}] }

The include_usage line matters. Without it a custom provider's stream carries no token counts,
and you can't put a price on a run.

I benchmarked four models from four labs: DeepSeek V4 Flash, MiniMax M3.1 Flash,
Qwen3-235B-A22B-Instruct-2507 and OpenAI's open-weight gpt-oss-120b. GLM-5.3-Flash also passed my
smoke test; I left it out of the full run to keep the bench small.

The harness: a short leash. R-CLI runs with --permission-mode bypass, because nobody is awake
to approve prompts. That is only acceptable because of what surrounds it. nightshift-sloth is about
1,500 lines of standard-library Python that:

makes a private clone holding only the base commit's history, on a fresh sloth/<chore>/<stamp>-<model> branch, with no remote. The agent can't see my other branches or move my refs. When the night is over, the harness fetches that one branch back;
gives R-CLI a throwaway HOME whose config knows exactly one provider, and an environment scrubbed down to PATH, locale and that one key. My shell has other API keys in it; the agent never sees them;
writes R-CLI deny rules for git push, git commit, branch switching, gh, curl, package installs and environment dumps. I read R-CLI's permission engine to check that deny rules outrank bypass mode, and the bench confirmed it: one agent tried git branch --all and was refused;
prices every usage event as it streams in, appends it to a ledger, and kills the run when it reaches its dollar cap (25 cents). The ledger also refuses to start a run that could push the project past its total cap (mine was $3);
kills the whole process group at timer + 2 minutes, because R-CLI's --timer is advisory: the model is told about it, but nothing gets cancelled;
stops a run after six tool errors in a row (more on why below);
runs the tests before and after, makes the one local commit itself, and writes the morning report.

It is not a sandbox. The agent can still run shell commands as my user. I treat it like a junior
colleague with no push access, not like a VM.

Measuring honestly. "Tests pass" means the agent graded its own homework. So for the bench every
chore also has a hidden acceptance check. It lives outside the repo, runs after the agent has gone,
and was validated against the untouched repo (must reject) and a reference solution (must accept).
For example:

test-parse-amount asks for unit tests without changing the function. The check confirms money.py is byte-for-byte unchanged, then runs the new tests against four mutants of parse_amount. The tests have to catch all four.
readme-usage-drift imports the real argparse parser and checks that every subcommand and every option shows up in the README's Usage section, and that the other sections were left alone.
utcnow-deprecation builds a ledger that mixes old naive timestamps with new timezone-aware ones, exactly as the chore describes, and runs every command on it.

The first wall, and the way round it. I started with the R-CLI binary from the official
download link (3.0.5 for macOS arm64, fetched on 24 October 2026). It refused my Nebius provider
with "No credentials found". That build's bring-your-own-key list is Anthropic, OpenAI, Google and
OpenRouter, and it has no --timer flag. Because R-CLI is open source, I could go and look: custom
providers and the timer had both landed on main two weeks earlier, in
51bee17
("Add per-turn timer and todo budget feedback"). I built that commit with Bun and everything
worked. One oddity if you try it: that main build calls itself 3.0.4, one below the release.

Why Does Open Innovation Matter?

Here is what open source made possible this week that a closed API or a closed agent would not have.

When the release was missing a feature, I built main. With a closed agent, "the shipped
build doesn't support my provider" ends the project. Here it cost a git checkout and
bun run build. Reading the source also told me things no doc did: that --timer doesn't cancel
anything, that deny rules beat bypass mode, and which event carries token usage. The harness's
safety and its cost meter are built on those three facts.

Swapping models is one line. The same agent, prompt and checks ran against four models from four
labs. Nothing in the harness is model-specific, and the provider is two lines of config. When a
better or cheaper open-weight model ships next month, I change one line and re-run the bench.

The price makes "every night" sensible. DeepSeek V4 Flash spent 1.3 cents on all six chores and
finished five of them. At a quarter of a cent per finished chore, a chore every night for a year costs
less than a dollar. MiniMax, the only model to get all six right, works out at about $4 a year. The
number that matters on the price page turned out to be the cached-input price: an agent re-sends the
same growing conversation on every call, so 70% to 98% of each model's input tokens were cache hits.
DeepSeek bills those at $0.01 per million tokens. Qwen gets no cache discount on this provider,
which is most of why it cost seven times as much as DeepSeek for the same six chores.

No account, no lock-in. I never created a Backboard account for this. The agent is MIT-licensed,
the models are open weights behind an OpenAI-compatible endpoint, and the only key is for that
endpoint. If my provider vanished tomorrow I would change base_url and keep going.

What I am not claiming: this is not offline or private. The code goes to a hosted inference
endpoint, as it would with a closed API. I also didn't run a closed model, so I can't tell you open
models are better at this. My claim is narrower: for a small chore, a model that costs a fraction of
a cent did the job, and when the tooling got in my way, open source let me fix it myself instead of
waiting.

My Agent Session

R-CLI keeps its own session log, and nightshift-sloth saves it for every run, so instead of DevRelay
I am linking the night-shift sessions directly. Each folder in
docs/runs has a readable
TRANSCRIPT.md next to the raw events.jsonl. Four are worth reading:

Green but wrong. DeepSeek V4 Flash on utcnow-deprecation:
four model calls, eight seconds, a two-line diff and a green suite. The chore said that old ledgers
hold naive timestamps and that list, report and export must keep working on a ledger mixing
old and new entries. DeepSeek reasoned about it instead of trying it, and its summary says
"downstream code (list, report, export) only reads e.when/sorts, so mixed ledgers keep
working." The hidden check tried it: all three commands crashed comparing naive and aware datetimes.
Qwen made the same two-line change.

Slow and right. MiniMax M3.1 Flash on the same chore:
twelve calls and almost two minutes. It normalised old timestamps to UTC on load, wrote tests for a
mixed ledger, and ran the suite a second time with -W error::DeprecationWarning, which nobody asked
it to do. It was the slowest model in the bench and the only one to finish all six chores.

The loop. gpt-oss-120b on fix-empty-report:
the fix was in place after 26 seconds, and it passes the hidden check. Then the model kept going:
92 model calls, and 79 of its 93 tool calls failed, with missing arguments, an empty tool name, and
harmony <|channel|> tokens leaking into tool names. It stopped when the provider returned an
error, at $0.14, ten times what DeepSeek spent on all six chores. That run is why the harness now stops
after six tool errors in a row. gpt-oss produced malformed tool calls in every run. I didn't work out
whether the model, the provider's translation layer or the agent's OpenAI adapter is at fault, so I
read its column as "this combination doesn't work yet", not as a verdict on gpt-oss.

The peek. An excluded run.
The first version of the harness used git worktree, which shares refs with your repo. Early in
the bench, MiniMax ran git log --oneline --all, found DeepSeek's attempt at the same chore
from a few minutes earlier, and read it with git show. A night-shift worker reading the day
shift's notes is fine; a bench contestant reading another's answer is not. So now every night works
in a private clone that holds only the base commit's history. I kept the original run as evidence,
re-ran it in a clean clone (MiniMax still got it right, in half the time), and counted only the
re-run.

The Lazy Payoff

Six chores on the demo app, and here is how the night shift did:

Model	Done	Tests green	Median time	Cost, all 6	Per done chore
MiniMax M3.1 Flash	6/6	6/6	50s	$0.068	$0.011
DeepSeek V4 Flash	5/6	6/6	13s	$0.013	$0.0026
Qwen3-235B-A22B-Instruct-2507	3/6	6/6	16s	$0.093	$0.031
gpt-oss-120b	3/6	3/6	11s	$0.216	$0.072

"Done" means the repo's tests pass and the hidden check passes. Every chore was done by at least
one model, so every chore now has a branch I would merge. Four branches had green tests and were
still wrong: you can see them in the gap between the "Done" and "Tests green" columns. The
chore-by-chore table
links every morning report.

The whole project, including smoke tests, the bench, one aborted run, the excluded run and the demo
recording, cost $0.45 according to the
ledger. That's 15% of
the $3 cap I set before starting.

Two honest caveats. This is one run per model per chore, so treat it as anecdotes with receipts, not
a benchmark. And the demo app and its chores were written for this project; the hidden checks were
validated before any model saw the chores, but this is not a year of nights on a real codebase.

What I took away is the reason the sloth never merges. If I had run only DeepSeek, five chores
would have been done for a little over a cent, and the sixth would have arrived marked READY FOR
REVIEW with green tests and a bug in it. The cheap model is good enough to do the work. It is not
good enough to skip reading the diff. That suits me: reading one short diff over coffee is the kind
of slow and steady I can keep up.

Prize Categories

Best Use of Backboard. nightshift-sloth is built on Backboard R-CLI as a component, not a chat
window: headless --print, the --format json event stream for live cost metering, --timer,
custom OpenAI-compatible providers and project deny rules. The bench compares four open-weight
models through the same R-CLI build. I did that through R-CLI's custom-provider config with my own
inference key, not a Backboard API key; I didn't create a Backboard account.

Things I ran into that might be useful to the R-CLI team:

The binary from the download link (3.0.5) is behind main: no custom providers, no --timer. And main reports itself as 3.0.4.
Custom providers report no token usage unless stream_options.include_usage is set in extraArgs. Turning that on by default for openai-chat providers would make cost tracking work out of the box.
--timer is advisory. A hard wall-clock limit or spend limit for --print runs would make unattended use safer; I had to build both outside the agent.
With gpt-oss-120b, malformed tool calls repeated until the provider gave up. An option to stop after N consecutive tool errors would have saved most of that run's cost.

AI use: I built nightshift-sloth with an AI coding assistant (Claude Code), which wrote most of the
harness, the demo app, the hidden checks and a first draft of this post, working to my brief. The
night-shift work in the bench was done by R-CLI with the open-weight models named above, and every
one of those sessions is in the repo.

chaptermill: a podcast's back catalog to show notes for 3¢ an audio hour, on an old PC I kept crashing on purpose

Bowen Rui — Thu, 15 Oct 2026 20:19:46 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Most podcasts that have been running for a few years have the same problem: hundreds of old episodes with no transcript, no chapters and no show notes worth reading. That archive is what new listeners stumble on through search, and it is the part nobody wants to pay per-hour SaaS prices to fix.

chaptermill takes a podcast's RSS feed and, on a machine you already own, turns the back catalog into the files a producer actually delivers:

a timestamped transcript (SRT, VTT, JSON),
chapters as a Podcasting 2.0 chapters.json and as YouTube chapter lines,
2–4 pull quotes, each checked word for word against the transcript before it is accepted,
Markdown show notes with a summary, topics and a credit line.

Who pays for it: freelance podcast producers (show notes are a standard deliverable) and independent shows with a big archive. The hosted tools price by the audio hour. On their pricing pages today, Castmagic's Hobby plan is $19/month billed annually for 30 transcribed hours in total, and its Team plan, 400 hours, is $139/month billed annually; Podsqueeze's Agency Lite plan is $89/month for 600 minutes per month. A 300-episode archive of half-hour shows is 150 hours. With chaptermill, at the rate of my demo run, the same archive costs about $4.40 in open-weight model calls and a day of runtime on a 2015-era desktop. The side hustle writes itself: a flat-fee "archive pass" for small shows, where the marginal cost is cents. For scale: Hacker Public Radio, whose episodes I used for the demo, numbers its episodes past 4,700. At the average length of the nine I processed (28.5 minutes) that is roughly 2,250 hours, or about $65 of model calls at my demo run's rate. That is an estimate from a sample of nine, not a quote.

To be fair to the incumbents: Apple Podcasts and Spotify now generate chapters for some shows inside their own apps. That helps listeners on those apps. It does not give the producer a file to put in the feed, on YouTube or on the show's website, and it does not touch the archive in one go.

Demo

Live: https://bowen1314.github.io/chaptermill/

Nine recent episodes of Hacker Public Radio (a community podcast released under CC BY-SA 4.0), processed on my home server (one run, plus a re-run of two episodes that I explain below), with nothing edited by hand. Pick an episode; press a chapter or a quote and the player jumps there; open the full transcript and the current line follows the audio. Every file is downloadable.

Further down the page is the part I am proudest of: the report from a second run during which a script SIGKILLed the worker 9 times and the Temporal server once, while the worker injected model-API 503s, cut-off downloads and failing tool calls on purpose.

Code

Bowen1314 / chaptermill

Turn a podcast back catalog into transcripts, chapters, verified pull quotes and show notes with open-weight models; Temporal makes it survive crashes.

chaptermill

Point it at a podcast RSS feed and leave it running on an ordinary PC. It turns the back catalog into transcripts, chapters, pull quotes that are checked word for word against the transcript, and show notes ready to paste (the demo run did 4.3 hours of audio in 44 minutes on a 2015-era desktop). Open-weight models do the work. Temporal makes it durable: a crash, a dropped connection or a rate-limited model API means it carries on from where it stopped, without redoing finished work or re-buying model replies it already has.

Live demo: https://bowen1314.github.io/chaptermill/ (nine Hacker Public Radio episodes, plus the report from a run where I kept killing the worker and the Temporal server)

Built for the DEV Hacktoberfest Open-Source AI Challenge, week 2 (Side Hustle), Temporal category.

Who it is for

Freelance podcast producers who write show notes for clients, and independent shows sitting on…

View on GitHub

MIT licensed. About 1,900 lines of Python plus 560 lines of tests, including workflow tests on Temporal's time-skipping test server.

How I Built It

Three open pieces do all the work:

Piece	What it does	License
faster-whisper `base.en`	transcription, locally, on 2 CPU threads	MIT
Qwen3-235B-A22B-Instruct-2507	the show-notes agent, through Nebius Token Factory's OpenAI-compatible API	Apache-2.0
Temporal (dev server + Python SDK)	durable orchestration, retries, signals, queries	MIT

The machine is a shared home server: an Intel i3-6100 (2 cores), 3.8 GB of RAM and one spinning disk, also running a dozen other services.

The shape of it

A CatalogWorkflow reads the feed and starts one child EpisodeWorkflow per episode, two at a time, inside a dollar budget. Each episode workflow downloads the audio, cuts it into 5-minute chunks (one activity per chunk, on its own task queue so Whisper never starves the rest), merges the transcript, then runs the agent loop and writes the files.

The agent gets a one-line-per-minute outline and three tools: read_transcript(start, end), search_transcript(query) and submit_notes(...). Every model call and every tool call is a separate Temporal activity, so each one has its own timeout and retry policy, and a crash in turn 6 resumes at turn 6:

for _turn in range(agent.MAX_TURNS):
    await self._wait_for_cap()            # durable wait for a raise_cap signal if over budget
    resp = await workflow.execute_activity_method(
        Activities.llm_turn,
        LlmRequest(model=inp.model, messages=messages, tools=agent.TOOLS, ...),
        start_to_close_timeout=timedelta(minutes=3),
        heartbeat_timeout=timedelta(seconds=30),   # a dead worker is noticed in 30 s
        retry_policy=LLM_RETRY,                    # 429/5xx retried, other 4xx fail fast
    )
    messages.append(resp.message)
    for call in resp.message.get("tool_calls") or []:
        if call["function"]["name"] == "submit_notes":
            v = await workflow.execute_activity_method(Activities.validate_notes, ...)
            out = "Accepted." if v.ok else "Rejected. Fix these and call submit_notes again:\n- " + "\n- ".join(v.errors)
        ...  # read_transcript / search_transcript: one activity each
        messages.append({"role": "tool", "tool_call_id": call["id"], "content": out})

Audio, transcripts and chunk results stay on disk; workflow history carries file paths and the agent's conversation, nothing bigger.

The model is checked, not trusted

submit_notes does not end the loop. A validator compares the submission with the transcript, and if anything is off, the error goes back to the model as the tool result. In the demo run it rejected 4 submissions; the model's next submission fixed each of them. Two of them are worth showing, because they are opposite failures:

HPR4749: the model's first quote silently cut a sentence out of the middle ("And from the website, I'm gonna quote it here.") and stitched the rest together. Rejected; the resubmission quoted the host exactly.
HPR4747: the model wrote "in place of cat", which is what the host said (the Unix command). Whisper had transcribed it as "CAD". The validator rejected the correct word, and the final quote says "CAD".

That second one is the honest limit of this design: quotes are guaranteed to match the transcript, not the audio. A custom vocabulary for Whisper is the obvious next step.

Making it durable, for real

Temporal's AI cookbook patterns map almost one to one onto this pipeline:

Retries in one place. The HTTP client's own retries are off. Errors are classified: 408/409/429/5xx are retryable, other 4xx fail fast, and a Retry-After header becomes the activity's next_retry_delay.
Idempotent model calls. Each reply is written to a ledger and to a cache keyed by a hash of the request before the activity returns, so if the worker dies a millisecond later, the retry reads the cache instead of buying the reply again.
Chunks as checkpoints. A finished chunk is a file. Whisper runs at temperature 0, so a re-run chunk produces the same text.
Heartbeats on every long activity, including a side thread that heartbeats while the CPU activity is stuck loading the model.
Budget as a durable wait. Episodes that would go over the run's budget wait in workflow.wait_condition until a raise_budget signal arrives, for as long as it takes; pause/resume signals and a progress query come for free.

Then I tried to break it. scripts/chaos.py started a three-episode catalog and, every 45 to 110 seconds, SIGKILLed the worker, kept it down for 5 to 20 seconds and started it again; on the third kill it took the Temporal server down too. On top of that, the worker failed 30% of model calls with a fake 503, cut off 50% of audio downloads and broke 15% of transcript tool calls, each only on an activity's first two attempts. The result:


Kills	9 SIGKILLs of the worker at random moments, one of them together with the Temporal server
Episodes finished	3 of 3, in 12.5 minutes, with no manual intervention
Transcripts vs. the undisturbed run	identical, segment for segment (584 segments)
Activity attempts	88 for 67 activities: 8 retried transcription chunks, 8 retried model calls, 3 retried downloads, 2 retried tool calls
Model replies received	18, none bought twice
Requests in flight at a kill	1: its reply died with the process and the retry asked again (the provider may have billed both)
Fake 503s absorbed	6
Model cost	$0.021

Each kill cost at most the one chunk or model call in progress. The unluckiest chunk, in HPR4740, was killed three times and finished on its fourth attempt; nothing before it was redone.

The box itself added chaos I had not planned. During the clean run, another service on the same disk was writing about 16 MB/s, and Linux reported I/O pressure between 40% and 87%. The Temporal dev server keeps everything in SQLite, so every history write queued behind that disk, and a few tasks were handed to my worker after the worker had stopped waiting for them. Temporal's answer was a timeout and a retry; the run finished anyway, with 20 workflow-task timeouts in its history. Halfway through I restarted the server with SQLite in WAL mode and synchronous=normal (still safe when the process is killed, which is the failure I care about here): before the switch 19 of 78 workflow tasks timed out, after it 1 of 222. Not a controlled experiment, but a big enough difference that the chaos script now starts the server that way. I also learned not to fsync inside an async activity: on that disk a single fsync blocked the event loop the workflows share for seconds. Disk-heavy activities now run as sync activities on a thread pool.

Numbers from the demo run


Episodes	9 Hacker Public Radio episodes, 4.3 h of audio, 38,318 words
Wall clock	44 minutes, two episodes at a time
Transcription	26 minutes for 4.3 h of audio (9.8× real time), Whisper `base.en` on 2 CPU threads
Agent	72 turns, 91 tool calls, 4 rejected submissions, all fixed
Model replies paid for	74, none bought twice: $0.125 in total, $0.029 per audio hour
Difference from HPR's own transcripts	12.6% of words (HPR's are also made with Whisper and not proofread, so this is two machine transcripts disagreeing, not an error rate)

One honest miss: the first run finished 7 of 9. On the two longest interviews the agent spent all ten of its turns reading the transcript window by window and never called submit_notes. I added a "three turns left, submit now" message and re-ran just those two. Every transcript chunk and the first nine model replies of each came straight from cache; only the last turn was new, at about a third of a cent each. The $0.125 above includes the wasted turns.

Why Does Open Innovation Matter?

Cost. The expensive half of this job, turning audio into text, costs nothing per minute when the model runs on your own CPU. OpenAI lists its Whisper API at an estimated $0.006 per minute, $0.36 per audio hour; my entire language-model bill was $0.029 per audio hour. Open weights move the dominant cost from "per minute, forever" to an old PC's spare time.

Privacy. The audio never leaves the machine. Only transcript text goes to the model endpoint, and because the model is open-weight, a producer whose clients will not allow even that can point chaptermill at their own server. (I want to be precise: for the demo I used a hosted endpoint; I did not run the 235B model at home.)

Reproducibility and no lock-in. The Whisper weights are a file on my disk that nobody can update or retire under me, so running an episode again gives the same transcript; the chaos run shows it, with transcripts identical segment for segment to the undisturbed run. The agent talks to any OpenAI-compatible server, so switching model or provider is a config change, not a rewrite. I did try Gemma 3 27B first and it ignored the tool definitions on that endpoint, so "any" means any model whose tool calling works.

Fine-tuning for a niche. The "cat"/"CAD" miss above is exactly the kind of error open weights let you fix yourself. A producer whose shows are full of jargon (Unix commands, drug names, player names) can fine-tune base.en on a few hours of their own corrected transcripts, or at least hand Whisper a vocabulary prompt, and run the result on the same CPU. A closed transcription API may let you pass a prompt, but you cannot retrain it on your show. I have not done this yet; it is next on the list.

The orchestration is open too. Temporal's dev server ran on the same old box, with a SQLite file, no account and no cloud.

Prize Categories

Temporal. The category asks for an agent that "survives failures, retries flaky tool calls, and picks up where it left off". chaptermill's agent is a Temporal workflow where every model call and tool call is an activity, and the demo page shows the evidence from a run where I killed the worker 9 times and the server once.

Which trees near me are turning this weekend? Forecasting fall color from 6,650 iNaturalist observations with TabPFN

Bowen Rui — Tue, 06 Oct 2026 19:55:14 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

Built with PriorLabs-TabPFN.

What I Built

Fall color doesn't show up on a fixed date. It comes when the nights turn cold, and every fall turns cold on its own schedule. Foliage maps can tell you that a region is "near peak". They can't tell you whether the red maples on your usual walk will be red on Saturday, or whether you should wait another week.

peakweek answers that question for the trees around you. You type a town in the US Northeast or Mid-Atlantic, wait about a minute, and get one card for the coming weekend:

for each of 8 common fall-color trees (red maple, sugar maple, sweetgum, northern red oak, American beech, black gum, sassafras, Norway maple), the chance that a tree of that species you come across will be green, in color, or bare;
a verdict per species (go, starting, wait, late) and the day in the next two weeks with the best chance of color;
one line on how to recognize each tree, so you know which one you're looking at once you're out there;
one plain sentence at the top that tells you what to do.

Here is the headline for New Brunswick, New Jersey, from a forecast made on Tuesday, October 6, for the weekend of October 10–11:

No tree here is in full color yet this weekend, but sassafras, black gums and sugar maples are starting to turn; come back around Mon, Oct 19 for northern red oaks.

And here it is for Burlington, Vermont, from the same day:

This weekend, go see red maples, sugar maples and black gums; come back around Tue, Oct 20 for northern red oaks.

Then the screen gets out of the way. Print card gives you a one-page, black-and-white card to take with you. Done, go outside shrinks the page to that one sentence. The footer asks for one thing after your walk: add what you saw to iNaturalist with the "Leaves" annotation. Those annotations are exactly what the model learns from, so every walk makes next fall's forecast a little better.

It's for anyone who likes to go look at leaves and would rather spend that time outside than refreshing a foliage map.

Demo

This is a real run, searching Burlington, Vermont: the server runs TabPFN on a 4-core CPU with no GPU, and the 100 seconds it spent on this forecast (the first one after the server started) are sped up 10x in the GIF. Full video.

On a phone, Use my location near New Brunswick gives the "not yet" version of the card:

And this is what Print card produces, one black-and-white page:

Try it: bowen1314.github.io/peakweek. A forecast takes about a minute of CPU, so instead of a server there's a GitHub Actions job: every morning it fetches the weather and runs TabPFN once for each 1-degree grid cell of the region that has training data (103 cells), and GitHub Pages serves the cards. The page shows the card for the cell around the place you search, made at that cell's weather point, so it's a 1° approximation (a cell is about 110 × 80 km), and every card says which cell it's for and the day it was made. Before anything is published, the build re-runs a few cells through the ordinary single-point forecast and stops unless the numbers match. For your exact spot, run it locally. The repo also includes saved real forecasts for New Brunswick, Burlington and Pittsburgh, so you can open the full UI without installing the model:

python3 server.py --fixture examples/forecast_new_brunswick.json   # http://127.0.0.1:8770/
python3 -m peakweek --fixture examples/forecast_burlington.json      # the same card, in the terminal

Code

Bowen1314 / peakweek

Which trees near me are turning this weekend? Fall color forecasts from iNaturalist + ERA5 weather with TabPFN

peakweek

Which trees near you are turning color this weekend, and which day should you go out?

Built with PriorLabs-TabPFN. peakweek forecasts fall color for 8 common trees in the US Northeast and Mid-Atlantic from 6,650 iNaturalist community-science observations and this season's weather, using the open TabPFN v2 weights on a plain CPU. The answer is one card you can print, and then you go outside.

The waiting part (about a minute of TabPFN on a 4-core CPU) is sped up in the GIF. Full video: docs/demo.mp4.

Live site

https://bowen1314.github.io/peakweek/: search a town or press Use my location; nothing to install.

Updated daily. Every morning (around 10:20 UTC, about 6:20 in New York) a GitHub Actions job (.github/workflows/pages.yml, scripts/build_site.py) fetches this season's weather from Open-Meteo, runs TabPFN once on the runner's CPU for every grid cell below, and publishes the cards to GitHub Pages…

View on GitHub

The app itself (server, page, card, verdict logic) uses only the Python standard library. The model side needs tabpfn and a CPU build of PyTorch. MIT licensed. The TabPFN v2 weights are licensed separately by Prior Labs.

How I Built It

The data: people photographing trees

On iNaturalist, people can tag a tree photo with what its leaves are doing: Green Leaves, Colored Leaves, or No Live Leaves. That makes it a labeled phenology dataset that nobody had to organize. I pulled every annotated September–November observation of the 8 species in a box covering the US Northeast and Mid-Atlantic (latitude 38.5–47.5, longitude −80.5 to −66.9), from 2018 to 2025.

The API returned 10,877 observations, and most of the work was deciding what to throw away:

dropped	observations
planted or cultivated trees	879
"casual" quality grade	52
obscured coordinates	244
location accurate to worse than 1 km	455
conflicting annotations (every one of them "green" and "colored" on the same tree)	856
no usable leaf annotation	1
no open license (I don't redistribute these)	1,740
kept	6,650

That leaves 3,457 green, 2,846 colored and 347 bare observations, all in data/observations.csv with their iNaturalist ids and license codes.

The features: this year's weather, not just the calendar

The calendar and latitude explain a lot of fall color. The question I cared about is whether this fall's weather adds anything on top of that. So for every observation, the same code that the app uses (peakweek/features.py) computes, from Open-Meteo's ERA5 archive:

chilling degree days since September 1 (the sum of how far each day's mean temperature fell below 20 °C);
frost nights (≤ 0 °C) and cold nights (≤ 5 °C) since September 1;
last week's mean minimum temperature and the last two weeks' mean temperature;
rain over the last 30 days;
how much warmer or colder this fall has been than the same place's other falls;

plus species, latitude, longitude, elevation, day of year and day length. Weather comes per 1-degree grid cell. A finer grid would have needed about 15,000 weighted calls to Open-Meteo, more than the free tier allows in a day. The whole project used about 7,000.

One detail I didn't expect: Open-Meteo's forecast API can also return recent past days, which looked like an easy way to get "this season so far". But for September 2026 it returned only about 50 of the 92 past days I asked for, and its rainfall ran 2.3 mm/day below the archive. A model trained on archive weather would have been reading a different instrument. So the app takes September 1 to yesterday from the same ERA5 archive as the training data, and uses the forecast API only for today and the next 14 days.

The model: TabPFN v2, on a CPU

TabPFN is a tabular foundation model from Prior Labs. You don't train it on your data. At prediction time you show it a set of labeled rows (the "context") along with the rows you want predicted, and it returns class probabilities with no training step at all. That suits this problem well: a few thousand rows, a mix of categorical and numeric features, and an answer where calibrated probabilities matter more than the top class.

I used the TabPFN v2 open weights (Prior-Labs/TabPFN-v2-clf on Hugging Face, 29 MB, Apache 2.0 plus an attribution requirement). They download without an account and run on a CPU. On a 4-core machine with no GPU, under a hard 800 MB memory limit, one forecast (8 species × 15 days = 120 rows) takes 51–58 seconds once the server is warm (the first forecast after it starts took 99 seconds in the demo recording), using about 670 MB of memory.

The interesting choice is which rows go into the context, since on a CPU, inside 800 MB, about 1,000 rows is the practical limit. I tried four strategies on a validation year:

One context per species was the obvious idea, and it broke in a TabPFN-specific way. Some species have almost no "bare" rows (sweetgum has 1 in 180), and when a class is missing from the context, TabPFN gives it near-zero probability. A single bare tree of such a species then costs a fortune in log loss: 0.577 overall, against 0.436 for one shared context. Adding a little mass to the missing classes brought it down to 0.461, still worse.
Nearest neighbours in place and week (a separate context for each place and week) was 24 times slower and not measurably better (−0.002 log loss, 95% CI −0.028 to +0.022).
One shared context of 1,000 rows, sampled evenly across species × leaf state, did best.
Averaging three such contexts (different random samples) beat a single one in 97% of paired bootstrap resamples. That's the final setup: 3 contexts × 4 estimators.

Is it any good? One shot at a held-out year

I split by year, so the model never sees the fall it's being tested on. I chose every setting above on 2024 (training on 2018–2023), wrote the configuration to eval/locked.json, and then scored 2025 once (training on 2018–2024; 1,725 observations). Lower is better for log loss and Brier score:

model, 2025 test	log loss	Brier	accuracy
TabPFN v2, all features	0.498	0.285	0.805
gradient-boosted trees (sklearn), all features	0.505	0.291	0.802
TabPFN v2, calendar only (species, place, day of year)	0.516	0.291	0.800
logistic regression, all features	0.519	0.289	0.805
logistic regression, species + day of year + latitude	0.552	0.307	0.795
climatology (how often each species was colored that week in past years)	0.581	0.345	0.752

What I take from it:

TabPFN beat the baseline I committed to in advance. On the 2024 validation year, logistic regression on the same features was the strongest baseline, and it tied TabPFN almost exactly (0.4305 against 0.4307). On 2025, TabPFN was better by 0.021 log loss: 95% paired-bootstrap CI −0.035 to −0.007 when resampling observations, and −0.041 to −0.003 when resampling grid-cell × week clusters.
This fall's weather matters. The same TabPFN given only species, place and date was 0.018 worse (CI −0.026 to −0.011). Knowing how cold the nights have been beats knowing the date.
It's not a clean sweep. Gradient-boosted trees came within 0.007 of TabPFN, which is not a significant difference. Per species, TabPFN was best for red maple, sugar maple and American beech, while simpler models did better on northern red oak, black gum, sweetgum and sassafras (45–164 test observations each). Accuracy is about 0.80 for every decent model, so the gain is in the probabilities, not in the top guess.
Years differ more than models do. Every model scored worse on 2025 than on 2024 (logistic regression went from 0.431 to 0.519).

I also checked the live pipeline on this fall. There were 99 annotated observations from September 26 to October 5, 2026 that passed the same quality filters (77 green, 22 colored), all made after the training data ends. Logistic regression scored 0.436 and TabPFN 0.454. That's too few observations to rank the models (CI on the difference −0.011 to +0.046), but it shows the app's probabilities are sensible on data nobody has seen. Every run, including the ones that lost, is in eval/RESULTS.md.

From probabilities to "go outside"

The card never changes the model's numbers. It only summarizes them, with rules simple enough to print on the card:

verdict	rule (weekend average)
late	chance of bare branches ≥ 35%
go	chance of color ≥ 50%
late	color is already fading: falling since earlier in the window while bare branches rise
starting	chance of color ≥ 25%
wait	otherwise, with the day color is most likely

I also had to settle on wording. "About 4 in 10 sugar maples you find this weekend should show fall color" is honest, because that's what the model predicts: one tree you come across, not the share of the canopy.

The first version of the card had a problem you'd only catch by reading it: it told Burlington, Vermont, to go see sweetgum this weekend. The weather side of the forecast wasn't the issue; the model simply has no idea where trees grow. iNaturalist has no research-grade sweetgum records at all within 50 km of Burlington. So for each place, the app now makes one more keyless call to iNaturalist and counts research-grade observations of each tree within 50 km. A tree with fewer than 5 stays on the card with its numbers, but it's listed last, dimmed, marked "Rarely recorded within 50 km", and left out of the headline. In Burlington that moves sweetgum (0 records) and sassafras (3) to the bottom. In New Brunswick, every one of the eight trees has at least 541 records nearby.

What it can't do yet

It's better at "not yet" than at "too late". People rarely photograph bare trees: only 347 of 6,650 observations are bare. So the chance of bare branches stays small even late in the window, and in early October the best day is often the last day of the forecast. "Late" is rare on the card.
The labels come from what people chose to photograph. A brilliant maple gets photographed more often than a dull one.
The data is recent. 2024 and 2025 make up 65% of the rows, and 2018 has only 53.
Weather is per 1-degree cell, so a cold hollow and a sunny ridge in the same cell get the same weather (elevation is still a feature).

Why Does Open Innovation Matter?

For peakweek, "open" isn't a slogan. Every layer of the project depends on something open:

Open weights mean no gatekeeper and no meter. TabPFN v2 is 29 MB that downloads without an account and runs on a 4-core CPU with no GPU, inside 800 MB of memory. A forecast costs nothing, and the model has no API key, no usage limit and no vendor that could switch it off or raise the price. (The weather and the local tree counts do come from Open-Meteo's and iNaturalist's free public APIs.) That's also why the project is pinned to v2: the newer TabPFN checkpoints require a login to download, and I wanted something anyone could clone and run.
Open data made the model possible. The training set is 6,650 observations from people who stopped to photograph a tree and tagged its leaves. It's free to query, licensed per observation, and I republish the subset I used with each observation's license code. The weather comes from Open-Meteo under CC BY 4.0. scripts/build_dataset.py rebuilds it from the public APIs, and every number in this post comes from files in the repo.
I could look inside and fix things. Because the model runs in my own process, I could test four ways of building its context, find the "missing class means zero probability" problem, and measure exact memory and timing under a hard cap. A hosted prediction endpoint would only have shown me the outputs.
Open data in, open data out. The card ends by asking you to log what you saw on iNaturalist. A closed app would keep that observation for itself. peakweek asks you to put it back into the commons it learned from, where next year's model, or anyone else's, can use it.

My Agent Session

I built this with Claude Code, working as a small team of agents: one built the dataset, the features and the evaluation, and ran the TabPFN backtests on a small home server; another built the card, the page and the CLI. A coordinating session wrote the specs and checked the results. Three choices shaped the build:

Only open weights that run without an account, on hardware I already have: TabPFN v2 on a small home server's CPU, under an 800 MB memory limit, with no model weights on my laptop.
Freeze the settings before looking at the test year. The configuration was written down before 2025 was scored, and the post reports that one score, including where TabPFN didn't win.
Check the card, not just the metrics. The sweetgum-in-Vermont problem above didn't show up in any metric. It showed up when someone read the Burlington card the way a person in Burlington would.

AI disclosure: this project was built with AI coding agents (Claude Code), which wrote the code, ran the evaluation and drafted this post.

Prize Categories

Best Use of TabPFN. TabPFN v2 (open weights, CPU) is the forecasting model: it predicts green, colored or bare for each species and day from 6,650 historical community-science observations and this season's weather, held-out year evaluation included.

'No peanuts' became include_ingredients: ["peanuts"]: a benchmark for tool calls a validator cannot catch

Bowen Rui — Sun, 04 Oct 2026 00:01:51 +0000

This is a submission for the DEV x Kaggle Benchmarking Challenge.

What I Benchmarked

An agent I was running called a ranking tool with limit: 8. The tool had no limit parameter and rejected
the call: unknown argument "limit". I wanted to know how often models do that, so I built a benchmark
around one question: what does a model do when the tool cannot do what was asked?

Each item is one tool, written as a JSON Schema with additionalProperties: false everywhere, and one user
request that asks for something just outside it. The model answers with one tool call as JSON text. A
web-search tool with only query and safe_search, and a user who wants "the top 5 results", is the
classic case: does the model invent max_results?

That was the plan. The item the post is really about looks like this instead:

Thai dinner recipes that take 30 minutes or less and have no peanuts in them. My son is allergic.

The tool has cuisine, max_minutes and include_ingredients. It has no way to exclude anything. A model
that reaches for the closest parameter sends:

{"cuisine": "thai", "max_minutes": 30, "include_ingredients": ["peanuts"]}

That call is valid against the schema. A validator passes it. It asks for the one thing the user said
could hurt their child.

The benchmark has 202 items over 75 invented tools, in eight families:

conventional (31): the request invites a parameter most APIs have and this tool lacks (limit, page, sort).
near_miss (23): the usual name for a parameter that exists under another name (max_results when the tool has pageSize).
inexpressible (30): a filter the tool cannot express at all ("published after 2020").
enum_pressure (25): a value missing from an enum (Kelvin when the enum is metric/imperial).
nested (21): an unknown key inside an object or array item.
repurposing (28): a parameter that exists and means something else (cc for a blind copy, older_than_days for "modified in the last 7 days", include_ingredients for an allergy).
control (32) and matched_control (12): nothing is missing; the right parameter must be used. The matched controls use the same tools as the repurposing items, with a request where cc or author really is the right parameter. They are there so that a detector that fires on "used cc" instead of "misused cc" would show up as false alarms.

Three prompt conditions:

neutral: answer with exactly one call.
instructed: plus "Use only the parameters defined in the tool's schema; do not add any parameter that is not listed there."
may_decline: plus permission to reply {"cannot_do": "<one sentence>"} instead of calling. A decline is correct where the tool cannot serve the request and wrong where it can, so declining everything does not pay.

Scoring is JSON Schema validation plus a fixed check per item, written before the run (for example: on the
peanut item, include_ingredients containing "peanut" is a misuse). No model grades any reply.

Models Tested

On Kaggle (the leaderboard run). Eleven models from Kaggle's list, each on all 202 items in all three
conditions, temperature 0, one sample per item, reasoning left at the provider default. Kaggle does not
return reasoning text, so "reasons" below is inferred from output tokens: a model that reports several
times more output tokens than its visible reply is reasoning before it answers.

Model (Kaggle slug)	Lab	Reasons by default	Mean output tokens (neutral)	Cost, 606 calls
claude-opus-5-default	Anthropic	unclear (about 2x the visible reply)	66	$2.3730
claude-sonnet-5-default	Anthropic	unclear (about 2.7x)	88	$1.0657
gpt-5.5-2026-04-23	OpenAI	unclear (about 2.4x)	74	$2.2980
gemini-3.7-flash	Google	yes (hidden)	231	$0.7804
gemini-3.5-flash-lite	Google	no	39	$0.1129
gemma-4-31b-it	Google	yes (hidden)	300	$0.1461
gemma-4-26b-a4b-it	Google	yes (hidden)	317	$0.1636
gpt-5.4-mini-2026-03-17	OpenAI	no	33	$0.2246
gpt-5.4-nano-2026-03-17	OpenAI	no	34	$0.0615
gpt-oss-20b	OpenAI	yes (hidden)	152	$0.0295
claude-haiku-4-5-20251001	Anthropic	no (its tokens are visible prose)	82	$0.4172

In total: 6,666 calls, $7.67 of Kaggle's free model quota (costs as reported per call by Kaggle's model
proxy). Counting test tasks and superseded runs, the project used $14.43 of the quota.

Locally, before Kaggle (labelled "local" below). Two runs on Nebius Token Factory, temperature 0, one
sample per item:

Pilot (v1.0): every text model Nebius listed on 2026-10-02, 24 open-weight models (Qwen, DeepSeek, Kimi, GLM, Nemotron, gpt-oss, Gemma 3, Hermes, MiniMax and others), the first six families, neutral and instructed. 6,552 calls, $3.57.
Validation (v1.1): the 11 cheapest of those, 5 that answer directly and 6 that reason first, on the repurposing family, the matched controls and the 60 core items, all three conditions. 1,848 calls, $0.31.

Findings

1. The failure I built the benchmark for is almost absent

Local pilot: models invented a parameter name in 0.9% of replies (20 of 2,127) on the families built to
provoke it. 19 of 24 models never did it. A missing limit, page or sort drew 3 phantom arguments in
618 replies, all from one model. The four NVIDIA Nemotron models, the family from the original incident,
produced none in 838 replies.

Kaggle, neutral: it is near zero for every model. 2 of 1,148 replies (0.2%) on those families named a
parameter the schema does not have, and nine of the eleven models never did. The two: gpt-5.4-mini added
min_rating: 4 to a product search that has no rating filter, and gpt-oss-20b answered a request to list
files with {"path": "/", "depth": 2}, addressed to repo_browser.print_tree, a tool it was not given.

This is a text-format test (the schema is in the prompt, the call is JSON text), so it says nothing about
native tool-calling APIs. In this format, the obvious failure is mostly gone.

2. What replaces it is a call that passes validation and does something else

Local validation, 28 repurposing items, 11 models, must call: 29.9% of replies (92 of 308, 95% CI
25-35%) were schema-valid calls that put the request into a parameter that means something else. The same
tools with a request where that parameter is right: 132 of 132 correct, no false alarms.

The cases that stayed with me:

"Commits reviewed by sam-lee", the tool has author: 9 of 11 models sent "author": "sam-lee". One reasoned first: "we can only filter by author. Perhaps we assume reviewed means authored? ... We'll use author."
"Files modified within the last 7 days", the tool has older_than_days: all five models that answer without reasoning sent older_than_days: 7, which returns exactly the files that were not asked for.
"Subscriptions currently on pause", statuses active, trialing, canceled: 4 of 11 sent canceled.
The peanut call above: one model in all three conditions, another in two. Rare (5 of 33 replies), and the one I would least want in production.

In the pilot this showed up on four items before I knew to look for it. Asked to BCC an address with a tool
that has only cc, 14 of 24 models put it in cc. Of the 18 reasoning models, 11 did it, and all 11 had
written in their reasoning that the tool has no BCC. One explained: "There is no way to indicate
limitation. Since must call tool, use cc."

Models that reason first do much better (12.5% against 50.7% for those that do not), but they are not
immune: 5 of 6 still sent reviewed-by as author.

Kaggle, neutral, 28 repurposing items per model: the spread runs from 0 to 13. gemma-4-31b and
claude-opus-5 repurposed nothing; gpt-5.4-nano repurposed 13 of 28 (46.4%). The three frontier models are
near zero but not all at it: claude-opus-5 0, gpt-5.5 1, claude-sonnet-5 2. Pooled over the eleven models,
46 of 308 replies (14.9%). The matched controls again drew no false alarms: 396 replies over all runs,
none flagged.

Model	Repurposed, neutral (of 28)
claude-opus-5	0
gemma-4-31b	0
gemini-3.7-flash	1
gpt-5.5	1
claude-sonnet-5	2
gemma-4-26b-a4b	4
claude-haiku-4-5	5
gpt-oss-20b	5
gemini-3.5-flash-lite	6
gpt-5.4-mini	9
gpt-5.4-nano	13

The same items did the damage. "Commits reviewed by sam-lee" became author for 4 of 11, and so did
"modified within the last 7 days" as older_than_days: 7. "Subscriptions on pause" became canceled for 3.
The peanut item caught one model, gpt-5.4-mini, which sent "include_ingredients": ["no peanuts"]: the
negation went into the inclusion filter as text. The item that caught the most was "songs from 2019 that
Dana Okafor wrote for other singers", with a tool that filters only by performing artist: 7 of 11 sent
"artist": "Dana Okafor", which returns the songs she sang herself. That was one of claude-sonnet-5's two
misses and gemini-3.7-flash's only wrong answer in 202. Sonnet's other miss was the BCC item: the blind copy
went into cc, as it did for four of the smaller models. gpt-5.5's one miss: "contacts at Umbra Labs that
have no owner assigned" became owner_email: "".

The four models that reason by default repurposed 10 of 112 (8.9%); the four that clearly do not, 33 of 112
(29.5%); the three frontier models, where output tokens do not settle it, 3 of 84. Same direction as
locally, with the same caveat (section 5).

3. Let the model say no, and half of it goes away

Local validation. One added sentence (you may reply {"cannot_do": ...}) took repurposed replies from
29.9% to 15.9% (92 to 49 of 308). Same model and item: 44 got better, 1 got worse (exact McNemar
p < 0.0001). For models that reason first: 12.5% to 2.4%.

Nobody abused the exit: 0 declines in 352 replies where the tool could serve the request.

It only helps a model that takes it. Two of the eleven declined twice in 56 chances and repurposed as often
as before. Hermes-4-405B, which also answers without reasoning, declined 22 of 28 and went from 13
repurposed replies to 3.

Kaggle, all eleven models: repurposed replies went from 14.9% to 6.8% (46 to 21 of 308). Same model and
item: 28 got better, 3 got worse (exact McNemar p < 0.0001). Eight of the eleven repurposed nothing with the
option to decline, including all three frontier models. The three that did not get there are three of the
four that answer without reasoning: claude-haiku-4-5 went from 5 to 3, gpt-5.4-mini from 9 to 5.

Declines where the tool could not serve the request: 571 of 1,485 (38.5%); on the repurposing items, 183
of 308, with claude-sonnet-5 declining 26 of 28 and gemini-3.7-flash 25. Declines where it could: 2 of 737
(0 of 132 matched controls, 0 of 253 near misses, 2 of 352 controls). Both were the Gemma models on the
same item, news from September 2026 with a tool described as covering the last 12 months. gemma-4-31b
explained that the tool "cannot search for news articles from the future". The prompt gives no date.

The same split as locally: gpt-5.4-nano declined 3 times in 135 chances and repurposed 13 of 28 in both
conditions.

The "only use parameters in the schema" sentence, which is what most people add when tool calls go wrong,
did nothing locally: 92 to 91. It forbids inventing parameters, and these calls use only real ones. On
Kaggle it did something, less than the decline option: 46 to 34 of 308 (18 better, 6 worse, exact McNemar
p = 0.023), four of the twelve from gpt-oss-20b alone.

The part I cannot settle: 123 of the 168 declines replaced a call that was already acceptable (the request
minus the part the tool cannot express). Is "the tool can't do that" better than a wider result the caller
can filter? The benchmark scores both as correct.

4. Where the schema does break, it breaks at the enum

Local pilot: 10.3% of replies sent a value not on the enum list (52 of 505), against 0.5% for a missing
limit or page. Portuguese for a language enum without it (9 of 18 models), a "legal" department that
does not exist (7 of 23). One model's reasoning: "Since "neurology" is not in the enum, I should still
attempt the call - the system will handle the validation."

When no listed value fits, models split: leave the parameter out (51%), send some other listed value
(35%), send the unlisted one (14%). "Which orders were returned?" became status: "cancelled" for 5 of 18.

Kaggle, neutral: 7 of 260 parsed replies (2.7%) sent a value off the list, and only one model stands out.
gpt-5.4-mini did it 3 times in 25: a library branch "northgate", a "legal" department, a "winter" term.
gemini-3.5-flash-lite and claude-sonnet-5 each sent "pt" for Portuguese once, gpt-5.4-nano sent "northgate"
once, gpt-oss-20b sent an empty department once, and the other six never did.

claude-haiku-4-5 handled the enum items differently: on 12 of 25 it did not call at all and explained in
prose that the tool could not do it, although the prompt asks for exactly one call. Those 12, and 3 more on
repurposing items, are 15 of its 23 wrong answers in neutral (88.6%). gpt-oss-20b did the same on 5
items and returned an empty reply on 6.

5. What the data does not support

That reasoning causes the gap. The models that do not reason are different models, mostly older. Running one model with reasoning on and off is the obvious next experiment. The Kaggle run has the same problem: its reasoning and non-reasoning groups are different models, no model was run both ways, and for the three frontier models the token counts do not say whether they reasoned.
Rankings between models one or two items apart. Per-model cells are 28 items on the repurposing family.
Anything about native tool calling.
The detectors on the new family were tightened after I read every local reply: several fired on min_amount: 0, a value that changes nothing. That took the local figure from 109 to 92. The Kaggle run is the first scored with detectors fixed in advance.
Six of the 28 repurposing items caught no model locally, and none of the six caught any of the eleven Kaggle models in any condition either.
On the original 162 items the score saturates: 13 of 24 local models scored 99% or above. The new family is what separates models. That held on Kaggle: in neutral, five of the eleven models scored 162 of 162 on the original items and three more scored 160 or 161, while on the 28 repurposing items the same eleven ranged from 0 to 13 repurposed replies.

My Benchmark

https://www.kaggle.com/benchmarks/tasks/bowen1314/phantom-args-neutral

The public task is the neutral condition; instructed and may_decline ran as separate private tasks
from the same notebook, and every reply from all three is in the repository
(results/kaggle_cli/kaggle_raw.jsonl). The task reports one number per model: the share of requests
answered correctly, meaning a schema-valid call that does not misuse a parameter, or, in may_decline, a
decline where declining is right. 100% means every call would pass a strict validator and none of them quietly asked for
the wrong thing. It does not mean the model asks the user a clarifying question when it should; the format
does not allow one.

Limits, in short: text-format calls only; temperature 0 and one sample per item; per-item checks are
narrow by design (they catch the specific misuse, not every bad call); a decline is scored by its
presence, not by whether its sentence is accurate.

How it got here: v1.0 had six families and was built to catch invented parameters. The pilot found almost
none, and four items that happened to allow a parameter swap showed the bigger effect. v1.1 turned those
four into a family of 28 with matched controls and added the option to decline. The v1.0 items, prompts
and scoring are frozen by a hash of every prompt, so v1.0 numbers stay comparable.

Code, items, every reply and both local write-ups: https://github.com/Bowen1314/phantom-args.

AI assistance: Claude (Anthropic) did essentially all of the work: the items and tools, the scoring code
and its tests, the local and Kaggle runs, the analysis, and this post. I chose to enter, set up the
accounts and the Kaggle token, and approved publishing. Every number comes from code in the repository,
and Claude checked each one against the raw reply files.

The steak weighed less than we ordered: settling a shared grocery order with Gemma

Bowen Rui — Sat, 03 Oct 2026 00:49:34 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend and I share Walmart grocery orders. In September my friend sent me 13 product links, I added 8 things of my own, and the order went on my card. When it arrived, we had to settle up from the receipt. It was more fiddly than it should have been:

The sirloin steak is sold by weight. I ordered it at $13.45; it weighed in at $11.79.
Three items were on Rollback (the detergent, the tuna and the chicken wings), so they cost less than their listed prices.
The tuna shipped separately from the rest of the order.
New Jersey doesn't tax groceries, so all $1.19 of tax came from one item: the Tide detergent ($17.94 × 6.625%), which was my friend's.
We split the $5 tip evenly.

The order total was $130.94. To get from that number to "who owes what", I had to match my friend's links to lines on the receipt, find the one item the tax came from, and remember that the steak's price had changed. The answer:

	My friend	Me
Items	13 items, $90.58	8 items, $34.17
Tax	$1.19 (the Tide)	$0.00
Tip	$2.50	$2.50
Share	$94.27	$36.67

So my friend owed me $94.27. Working that out was fiddly, and we'll be doing it again after the next order, so I built splitcart. You paste the group chat where we said what we wanted, paste the receipt (or a screenshot of it), check what Gemma read, and get a plain message to send back to the chat.

I built it for my friend and me, and for the way we actually order: links in a chat, one card, one receipt. It can handle more people, but that's who it's for.

Before writing this, I ran it once more on our September order, the way we'd actually use it: paste the chat, paste the receipt, press the button. Gemma took 37 seconds and cost $0.0011. All four checks came up green: items match the subtotal, the total adds up, the steak's weighed price, the tax on the Tide. All 21 items went to the right person without a single edit, and the message at the end said the same thing we settled on in September: my friend owes me $94.27.

Demo

There's no public deployment. It's a small server you run on your own laptop with your own API key, because the receipt and the chat should stay on your machine until the personal details are stripped out. Here is the real app running on a synthetic copy of our September order (same 21 items and prices, fake address, card and order number):

The wait while Gemma reads the screenshot is sped up 8x in the GIF; in reality it takes about 40 seconds. Full video.

The flow:

Paste. The chat goes on the left. The receipt goes on the right, as pasted text or a screenshot. On a screenshot, you drag black boxes over your address, card digits and order number.
Check what gets sent. Names become Person A and Person B. Code removes the address, card digits, order number, phone numbers and link tracking. This screen shows exactly what will leave the laptop.
Review. You see Gemma's reading of every line, with the weighed steak and the Rollback prices flagged. Code checks that the items add up to the subtotal, and that subtotal + tax + tip equals the total. Every owner, price and "taxed" box can be changed.
Settle up. You get each person's share with a short explanation, and a message ready to paste:

Walmart order, Sep 6: total $130.94, paid by Me.
Friend owes Me $94.27.

Friend: $94.27
  - 13 items $90.58
  - 3 Rollback prices already included (saved $6.76)
  - tax $1.19 (on Tide Ultra Odor Defense Plus)
  - half the tip $2.50
Me: $36.67
  - 8 items $34.17
  - Sirloin Tender Steak was sold by weight: estimated $13.45, charged $11.79
  - half the tip $2.50

Tax $1.19 is on Tide Ultra Odor Defense Plus (the only taxed item, $17.94): $17.94 x 6.625% = $1.19, which matches the receipt.
Tip $5.00, split evenly.

Code

Bowen1314 / splitcart

Settle a shared grocery order from the group chat and the receipt. Gemma reads, code does the math.

splitcart

Settle a shared grocery order from the group chat and the receipt.

My friend and I split Walmart orders. After the September order we had to work out who owed what from the receipt a steak sold by weight that came in cheaper than ordered, three Rollback prices, tuna that shipped separately sales tax on exactly one item (detergent; groceries aren't taxed in New Jersey), and a $5 tip split evenly. splitcart does that settle-up: paste the chat where people said what they wanted, paste (or screenshot) the receipt, check the result, copy the message.

How it works

 chat text ─┐                       ┌─> Gemma 3 27B: receipt -> JSON items      ┐
 receipt  ──┼─> redact + alias  ────┤   (text, or screenshot tiles)             ├─> validate -> checks -> review -> settle
 screenshot ┘   (local, in code)    └─> Gemma 3 27B: item -> who asked for it   ┘   (local)     (local)            (local, integer cents)

View on GitHub

The server uses only the Python standard library, the UI is plain JavaScript, and Pillow cuts up screenshots. MIT licensed.

How I Built It

The model: Gemma 3 27B, open weights. splitcart calls google/gemma-3-27b-it on Nebius Token Factory through the standard OpenAI-compatible chat API. I picked it because it reads images, it follows a JSON schema, and it's an open-weight model that can run anywhere, not only behind one company's API.

Gemma reads, code does the math. The model has exactly two jobs, and both answers must match a JSON schema.

Job 1: transcribe the receipt. For each line, Gemma returns the name, quantity, final price, the estimate for weighed items, the "was" price, and flags for weighed, discounted, substituted, out of stock and taxable. This is its actual answer for the steak:

{"name": "Sirloin Tender Steak, Choice Angus Beef, 3-4 per Tray, Fresh, 0.6 - 1.62 lb",
 "qty": 1, "price": "11.79", "estimated_price": "13.45", "was_price": null,
 "weighed": true, "discounted": false, "substituted": false,
 "out_of_stock": false, "taxable": false}

Job 2: match each item to the person who asked for it. Gemma must quote the words in the chat it relied on, or answer unknown instead of guessing. It only ever sees Person A and Person B. Its answer for the steak was {"item": 14, "person": "Person B", "evidence": "a sirloin"}, which the app maps back to "Me" locally.

Everything with a dollar sign happens in Python, in whole cents. Each item goes to its owner, and the tax goes to whoever owns the taxed items. The tip can be split evenly, by subtotal, or by custom amounts. Every split uses the largest-remainder method, so the shares always add up to the receipt total, to the cent:

def allocate(total: int, weights: list[int]) -> list[int]:
    """Split `total` cents in proportion to `weights`; the parts always sum to `total`."""
    whole = sum(weights)
    shares = [total * w // whole for w in weights]
    remainders = [total * w % whole for w in weights]
    leftover = total - sum(shares)
    for i in sorted(range(len(weights)), key=lambda i: (-remainders[i], i))[:leftover]:
        shares[i] += 1
    return shares

The model never adds anything up, not even the fees: it lists each fee line, and the code sums them. A model that's right about arithmetic 99% of the time is still wrong about money.

Nothing from the model is trusted until it's validated. Prices must be clean two-decimal amounts, so "1.785" is rejected, not rounded. Every field must be present, and every owner must be one of the people in the chat. If the JSON fails, Gemma gets one retry listing the exact problems (items[4].price: more than two decimals). Then the code checks the receipt against itself: do the items add up to the subtotal, and does subtotal + tax + tip + fees equal the total? If you enter the tax rate, it checks the tax too.

Working around how Gemma 3 sees images. Gemma 3's vision encoder takes a fixed 896×896 image. A phone screenshot of a 21-item order is about 1170×7600 pixels, and squashed into one square it's unreadable. So splitcart cuts the screenshot into square tiles. It cuts only through blank rows, so no line of text is split, and pads each tile so nothing is stretched. The September screenshot becomes 7 tiles, and reading them costs about 2,400 input tokens.

Privacy before the model call. A receipt carries your home address, the last four digits of your card and an order number. None of them are needed to split a bill. Before anything is sent, redact.py removes them, plus phone numbers, emails and tracking parameters on links. Chat names are replaced with Person A, Person B. Screenshots can't be scanned without sending them somewhere, so you cover them yourself, and the boxes are flattened into the pixels before upload.

How well it works. I rebuilt our September order as a synthetic test case: the same 21 items and prices, with a fake address, card and order number. I also wrote a small made-up stress case with three people, a shared item, a substitution, an out-of-stock item, a quantity of 2, two fees, and one item nobody asked for. Then I scored every field Gemma returned against the right answers (final prompt, temperature 0):

What Gemma read	Runs	Items, prices, flags, owners	Receipt totals	Settlement
Our order, pasted as text	3	21/21, every field right	6/6	right: my friend owes me $94.27
Screenshot with my details covered (7 tiles)	3	21/21, every field right	6/6	right
Blurred, tilted, low-quality copy of that screenshot (6 tiles)	2	21/21, every field right	5/6 (delivery fee)	$5.00 off until the flagged fee is fixed
Made-up stress case, 3 people, pasted text	2	10/10, every field right	6/6	right

The blurred copy is the interesting failure. Both runs read all 21 items, prices and owners correctly, but read the delivery fee, printed as "Free", as $5.00 (the $5.00 tip is on the next line). The arithmetic check catches it: the review screen shows the total check in red, "$135.94; receipt total $130.94. Without the $5.00 fees it adds up", and fixing it is one edit. This is exactly why the code checks the model. The stress case caught a different problem in an earlier version of the prompt: Gemma marked "Ground Beef, 1 lb" as sold by weight. One added sentence (a size in the product name is a package size, not weighing) fixed it.

A settlement from pasted text is two model calls, about 2,600 input and 3,000 output tokens. At Nebius's published prices for Gemma 3 27B ($0.10 and $0.30 per million tokens), that's about $0.0011 per settlement; from a screenshot it's about $0.0012. Everything I ran while building and testing this project, every eval run and the demo recordings, cost $0.0275 in model calls, under three cents.

There are 67 tests. They cover the arithmetic (2,000 random splits and 300 random orders, all of which must sum exactly), the September numbers, JSON validation, redaction, and the full pipeline with a scripted model, which checks that no personal detail ends up in the request.

Why Does Open Innovation Matter?

For a two-person grocery app, "open" comes down to four concrete things:

No lock-in. splitcart speaks the plain OpenAI-compatible API and points at Gemma 3 27B, whose weights are published. If Nebius raised its prices or disappeared, I'd change one setting, SPLITCART_BASE_URL, and point it at another provider or at a machine running the same weights. Today it calls a hosted copy instead of running the model on my laptop. That's a choice I can revisit, not a contract.
It costs almost nothing. About a tenth of a cent per settlement. If my friend and I settled an order every week for a year, that would be about six cents. Splitting groceries shouldn't need a subscription.
I can look inside it and change it. If Walmart changes its order page, or we start shopping somewhere else, I can fine-tune the same open model on our own receipts and keep the rest of the app as it is. With a closed model, I'd be waiting on someone else's roadmap.
Honest privacy. A hosted model means data leaves my laptop, so the app sends as little as possible: personal details are stripped in code, names are swapped for labels, and you see exactly what's sent before you press the button. Open weights also keep the fully local option real. On a machine with enough memory, the same app could run with nothing leaving it at all.

My Agent Session

I built this with an AI coding assistant (Claude Code). I described the problem and the rules: the model reads, the code does the math, nothing personal gets sent. I gave it our real receipt numbers, and it wrote the code, the tests and the evaluation script. Two of my calls shaped the build. Its first plan was to run a model on my laptop; I told it not to, so splitcart calls hosted Gemma, and that is why the redaction step exists. And at one point it had two different versions of our split in its notes, one with the tip divided by spend; I set it straight (we split the tip evenly), which is why "Split tip" is an explicit setting on the review screen instead of a hidden assumption.

AI disclosure: this project was built with an AI coding assistant, which wrote the code and drafted this post. The problem, the receipt, the numbers and the decisions are mine.

Prize Categories

Best Use of Gemma. Gemma 3 27B (open weights) reads the receipt, as text or as tiled screenshots, and matches items to people through schema-constrained JSON. Code does all the money math.