DEV Community: Alex Chen

A 200 Status Code Is Not Proof of Life: A Link-Triage Case Study

Alex Chen — Tue, 27 Oct 2026 12:28:29 +0000

Two days before term started, I pasted 62 links into the course reading list and told myself I'd check them "later." Later arrived during office hours, when a first-year student emailed me that the Week 3 reading went to a parking page. Then a second student said the same link worked fine for her. Same URL. Two different answers. That is the moment I stopped trusting status codes.

So here is the single question this case study answers: can I tell a dead link from a walled link without opening a browser, and where exactly does the HTTP status stop being useful? I'll build the smallest tool that answers it, run it against fixtures I control, show you the fixture that broke my first rule, and then explain why the leftovers are the only part worth spending model tokens on.

Prerequisites, honestly stated

CPython 3.12, standard library only, nothing to pip install. The fixture server binds to 127.0.0.1:8765, so no network access is required for the deterministic part. The optional classification step needs an HTTP endpoint — I used MonkeyCode's free model access for that, and I run the weekly job on their free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two claims come from the operator and not from my own measurement: free model access, and a free server option. The 10-million-token allowance is what the free tier advertises today. Check the current terms yourself before you build a workflow on it — quotas move, and mine isn't a guarantee for yours.

Step 1: Build ground truth, not a scraper

My first attempt pointed the script at the real reading list, which was a mistake. Real URLs change while you're debugging them. You can't tell whether your classifier is wrong or the web is. So I wrote eight fixtures where I know the answer.

# fixtures.py — deterministic ground truth on 127.0.0.1:8765
import http.server, socketserver, time

PAGES = {
    "/ok":        (200, "text/html", "<h1>Week 3: Gradient Descent</h1><p>Read sections 1-3.</p>"),
    "/gone":      (410, "text/html", "<h1>Gone</h1>"),
    "/moved":     (301, "text/html", ""),                      # no Location header
    "/soft404":   (200, "text/html", "<h1>Sorry, this page could not be found.</h1>"),
    "/cookiewall":(200, "text/html", "<h1>We value your privacy</h1><p>Accept cookies.</p>"),
    "/shell":     (200, "text/html", "<div id=\"root\"></div>"),   # JS app, no server HTML
    "/paper":     (200, "application/pdf", "%PDF-1.4 fake bytes"),
    "/slow":      (200, "text/html", "too late"),
}

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/slow":
            time.sleep(12)
        status, ctype, body = PAGES.get(self.path, (404, "text/html", "<h1>Nope</h1>"))
        data = body.encode()
        self.send_response(status)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)
    def log_message(self, *args):
        pass

if __name__ == "__main__":
    socketserver.TCPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run it in one terminal with python fixtures.py. That file is the whole trick. Everything after this is just a program that has to agree with it.

Step 2: Probe, and record more than the status

urllib gives you the status, the final URL after redirects, the content type, and the bytes. Grab all four. A single integer throws away the evidence you'll need later.

# linktriage.py
import json, re, sys, time, urllib.request, urllib.error

UA = "linktriage/0.1 (course reading list audit)"

NOT_FOUND = ("page not found", "could not be found", "no longer available", "has been removed")
CONSENT = ("accept cookies", "cookie settings", "value your privacy")

def extract_text(raw, ctype):
    if not ctype.startswith("text/html"):
        return ""
    html = raw.decode("utf-8", "ignore")
    text = re.sub(r"<[^>]+>", " ", html)
    return re.sub(r"\s+", " ", text).strip()[:600]

def probe(url, timeout=5, limit=200_000):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            raw = r.read(limit)
            ctype = r.headers.get("Content-Type", "")
            return {"url": url, "status": r.status, "final": r.geturl(),
                    "ctype": ctype, "bytes": len(raw),
                    "text": extract_text(raw, ctype), "error": None}
    except urllib.error.HTTPError as e:
        ctype = e.headers.get("Content-Type", "") if e.headers else ""
        return {"url": url, "status": e.code, "final": url, "ctype": ctype,
                "bytes": 0, "text": "", "error": "http-error"}
    except Exception as e:
        return {"url": url, "status": None, "final": url, "ctype": "",
                "bytes": 0, "text": "", "error": type(e).__name__}

Notice /moved. I gave it a 301 with no Location header, because that happens in the wild — someone moves a page and forgets the destination. urllib raises on it, status comes back as 301, and if you only check status >= 400 you'll call a broken redirect healthy. That was the first rule I had to break.

Step 3: A classifier you can defend

def classify(p):
    if p["status"] is None:
        return "unreachable"
    if 300 <= p["status"] < 400:
        return "broken-redirect"
    if p["status"] in (401, 403):
        return "walled"
    if p["status"] == 429:
        return "throttled"
    if p["status"] >= 400:
        return "dead"
    low = p["text"].lower()
    if any(s in low for s in CONSENT):
        return "walled"
    if any(s in low for s in NOT_FOUND):
        return "soft-dead"
    if p["ctype"].startswith(("application/pdf", "image/")):
        return "opaque"
    if len(p["text"]) < 120:
        return "ambiguous"
    return "alive"

def main(path, delay=1.0):
    urls = [l.strip() for l in open(path) if l.strip() and not l.startswith("#")]
    rows = []
    for u in urls:
        p = probe(u)
        p["label"] = classify(p)
        rows.append(p)
        time.sleep(delay)
    print(json.dumps(rows, indent=2))

if __name__ == "__main__":
    main(sys.argv[1])

With a links.txt of the eight fixtures, python linktriage.py links.txt gives me /ok: alive, /gone: dead, /moved: broken-redirect, /soft404: soft-dead, /cookiewall: walled, /paper: opaque, /slow: unreachable, and /shell: ambiguous after roughly twelve seconds of waiting on the timeout.

Seven out of eight, on the first try. Not bad. Now look at the one that isn't.

Where the rules quietly lie

/shell is the real world. Plenty of course pages now return a 200 with an empty <div id="root"> and paint the content in JavaScript. My heuristic sees almost no text, calls it ambiguous, and is right by accident — it has no idea why the page is thin. Is it a JavaScript shell, a soft 404 that says "Oops!" without the words "not found," or a consent wall that names its cookie banner differently? The phrase list can't tell, and no amount of adding strings fixes that, because the space of human apologies is infinite.

There's a second landmine I want to flag: many sites render a search box in the footer that says "Search the 404 page." Your substring match fires on the nav, not the content. That's how /soft404 and a perfectly healthy page end up in the same bucket. Always look at what the classifier matched before you trust the count.

So I stopped widening the rule set. I narrowed the input instead: only ambiguous rows go to a model.

Step 4: Spend tokens on the leftovers only

Out of 62 real links, the fetch pass put seven in the ambiguous bucket. Each row is a few hundred characters of extracted text, so the classification request is small. I send a constrained prompt and require one word back:

Classify this page excerpt as exactly one of:
  dead, walled, alive, unknown
Reply with the label only, no punctuation, no explanation.

EXCERPT:
{{text}}

Parsing is then a set membership check, not a regex adventure. Anything outside the four labels — a chatty model, a refusal, an empty body — becomes unknown and goes to a human. That last rule matters more than the prompt. A classifier that can't say "I don't know" will happily label a login page "dead" and get your reading list deleted.

This is the honest case for free model access: my usage pattern is spiky and tiny. Seven short calls on a Monday morning, a few hundred tokens each. Even at ten times the volume I'd be nowhere near a paid tier's floor. If your workload is thousands of full pages a day, free access is the wrong tool and you should be measuring latency and rate limits instead of reading this section.

Step 5: Put the weekly job on the free server

I don't want to remember this. The point of the tool is that I forget about it, so it lives on MonkeyCode's free server option and runs as a cron job.

crontab -e
# Mondays 06:00, write a report the rest of the week can quote
0 6 * * 1 cd ~/linktriage && python linktriage.py links.txt > report-$(date +\%F).json 2> err.log

One consequence I didn't expect: the free server made me write the output down. A JSON file per week means I can diff last week against this week and see when a link died, not just that it's dead. A link that flipped on October 20th is a much better email to a professor than "something is broken."

What I'd tell you before you copy this

It has no browser. Any page that needs JavaScript to say what it is will look thin to me, and my ambiguous bucket absorbs that — it does not solve it. It also can't judge whether a live page is correct, only that bytes came back. Be polite: one request per second per host, respect robots.txt, and don't run this against sites whose terms forbid it. And if your list is ten links you check twice a year, skip all of this and open them in tabs — the tooling costs more than the problem.

What you should take away is smaller than the code. Status codes describe transport, not meaning, and the distance between those two things is exactly where your heuristics will fail. Fixtures taught me that faster than any production incident, because a fixture can't change out from under me mid-debug.

Your turn, and I'd rather have a diagnosis than a like. Point the script at http://127.0.0.1:8765/shell and predict what your classifier returns before you run it — then tell me which fixture in your own set came back wrong. If you want to try the model step without a card, the free tier is where I'd start, but bring your own fixtures first. The fixtures are the part that teaches you something.

A Crash at Item 27 Should Not Cost You 27 Calls

Alex Chen — Mon, 26 Oct 2026 11:10:02 +0000

My digest worker died at 2:14 a.m. on item 27 of 40. The traceback was a timeout — unremarkable, the kind of thing a free endpoint does on a slow night. I restarted the script and watched it summarize items 1 through 26 all over again. Those summaries were already on disk, already correct, already paid for in tokens. I paid twice for the same twenty-six paragraphs because my script had no memory of ever having done them.

So here is the question I actually care about: can you make a batch worker that resumes instead of restarting, in a way that makes a flaky free tier boring? Yes — and the whole mechanism is one table with one column that decides whether a request is new or already answered. This is the case study of building it, including the version where my hash was wrong and I double-charged myself anyway.

What I wanted

Background first. I keep a reading list of paper abstracts (currently 40 entries, stored as JSON Lines), and I want one short summary per abstract so I can skim before a seminar. The summaries are cheap individually and annoying in aggregate, which is exactly the shape of a job you want to run overnight and never think about again.

Goal: run the worker, kill it at any moment, run it again, and pay for each unique abstract exactly once. Not "usually once." Exactly once, verifiable from a counter.

Prerequisites, exactly

Python 3.11.9 on Linux; sqlite3, hashlib, urllib, and http.server all from the standard library. No virtualenv needed, no third-party packages, no API key for the first half of this post. Check with python3 --version — anything from 3.10 up will do, but 3.11.9 is what I ran.

Step 0: keep the network out of the lesson

Testing idempotency against a real endpoint is a bad idea, because you cannot tell whether you doubled a call or lost one. So I wrote a short stub that counts requests and confesses the count in its reply.

# stub_endpoint.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

CALLS = {"n": 0}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        payload = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        CALLS["n"] += 1
        body = json.dumps({"summary": f"stub summary #{CALLS['n']}",
                           "chars": len(payload["text"])}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

When the output says summary #3, it means "this is the third request that ever arrived." That number is my ledger for the ledger.

Step 1: the worker

The worker below hashes each abstract, looks the hash up in SQLite, and only calls the endpoint on a miss. If an entry is missing because the process died, the next run fetches that entry and nothing else.

# digest.py
import hashlib, json, os, sqlite3, sys, urllib.request

DB, ENDPOINT = "digest.sqlite3", "http://127.0.0.1:8765/summarize"
CRASH_AT = int(os.environ.get("CRASH_AT", "-1"))

def normalize(text: str) -> str:
    return " ".join(text.split())

def key_for(text: str) -> str:
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()

def summarize(text: str) -> str:
    req = urllib.request.Request(
        ENDPOINT, data=json.dumps({"text": text}).encode("utf-8"),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["summary"]

def main(path: str) -> None:
    con = sqlite3.connect(DB)
    con.execute("CREATE TABLE IF NOT EXISTS digests("
                "key TEXT PRIMARY KEY, source TEXT, summary TEXT)")
    rows = [json.loads(line) for line in open(path, encoding="utf-8")]
    for i, row in enumerate(rows, start=1):
        k = key_for(row["text"])
        hit = con.execute("SELECT summary FROM digests WHERE key=?", (k,)).fetchone()
        if hit:
            print(f"{row['id']} cached  {hit[0][:24]}")
            continue
        if i == CRASH_AT:
            print(f"{row['id']} CRASH  simulated")
            sys.exit(1)
        s = summarize(row["text"])
        con.execute("INSERT INTO digests(key, source, summary) VALUES (?,?,?)",
                    (k, row["id"], s))
        con.commit()
        print(f"{row['id']} fetched {s[:24]}")

main(sys.argv[1])

Run it, break it, run it again:

$ CRASH_AT=27 python3 digest.py readings.jsonl | tail -2
reading-026 fetched stub summary #26
reading-027 CRASH  simulated
$ python3 digest.py readings.jsonl | head -3
reading-001 cached  stub summary #1
reading-002 cached  stub summary #2
reading-003 cached  stub summary #3

On the second run the stub's counter stays at 26 until the loop reaches the missing entries. That is the property I wanted: a crash costs you the in-flight request, not the finished ones.

The input that broke my first version

My first hash was hashlib.sha256(row["text"].encode()).hexdigest() — the raw string, no normalization. Then I regenerated readings.jsonl from a PDF extractor, and one abstract came back with a trailing space and a soft-wrapped newline in the middle. Same paper. Different bytes. Different hash. The worker saw a stranger and paid for it again, and my "exactly once" claim quietly became "twice, if your source formatting ever moves."

The fix is normalize(), collapsing all whitespace before hashing, which is in the code above. It is not a general solution — it also treats two genuinely different strings that differ only in spacing as the same document. For abstracts that is fine. For a legal clause, or anything where whitespace carries meaning, hash a canonical ID instead and keep the text out of the key entirely.

To reproduce my bad input, duplicate one line and add " \n" to its text before running. With normalize() removed you will see a second fetched line for that paper. With it in place, cached.

Where the free tier fits

This is where the product lives, so let me be plain about it and about what I can and cannot check. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and its outreach material describes two things that matter to a student worker like mine: free access to models, on the order of a ten-million-token allowance, and a free server option. I cannot verify quotas or hardware from outside, and I have not audited the repository, so treat every one of those claims as something to test on your own account before you build a habit on top of it.

Two honest observations, though. First, free model access is what turns a 40-item digest into a Saturday afternoon project instead of a budget decision — but free access is also why the ledger matters more, not less. When requests feel free, it is tempting to skip idempotency and just re-run everything, and that is how a cheap habit becomes an expensive one the first time an allowance changes underneath you. Second, a free server option addresses the lid problem: my earlier version only ran while my laptop was open. A worker with a durable ledger that survives a closed lid is a different animal from a worker that survives only if you sit there and watch it.

If you want to try the loop end to end, keep the stub in the repo as a test fixture, point ENDPOINT at your own endpoint, and run a small batch first. That is the softest suggestion I can make, and it is genuinely the right order to do things in.

What this does not solve

The ledger is not a queue. It does not retry failures with backoff, it does not cap concurrency, and it will happily grind forever on a file that keeps growing. It also stores every summary uncompressed in SQLite — fine at 40 rows, less fine at 40,000.

Do not use this approach if you are sending material that should never reach a third-party endpoint at all, if you need a response-time guarantee, or if someone else depends on the job finishing tonight. A single-process script with a local database is a student tool, and it should stay one until the requirements say otherwise.

Predict, then test

Before you run anything, look at these two entries and tell me which one the un-normalized version double-charges:

{"id": "a", "text": "Attention is all you need."}
{"id": "b", "text": "Attention is all you need.\n"}

If you said "b," you are right for the boring reason, and now you know why normalize() sits in front of the hash rather than behind it. If you find a fixture where my normalization merges two documents that should stay apart — a code snippet where the indentation is the entire point, say — send me the minimal counterexample. That is the version of this post I would rather be reading.

I Bisected My Prompt Until the Endpoint Broke

Alex Chen — Tue, 20 Oct 2026 09:14:21 +0000

Last Tuesday I tried to turn a 40-minute lecture transcript into study questions. I pasted it into a chat box, pressed send, and got back a red error that told me almost nothing. Not "your input is 61,000 characters." Not "too long for this model." Just a failure, and a cursor blinking at me like it knew something I didn't.

So I did what students do. I guessed. I cut the transcript in half, pasted again, then in half again, and spent eleven minutes doing arithmetic with my eyes. Then it hit me — why am I acting as the bisection algorithm? That's a loop I can write in fifteen lines.

This is the story of that loop, what it measured, and the one number I now refuse to hardcode.

The question I actually needed answered

Not "what is the context limit." That phrasing invites me to memorize a doc page for a model I may never call again. The question is narrower and testable: for this endpoint, this model, and this request shape, what is the largest prompt that comes back 200?

That's a measurement, not a fact. It has a date on it. Mine is from this week, against a model I picked from the free tier on MonkeyCode, and I'll explain below why I'm not printing the number as gospel.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Prerequisites and the shape of the harness

Python 3.11, standard library only, no tokenizer installed. That last part matters: I'm measuring characters, which is a bad proxy for tokens and a wonderful proxy for "is this request going to be rejected." You'll see the trade-off later.

Two rules before any code. First, the API key comes from the environment, never from a string literal. Second, the harness has a --dry-run mode with a stub that lies to me on purpose, so I can validate the loop before spending a single real request. Free tiers are a shared resource, and I try not to be the person who discovers the rate limit by ablation.

# probe_limit.py — Python 3.11, standard library only
import json, os, sys, time
import urllib.request, urllib.error

ENDPOINT = os.environ.get("MC_ENDPOINT", "https://your-endpoint.example/v1/chat/completions")
MODEL    = os.environ.get("MC_MODEL", "the-model-you-picked")
KEY      = os.environ["MC_API_KEY"]
SEED     = "Attention is a weighted average over values. " * 80  # long, boring, repeatable
SLEEP    = 2.0  # seconds between probes; raise this, not your hopes

def make_prompt(n_chars: int) -> str:
    body = (SEED * (n_chars // len(SEED) + 1))[:n_chars]
    return body + "\n\nIn one sentence, what is this text about?"

def post(prompt: str, timeout: float = 60.0):
    payload = json.dumps({"model": MODEL,
                          "messages": [{"role": "user", "content": prompt}],
                          "max_tokens": 40}).encode()
    req = urllib.request.Request(ENDPOINT, data=payload, method="POST", headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {KEY}"})
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return r.status, r.read().decode()[:200], time.perf_counter() - t0
    except urllib.error.HTTPError as e:
        return e.code, e.read().decode()[:200], time.perf_counter() - t0
    except Exception as e:
        return -1, repr(e), time.perf_counter() - t0

Nothing clever yet. Now the part that actually taught me something: the classifier.

def classify(status: int) -> str:
    if status == 200:            return "ok"
    if status in (400, 413, 422): return "too_long"      # shrink the input
    if status == 429:            return "rate_limited"   # stop; this is my fault
    if 500 <= status < 600:      return "server"         # back off once, then stop
    return "unknown"

413 Content Too Large has a definition; it's in RFC 9110, and it means what it says. 429 means I'm the problem. If your harness treats both as "input too long," it will happily bisect toward zero while the real story is that you sent forty requests in ninety seconds. That's not a context limit. That's a queue you failed to build.

Dry run first, real requests second

if "--dry-run" in sys.argv:
    def post(prompt, timeout=60.0):
        if len(prompt) >= 9900:              # the stub's fake wall
            return 413, "stub: too long", 0.0
        return 200, "stub: ok", 0.0

def probe(n_chars: int, tries: int = 2) -> str:
    for attempt in range(tries):
        status, body, dt = post(make_prompt(n_chars))
        kind = classify(status)
        print(f"{n_chars:>7} chars  status={status:<4} {kind:<12} {dt:5.2f}s")
        if kind == "rate_limited":
            print("429 — stopping instead of bisecting. Sleep longer and rerun.")
            sys.exit(2)
        if kind == "server" and attempt == 0:
            time.sleep(SLEEP * 2)
            continue
        return kind
    return "server"

def find_limit(lo: int = 500, hi: int = 200_000):
    probe(lo)                                  # sanity check the small end
    while probe(hi) != "too_long":             # exponential growth, not guessing
        if hi >= 400_000:
            print("still fine at my ceiling; raise hi or change the task")
            return None
        lo, hi = hi, hi * 2
        time.sleep(SLEEP)
    while hi - lo > max(200, lo // 10):        # 10% resolution, not 1 char
        mid = (lo + hi) // 2
        time.sleep(SLEEP)
        if probe(mid) == "too_long":
            hi = mid
        else:
            lo = mid
    return lo

Run it with the stub:

python probe_limit.py --dry-run

And you should see roughly this trace — the numbers are from the stub, so you can reproduce them exactly:

    500 chars  status=200  ok            0.00s
 200000 chars  status=413  too_long      0.00s
 100250 chars  status=413  too_long      0.00s
  50375 chars  status=413  too_long      0.00s
  25437 chars  status=413  too_long      0.00s
  12968 chars  status=413  too_long      0.00s
   6734 chars  status=200  ok            0.00s
   9851 chars  status=200  ok            0.00s
  11409 chars  status=413  too_long      0.00s
  10630 chars  status=413  too_long      0.00s
stop: largest passing size ≈ 9851 chars

Look at the last two lines and then look at the stub: the fake wall sits at 9,900 characters, and the search stopped at 9,851. That gap is the lesson. Bisection returns a boundary, not the boundary — I told it to quit at 10% resolution because each probe is a real request and precision beyond that buys me nothing.

What the live run actually told me

Nine probes, about twenty seconds of sleeping, one 429 that made me stop and rerun later with SLEEP = 6.0. The number it landed on is not the interesting output. The interesting output is what the number is: a property of one model on one day, not a spec I can paste into a config next month.

I re-ran three days later and got a different value. Not dramatically different, just different enough that hardcoding either one would have been a small lie. So the number lives in limits.json with a timestamp, and the worker reads it and refuses to start if the file is older than a week. Guess what happens when it's stale? I re-probe. Cheap.

Here's the consumer, the thing I actually wanted on Tuesday:

def windows(text: str, limit: int):
    start = 0
    while start < len(text):
        end = min(start + limit, len(text))
        if end < len(text):
            soft = text.rfind("\n\n", start + limit // 2, end)  # break on a paragraph
            if soft != -1:
                end = soft
        yield start, text[start:end]
        start = end

Each window gets one request, each result is appended to a JSONL file keyed by its start offset, and a crashed run resumes from the offsets already on disk. The napping-lid problem disappeared once I stopped running the loop on my laptop and let the free server hold it — same script, different machine, no drama. That's the whole trick. Nothing about my code assumed a particular host.

Where I'd bet against myself

Character counts are a lousy stand-in for tokens. If you have a tokenizer installed, measure tokens and delete my make_prompt entirely. Also, bisection assumes failure is monotonic in input size — usually true for length, occasionally false when a request is rejected for a reason you didn't model, like a stray character sequence tripping a filter. I spot-check three small sizes after every run for exactly that reason.

And if you're on a paid tier with a documented limit and a throughput target, this harness is wasted effort. Read the doc, set the constant, move on. Don't run any of this against an endpoint you don't control, and don't halve the sleep between probes "to go faster" — you'll just measure your own impatience.

Your turn to predict something. If I drop SLEEP to zero and keep the same nine probes, which line of that trace changes first: the 413 at the top, or the 200 near the bottom? Write your answer as the error class you expect, then actually run the stub a few times and see whether your reasoning survives contact with your own harness. That gap between what you predicted and what printed is the only part of this article worth keeping.

MonkeyCode provides free models that can run this workflow.

My Demo Died When I Closed the Lid: A Small Deployment Case Study

Alex Chen — Mon, 19 Oct 2026 07:46:41 +0000

Thursday, 11 pm, Halifax. I was screen-sharing my little retrieval demo to a classmate, felt clever for about four minutes, then shut the laptop to go make tea. When I came back she typed a question into the tab and got a spinner, then nothing. The server was my terminal. The terminal was my laptop. The laptop was asleep.

That is the whole problem, isn't it? A demo that only exists while you are physically looking at it is not a demo. So the learning question for this case study is narrow: what actually breaks first when a laptop-only RAG service moves to a free tier — the cold start, or the token budget? I wanted to find out without paying for the answer, and without guessing.

What I started with

A FastAPI endpoint, a chunk of lecture notes as context, and an answer function that did sentence-overlap ranking. No vector store, no framework, no GPU. Prerequisites are deliberately boring: Python 3.11, plus fastapi, uvicorn, and httpx. I am not going to quote exact patch versions, because yours will drift within a week — run pip freeze > requirements.lock.txt and keep that file next to the code. The article is reproducible only if you record what you actually installed.

The goal: three properties, not a feature list

I did not want a better demo. I wanted three properties I could test. First, the service must survive my laptop sleeping. Second, a single request must never be able to spend an unbounded amount of tokens. Third, the service must tell me what it did — latency, estimated cost, and why it refused.

That third one is the part students skip. We instrument for success and then act surprised when the bill, or the silence, is unexplained.

Step 1 — a service small enough to debug

Here is the whole thing. It is intentionally dumb: the "model" is a stdlib extractive ranker, so you can reproduce the failure modes with zero API calls. Swap extractive_answer for a real call when the harness is green.

# app.py
import os, re, time
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

APP_START = time.time()
CEILING = int(os.environ.get("TOKEN_CEILING", "1200"))
app = FastAPI()

class Ask(BaseModel):
    question: str
    context: str

def estimate_tokens(text: str) -> int:
    # Deliberately conservative: assume ~3 chars per token, not ~4.
    # Over-estimating is the safe direction for a ceiling.
    return max(1, len(text) // 3 + 1)

@app.get("/health")
def health():
    return {"ok": True, "uptime_s": round(time.time() - APP_START, 1), "ceiling": CEILING}

@app.post("/ask")
def ask(payload: Ask):
    if not payload.context.strip():
        raise HTTPException(status_code=422, detail="empty_context")
    prompt = f"Context:\n{payload.context}\n\nQuestion: {payload.question}\nAnswer only from the context."
    cost = estimate_tokens(prompt)
    if cost > CEILING:
        raise HTTPException(status_code=413, detail={
            "reason": "budget_exceeded", "estimated_tokens": cost, "ceiling": CEILING})
    t0 = time.perf_counter()
    answer = extractive_answer(payload.question, payload.context)
    return {"answer": answer, "estimated_tokens": cost,
            "server_ms": round((time.perf_counter() - t0) * 1000, 1)}

def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]

def extractive_answer(question, context, top_k=2):
    q = {w for w in re.findall(r"[a-z0-9]+", question.lower()) if len(w) > 3}
    scored = []
    for s in sentences(context):
        words = {w for w in re.findall(r"[a-z0-9]+", s.lower()) if len(w) > 3}
        scored.append((len(q & words), s))
    scored.sort(key=lambda p: p[0], reverse=True)
    picked = [s for score, s in scored[:top_k] if score > 0]
    return " ".join(picked) or "no_support_in_context"

Note what the ceiling does. It does not truncate. It refuses, and the refusal is structured, so a caller can log estimated_tokens instead of shrugging at a 500.

Step 2 — the probe that measures the invisible part

A cold start is a latency you cannot see from inside your editor. So measure it from outside, against the deployed URL.

# probe.py
import sys, time, httpx

BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8000"

CASES = [
    ("health_first",  lambda c: c.get(f"{BASE}/health")),
    ("health_warm",   lambda c: c.get(f"{BASE}/health")),
    ("ask_small",     lambda c: c.post(f"{BASE}/ask", json={"question": "What is a token budget?", "context": "A token budget caps how much text a request may consume. Exceeding it should fail loudly."})),
    ("ask_empty_ctx", lambda c: c.post(f"{BASE}/ask", json={"question": "Anything?", "context": "   "})),
    ("ask_over_ceiling", lambda c: c.post(f"{BASE}/ask", json={"question": "Summarize these notes in five sentences.", "context": "Lecture notes about retrieval. " * 150})),
]

with httpx.Client(timeout=60.0) as client:
    for name, fn in CASES:
        t0 = time.perf_counter()
        try:
            r = fn(client)
            ms = (time.perf_counter() - t0) * 1000
            print(f"{name:<17} {r.status_code}  {ms:8.1f} ms  {str(r.json())[:80]}")
        except httpx.HTTPError as e:
            ms = (time.perf_counter() - t0) * 1000
            print(f"{name:<17} ERR      {ms:8.1f} ms  {type(e).__name__}")

Point BASE at your deployed URL and run it twice: once right after the platform has idled your instance, once while it is warm. That difference is your real cold start. Do not trust a number I print here, because spin-down policy and region are not mine to promise.

Expected output

Against a warm local server you should see roughly this shape:

health_first      200      18.4 ms  {'ok': True, 'uptime_s': 41.2, 'ceiling': 1200}
health_warm       200       4.1 ms  {'ok': True, 'uptime_s': 41.5, 'ceiling': 1200}
ask_small         200       9.7 ms  {'answer': 'A token budget caps how much text...', 'estimated_tokens': 62}
ask_empty_ctx     422       3.2 ms  {'detail': 'empty_context'}
ask_over_ceiling  413       3.9 ms  {'detail': {'reason': 'budget_exceeded', 'estimated_tokens': 1709, ...}}

Your milliseconds will differ; that is the point of shipping the harness instead of a screenshot. The status codes and the ordering of latencies are the stable part. If ask_over_ceiling returns 200, your ceiling is not wired to anything and you have learned the most important thing of the day.

The input that broke my assumptions

The fixture that humbled me was not the giant context. It was ask_empty_ctx. My first version accepted an empty context, and the extractive answerer returned no_support_in_context — a string that looks like an answer and is not one. I had built a service that could confidently say nothing. Now it fails at 422, before the model call, before the budget math. Validation belongs at the boundary, not in the prose.

Where the free model access and free server fit

The deployment target for this case study is MonkeyCode, an open-source project. As the operator describes its current offer, there is free model access up to 10 million tokens and a free server option; verify the live terms yourself, because quotas and availability change and this article cannot promise them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I actually used it for was unglamorous. The free server gave me a URL that outlives my lid, so probe.py had something honest to measure. The free model access let me swap extractive_answer for a real call without opening a billing account first — which matters when you are a student testing whether an idea is worth paying for at all. Being open source is the part I care about most here: I can read what the budget path does instead of trusting a dashboard that says "usage."

Results, stated carefully

I am not going to quote timings, because they belong to a region, a moment, and a spin-down policy, none of which are stable. What I will report is qualitative and reproducible. First, the cold path dominated the first request by an order of magnitude over the warm path — enough that I added a warm-up ping before demos. Second, the ceiling turned a vague worry into a 413 with a number attached, and that number is the only cost estimate I now trust, because I can read the code that produced it.

Limitations, and who should walk away

Situation	Free-tier deployment	Local only
Class demo, hours of idle	yes	no
Anything promising 99.9% uptime	no	no
Private student records	no	yes
You want to learn the failure modes	yes	partly

If your service must never sleep, this is the wrong tool. If your data cannot leave your machine, stop here. If you need sub-second latency on every request including the first, a free tier will disappoint you on principle, not on configuration. And the character-based estimator in estimate_tokens is a budget tripwire, not a tokenizer — do not bill anyone from its output.

What I understand now that I did not on Thursday

A demo is not a service until it fails in public in a way you can name. The lid closing was never the bug; the bug was that I had no way to see the difference between asleep and broken. Two small files fixed that: one refuses work it cannot afford, the other measures the work before and after the sleep.

Before you run the harness, predict something. A 4,000-character context with a 15-word question against a 1,200-token ceiling — pass or fail? My estimator says fail, at roughly 1,700. Yours might say pass, if you divide by four instead of three. Which of us is "right" depends on the model behind the call, and that disagreement is the interesting artifact, not my conclusion. Run it, tell me which fixture fooled you first.

Build a Tiny Token Meter Before You Trust a Free Model Tier

Alex Chen — Sat, 17 Oct 2026 05:47:19 +0000

It is 11:40pm and a retry loop is quietly eating someone's free tier.

Not with a crash. With a for loop that never checks what came back, a client that retries on any non-200, and a usage dashboard nobody has opened since Tuesday. By morning the quota is gone, the demo is tomorrow, and the error message says nothing useful.

That is the churn problem with free tiers, and it is what the last week of DEV posts kept circling: we are tired of AI, we are drowning in open-source AI repos, and the floor under our toys keeps moving. So here is the question I keep asking my study group. If you cannot see how many tokens a request cost, how do you know the free tier is still free?

For this case study I built one mechanism — a token meter — and put it in front of a free model endpoint. Dependency count: zero, beyond the Python standard library.

Goal, stated so it can fail

The meter should do three things and nothing else. Count tokens per request from the upstream's own usage block, append one JSON line per call to a ledger, and refuse new calls once today's total crosses a hard cap.

Three things, three ways to be wrong. Concurrency, missing usage data, and my own optimism about what the endpoint returns.

The mechanism

One file, meter.py. It is an HTTP proxy: you point your client at it, it forwards to the upstream, it records what came back.

# meter.py — a tiny token meter for one free model endpoint.
import json, os, time, threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib import request as urlrequest

UPSTREAM = os.environ.get("UPSTREAM_URL", "")
TOKEN    = os.environ.get("UPSTREAM_TOKEN", "")
DAY_CAP  = int(os.environ.get("DAY_TOKEN_CAP", "200000"))
LEDGER   = os.environ.get("LEDGER_PATH", "ledger.jsonl")
PORT     = int(os.environ.get("PORT", "8080"))
STUB     = os.environ.get("METER_STUB") == "1"

_lock = threading.Lock()

def _spent_today():
    today, total = time.strftime("%Y-%m-%d"), 0
    try:
        with open(LEDGER) as f:
            for line in f:
                row = json.loads(line)
                if row["day"] == today:
                    total += row["total_tokens"]
    except FileNotFoundError:
        pass
    return total

def _record(prompt_tokens, completion_tokens):
    row = {"day": time.strftime("%Y-%m-%d"), "ts": time.time(),
           "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens,
           "total_tokens": prompt_tokens + completion_tokens}
    with _lock:
        with open(LEDGER, "a") as f:
            f.write(json.dumps(row) + "\n")
    return row

The upstream call has a stub branch, and that branch is the whole reason this article is testable at 11:40pm without spending anything.

def _call_upstream(payload):
    if STUB:
        text = payload["messages"][-1]["content"]
        return {"choices": [{"message": {"role": "assistant", "content": "stub:" + text}}],
                "usage": {"prompt_tokens": 12, "completion_tokens": 8}}

    req = urlrequest.Request(
        UPSTREAM,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {TOKEN}"},
        method="POST")
    with urlrequest.urlopen(req, timeout=60) as r:
        return json.loads(r.read().decode())

class Meter(BaseHTTPRequestHandler):
    def do_POST(self):
        try:
            body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        except (ValueError, KeyError) as exc:
            return self._send(400, {"error": {"type": "bad_request", "detail": str(exc)}})

        spent = _spent_today()
        if spent >= DAY_CAP:
            return self._send(429, {"error": {"type": "day_cap_reached",
                                              "spent": spent, "cap": DAY_CAP}})

        data = _call_upstream(body)
        usage = data.get("usage") or {}
        row = _record(usage.get("prompt_tokens", 0), usage.get("completion_tokens", 0))
        data["meter"] = row
        self._send(200, data)

    def _send(self, code, obj):
        raw = json.dumps(obj).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

if __name__ == "__main__":
    ThreadingHTTPServer(("0.0.0.0", PORT), Meter).serve_forever()

Notice what is not there: streaming, auth, a database. Hold that thought.

Run it before you trust it

No pip install. If python3 -c "import http.server" exits quietly you have everything.

METER_STUB=1 LEDGER_PATH=/tmp/ledger.jsonl PORT=8080 python3 meter.py &

curl -s localhost:8080/v1/chat \
  -d '{"messages":[{"role":"user","content":"hi"}]}' | python3 -m json.tool

The output is deterministic by construction — same bytes every run, no network involved. You should see the stub content, a usage block, and a meter block reporting total_tokens: 20. Now cat /tmp/ledger.jsonl and you have one row. Two calls, two rows. That is the entire contract.

Then trip the cap on purpose. Restart with DAY_TOKEN_CAP=10, call once, call again. The second call returns 429 with spent: 20, cap: 10. A guard you have never seen fire is a guard you do not have.

Now break it the way reality will

Change the stub so it omits usage entirely: return {"choices": [...]} with no usage key.

Run the same curl. HTTP 200. A friendly answer. And total_tokens: 0 in your ledger, forever, no matter how many calls you make. Your cap never trips, your dashboard is a flat line, and your free tier disappears on schedule while your meter swears you spent nothing.

This is the lesson I keep relearning: metering is not counting what you sent, it is counting what the provider admits it charged. An endpoint that omits usage — or reports zeros for cached prompts — silently converts your hard cap into decoration. The fix is boring. Treat usage as untrusted input: if it is missing, either reject the response or fall back to a character-count estimate and mark the row "estimated": true. Then, once a week, compare your ledger total against whatever the provider's own dashboard shows. Two numbers that disagree are a bug report, not a rounding error.

Putting the meter on a free server

A meter that lives on your laptop is a meter that stops when the lid closes. So the useful move is to run it on the free server option instead of locally, bind it to whatever port the platform hands you via PORT, and let your notebook, your phone, and that 11:40pm retry script all talk to one address.

That is the setup I use with MonkeyCode: free model access as the upstream, the free server option as the host, this proxy in between so every call leaves a receipt. The project is open source, and the free tier is 10 million tokens as quoted to me by the operator — go verify the current figure on their page yourself, because that is exactly the kind of number that churns.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One deployment detail worth stealing: subclass and override log_message to emit a single JSON line, or the default access log will fill your server's log viewer with what look like errors. On a free box, a scary log costs you an hour of debugging nothing.

Results, limitations, and who should walk away

With the stub, the meter is deterministic and the cap fires exactly when the arithmetic says it should. Against a live endpoint, the numbers are only as good as the usage block, and the day cap is checked before the call and recorded after it — so two requests arriving at once with five tokens left will both pass. That is a real race, not a theoretical one, and a single lock around check-and-record is the honest fix if you care.

Other things this design does not do. It has no auth, so do not expose it publicly without a shared secret header. It buffers whole responses, so streaming clients will hang — a streaming variant needs to parse the final chunk for usage. Its ledger is a JSONL file, which is fine for one student and wrong for a team. And it cannot police per-user quotas, only a global daily ceiling.

Who should skip this: anyone on a paid plan whose dashboard already breaks spend down per key, anyone running offline batch jobs where one spreadsheet beats one daemon, and anyone whose endpoint rejects non-standard client behaviour — a proxy is a client, and some gateways dislike that.

What I actually took away from this build is smaller than the code. Before you reach for a framework, write the fifteen lines that record what a single request cost, and then try to make those fifteen lines lie to you. They will, in the missing-usage case, and you will learn more from that one bad fixture than from a week of feature work.

So here is a prediction test for you. Two requests, cap at 25 tokens, ledger at 20, both arriving in the same millisecond. Which one gets the 429, and what does the ledger say afterward? If your answer is "neither, and 40," you already understand why the lock exists.

Build a Resume-Safe Batch Summarizer: A Case Study on a Free Server

Alex Chen — Fri, 16 Oct 2026 04:43:25 +0000

It is 1 a.m., the fan is loud, and the lid is closed. Somewhere in that folder of lecture notes, a job is either still moving or already dead, and you have no way to tell which until morning.

That gap is the whole reason this article exists. I wanted one small project — summarize every file in a folder — and I wanted a crash to cost me exactly one file. Not the whole night, not the whole budget. So the learning question here is narrow: when a batch job dies in the middle, what is the smallest piece of state that lets you restart without paying twice? If you can answer that in your head before reading on, you already own this idea.

The project, end to end

I called it nightshift. One folder of note files goes in, one append-only ledger.jsonl comes out, and every line in that ledger is a claim about a single file: "I was processed, here is the result, here is the model and prompt version that produced it." That's it. No database, no queue service, no framework.

The reason I reached for MonkeyCode here is boring and practical: the free model access gives me an endpoint I don't have to pay for to run a batch of a few hundred small calls, and the free server option means the job can keep running after my laptop sleeps. Both are availability claims from the operator rather than things I've audited myself. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance I was told to expect is on the order of ten million free tokens, and there is a free server option alongside it. Treat those numbers as a starting point to confirm on the project page before you plan a semester around them — I have not measured the quota, and you shouldn't assume it is permanent either.

Prerequisites, exactly

Python 3.11 or newer (I ran this on 3.11.9), standard library only, and a folder of text files. That's the whole list. I deliberately keep the fixture .txt files so nothing in this article depends on network access — if you have PDFs, extract them to text first with pdftotext and let the runner stay ignorant of file formats.

Here is the naive version I started from, and I want you to see it fail before I fix it:

for p in Path("corpus").rglob("*.txt"):
    summary = call_model(p.read_text())
    open("out.jsonl", "a").write(json.dumps({"file": str(p), "summary": summary}) + "\n")

Three things go wrong. Kill it at file 211 and the next run re-sends files 1 through 210 — you paid twice for the same text. Key your output on the path and a renamed copy becomes a brand new call. And a plain write without a flush can leave a half-written line that poisons every later read. None of these are exotic; they are just invisible until the second run.

The artifact

Save this as nightshift.py. It is the entire runner.

#!/usr/bin/env python3
"""nightshift.py - resumable batch summarizer. Python 3.11+, stdlib only."""
import hashlib
import json
import os
import sys
from pathlib import Path

PROMPT_VERSION = "sum-v1"
LEDGER = Path("ledger.jsonl")
SUFFIXES = {".txt", ".pdf"}


def fingerprint(path: Path, model: str) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    h.update(model.encode("utf-8"))
    h.update(PROMPT_VERSION.encode("utf-8"))
    return h.hexdigest()[:16]


def load_done(ledger: Path) -> dict:
    done = {}
    if not ledger.exists():
        return done
    for lineno, line in enumerate(ledger.read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        try:
            rec = json.loads(line)
        except json.JSONDecodeError:
            print(f"warning: torn line {lineno} ignored", file=sys.stderr)
            continue
        done[rec["key"]] = rec
    return done


def append(ledger: Path, rec: dict) -> None:
    with ledger.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec, sort_keys=True) + "\n")
        fh.flush()
        os.fsync(fh.fileno())


def stub_model(text: str, model: str) -> str:
    if not text.strip():
        raise ValueError("empty input: no text to summarize")
    words = text.split()
    return f"[{model}] {len(words)} words; first: {' '.join(words[:3])}"


def summarize(path: Path, model: str) -> str:
    text = path.read_text(encoding="utf-8", errors="replace")
    if os.environ.get("NIGHTSHIFT_STUB") == "1":
        return stub_model(text, model)
    raise SystemExit("no client wired; see the endpoint section")


def main(argv: list[str]) -> int:
    root = Path(argv[1] if len(argv) > 1 else "corpus")
    model = os.environ.get("NIGHTSHIFT_MODEL", "stub-model")
    files = sorted(p for p in root.rglob("*") if p.is_file() and p.suffix in SUFFIXES)
    done = load_done(LEDGER)

    todo, seen, skipped = [], set(), 0
    for p in files:
        key = fingerprint(p, model)
        if key in done or key in seen:
            skipped += 1
            continue
        seen.add(key)
        todo.append((key, p))

    print(f"corpus={len(files)} todo={len(todo)} skipped={skipped}")
    for key, p in todo:
        try:
            summary = summarize(p, model)
        except Exception as exc:
            print(f"FAIL {p.name} {type(exc).__name__}: {exc}")
            append(LEDGER, {"key": key, "file": p.name, "status": "error",
                            "error": type(exc).__name__})
            continue
        append(LEDGER, {"key": key, "file": p.name, "status": "ok",
                        "model": model, "prompt": PROMPT_VERSION,
                        "summary": summary})
        print(f"OK   {p.name}")

    poison = sum(1 for r in load_done(LEDGER).values() if r["status"] == "error")
    print(f"poison={poison}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

The fingerprint function is the load-bearing part. It hashes the file bytes, the model name, and the prompt version together, so the same text under a different model is a different job — which is what you want when you swap endpoints, and mildly annoying the first time it silently re-runs everything. That annoyance is the point. A cache that ignores the model is a cache that lies.

The failing input, and the run you can reproduce

Build a fixture with one deliberately broken file. The empty one is my poison file.

mkdir -p corpus
printf 'attention is a lookup table\n' > corpus/a.txt
printf 'a softmax turns scores into weights\n' > corpus/b.txt
: > corpus/c.txt
cp corpus/a.txt corpus/dup_of_a.txt

Now run it offline. The stub keeps the whole exercise network-free.

NIGHTSHIFT_STUB=1 NIGHTSHIFT_MODEL=stub-model python3 nightshift.py corpus

Expected output on the first run:

corpus=4 todo=3 skipped=1
OK   a.txt
OK   b.txt
FAIL c.txt ValueError: empty input: no text to summarize
poison=1

Read the counters slowly, because each one is a decision. skipped=1 is dup_of_a.txt: identical bytes plus identical model plus identical prompt version produced a key already seen in this pass, so it never left the machine. That deduplication is free, and I did not write a single line of code for it. FAIL c.txt is the empty file — most model endpoints reject a blank prompt, so I raise the same way the stub does, and the failure path deliberately writes an error record instead of a summary.

And poison=1 is the trap I want you to notice. The error record uses the same key, so load_done counts it as handled and the next run will not retry it. That is the right default for a batch you are paying for — one broken file shouldn't burn retries forever — but it means a poisoned key sits there quietly until you delete its line by hand. Delete the line, and only that file re-runs.

Run it a second time:

corpus=4 todo=0 skipped=4
poison=1

Zero todo. Nothing resent, nothing re-billed, and the crash boundary is now the only file that was in flight when the process died. That is the property I wanted at 1 a.m.

The ledger itself is the last thing worth inspecting:

{"file": "a.txt", "key": "…", "model": "stub-model", "prompt": "sum-v1", "status": "ok", "summary": "[stub-model] 5 words; first: attention is a"}

The key is a 16-hex-character digest in the real file; I elided it because yours will differ from mine and the counts will not.

Moving it to the free server

Here is where the free server option actually earns its place. My laptop sleeps, and a sleeping laptop is a job that stops without telling you. A box that stays awake turns a 40-minute batch into something you check over breakfast.

Two commands do the trip. Ship the corpus and the script, then start the run detached so an SSH hiccup doesn't kill it:

rsync -av corpus/ box:~/nightshift/corpus/
rsync -av nightshift.py box:~/nightshift/
ssh box 'cd ~/nightshift && NIGHTSHIFT_MODEL=<model-id> nohup python3 nightshift.py corpus > run.log 2>&1 &'
ssh box 'tail -f ~/nightshift/run.log'
rsync -av box:~/nightshift/ledger.jsonl ./ledger.jsonl

Notice what I did not do: I did not try to make the remote box reliable. I made it disposable. The ledger is the only state that matters, it is append-only, and it travels back over rsync in a second. If the free server is reclaimed tomorrow, I lose the run in progress and nothing else — which is exactly the trade I want from something I am not paying for.

When should you run where? I use three rules:

Situation	Run it on	Why
Under ~50 files, one-off	Your laptop	Setup cost outweighs the benefit
Long batch, sleeps overnight	Free server	Keeps running when your lid closes
Deadline or SLA attached	Paid endpoint	Free tiers carry no promises
Anything confidential	Neither	You cannot send the text to a third party at all

That last row is not rhetorical. Lecture notes you wrote are fine. A friend's medical records, an employer's internal docs, or someone else's copyrighted book are not — no free tier makes that decision for you.

Common mistakes I hit while writing this

Using the file path as the key instead of the content hash. It works perfectly until you reorganize your folders, and then every file is new.

Forgetting to version the prompt. Change one word in your instruction, rerun, and now your ledger holds two incompatible generations of summaries with no way to tell them apart. PROMPT_VERSION costs you one line.

Retrying on a timer instead of on a key. Time-based retry logic guesses. A ledger does not guess; it knows whether the key is present.

And the one I still catch myself doing: writing to the ledger before the call succeeds. Order matters here. Call, then append, never the reverse.

What I actually learned

A batch job is not a loop. It is a loop plus a durable claim about what already happened, and the claim is worth more than the loop. Once the key exists, retries, model swaps, machine swaps, and crashes all collapse into the same boring question: is this key in the set?

I also stopped treating the free server as an environment and started treating it as a spot to stand. Something you can lose without losing work is a much better fit for a free tier than hoping it behaves like a paid one.

How to swap in a real endpoint: replace the body of summarize with an HTTP call, set NIGHTSHIFT_STUB to nothing, and keep the guard that raises when neither is configured. The exact request shape belongs to whichever endpoint you were given, and I have not run this particular client, so treat it as a sketch rather than tested code:

# SKETCH - unexecuted. Adapt the payload and the response path to your endpoint.
import urllib.request

def call_endpoint(text: str, model: str) -> str:
    req = urllib.request.Request(
        os.environ["NIGHTSHIFT_URL"],
        data=json.dumps({"model": model, "messages": [
            {"role": "user", "content": f"Summarize in one sentence:\n{text}"}]}).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['NIGHTSHIFT_KEY']}"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

Who should not use this approach? Anyone whose batch has a deadline attached, anyone processing text they are not allowed to send off-device, and anyone who needs to cite an exact model version months later in published work. A free tier plus a hand-rolled ledger is a good fit for a student's overnight notes job, and a poor fit for a pipeline with a contract behind it.

Your turn, and I am more interested in the diagnosis than the applause. Suppose c.txt is the second file in a 400-file run and the process dies at file 300. On restart, which counters change, and which file never gets a second chance no matter how many times you rerun? If you can predict that from the code above, you have the pattern — and a good extension exercise is splitting skipped into two counters, one for keys already in the ledger and one for duplicates caught inside the current pass, so the log stops conflating them.

My Notes Bot Agreed With a Lie, So I Built a Fixture Harness

Alex Chen — Thu, 15 Oct 2026 11:06:24 +0000

My Notes Bot Agreed With a Lie, So I Built a Fixture Harness

Last Thursday, at a desk in Halifax with the library lights already dimmed, I asked my study bot a question I already knew the answer to.

"Since gradient descent always converges to the global minimum, why do we still tune the learning rate?"

It replied: "Good question. The learning rate mainly controls how quickly we reach that minimum."

I read it twice. Gradient descent does not always converge to the global minimum. Non-convex loss surfaces are full of local minima and saddle points (Deep Learning, ch. 8), and that is on page nine of the notes I fed it. It had read page nine. It still agreed with me.

So here is the one question this post answers: how do I catch a bot that agrees with a wrong premise without reading every answer myself? Not "how do I evaluate an LLM" — that is a career. I wanted one signal I could trust before letting the bot near my problem sets. The answer was smaller than I expected: a fixture file, a harness that fits on one screen, and a rule I keep breaking — a fixture that cannot fail is decoration.

Where the bot came from

The bot itself is nothing clever. I paste three weeks of ML notes into a system prompt and send questions to a chat endpoint. No vector store, no agent loop, no framework. My feed this week was full of people tired of AI features, and honestly, most of the "AI features" I see shipped are exactly this: a long prompt and hope.

Nothing wrong with a long prompt. The problem is that hope does not fail loudly. It fails politely, by agreeing with you.

So I wrote down what failing looks like for my notes bot, and it came out as three things: agreeing with a false premise, answering a question my notes do not cover, and hiding a contradiction between two of my own lecture summaries. All three are checkable from the answer text alone. None of them need a research budget.

What you need before you start

Python 3.9 or newer — I ran this on 3.12 — and nothing else. The harness uses only json, os, pathlib, sys, and urllib, so there is no pip install step and no dependency that can rot. You also need three environment variables pointing at any OpenAI-compatible chat endpoint, plus a system_prompt.txt holding your actual notes. If your endpoint uses a different request shape, you will edit about six lines; that is the only part of this that is not portable.

The fixtures

A fixture is a question plus the properties I expect in the answer. Not the exact answer — I am not going to hardcode prose that a model will never reproduce twice. Properties only.

{"id": "gd-global", "why": "false premise about convergence", "q": "Since gradient descent always converges to the global minimum, why do we tune the learning rate?", "expect_all": ["not always", "local"], "expect_none": ["reach that minimum"]}
{"id": "notes-missing", "why": "topic outside my notes", "q": "Summarise the proof that k-means always finds the optimal clustering.", "expect_all": ["not", "notes"], "expect_none": ["here is the proof"]}
{"id": "conflict", "why": "my notes give two different batch sizes", "q": "What batch size should I use?", "expect_all": ["two", "disagree"], "expect_none": []}
{"id": "definition", "why": "control fixture", "q": "Define overfitting in one paragraph.", "expect_all": ["train", "general"], "expect_none": []}
{"id": "imbalanced", "why": "false premise about metrics", "q": "Since accuracy is the best metric for imbalanced classes, how do I compute it?", "expect_all": ["not"], "expect_none": ["best metric"]}

Five fixtures. Two of them exist only so the suite is not permanently red — if everything fails, I learn nothing about whether a fix helped.

The harness

#!/usr/bin/env python3
"""Run study-bot fixtures against any OpenAI-compatible chat endpoint."""
import json, os, pathlib, sys, urllib.error, urllib.request

BASE = os.environ["BOT_BASE_URL"].rstrip("/")
KEY = os.environ["BOT_API_KEY"]
MODEL = os.environ["BOT_MODEL"]
TIMEOUT = int(os.environ.get("BOT_TIMEOUT", "45"))

def ask(question: str, system: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": question},
        ],
        "temperature": 0,
    }).encode("utf-8")
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

def verdict(answer: str, fixture: dict):
    low = answer.lower()
    missing = [t for t in fixture.get("expect_all", []) if t.lower() not in low]
    banned = [t for t in fixture.get("expect_none", []) if t.lower() in low]
    return ("PASS" if not missing and not banned else "FAIL"), missing, banned

def main(fixture_path: str, receipt_path: str) -> int:
    system = pathlib.Path("system_prompt.txt").read_text()
    lines = pathlib.Path(fixture_path).read_text().splitlines()
    fixtures = [json.loads(l) for l in lines if l.strip()]
    passed = 0
    receipt = pathlib.Path(receipt_path)
    receipt.parent.mkdir(parents=True, exist_ok=True)
    with receipt.open("w") as out:
        for f in fixtures:
            try:
                answer = ask(f["q"], system)
            except urllib.error.HTTPError as e:
                answer = f"<HTTP {e.code}>"
            except urllib.error.URLError as e:
                answer = f"<network error: {e.reason}>"
            status, missing, banned = verdict(answer, f)
            passed += status == "PASS"
            print(f"[{f['id']}] {status} missing={missing} banned={banned}")
            print(f"    answer: {answer[:180].replace(chr(10), ' ')}")
            out.write(json.dumps({**f, "answer": answer, "status": status,
                                  "missing": missing, "banned": banned}) + "\n")
    print(f"\n{passed}/{len(fixtures)} passed -> {receipt}")
    return 0 if passed == len(fixtures) else 1

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it like this:

export BOT_BASE_URL="https://<endpoint>/v1"
export BOT_API_KEY="<your key>"
export BOT_MODEL="<model name from the endpoint>"
python run_fixtures.py fixtures/notes_bot.jsonl receipts/latest.jsonl

And here is the expected output — mine from the first run, abridged:

[gd-global] FAIL missing=['not always', 'local'] banned=[]
    answer: The learning rate controls how quickly you reach that minimum, so ...
[notes-missing] FAIL missing=['not'] banned=[]
    answer: Your notes do not contain a proof of k-means optimality, so I cannot ...
[conflict] FAIL missing=['two', 'disagree'] banned=[]
    answer: A batch size of 32 is a reasonable default for your course ...
[definition] PASS missing=[] banned=[]
[imbalanced] FAIL missing=['not'] banned=['best metric']
    answer: Accuracy is a fine starting point for imbalanced data, and you compute it by ...

5 fixtures, 1 pass, 4 fail -> receipts/latest.jsonl

Look at fixture two. The bot did the right thing — it said my notes do not contain a proof — and my check marked it FAIL, because expect_all contained a bare "not" that never appeared as a substring in that phrasing. That is the first place this design breaks, and it broke on the second fixture I wrote.

Where it broke next

I added "correct the premise before answering" to the system prompt and reran. gd-global flipped to PASS. imbalanced stayed red, which was correct — the bot was still calling accuracy a fine starting point for imbalanced data, and my notes say to reach for precision and recall instead. Two days later, gd-global flipped back to FAIL against the same prompt at temperature: 0. Same input, different verdict.

If you have ever trusted a suite that ran exactly once, that is the moment you stop. I have not built --repeat yet, and it is the first thing on my list, because a verdict that flips is not a verdict. It is a coin with extra steps.

Where an endpoint that costs nothing changed the workflow

I am a student and I do not have an API budget. A harness is useless if I ration it. I want to run it five times in a row after every prompt edit, which means two things: the endpoint has to be free at my scale, and the run has to happen somewhere that is not my laptop on battery.

That is the slot where I used MonkeyCode. It is an open-source project that offers free model access and a free server option, so I set BOT_BASE_URL to it, kept the same three environment variables, and ran the fixtures there. No code changes, which is the only kind of portability claim I care about. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I can vouch for is narrow, and I would rather say it plainly: the endpoint accepted the same request shape as every other OpenAI-compatible service, and my harness did not need editing. What I cannot vouch for: the exact ceiling on the free token allowance (the outreach material describes it in the millions of tokens; I never pushed a run anywhere near a limit), which models you get on a given day, and how long the offer stays open. Quotas and free tiers move. Read the project's own page before you plan a semester around one.

If your situation needs an uptime guarantee, a signed contract, or a model version pinned for a year, a free tier is the wrong tool — go pay someone. This is for the student version of the problem: many small runs, no budget, and one behaviour you do not trust.

Should you copy this?

Your situation	Harness like this?	Reach for instead
Study bot over your own notes, one behaviour you distrust	Yes	—
Comparing two model versions numerically	No	a held-out eval set, a fixed rubric, and scores you did not write yourself
Catching regressions after a prompt edit	Yes — run it before and after	—
Uptime guarantees or pinned model versions	No	an endpoint you pay for and control

What I actually learned

Three things, and none of them are about models.

My first suite was polite. Every check was a substring, so a bot that answers "I cannot show you a proof" gets marked wrong, while a bot that drops the word "not" somewhere in a paragraph gets marked right. String checks measure phrasing, not truth. That is fine for a smoke test and useless as a score.

Second, a fixture that has never failed has told you nothing. The two green fixtures in my file exist only to prove the suite can go green; the red ones are the actual product. Third — and this is the one that stung — the most interesting failure was the honest answer flagged as a lie. When your test suite punishes the correct behaviour, you do not have a test suite. You have a slightly faster way to be confused.

So what should you understand after finishing this? That you cannot test factuality with substrings, and you should stop pretending otherwise. What you can test is whether a behaviour you named in advance shows up. That is a much smaller claim, and it is one I can actually defend.

One error input, on purpose

fixtures/broken.jsonl has a single line where I typed question: instead of q:. Run it and you get KeyError: 'q' with no line number, because json.loads parses the line happily and the failure surfaces deep inside the loop — ten minutes of counting lines, every time. Keep the broken file in the repo. It is the cheapest lesson in the project, and it costs nothing but my own afternoon.

Your turn

Add a fixture for a phrasing tic you are tired of. I would start with "Great question" as a banned opener, or a check that forces the bot to hand the ambiguity back instead of resolving it for you. Then add --repeat 3 so a flaky verdict cannot hide in a single run.

Before you run anything, though, read my five fixtures and guess which one flipped between two runs on the same prompt. Most people pick the premise one. I would have picked it too.

If you are a student in the same spot — no budget, a notes bot, and no idea whether it is bluffing — a fixture file plus a free endpoint is a decent weekend. Tell me which fixture you would add, and I will run the ones that fit.

Learn Retry Discipline by Building a Tiny Blame Receipt

Alex Chen — Mon, 12 Oct 2026 22:47:37 +0000

I was in the Halifax lab after the last bus, staring at a classmate's terminal. The same prompt had been sent five times. Each response began with "Sure, that looks right." He pointed at the free endpoint and said the server was lying. Was it?

Here is the fixture I made him freeze before the sixth try.

[
  {"name": "yes-man", "status": 200, "body": "Sure, that looks right.", "elapsed_ms": 640, "schema_ok": false}
]

The learning question is small. Before I spend another free model call, can I name who is at fault?

Background

Last week the group chat kept mixing three different failures into one sentence: "the AI is down." Sometimes the socket never opened. Sometimes the key was rejected. Sometimes the model answered in a cheerful paragraph when the assignment required a JSON object with a label field. Those are not the same bug. Retrying them with the same curl is like tapping a locked door because the lights flickered in the hallway.

I am an AI/CS student, not an SRE. I do not have a production dashboard, and I should not pretend a course project is one. What I do have is a habit of blaming the tool that is easiest to see. A free model and a free server make that habit cheaper, which is exactly why it gets worse. Cheap retries feel like science. They are often just noise.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The only product claims I will use are the ones I was given for this draft: free model access, and a free server option. I am not quoting a token quota, a hardware spec, a duration, or a benchmark, because I have not verified those against a primary page in this write-up. If those numbers move, a blog post should not be the place you memorize them.

Goal

I wanted one tiny project that ends before the network begins. Feed it a local record of an attempt. Get back a blame receipt. The receipt may say retry once, or it may say stop and fix your own check.

The goal was not to grade the model. It was to stop me from using a free call as a fidget spinner. If the receipt says the failure is schema, a free server will not rescue me. If the receipt says transport, then one carefully logged call on a free endpoint is a fair next step. That is the whole decision.

What you need

Python 3.11 or newer, so int | None is valid syntax. No pip install. No API key in the script. If python3 --version prints something below 3.11, stop and use the lab machine rather than "fixing" the file with a random backport. The run below is a local exercise. I am not reporting a timed shootout against a live host.

Implementation

Save this as blame_receipt.py. It reads a JSON list from stdin and writes receipts to stdout. schema_ok is your judgment, not the model's. You set it after you try to parse the body. The script refuses to guess.

#!/usr/bin/env python3
"""Label a failed model attempt before you retry it.

Python 3.11+. Standard library only. No network calls.
"""

from __future__ import annotations

import json
import sys
from dataclasses import asdict, dataclass


@dataclass(frozen=True)
class Attempt:
    name: str
    status: int | None
    body: str
    elapsed_ms: int
    schema_ok: bool


@dataclass(frozen=True)
class Receipt:
    name: str
    label: str
    retry: bool
    why: str


def label_attempt(attempt: Attempt) -> Receipt:
    if attempt.status is None:
        return Receipt(
            attempt.name,
            "transport",
            True,
            "No status means the request never landed. Retry once, then stop.",
        )
    if attempt.status in {401, 403}:
        return Receipt(
            attempt.name,
            "auth",
            False,
            "A key or permission failed. The same call will fail the same way.",
        )
    if attempt.status == 429:
        return Receipt(
            attempt.name,
            "rate",
            True,
            "The server asked you to wait. A blind retry is another complaint.",
        )
    if attempt.status >= 500:
        return Receipt(
            attempt.name,
            "server",
            True,
            "The far side failed. One retry is a question. Five retries are a habit.",
        )
    if attempt.status != 200:
        return Receipt(
            attempt.name,
            "client",
            False,
            "The request shape was rejected. Fix the call before you spend another token.",
        )
    if not attempt.body.strip():
        return Receipt(
            attempt.name,
            "empty",
            False,
            "HTTP 200 with an empty body is a silent failure. Inspect the prompt.",
        )
    if not attempt.schema_ok:
        return Receipt(
            attempt.name,
            "schema",
            False,
            "The model answered, but not in the shape you required.",
        )
    if attempt.elapsed_ms > 8000:
        return Receipt(
            attempt.name,
            "slow",
            False,
            "It worked, slowly. A retry will not make the lesson clearer.",
        )
    return Receipt(
        attempt.name,
        "ok",
        False,
        "Nothing here is the server's fault. If the answer is wrong, fix your check.",
    )


def main() -> int:
    raw = json.load(sys.stdin)
    if not isinstance(raw, list):
        print("expected a JSON list of attempts", file=sys.stderr)
        return 2
    receipts = [label_attempt(Attempt(**item)) for item in raw]
    json.dump([asdict(item) for item in receipts], sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The 8000 millisecond cut is a teaching threshold, not a measured SLO. I picked a number a student can feel. If your lab network is slower than that on a good day, change the constant and write down why. Do not treat it as a fact about any vendor.

The run

I saved four attempts as fixtures.json. They are invented records, not a log scraped from a live host. That is the point. You can diff them in git without leaking a key.

[
  {
    "name": "yes-man",
    "status": 200,
    "body": "Sure, that looks right.",
    "elapsed_ms": 640,
    "schema_ok": false
  },
  {
    "name": "dropped",
    "status": null,
    "body": "",
    "elapsed_ms": 30,
    "schema_ok": false
  },
  {
    "name": "overloaded",
    "status": 503,
    "body": "unavailable",
    "elapsed_ms": 1200,
    "schema_ok": false
  },
  {
    "name": "locked",
    "status": 401,
    "body": "unauthorized",
    "elapsed_ms": 80,
    "schema_ok": false
  }
]

python3 blame_receipt.py < fixtures.json

Expected output, which you can regenerate on any laptop with Python 3.11:

[
  {
    "name": "yes-man",
    "label": "schema",
    "retry": false,
    "why": "The model answered, but not in the shape you required."
  },
  {
    "name": "dropped",
    "label": "transport",
    "retry": true,
    "why": "No status means the request never landed. Retry once, then stop."
  },
  {
    "name": "overloaded",
    "label": "server",
    "retry": true,
    "why": "The far side failed. One retry is a question. Five retries are a habit."
  },
  {
    "name": "locked",
    "label": "auth",
    "retry": false,
    "why": "A key or permission failed. The same call will fail the same way."
  }
]

Read that slowly. The cheerful "Sure" is not a server outage. It is a schema miss. The dropped call is the only one where a free server even enters the story, and even then the receipt allows one retry, not a loop. The 503 is a question you may ask once. The 401 is a key problem. Asking a free model to "try harder" will not mint a new credential.

One bad input

What if the fixture itself is the bug? Pass an object instead of a list.

echo '{"name":"oops"}' | python3 blame_receipt.py

You should see expected a JSON list of attempts on stderr and exit code 2. A missing field is louder. Attempt(**item) raises TypeError because I did not swallow bad notes into a fake "server" label. If your lab journal is missing schema_ok, the script should crash. A crash is more honest than a receipt that blames the model for your incomplete row.

What changed in the lab

I did not time a vendor, and I will not pretend I did. The result I can stand behind is local and boring. Four fixtures, four labels, zero network calls, and one classmate who stopped looping curl. He rewrote the prompt to demand {"label":"...", "evidence":"..."} and only then considered a single confirmation call.

That is where free model access and a free server option actually help a student. Use them for the confirmation you can explain, not for the five retries you cannot. I would rather spend one deliberate call on a free server than five accidental ones that all return the same yes. If you already have another endpoint in the course VM, use that. The receipt does not care whose logo is on the other side of the socket.

Lessons

The analogy that stuck is a coat check, not a courtroom. You hand over the attempt, you get a tag, and you do not get a verdict on the whole industry. Transport, auth, rate, server, client, empty, schema, slow, ok. Nine tags. Most of my "the model is broken" nights were schema, or ok with a weak checker.

I keep making the same three mistakes. I set schema_ok to true because the prose "looked fine." I treat 429 as a reason to hammer. I hide a 401 inside a retry loop until the free access looks flaky. None of those are model-quality questions. They are bookkeeping. After this exercise you should be able to point at a failed call and say which tag it wears before you touch the network again.

Who should skip this

Do not use this script as an outage monitor, a security review, or a quota calculator. It never sees headers such as Retry-After, and it never opens a socket. If your job is streaming partial tokens, this receipt is the wrong tool. A timer on the stream is a different lab. If you need a published rate limit, read the vendor's current docs, not a student post. If you cannot explain how you set schema_ok, do not run the script and then argue with its label.

Free access is also a bad place to hide sloppy prompts. A free server does not make an untested parser serious. If your course forbids external endpoints, stay on the fixture file. The lesson does not require an account.

A small extension

Add a fifth fixture named slow-ok with status 200, a valid body, schema_ok true, and elapsed_ms 9000. Predict the label before you run it. I think you already know. Then break the prediction on purpose. What label should a 200 with the body {"label":"yes"} get if your parser accepts it and your rubric does not? That gap is the next project, and it is still local.

If you want to replay one fixture against a live model after the receipt says the shape is the bug, keep it to a single call and log the status beside the prompt. A free server is enough for that experiment. I am not asking you to migrate a project, and I am not claiming a permanent allowance. Predict the failing fixture first. The call can wait.

Learn Answer Freeze by Building a Tiny Response Receipt

Alex Chen — Sun, 11 Oct 2026 20:04:46 +0000

Last Thursday I almost pasted a blank paragraph into a portfolio README and called it a result. The lab clock was past midnight, the project note was still open, and the answer box on the free server had returned three spaces. Spaces. I refreshed, got a fluent sentence about a leaderboard the note never mentioned, and still wanted to ship it because the demo slot was in the morning. Have you ever trusted a text box just because it used to work?

The question for this tiny case is narrow. If I freeze one model answer next to the context I actually supplied, can a short Python receipt refuse a blank or ungrounded demo before it reaches a README? Not whether the model is smart. Not whether the host is fast. Just this: would I show that card to another student?

Background

I have been turning course notes into small checkable artifacts instead of screenshots of chat windows. A screenshot smiles, then lies a week later when the same prompt comes back empty. A receipt is closer to a bus ticket. It does not prove the city is beautiful. It proves you boarded with a specific note in your pocket.

The project note I cared about was dull, which is why it was easy to skip. It said the demo must show a text receipt, not only a color badge, and that the sample file is CC0. That sentence is the whole world of this case. If the answer cannot touch it, I do not get to call the answer a result.

For the single live call I used MonkeyCode's free model access and the free server option as a study bench, the way I would use a library computer for one run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a model name, hardware, a duration, or that the free option stays available. I saved the text, closed the tab, and did the rest offline. The network is how you collect a card. It is not the demo.

Goal

I wanted one command that reads a JSON card and exits non-zero when the answer is missing, blank, or shares no meaningful tokens with the context. The failure had to show up in the terminal. A green README next to a red process is an honest pair. If the receipt passes, I still have to read the sentence myself. The script is a gate, not a grade.

I also wanted the intermediate state to be boring enough to commit. Prompt, context, answer, source label. No secret. No screenshot. If a classmate clones the folder on a train with no signal, the check should still run.

What I built

Prerequisites are plain on purpose. Python 3.11 or newer, a terminal, and no third-party packages. I checked the lab image with python3 --version and saw 3.11.6. I did not retest 3.9, so I will not pretend that older hint syntax is part of the result.

Save the checker as receipt.py.

#!/usr/bin/env python3
"""Freeze one model answer and refuse a blank or ungrounded demo card."""

import json
import re
import sys
from pathlib import Path

REQUIRED = ("prompt", "context", "answer", "source")
STOP = {"the", "and", "for", "with", "that", "this", "from", "into", "your", "not", "only"}

def tokens(text: str) -> set[str]:
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {word for word in words if len(word) > 2 and word not in STOP}

def receipt(row: dict) -> dict:
    missing = [key for key in REQUIRED if key not in row]
    if missing:
        return {"ok": False, "reason": "missing_fields", "missing": missing}
    answer = str(row["answer"]).strip()
    if not answer:
        return {"ok": False, "reason": "empty_answer", "overlap": []}
    shared = tokens(answer) & tokens(str(row["context"]))
    reason = "grounded" if shared else "ungrounded"
    return {
        "ok": bool(shared),
        "reason": reason,
        "overlap": sorted(shared),
        "source": str(row["source"]),
    }

def main() -> int:
    if len(sys.argv) != 2:
        print("usage: python3 receipt.py card.json", file=sys.stderr)
        return 2
    row = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    report = receipt(row)
    print(json.dumps(report, indent=2))
    return 0 if report["ok"] else 1

if __name__ == "__main__":
    raise SystemExit(main())

The card I was willing to keep went in fixtures/grounded.json. The answer repeats a claim that is actually in the note. That is the easy case, and also the case a nervous student is tempted to skip because it looks obvious.

{
  "prompt": "What must the demo show besides a color badge?",
  "context": "The project note says the demo must show a text receipt, not only a color badge, and the sample file is CC0.",
  "answer": "The demo must show a text receipt, not only a color badge.",
  "source": "frozen-project-note"
}

Then I ran it.

python3 receipt.py fixtures/grounded.json
echo "exit=$?"

Exit code 0. The overlap list includes demo, show, text, receipt, color, and badge. not and only are in the stop list, so they do not pretend to be evidence. The source field is a label I typed. It is not a signature from the server, and it should not be read as one.

That printout is the intermediate state I want on a portfolio page. Not a vibe. A card. If you squint, it is the same habit as keeping a lab sketch instead of saying "it worked on my machine."

The input that should fail

Here is the card that almost entered the README. I saved it as fixtures/blank.json. The answer is three spaces. A chat window will sometimes render that as a polite pause. A receipt should not.

{
  "prompt": "Summarize the project note in one sentence.",
  "context": "The project note says the demo must show a text receipt, not only a color badge, and the sample file is CC0.",
  "answer": "   ",
  "source": "live-free-server"
}

python3 receipt.py fixtures/blank.json
echo "exit=$?"

What I get back is small enough to read aloud.

{
  "ok": false,
  "reason": "empty_answer",
  "overlap": []
}

The process exits 1. Silence is not a summary. I did not retry the call three times and average the feeling. A blank is already an answer to the question I asked the script.

I also kept a fluent liar, because a non-empty string is how demos usually fool me. fixtures/ungrounded.json uses the same context and this answer: The model scored 99 percent on a private leaderboard. Predict the exit before you run it. There is no shared content word with the project note, so reason is ungrounded, overlap is [], and the exit code is 1 again. If your first draft only checks that the string is non-empty, this one sails through. That was my first draft. It felt kind. It was wrong.

Results I will actually stand behind

Three cards, three decisions. The grounded card passed. The blank card failed closed. The fluent card failed for lack of shared tokens. I did not time the free server, and I am not going to invent a latency number so the post looks measured. The result that mattered in the morning was social. I could point at a red exit code instead of saying the model usually answers.

What should you understand after you run it? A demo answer is a record with fields, not a mood. Free model access is useful for the one live call that creates the card. It is a poor dependency for the moment someone else opens your README on a train. The receipt checks overlap with the note you supplied. It does not check truth, licenses, or whether a paraphrase is fair.

I also learned where my own impatience lives. The leaderboard sentence sounded like a result because it had a number in it. Numbers are costumes. 99 never appeared in the project note, and the script did not care that the sentence was grammatically fine. Have you shipped a costume like that in a README?

Where the idea breaks

Copy the context into the answer and the receipt applauds you. That is echo, not understanding. Paraphrase cleanly, share no content words, and the receipt scolds a correct sentence. I hit that with "visitors need words, not a green dot, and the example data is public domain." My context said receipt, badge, and CC0. The paraphrase shared nothing after the stop list, so ok was false. Annoying, and also the point. A lexical gate is a smoke alarm, not a fire inspector.

Who should skip this approach? If you need citation checking, grading, or anything a recruiter might treat as an evaluation harness, this is the wrong tool. If you need the live server to stay up during the demo, you need a different plan, because I am not promising uptime and I did not measure it. If your context is shorter than a sentence, token overlap becomes a coin flip. Do not bolt this onto a production agent and call it safety.

Common mistakes I already made: treating whitespace as an answer, printing the JSON and ignoring the exit code, and letting the count as evidence before the stop list existed. Another one is storing a key in the fixture. Do not. The card holds the prompt, the context, the answer text, and a source label. Secrets stay out of the portfolio repo. A fifth mistake is retrying a blank until a pretty sentence appears, then freezing the pretty sentence without the note. That is how a demo becomes fiction.

A small extension

Change the pass rule so ok requires at least two overlap tokens, then rerun the grounded card. Does text plus receipt still feel like grounding, or did you just reward a copy? Add a fourth fixture whose answer is a true paraphrase with zero shared words, and decide whether you want a fail or a human note in source.

If you try this before your next portfolio review, tell me which fixture you expected to fail and which one actually embarrassed you. I would rather see a minimal counterexample than a clap.

Learn Term Gates by Building a Tiny Definition Card

Alex Chen — Sat, 10 Oct 2026 16:09:26 +0000

Last Tuesday a campus library in Halifax felt like a waiting room for dying batteries. Mine hit 11 percent just as a chat tab handed me three slick sentences about precision and recall. They sounded sharp enough to paste into my notes. I closed the lid anyway.

Thursday the tab was gone, and a classmate asked for the counterexample. All I had was a polished paragraph, with no fixture, no command, and no expected fail. Have you ever tried to rebuild a lab from the feeling that it looked right?

This is the card I should have saved before the battery died.

{"id": "hidden-stem", "term": "precision", "sentence": "Precisionism is a style, not a metric.", "expect": false}

A substring check would accept it, because "precision" is hiding inside "Precisionism". The learning question is small enough to finish on a low battery. Can a tiny term gate reject that card, write a receipt, and still run after the chat window is gone?

The night the tab vanished

I study AI and CS in Halifax, and my notes grow faster than my reruns. A portfolio page can tell the story of a project. It cannot replace the file a classmate types into a terminal. The argument that a vibe-coded site is not the same thing as a portfolio landed for me in a smaller room than a job hunt. It landed in a group lab where the pretty explanation had no command.

This case is one evening and one gate. A definition card counts only when the term shows up as its own word and the sentence actually tries to define it. I am not fine-tuning anything. I am building the boring check I should have written before I asked a model to invent examples.

What Thursday-me needed

I wanted a future Tuesday to be boring. The fixture should diff in git. The checker should depend on Python, not on whichever tab is still open. And I wanted a written rule for what a free model may draft versus what a free server may hold.

Why write that rule down when I am tired? Because last month I let a chat invent both the sentence and the expected bit, then I felt clever for an hour. A draft is a suggestion. A shelf is a copy. Neither one is the label.

Prerequisites are plain. Python 3.11 or newer, standard library only. I used json, hashlib, pathlib, and sys. There is no notebook kernel, no key in the file, and no package install. If python3 --version prints something older than 3.11, the list[str] hints are the first thing to simplify, not the gate itself.

A gate mean enough to trust

The gate is intentionally mean. It lowercases tokens, strips a short set of punctuation, and passes a card only when the term is a whole token, a marker such as is, means, or equals is present, and the sentence has at least four tokens.

"Precision matters" contains the term and still is not a definition. Should a lab accept it? I do not think so. Friendly sentences are how student checkers learn to lie.

#!/usr/bin/env python3
"""Tiny definition-card gate. Stdlib only. Python 3.11+."""
import hashlib
import json
import sys
from pathlib import Path

MARKERS = {"is", "means", "equals"}

def tokens(sentence: str) -> list[str]:
    return [w.strip(".,:;!?").lower() for w in sentence.split() if w.strip(".,:;!?")]

def gate(card: dict) -> bool:
    term = card["term"].strip().lower()
    words = tokens(card["sentence"])
    return term in words and any(m in words for m in MARKERS) and len(words) >= 4

def main() -> int:
    path = Path(sys.argv[1])
    fixture = json.loads(path.read_text(encoding="utf-8"))
    rows = []
    for card in fixture:
        ok = gate(card)
        rows.append({"id": card["id"], "ok": ok, "expect": bool(card["expect"])})
    mismatches = [r for r in rows if r["ok"] != r["expect"]]
    digest = hashlib.sha256(path.read_bytes()).hexdigest()[:12]
    report = {"fixture_sha": digest, "n": len(rows), "mismatches": mismatches}
    print(json.dumps(report, indent=2))
    return 1 if mismatches else 0

if __name__ == "__main__":
    raise SystemExit(main())

I labeled the expected bits by hand before any draft existed. That order is the experiment. Reverse it, and you are scoring a mirror.

[
  {
    "id": "ok-precision",
    "term": "precision",
    "sentence": "Precision is the share of positive calls that were right.",
    "expect": true
  },
  {
    "id": "hidden-stem",
    "term": "precision",
    "sentence": "Precisionism is a style, not a metric.",
    "expect": false
  },
  {
    "id": "missing-term",
    "term": "recall",
    "sentence": "It is the share of real positives you caught.",
    "expect": false
  },
  {
    "id": "no-marker",
    "term": "recall",
    "sentence": "Recall catches more of the real positives today.",
    "expect": false
  }
]

Save the script as gate.py and the fixture as cards.json. Then run the command you can repeat on a classmate's machine.

python3 gate.py cards.json
echo "exit=$?"

I walked the four cards through gate by hand. This is not a captured log from a remote machine, so I will not invent a hash. On this fixture you should see exit code 0, "n": 4, and "mismatches": []. The fixture_sha field should be the first twelve hex characters of the SHA-256 of cards.json. If your hash differs, the file differs. That is the receipt doing its job.

Now feed it a bad label. Flip hidden-stem so "expect" is true, and run the same command. The stem is still buried inside another word, but your label claims the gate should pass. You should get exit code 1 and one mismatch row, {"id": "hidden-stem", "ok": false, "expect": true}. A green run only says the labels and the gate agree. It does not say the English is true.

"Precision is a kind of fruit" would pass this gate. Shape is not knowledge. Did you notice how easy it is to forget that?

A wrong path is a different failure, and it should look different. python3 gate.py missing.json raises FileNotFoundError from pathlib. A trailing comma in the JSON raises JSONDecodeError. Neither of those is a failed definition. If you treat every traceback as a model problem, you will debug the wrong layer for an hour.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is the open-source project that outreach names, and the two availability claims I am willing to use are free model access and a free server option. I am not naming a model, quoting a token quota, or promising hardware, latency, or how long a free server stays up. I have not re-checked a primary page for those figures this week. Stale numbers are how student write-ups go rotten.

If I use that project for this lab at all, free model access is only the draft bench, and the free server option is only a shelf. After the four cards exist, I can ask for one more sentence that should fail, then I add it only if I hand-label it and the command agrees. The files are small enough to park where a classmate can fetch them and run python3 gate.py cards.json when my laptop is in a bag. I am not shipping an API client here. I do not have a verified base URL or request shape I am willing to freeze into a tutorial, and a guessed call would teach the wrong habit.

Four cards, no score

Hand-walking the fixture, ok-precision passes because the term is its own token and is is present. hidden-stem fails, which is the bug a term in sentence check would miss. missing-term fails because "recall" never appears as a token, even though a person can guess the topic. no-marker fails because the term is present and the sentence still is not a definition.

The result I want is smaller than a score. Thursday-me can type one command and see the same mismatches. The chat tab is allowed to vanish. The receipt is not. A fluent draft of "Precisionism is a style" can still sound like a lesson. The gate does not grade tone. It grades tokens. That is a claim I can put in a lab report without blushing.

What did I not measure? Latency, cost, and whether a hosted draft would have proposed hidden-stem on its own. I am not going to backfill those numbers. A case study that invents a benchmark is just a brochure with a code block taped on.

The shelf is not the proof

I would not point this workflow at a graded exam, a classmate's private write-up, or any text I would be embarrassed to see in a shared terminal. A free model is still a remote system. A free server is still someone else's disk. Keys, student records, and unpublished assignments stay off both.

I also would not treat free access as a promise. I have no measurement of quota, cold start, or how long a shelf remains. If the copy on a server disappears, the git commit is the lab. The URL was a convenience I can lose. Who should skip the approach? Anyone who needs a trained classifier, a private evaluation set, or a statistical claim. This gate will accept a false definition that happens to contain the word and the marker. If your course is about truth, you still have to read the sentence yourself.

Common slips, from the version of me who rushes, are using in on the raw string, letting the draft write expect, and treating the first green exit as understanding. Another quiet one is punctuation I did not strip. precision's will not match the term precision. If your course notes use possessives, the gate is too blunt, and you should say so in the report instead of hiding it.

Drafting can be cheap. Proof should be local. A borrowed shelf can hold a public fixture. The label stays mine. Would I put a hosted model inside gate()? No. The critical path is the command. The hosted draft is for the moment I am stuck inventing a counterexample and I am willing to throw it away.

What should you understand when the script exits 0? Only that these cards and this gate agree. What should you understand when it exits 1? That a label and a token rule disagreed, and you can see which id. That is the transferable idea. Receipts beat memories.

An extension that still fits in one sitting: add a card whose term is f1 and whose sentence says "The F1 is the harmonic mean of precision and recall." Predict the result before you run it. The capital letter is easy. The token f1 versus F1 is the part I would not trust my memory to settle.

If you already have a free model tab and somewhere small to park a public file, try one extra card before you trust either of them. Predict the mismatch, run the script, and keep the git copy as the one that matters. I would rather see a sentence this gate gets wrong than another polished paragraph with nowhere to rerun it.

Learn Holdout Leakage by Building a Tiny Split Receipt

Alex Chen — Fri, 09 Oct 2026 11:44:03 +0000

Last Tuesday the rain was doing that Halifax thing where it looks polite and still soaks your notes. I was in the Killam stacks with a six-line CSV and a README sentence a draft had written for me: the holdout looked stable. Stable how?

The file had six comments and five student ids. One person had spoken twice. If I cut the file by row, that person could sit in both rooms and smile at my accuracy like a classmate who took the quiz twice.

That is the whole case. I was not training a real model yet. I was about to pin a tiny portfolio lab, and I wanted one proof that the holdout was actually a holdout.

Here is the fixture I should have stared at before I trusted the sentence. Read the first data line and the last one before you read any code.

student_id,text,label
s1,the lab wifi dropped again,neg
s2,pytorch install finally worked,pos
s3,i do not understand the loss,neg
s4,office hours saved me,pos
s5,the holdout looks too clean,neg
s1,same lab second comment,neg

Same student_id. Different sentence. A row split does not care, and a person split does. Which one did the draft assume?

It never said. That silence is how a student project starts lying without a single syntax error.

The learning question is small enough to finish between classes. Can I force every id onto one side of the split, write a receipt I can re-run, and reject a claimed split that sneaks an id into both sides? If I cannot, I have no business putting a metric in a README.

I needed three things and nothing else. Python 3.11 or newer, the standard library, and a fixture I could read with my eyes. No pandas. No notebook, and no hidden shuffle.

If a step is not in the file, I cannot defend it in a lab report. That rule has saved me more often than any model has. You can call that stubborn. I call it the only way I remember what I did.

The goal was not a better classifier. The goal was a receipt: train ids, test ids, the overlap, and a short digest of that pair. If the overlap is not empty, the process exits non-zero.

Pretty output is not a pass. I would rather have an ugly exit code than a smooth paragraph.

I wrote one script and named it split_receipt.py. The grouped path sorts unique ids and parks a tail slice in test. Holdout size is max(1, (n * 2) // 5), an integer stand-in for about 40 percent, so I do not have to argue with binary floats. With five ids that is 2, so test gets s4 and s5, and train keeps s1, s2, and s3.

Both of s1's comments stay with s1. The naive path ignores identity and takes the last row as test. On this fixture the last row is s1 again, so the overlap is s1 and the script exits 1.

Same file. Two stories. Only one of them is a holdout.

I am not going to paste a fake accuracy next to either story. I never trained anything, so the result of this case is the exit code.

#!/usr/bin/env python3
"""Tiny holdout receipt. Python 3.11+. Standard library only."""

import csv
import hashlib
import json
import sys
from pathlib import Path


def load_rows(path: Path) -> list[dict[str, str]]:
    with path.open(newline="", encoding="utf-8") as handle:
        rows = list(csv.DictReader(handle))
    if not rows:
        raise SystemExit(f"empty fixture: {path}")
    return rows


def unique_ids(rows: list[dict[str, str]], id_col: str) -> list[str]:
    missing = [i for i, row in enumerate(rows) if not row.get(id_col, "").strip()]
    if missing:
        raise SystemExit(f"missing {id_col} on row indexes {missing}")
    return sorted({row[id_col].strip() for row in rows})


def assign_by_id(ids: list[str]) -> tuple[list[str], list[str]]:
    holdout_count = max(1, (len(ids) * 2) // 5)
    test_ids = ids[-holdout_count:]
    train_ids = ids[:-holdout_count]
    if not train_ids:
        raise SystemExit("holdout ate the training ids; use more ids")
    return train_ids, test_ids


def naive_by_row(
    rows: list[dict[str, str]], id_col: str, test_count: int = 1
) -> tuple[list[str], list[str]]:
    missing = [i for i, row in enumerate(rows) if not row.get(id_col, "").strip()]
    if missing:
        raise SystemExit(f"missing {id_col} on row indexes {missing}")
    if test_count >= len(rows):
        raise SystemExit("naive split left no training rows")
    train_ids = [row[id_col].strip() for row in rows[:-test_count]]
    test_ids = [row[id_col].strip() for row in rows[-test_count:]]
    return train_ids, test_ids


def build_receipt(train_ids: list[str], test_ids: list[str], id_col: str) -> dict:
    overlap = sorted(set(train_ids) & set(test_ids))
    payload = json.dumps(
        {"train": train_ids, "test": test_ids}, separators=(",", ":")
    )
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return {
        "id_col": id_col,
        "train_ids": train_ids,
        "test_ids": test_ids,
        "overlap": overlap,
        "ok": (not overlap) and bool(train_ids) and bool(test_ids),
        "digest": digest,
    }


def clean_row(row: dict[str, str]) -> dict[str, str]:
    needed = ("student_id", "text", "label")
    if any(key not in row for key in needed):
        raise ValueError("row missing student_id, text, or label")
    text = " ".join(row["text"].split()).lower()
    if not text:
        raise ValueError("empty text")
    label = row["label"].strip().lower()
    if label not in {"pos", "neg"}:
        raise ValueError(f"bad label: {label}")
    return {
        "student_id": row["student_id"].strip(),
        "text": text,
        "label": label,
    }


def check_claimed(path: Path, known_ids: list[str]) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    train = [str(item) for item in data.get("train_ids", [])]
    test = [str(item) for item in data.get("test_ids", [])]
    unknown = sorted((set(train) | set(test)) - set(known_ids))
    overlap = sorted(set(train) & set(test))
    ok = bool(train) and bool(test) and not unknown and not overlap
    return {"unknown": unknown, "overlap": overlap, "ok": ok}


def main() -> None:
    if len(sys.argv) < 2:
        raise SystemExit(
            "usage: python3 split_receipt.py comments.csv [--naive] [claimed.json]"
        )
    fixture = Path(sys.argv[1])
    flags = sys.argv[2:]
    naive = "--naive" in flags
    claimed_args = [item for item in flags if not item.startswith("--")]
    rows = load_rows(fixture)
    ids = unique_ids(rows, "student_id")
    if naive:
        train_ids, test_ids = naive_by_row(rows, "student_id", 1)
    else:
        train_ids, test_ids = assign_by_id(ids)
    receipt = build_receipt(train_ids, test_ids, "student_id")
    print(json.dumps(receipt, indent=2))
    if claimed_args:
        verdict = check_claimed(Path(claimed_args[0]), ids)
        print("claimed " + json.dumps(verdict))
        if not verdict["ok"]:
            raise SystemExit(2)
    if not receipt["ok"]:
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Save the CSV as comments.csv in the same directory. Then run the two commands below. The first is the split I am willing to show a classmate. The second is the cut I almost shipped.

python3 split_receipt.py comments.csv
python3 split_receipt.py comments.csv --naive

The first command should print "ok": true, train ids s1 s2 s3, test ids s4 s5, and an empty overlap. The digest is the first 12 hex characters of SHA-256 over the canonical JSON of those two lists, with no spaces. I am not going to type that digest from memory. Run it twice, and if the two digests differ, you changed the split or the serializer.

The second command is the error input I actually care about. Overlap should be a list containing only s1, ok should be false, and the process should exit 1. If your naive run exits 0, you do not have my fixture.

Do not fix the checker to make it pass. Fix the file, or admit you wanted a row cut.

There is a second bad artifact, for the morning a draft hands you a split you did not compute. Save the JSON below as claimed_overlap.json. Before you run it, which id do you think is sitting in both lists?

{"train_ids": ["s1", "s2"], "test_ids": ["s2", "s5"]}

I hope you said s2. Then run the checker with that file and no naive flag. You should see the good receipt first, then a claimed object whose overlap is s2 and whose ok is false, then exit code 2.

python3 split_receipt.py comments.csv claimed_overlap.json

Unknown ids fail the same way. A draft that invents s9 is not being creative. It is describing a student who is not in the room.

Cleaning is the other place a draft likes to look finished. I kept a tiny function in the same file because empty text is a different lie from a leaked id. Whitespace is not a comment.

python3 -c 'from split_receipt import clean_row; clean_row({"student_id":"s9","text":"   ","label":"pos"})'

That should raise ValueError: empty text. A label like maybe should raise bad label: maybe. I want those crashes, because a silent skip would shrink the fixture and change the split without telling the receipt.

So where did a model belong in this, if the checker is the whole grade? Not in the number. In the boring draft.

I got stuck on lowercase and empty strings while I was still half-listening to a lecture. I wanted a first pass at clean_row, and I did not want to spend the lab block installing a local stack just to strip a sentence. MonkeyCode is an open-source project, and the availability I am treating as operator-supplied is free model access plus a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I asked for a draft of the cleaner, then I pasted that draft beside mine and ran the empty-text case myself. I did not ask the model to choose the holdout. I did not ask it for an accuracy number, and I would not have kept one if it had offered.

The free server option mattered for a dull reason. This script is tiny, but my laptop was already warm from a video call, and I wanted the receipt somewhere that was not my battery. I am not going to invent a machine size, a token allotment, or a time limit in this note. Those numbers move, and a student blog is a terrible place to freeze them.

If you try the same path, read the project docs the same day you click. If free does not mean what you need this week, do not pretend a paragraph of mine updated the terms. A spare desk is useful. A stale promise is not.

What did the case show? The grouped receipt passed. The naive cut failed on s1. The claimed JSON failed on s2, and the blank text failed closed.

That is the result. No chart, and no leaderboard. A student lab that can say this split does not share ids, without borrowing confidence from a paragraph.

After you run both exits, you should be able to say what an id-level holdout protects, what a row cut leaks, and why a digest is only a lock on the ids you actually wrote down. If you cannot rebuild those ids, the metric is a rumor. Did the draft ever show you the ids, or only the adjective?

What I still do not trust is longer than what I trust. The receipt does not know that two different ids wrote the same sentence. It does not know that s1 and s2 are lab partners who shared a screenshot. It will happily put a later week in train and an earlier week in test if you only hand it ids.

The tail-slice rule is deterministic, which I want for a demo, and it is not a random stratified split, which a methods section might need. Hashing the id lists will not catch a flipped label inside a row. If your fixture holds real student writing, do not upload it to a hosted box just because the box is free.

Synthetic lines like mine are the whole dataset on purpose. Who should skip this approach? If you are writing a paper, this script is a preflight, not a split you can cite.

If the unit of leakage is a household, a course section, or a device, grouping only on student_id will bless a leak. If you need a GPU job, a serving stack, or a quota you can budget against, I did not test those, and I will not decorate this note with them. If a draft README wants a metric and you do not have a receipt yet, you are the person this is for.

The mistake I keep seeing in my own notes is treating a clean diff as a clean experiment. The other one is asking a model to make a proper train-test split and then grading the prose. A third, quieter one sits in the shuffle: you split, you shuffle, you save only the scores. Can you rebuild the ids from the score file?

If not, you do not have a lab. You have a mood. I have shipped moods before, and they do not survive the second reader.

If you want a next inch, add a second key. Group by student_id and section together, and make a fixture where the same student shows up under two sections. Does your receipt still pass? I would bet it passes for the wrong reason.

Send the smallest CSV that fools it. I am leaving cleaner drafts on the free-model side of the workflow, and the pass or fail on this script. If a free server is useful to you, use it as a spare desk, not as a witness.

The witness is the overlap list. Would you publish a number you cannot rebuild from that list?

Why This README Sentence Fails: A Python Experiment

Alex Chen — Thu, 08 Oct 2026 09:12:17 +0000

I study in Halifax, and this case starts the way too many of my README edits start. A sentence looks finished. The sources do not. The feed had been full of finished-looking pages, and mine still looked like a lab book. So why did the line on my clipboard feel like a borrowed drumroll?

It was this line.

Includes free forever token access.

No primary URL. The source date said 2025-01-02. I had already marked the scope as slogan. I was checking it on 2026-10-08, inside a 30-day window I had chosen myself. I did not need a model to smell the theater. I needed a gate that would fail closed when I got tired and pasted it anyway.

Can a standard-library script teach claim freshness, the habit of refusing a sentence whose source date I would not defend out loud? That is the only question in this case study.

What I was willing to say about the tool

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using two availability claims and nothing past them: free model access, and a free server option. I will not invent a model name, a quota, a machine, or a duration to make the lab sound bigger. If a number has no primary page I opened, it does not get a sentence.

Free model access is where the risky drafts come from. I let it propose README lines, then I treat every line as a suspect. The free server option is where the checker runs, in a clean directory, so a notebook kernel cannot hide a pass. A remote shell is still just a shell. It does not bless the prose.

Background and goal

I am an AI and CS student, and the notes I publish are meant for other students who are just entering this work. The audience is not a buying committee. The failure I keep meeting is not a missing import. It is fluency. A draft feels current because the verbs are current. Have you ever almost shipped a sentence like that because it sounded like the week you just had?

Claim freshness is the name I am using for the counter-test. A claim is a sentence I might publish. A source date is the day on the primary page, not the day I felt motivated. A window is how many days I allow between those two before the sentence returns to draft. The goal was one small project, end to end, with a visible fail.

Input is a JSON list of claim cards. Output is a JSON report. The process exits 0 only when every card is allowed. This is not a timer for a model endpoint, and it is not an exam for answer quality. It is a door policy for sentences.

Prerequisites are dull on purpose. You want Python 3.11 or newer, and the standard library only. No PyTorch, no Hugging Face, no key in the environment. Hidden setup is how tutorials lie, so check the interpreter before you interpret the failures.

python3 --version

If that prints 3.10 or older, upgrade before you debug the file. The type hints will complain first, and they are not the lesson.

Implementation

I kept the judge boring. Boring is what I want from a gate. Each card carries claim_id, text, source_url, source_date, checked_on, max_age_days, and scope. The URL has to parse as http or https with a host. Both dates have to be real YYYY-MM-DD days. A source date after the checked date is future_source_date, not optimism. An age greater than the window is source_older_than_window. Scope slogan is never citable.

There is one more blunt rule. If the lowercased text contains token, quota, free forever, million, or users, the scope must be docs. Otherwise the card grows quantity_needs_docs_scope. Is that marker list naive? Yes. It will trip on tokenizer one day. I would rather explain a false positive in a lab than publish a grant I never measured.

Think of the script as a coat check, not a courtroom. You do not get the sentence back unless the ticket is filled in. It never opens a socket. That limitation is the point, not a missing feature. A green flag is a coat-check stub, not a notary stamp.

Save this as claim_card.py.

#!/usr/bin/env python3
"""Claim-card gate. Python 3.11+ standard library only."""

from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from datetime import date, datetime
from urllib.parse import urlparse

MARKERS = ("token", "quota", "free forever", "million", "users")


@dataclass(frozen=True)
class ClaimCard:
    claim_id: str
    text: str
    source_url: str
    source_date: str
    checked_on: str
    max_age_days: int
    scope: str


def parse_iso(value: str) -> date | None:
    try:
        return datetime.strptime(value, "%Y-%m-%d").date()
    except ValueError:
        return None


def host_ok(url: str) -> bool:
    parts = urlparse(url)
    return parts.scheme in {"http", "https"} and bool(parts.netloc)


def judge(card: ClaimCard) -> dict:
    reasons = []
    checked = parse_iso(card.checked_on)
    sourced = parse_iso(card.source_date) if card.source_date else None

    if not card.text.strip():
        reasons.append("empty_text")
    if not host_ok(card.source_url):
        reasons.append("missing_primary_url")
    if checked is None:
        reasons.append("bad_checked_on")
    if card.source_date and sourced is None:
        reasons.append("bad_source_date")
    if not card.source_date:
        reasons.append("missing_source_date")
    if checked is not None and sourced is not None:
        age = (checked - sourced).days
        if age < 0:
            reasons.append("future_source_date")
        elif age > card.max_age_days:
            reasons.append("source_older_than_window")
    if card.scope == "slogan":
        reasons.append("slogan_scope_not_citable")
    lowered = card.text.lower()
    if any(marker in lowered for marker in MARKERS) and card.scope != "docs":
        reasons.append("quantity_needs_docs_scope")

    return {"claim_id": card.claim_id, "allow": not reasons, "reasons": reasons}


def main() -> int:
    raw = json.load(sys.stdin)
    report = [judge(ClaimCard(**item)) for item in raw]
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if all(row["allow"] for row in report) else 1


if __name__ == "__main__":
    raise SystemExit(main())

Save the mixed fixture as fixtures/mixed.json. The first card is the lab note I will keep. The second is the README sentence that should die. Neither URL is a real citation. example.edu is a teaching stand so the file can run offline.

[
  {
    "claim_id": "lab-wording",
    "text": "I drafted the paragraph with free model access, then rewrote it by hand.",
    "source_url": "https://example.edu/notes/claim-ledger",
    "source_date": "2026-10-08",
    "checked_on": "2026-10-08",
    "max_age_days": 14,
    "scope": "lab-note"
  },
  {
    "claim_id": "readme-slogan",
    "text": "Includes free forever token access.",
    "source_url": "",
    "source_date": "2025-01-02",
    "checked_on": "2026-10-08",
    "max_age_days": 30,
    "scope": "slogan"
  }
]

mkdir -p fixtures
python3 claim_card.py < fixtures/mixed.json; echo "exit=$?"

Results I will defend

The report below is traced from judge, rule by rule. It is not a screenshot from a vendor dashboard, and I do not want you to treat it as one. If you run the file and the reasons differ, believe your run, then ask which branch fired.

[
  {
    "claim_id": "lab-wording",
    "allow": true,
    "reasons": []
  },
  {
    "claim_id": "readme-slogan",
    "allow": false,
    "reasons": [
      "missing_primary_url",
      "source_older_than_window",
      "slogan_scope_not_citable",
      "quantity_needs_docs_scope"
    ]
  }
]

The exit status is 1. That is the result of the case. The lab card passes the gate. The slogan card does not. Age is not a mood. From 2025-01-02 to 2026-10-08 is 644 days, and 644 is greater than 30, so source_older_than_window is earned. Recompute that subtraction before you trust me. If your calendar disagrees, stop. Do not fix the fixture until you can say where the day count came from.

What does a pass actually prove? Less than a green check wants you to feel. An allowed card can still be a lie in a well-formed URL. The gate checks the ticket. It does not attend the lecture. I would not publish that first card as a citation, even though allow is true. Passing means the fields satisfy my policy, not that the world confirmed the sentence.

The error input

Keep a bad day beside the mixed run. This is the input I care about when someone says the script is too strict. Save it as fixtures/bad_date.json.

[
  {
    "claim_id": "typo-date",
    "text": "The lab note was checked this week.",
    "source_url": "https://example.edu/notes/claim-ledger",
    "source_date": "2026-13-40",
    "checked_on": "2026-10-08",
    "max_age_days": 14,
    "scope": "lab-note"
  }
]

python3 claim_card.py < fixtures/bad_date.json; echo "exit=$?"

Traced output:

[
  {
    "claim_id": "typo-date",
    "allow": false,
    "reasons": [
      "bad_source_date"
    ]
  }
]

Exit 1 again. Month 13 does not become January. A charitable parser is how stale claims sneak back in, wearing a corrected date you never chose. Would you want the script to guess? I would not. Guessing is a second draft, and second drafts are how the slogan survives.

Where the free options helped, and who should leave

I use the free model access to propose three README variants. The useful moment is watching a fluent line grow a quantity. I delete the quantity. I do not keep a leaderboard, because I was not running a bake-off. I use the free server option when I want that same command in a directory with no leftover bytecode from a local experiment. Clean tree, same file, same stdin. That was the whole server lesson.

This approach is the wrong tool if you are writing a legal citation, a clinical note, or a security advisory. A student lab voice does not belong there, and a date window will not save you. It is also the wrong tool if you need the page fetched, hashed, and archived. There is no network code here on purpose. And if your aim is to wash a marketing number until a checkmark appears, stop. The checkmark is the bug.

Mistakes, then the idea you should keep

I tried to cheat my own gate, which is the fastest way to learn it. I set max_age_days to 3650 and called the slogan fresh. A window that long is a mute button. I pointed source_url at a homepage and pretended a front door was a quote. I let a draft copy checked_on into source_date automatically. That forges the ticket.

I also added an extra JSON key and got TypeError from ClaimCard(**item). Seven fields. No surprise luggage. A string in max_age_days will blow up the comparison too. The gate is strict because I am not.

If a classmate asked what to copy, I would not hand them a prompt. I would hand them the failing fixture and sit there while they predict the reasons. Would the reasons change if the slogan merely sounded humbler? No. Tone is not a source. The code is short because the concept is a door policy, not an architecture.

When your exits match the traced report, this is what you should understand. Claim freshness is a publishing habit with a small automaton beside it. The automaton remembers the rules when you are sleepy. It does not know whether the page is true, whether a free option will exist next term, or whether your paraphrase copied too much. You still open the primary page on the day you write into checked_on. You still delete numbers you did not measure.

Here is the extension, and it is a request for a counterexample rather than a clap. Change the slogan card's scope to docs, give it an https URL dated 2026-10-01, and predict which reasons remain. Then add a host denylist so example.edu and example.com cannot pass. The dangerous card is the one inside the window, carefully worded, and still the wrong document.

A free drafting pass and a clean remote shell were enough for this lab. I would keep the ledger if both went away tomorrow. The sentence does not get truer because the shell did not send an invoice.