DEV Community: Jordan Liu

Split the Clock Before You Blame the Model

Jordan Liu — Tue, 27 Oct 2026 13:11:28 +0000

Wall clock is the most popular wrong number in AI tooling. You send a prompt, you stopwatch the answer, you blame the model. But three things live inside that number — queueing, the wire, and generation — and the model owns exactly one of them.

Which means a lot of "this model is slow" verdicts are really "this path is slow" verdicts. So why do we keep publishing the first as if we'd proven the second?

Here's the move I keep coming back to: record two timestamps, then report a ratio instead of a duration. A ratio survives laptops, months, and model swaps. A raw millisecond count does not.

The split

Time to first token tells you how long you waited for the system. Time from first token to last token tells you how fast the model actually wrote. Everything before the first token — DNS, TLS, auth, scheduling, batching — is what I call path tax. Everything after it is generation.

Add them and you get the number that ends up in your README, your ticket, your angry post. Only one of the two is the model's fault. Timing a model by total wall clock is like timing a phone call by how long your finger was on the button.

The harness

One file, no framework, about forty lines. Streaming stays on, because non-streaming hides the split completely — you get one blurry total and nothing to argue with.

# probe.py — run: TAG=laptop python probe.py --runs 30 --out ledger.jsonl
import argparse, hashlib, json, os, time
import requests

URL   = os.environ["CHAT_URL"]     # your endpoint, not mine
MODEL = os.environ["MODEL_ID"]
KEY   = os.environ["API_KEY"]
TAG   = os.environ.get("TAG", "local")

PROMPT = "Write a Python function that dedupes a list while preserving order."

def one_run(i, prompt=PROMPT, max_tokens=256):
    body = {"model": MODEL, "stream": True, "max_tokens": max_tokens,
            "messages": [{"role": "user", "content": prompt}]}
    t0  = time.monotonic_ns()
    r   = requests.post(URL, json=body, stream=True, timeout=120,
                        headers={"Authorization": f"Bearer {KEY}"})
    first = None
    for line in r.iter_lines(decode_unicode=True):
        if not line or not line.startswith("data:"):
            continue
        if line[5:].strip() == "[DONE]":
            break
        if first is None:
            first = time.monotonic_ns()
    t_end = time.monotonic_ns()
    if first is None:
        return {"run": i, "ok": False, "status": r.status_code}
    total_ms = (t_end - t0) / 1e6
    gen_ms   = (t_end - first) / 1e6
    return {"run": i, "ok": True, "tag": TAG, "model": MODEL,
            "ttft_ms": round((first - t0) / 1e6, 1),
            "gen_ms": round(gen_ms, 1), "total_ms": round(total_ms, 1),
            "path_tax": round(1 - gen_ms / total_ms, 3),
            "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:12],
            "max_tokens": max_tokens, "status": r.status_code}

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--runs", type=int, default=30)
    ap.add_argument("--out", default="ledger.jsonl")
    a = ap.parse_args()
    with open(a.out, "a") as f:
        for i in range(a.runs):
            row = one_run(i)
            f.write(json.dumps(row) + "\n")
            print(row)

Two details matter more than they look. time.monotonic_ns() instead of time.time(), because a clock correction mid-run will quietly poison your p90. And the ledger stores a hash of the prompt rather than the prompt itself, so a rerun next month is comparable without dragging your test data around.

Notice what's missing: token counts. If your stream reports usage, capture it. If it doesn't, estimate locally and label it an estimate in the ledger. An honest estimate beats a confident blank, and a blank is what most teams actually have.

Report the tax, not the speed

# report.py — run: python report.py ledger.jsonl
import json, sys

rows = [json.loads(l) for l in open(sys.argv[1])]
rows = [r for r in rows if r.get("ok")]

def q(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(p * (len(xs) - 1))))]

tax = [r["path_tax"] for r in rows]
ttft = [r["ttft_ms"] for r in rows]
tot  = [r["total_ms"] for r in rows]
print(f"n={len(rows)}")
print(f"ttft  p50={q(ttft,.5):7.0f}ms  p90={q(ttft,.9):7.0f}ms")
print(f"total p50={q(tot ,.5):7.0f}ms  p90={q(tot ,.9):7.0f}ms")
print(f"path tax p50={q(tax,.5)*100:5.1f}%  p90={q(tax,.9)*100:5.1f}%")

path tax = 1 - gen_ms / total_ms. One number, and it tells you what you were actually measuring. If it's small, your model comparison is real. If it's large, you spent the afternoon benchmarking a load balancer.

Path tax (p50)	What it means	What it's good for
under 15%	model-bound; numbers travel well	streaming UIs, latency budgets
15–40%	mixed; a closer vantage point buys real speed	agent loops, batch jobs
40–70%	path-bound; model choice is noise	offline evals only
over 70%	you are measuring infrastructure	nothing latency-sensitive

Keep max_tokens and the prompt byte-identical across arms, run at least thirty times per arm, and record the very first run but analyze it separately — cold starts deserve their own row, not a smudge on your median.

Where the free server changes the experiment

This is the part that made me build the rig properly. MonkeyCode offers free model access and a free server option, which means you can run the same probe from two places instead of one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I haven't handed you my numbers for it, and I won't — the endpoint, the time of day, and the load are yours, not mine. What I'm handing you is the thing that produces them.

So: TAG=laptop and TAG=freesrv, same prompt, same max_tokens, thirty runs each, then diff the path-tax distribution. If the tax collapses, the win came from the vantage point, not from the model — and knowing that is worth more than any leaderboard. The advertised allowance is ten million tokens, which is plenty of room for a run this size; treat it as a promotional claim to re-check later, not as a line in your budget spreadsheet.

What this rig cannot tell you

The ledger measures one prompt shape. Short prompt plus long output is a different experiment from long prompt plus short output, and tool-calling workloads with fat JSON payloads are a third one. Ten runs of one prompt is an anecdote wearing a lab coat.

You also can't compare across days without pinning the model identifier and logging it — models get quietly rerouted, and a silent version bump will read as a performance regression. If your product needs a hard latency SLO, or your compliance rules demand provenance you can prove, this workflow is a starting point rather than a home; go buy the guarantee. And if you're hoping a benchmark will settle an argument, it won't. It'll only tell you which of the three clocks you were arguing about.

Run it from both places and you'll learn something about the system — or something about your assumptions. Both are useful. If you do run it, drop your path-tax number in the comments; I'd rather compare distributions than adjectives.

Free Tokens Don't Cut the Bill. They Move It.

Jordan Liu — Mon, 26 Oct 2026 11:41:34 +0000

Free tokens don't cut the bill. They move it.

That is the whole finding, and it is why I rebuilt my evaluation harness before I sent a single prompt through a free model endpoint. When generation costs nothing, the cost does not disappear. It relocates to the part of the pipeline that decides whether the output is any good. And that part is still paid for in seconds, CI minutes, and your own attention.

So the interesting question is not "how many tokens do I get?" It is "what does one accepted output cost me now?" Those are different numbers, and only one of them shows up on a pricing page.

The experiment I actually run

I split the work into three arms. Arm A generates and ships, no gate. Arm B generates once and runs a deterministic oracle against the candidate. Arm C generates, checks, and regenerates a single time on failure before giving up. Everything else is held constant: same task set, same prompt template, same oracle, same machine.

The oracle is the load-bearing piece. It has to be something a machine can settle without judgment — a unit test suite, a JSON schema linter, a compile step, a regex contract on a log line. If you cannot write the oracle, you are not evaluating, you are reading vibes.

The free server matters here for a boring reason. The oracle should run on the same class of infrastructure where the gate will eventually live, not on your laptop, or your verification seconds are fiction. Running the check step on a free server option keeps the measurement honest without renting a bench.

The operator currently lists a free allotment of 10M tokens plus a free server option. Treat that as a claim to verify in their docs before you size anything, not as a spec.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness

Below is the skeleton I use. It is unexecuted here and the URLs and model ids are placeholders on purpose — take the real ones from the operator's documentation, because names and limits change and I will not hardcode someone else's quota into your repo.

import argparse, csv, json, subprocess, time, urllib.request
from dataclasses import dataclass, asdict

@dataclass
class Row:
    arm: str
    item: str
    tokens_in: int
    tokens_out: int
    gen_s: float
    verify_s: float
    passed: bool

def generate(base_url, model, token, prompt, timeout=60):
    body = json.dumps({'model': model,
                       'messages': [{'role': 'user', 'content': prompt}],
                       'temperature': 0}).encode()
    req = urllib.request.Request(base_url, data=body,
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + token})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    text = payload['choices'][0]['message']['content']
    usage = payload.get('usage', {})
    return text, usage.get('prompt_tokens', 0), usage.get('completion_tokens', 0), time.perf_counter() - t0

def oracle(candidate, test_cmd):
    t0 = time.perf_counter()
    proc = subprocess.run(test_cmd, shell=True, input=candidate,
                          text=True, capture_output=True, timeout=120)
    return proc.returncode == 0, time.perf_counter() - t0

Then the loop, which is where the accounting lives:

for item in tasks:
    text, tin, tout, gen_s = generate(base_url, model, token, item['prompt'])
    ok, verify_s = oracle(text, item['test_cmd'])
    attempts = 1
    if not ok and arm == 'C':
        text, tin2, tout2, gen_s2 = generate(base_url, model, token, item['prompt'])
        ok, verify_s2 = oracle(text, item['test_cmd'])
        tin, tout, gen_s, verify_s = tin + tin2, tout + tout2, gen_s + gen_s2, verify_s + verify_s2
        attempts = 2
    rows.append(Row(arm, item['id'], tin, tout, gen_s, verify_s, ok))

Run it with:

python eval_arms.py --arm B --n 30 --out b.csv
python eval_arms.py --arm C --n 30 --out c.csv

The two numbers that decide it

One number is tokens per accepted output: total tokens across the run divided by the count of candidates the oracle passed. The other is verify-seconds per accepted output. Compute both, per arm, and the free tier stops looking free.

If tokens per accepted output is high, the model is producing candidates the oracle keeps throwing away, and the token meter is not your problem — the acceptance rate is. If verify-seconds per accepted output climbs when you move from B to C, the second attempt is buying you candidates at a worse price than the first one did. And if arm A looks great on tokens and terrible on everything downstream, you have just measured the cost of trusting output.

My rule: never publish a single number for a free tier. Republish the run and check whether the median and the spread agree. A wide spread on verify-seconds means the gate has a tail, and a tail is where a batch job turns into a morning.

Where each arm earns its keep

Task shape	Arm	Why
Output is consumed by a human reviewer anyway	A	The reviewer is the oracle; extra checks buy nothing
Output must satisfy a written contract (schema, tests, compile)	B	One deterministic gate captures most of the value
Failure rate is high but failures are cheap to retry	C	A second attempt is cheaper than a human adjudication
Failure is expensive, rare, and silent	B plus a replay set	Retries hide the distribution you need to study
No writable oracle exists	none of these	Go collect labels first

The last two rows are where teams get hurt. Free generation tempts you to skip the oracle entirely and read everything yourself, which is the most expensive gate ever built.

Limitations, and who should walk away

A deterministic oracle only covers tasks that can be settled deterministically. Fluent prose, UI taste, and product judgment do not fit, and mocking up a fake scorer to make the table look complete is worse than leaving the row empty.

My sample sizes here are small and machine-specific, and free tiers change their terms and their hardware without asking me. If you need a contractual SLA, a latency guarantee, or an audit trail you can hand to a regulator, a free endpoint is not the right substrate for the decision path — put the gate on infrastructure you control and use the free tier for the exploratory arms only.

Also skip this workflow if your accept rate is already high. Below a certain failure rate the harness costs more to maintain than it saves, and you should be spending that time on the product.

If you want to try it, pull the endpoint and the model id from the operator's own docs, run arm B first, and trust the verify-seconds column before the token one.

Forty Trials Can Kill a Free Tier. They Can't Crown It.

Jordan Liu — Tue, 20 Oct 2026 09:50:26 +0000

You do not need two hundred trials to find out that a free model can't hold your JSON schema. You need about forty.

What you cannot do with forty trials — or four hundred — is prove that it can. That is not a test. That is a shadow deployment, and it costs more than the tokens ever will.

Most free-tier evaluations I read are built to certify. Fifty prompts, a pass rate, a green checkmark, ship it. But a pass rate from a small sample is a point estimate with a wide shadow behind it, and the shadow is the only part that matters when your retry loop hits production.

So build the test to reject. Here's the arithmetic that forces it.

The shadow is wider than you think

Suppose forty trials come back at 85% pass. The 95% Wilson interval for that is roughly 0.71 to 0.93. You have not learned that the model is good. You have learned that it is somewhere between "needs a fallback" and "fine" — which is not a decision, it's a coin.

Push the sample size and watch the returns die:

trials	95% interval at p̂ = 0.85	width
20	0.64 – 0.95	31 pts
40	0.71 – 0.93	22 pts
60	0.74 – 0.92	18 pts
100	0.77 – 0.91	14 pts
200	0.79 – 0.89	10 pts

Five times the tokens buys you twice the precision. That is a terrible trade on a metered endpoint and a merely bad one on a free one.

The harness: block-sequential, reject-only

The rule is simple. State the pass rate you require before you run anything. Then stop the moment the upper bound of the interval drops below it. If you run out of blocks without that happening, the answer is not "pass" — the answer is "inconclusive," and you go shadow instead of shipping.

# harness.py — rejection-first free-tier test. Python 3.10+, stdlib only.
import json, math, os, time, urllib.request

Z = 1.96
BLOCK = 10          # trials per decision checkpoint
MAX_TRIALS = 60     # hard stop: past this you are not testing, you are shopping
REQUIRED_RATE = 0.90

def wilson(k, n, z=Z):
    if n == 0:
        return 0.0, 1.0
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = (z / denom) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return max(0.0, centre - half), min(1.0, centre + half)

REQUIRED = {"file", "line", "fix"}

def score(text):
    try:
        obj = json.loads(text)          # strict: fences and prose are failures
    except json.JSONDecodeError:
        return False, "unparseable"
    if set(obj) != REQUIRED:
        return False, "schema"
    if not isinstance(obj.get("line"), int):
        return False, "type"
    return True, "ok"

def call_model(prompt, base_url, key, model):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=90) as r:
        payload = json.loads(r.read())
    return payload, time.perf_counter() - t0

def run(prompt, base_url, key, model, n_max=MAX_TRIALS):
    k = n = 0
    spent = 0
    failures = {}
    while n < n_max:
        for _ in range(BLOCK):
            payload, _ = call_model(prompt, base_url, key, model)
            n += 1
            spent += payload["usage"]["total_tokens"]
            ok, why = score(payload["choices"][0]["message"]["content"])
            if ok:
                k += 1
            else:
                failures[why] = failures.get(why, 0) + 1
        lo, hi = wilson(k, n)
        if hi < REQUIRED_RATE:
            return {"decision": "REJECT", "k": k, "n": n, "upper": hi,
                    "cost_per_success": spent / max(k, 1), "failures": failures}
    lo, hi = wilson(k, n)
    return {"decision": "INCONCLUSIVE", "k": k, "n": n, "lower": lo,
            "upper": hi, "cost_per_success": spent / max(k, 1),
            "failures": failures}

The endpoint shape above is the OpenAI-compatible chat-completions form. Check the current docs for the base URL and the model ids before you run it — those move, and a stale string will look exactly like a model failure.

Validate the harness before you spend a token

Here is the cheapest step in the whole article and the one almost nobody does. Stand up a fake endpoint with a known pass rate and point the harness at it.

# stub.py — a model with a pass rate you control. STUB_P=0.80 python stub.py
import json, os, random
from http.server import BaseHTTPRequestHandler, HTTPServer

P = float(os.environ.get("STUB_P", "0.80"))
GOOD = {"file": "app.py", "line": 12, "fix": "close the cursor before returning"}

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        content = json.dumps(GOOD) if random.random() < P else "Sure! Here's the fix:\n```

json\n{}\n

```"
        body = json.dumps({"choices": [{"message": {"content": content}}],
                           "usage": {"total_tokens": 1200}}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a):
        pass

HTTPServer(("127.0.0.1", 8787), H).serve_forever()

Run it twice. With STUB_P=0.80, a working harness should reject the 0.90 claim inside the first few blocks. With STUB_P=0.97, a working harness should never reject, and should end at INCONCLUSIVE. If the 0.97 stub gets rejected, your scorer is buggy and you were about to blame a vendor for your own parser.

I ran it against the stub, not against a live endpoint, on purpose. Pass rates on hosted free models move from week to week; a number I can't re-verify tomorrow isn't evidence, it's a screenshot.

The line is 32 out of 40

Do the work once and you can stop redoing it. At 40 trials, 32 successes is the decision boundary: the upper Wilson bound lands at about 0.895, just under 0.90. Thirty-three successes and you can no longer reject — the interval reaches 0.913.

So the honest reading of a 40-trial run is one sentence long. Thirty-two or fewer passes: kill it, and spend your afternoon on something else. Anything above that: you have failed to reject it, which is not the same as approval.

Why does that matter? Because "failed to reject" is exactly the state in which teams ship, and it's the state in which a 78% model with a friendly first ten outputs looks identical to a 93% model.

Cost per success, not cost per call

Free tokens make people forget the denominator. What you care about is tokens spent per usable output, and retries inflate that fast.

A 1,200-token task at an 0.80 pass rate costs 1,500 tokens per success. At 0.60, it costs 2,000 — a 25% drop in quality is a 33% jump in true cost. Against an allowance of ten million tokens, that is roughly 6,600 successful tasks versus 5,000. Same "free" tier. Same prompt. A third of the work gone.

This is the calculation I actually want before I trust an allowance number, and it's three lines in the harness above: spent / successes. Two numbers, no dashboard.

Keep the model and the host on separate records

This is where a free server will lie to you. A shared host gives you queueing noise, and queueing noise looks exactly like a slow model. Measure it and you'll see the tail separate: a p95/p50 wall-time ratio above about 3 on a cold endpoint usually means you're waiting on someone else's job, not on inference.

So run one small latency-only pass before the accuracy pass, and write the ratio down next to the results. If it's ugly, cap your concurrency at one and re-measure before you conclude anything about the model. Never put shared-host tail latency on a model's permanent record.

When I need a token-hungry harness run and a box to run it on, this is the case where MonkeyCode fits the shape of the problem: free model access for the trial loop, and a free server option for the harness itself. The operator lists a ten-million-token allowance and a free server tier; I'm quoting their numbers, not re-auditing them, and everything above stays valid if you point BASE at any other OpenAI-compatible endpoint instead.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not do this

If the output is judged by a human or by another model on a five-point scale, strict pass/fail scoring collapses and this whole design goes with it — use pairwise comparison instead. If your task is multi-turn agentic work, per-success token accounting gets muddy enough that you should measure cost per completed session, not per reply. And if you genuinely need to distinguish 92% from 97%, forty trials will not do it, four hundred barely will, and the right instrument is a shadow deployment behind a flag, not a test suite.

Everything else — schema adherence, tool-call shape, refusal rate on your own prompts — dies or survives inside about forty trials. The test was never going to tell you yes. It was only ever going to tell you no, and no is the answer you can act on today.

Your Free Tier Has a Concurrency Knee. Go Find It.

Jordan Liu — Mon, 19 Oct 2026 08:31:16 +0000

The free endpoint didn't fail me with a wrong answer. It failed me with a right answer, eleven seconds late, and that is a much harder bug to notice.

So this is an evaluation of MonkeyCode's free model access and its free server option — both as the operator describes them, not as a benchmark I can hand you. Disclosure: This article was prepared as part of MonkeyCode's product outreach. As of this writing the offer is advertised as a large free token allowance plus a free server; I'm treating every number on that page as a claim to re-check on the day you read this, because free-tier terms move and stale numbers are worse than no numbers. What I can give you is a measurement you run yourself in about an hour, and a rule for reading it.

The mistake is counting tokens

Everyone audits a free tier by watching the quota drain. That's the visible meter, so it gets the attention. But the meter rarely decides whether your job finishes — the queue does. Two requests with identical prompt sizes can land a second apart or thirty seconds apart depending on how many other people hit the same free capacity at that moment. Which means the question isn't "how many tokens do I get?" It's "at what concurrency does the shape of the response stop being mine?"

That threshold has a name in capacity work: the knee. Below it, latency grows gently and errors are rare. Above it, p95 detaches from p50, and the kind of failure changes — first slow, then 429, then truncated bodies, then connection timeouts. Your retry logic behaves completely differently on each side of that line, and if you never located the line you tuned your retries against a phantom.

A harness that produces one number

The experiment is deliberately boring: hold the request byte-identical, step concurrency 1 → 2 → 4 → 8 → 16 → 32, and record latency percentiles plus a failure shape for every call. Failure shape is the part people skip. A 429 is not a timeout is not a 200 with an empty choices array, and lumping them into "error" throws away the only signal that tells you which fix to apply.

# knee.py — find the concurrency where latency stops being yours
import asyncio, json, os, statistics, time, httpx

BASE  = os.environ["FREE_BASE_URL"].rstrip("/")
KEY   = os.environ["FREE_API_KEY"]
MODEL = os.environ["FREE_MODEL"]
PROMPT = "Return the SHA-256 of the string 'knee' plus one sentence explaining it."

def body():
    return {"model": MODEL, "temperature": 0,
            "messages": [{"role": "user", "content": PROMPT}]}

async def one(client, sem, out):
    async with sem:
        t0, shape = time.perf_counter(), "ok"
        try:
            r = await client.post(f"{BASE}/chat/completions", json=body())
            dt = time.perf_counter() - t0
            if r.status_code == 429:            shape = "429"
            elif r.status_code >= 500:          shape = f"5xx_{r.status_code}"
            elif r.status_code != 200:          shape = f"http_{r.status_code}"
            else:
                try: j = r.json()
                except json.JSONDecodeError:    shape = "bad_json"
                else:
                    if not j.get("choices"):    shape = "empty_choices"
            out.append((dt, shape))
        except httpx.ReadTimeout:      out.append((time.perf_counter()-t0, "read_timeout"))
        except httpx.ConnectTimeout:   out.append((time.perf_counter()-t0, "connect_timeout"))
        except Exception as e:         out.append((time.perf_counter()-t0, type(e).__name__))

async def step(c, n):
    out, sem = [], asyncio.Semaphore(c)
    timeout = httpx.Timeout(60.0, connect=10.0)
    async with httpx.AsyncClient(timeout=timeout) as client:
        t0 = time.perf_counter()
        await asyncio.gather(*(one(client, sem, out) for _ in range(n)))
        wall = time.perf_counter() - t0
    lat = sorted(d for d, _ in out)
    shapes = {}
    for _, s in out: shapes[s] = shapes.get(s, 0) + 1
    return statistics.median(lat), lat[int(0.95*(len(lat)-1))], wall, shapes

async def main():
    for c in (1, 2, 4, 8, 16, 32):
        p50, p95, wall, shapes = await step(c, max(8, c * 4))
        print(json.dumps({"c": c, "p50": round(p50, 2), "p95": round(p95, 2),
                          "wall": round(wall, 2), "shapes": shapes}), flush=True)
        bad = sum(v for k, v in shapes.items() if k != "ok")
        if p95 > 4 * p50 or bad > 0.05 * sum(shapes.values()):
            print(f"KNEE at concurrency {c}")
            break

asyncio.run(main())

Run it under tee so each day's numbers are a file, not a memory:

FREE_BASE_URL=... FREE_API_KEY=... FREE_MODEL=... \
  python knee.py | tee knee-$(date +%F).jsonl
jq -s 'map({c, ratio: (.p95 / .p50)})' knee-*.jsonl

Two definitions make the output actionable instead of decorative. The latency knee is the first rung where p95 exceeds four times p50 — that's where a shared queue starts deciding your user experience. The error knee is the first rung where more than five percent of calls aren't plain ok. They are almost never the same rung, and the gap between them is your entire safety margin. If you ship at the latency knee because the errors haven't started yet, you've shipped into the gap.

Then run it twice, from two places

Here's where the free server earns its keep as an instrument, not just a host. Run the identical script from your laptop, then from the free server. Same prompt, same code, different client. If p50 barely moves but p95 collapses, your problem was the path from your network to the endpoint, not the endpoint. If both move together, you're looking at shared capacity and no amount of moving hosts will help. That's a fifteen-minute experiment that replaces a week of guessing, and it's the reason I'd rather have a free box next to the endpoint than a bigger quota far away from it.

What the number tells you to build

Workload	Verdict after the knee test
Single-user CLI or dev loop	Fine if c=1 p95 stays under your patience
Latency-blind batch backfill	Fine one rung below the error knee, with a retry budget
Chat UI with a human staring at it	Only if p95 at your real peak stays under budget
Anything with a contract attached	Measure, then don't promise it

Set your production concurrency to the error knee minus one, not to the biggest number that didn't explode. Set client timeouts slightly above the p95 at that rung, because a timeout below your own tail is a self-inflicted error generator. And log the shape, always — "it failed" cost me hours; 429 costs me minutes.

Where this approach breaks

It breaks when your real traffic isn't this uniform. One long-context request in a batch of short ones drags the tail in ways a fixed prompt never shows, so re-run with your heaviest real payload before you trust the rung. It breaks when the free capacity changes with the crowd, which is why the tee files matter more than any single run. And it breaks completely if you need a number you can put in a contract — a shared free queue cannot give you a p95 guarantee, and asking it to is a category error, not a tuning problem.

Who shouldn't bother? Anyone whose workload is already one request at a time, anyone who can't spend the hour, and anyone who needs an SLA on the response. For the rest of us — people prototyping agents on free capacity and quietly hoping the tail behaves — the knee is the only number worth knowing, and it takes one script to find. If you want to run it against the free models and free server, the project page has the setup; just check today's terms before you budget against them.

Run the Same Prompt Twice Before You Trust a Free Model

Jordan Liu — Sun, 18 Oct 2026 07:11:57 +0000

The first number your eval should report is not accuracy. It's whether your eval says the same thing twice.

I keep seeing the same review pattern in October: someone swaps a paid endpoint for a free one, runs their test suite once, and posts a verdict. Green, or red, and then a blog post. But a single run of a nondeterministic system is not a measurement. It's a sample. And if your free tier samples differently than your paid tier, your comparison is two coin flips wearing lab coats.

So before you benchmark anything, run an A/A test. Same prompt. Same parameters. Nothing changes. Then ask how many times you got the same answer back.

What an A/A test actually buys you

An A/B test compares two things, and only means something if the harness is quiet. An A/A test is the control: it tells you how loud the harness is on its own. If five identical calls return five different texts, your diff-based regression suite is measuring sampling noise and calling it a regression.

Temperature 0 does not save you here. Shared serving stacks batch and reorder work, kernels reduce in different orders, and plenty of providers rotate the model behind a stable alias without telling you. Determinism at temperature=0 is a common assumption, not a guarantee.

That last point is a gift, by the way. A fingerprint probe also works as a tripwire for silent model swaps — if your agreement suddenly collapses on unchanged inputs, something upstream changed, and it wasn't your prompt.

The probe

Here's the whole experiment. One small script, one prompt, k repeats, one JSON line out. It assumes an OpenAI-compatible /chat/completions shape; adjust the request body if your endpoint differs.

#!/usr/bin/env python3
"""aa_probe.py - A/A test: one prompt, k samples, measure self-agreement."""
import argparse, collections, hashlib, json, os, re, urllib.request


def post(base, key, payload):
    req = urllib.request.Request(
        f"{base.rstrip('/')}/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"},
    )
    with urllib.request.urlopen(req, timeout=180) as r:
        return json.load(r)


def normalize(text):
    t = text.strip().lower()
    t = re.sub(r"```

.*?

```", " ", t, flags=re.S)  # drop code fences first
    t = re.sub(r"[^a-z0-9 ]+", " ", t)
    return re.sub(r"\s+", " ", t).strip()


def fingerprint(text):
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()[:16]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--k", type=int, default=5)
    ap.add_argument("--temp", type=float, default=0.0)
    ap.add_argument("--prompt", default=os.environ.get("AA_PROMPT", ""))
    a = ap.parse_args()
    if not a.prompt:
        raise SystemExit("pass --prompt or set AA_PROMPT")

    base, key = os.environ["LLM_BASE_URL"], os.environ["LLM_API_KEY"]
    model = os.environ["LLM_MODEL"]
    fps, totals = [], []

    for _ in range(a.k):
        data = post(base, key, {
            "model": model,
            "temperature": a.temp,
            "messages": [{"role": "user", "content": a.prompt}],
        })
        fps.append(fingerprint(data["choices"][0]["message"]["content"]))
        usage = data.get("usage") or {}
        if "total_tokens" in usage:
            totals.append(usage["total_tokens"])

    counts = collections.Counter(fps)
    print(json.dumps({
        "model": model, "temp": a.temp, "k": a.k,
        "modal_share": round(counts.most_common(1)[0][1] / a.k, 3),
        "distinct": len(counts),
        "usage_total_tokens": sum(totals) if len(totals) == a.k else None,
    }))


if __name__ == "__main__":
    main()

modal_share is the number that matters: the fraction of runs that landed on the single most common answer. Normalization is deliberately brutal — lowercase, punctuation stripped, whitespace collapsed — because you're not testing wording, you're testing behavior.

Make it a week-long record, not a one-off

One probe run tells you about one moment on shared hardware. Run it on a schedule and you get something better: a time series of how honest your harness is.

#!/usr/bin/env bash
set -euo pipefail
PROMPT_FILE="prompts/cache-key.txt"
export AA_PROMPT="$(cat "$PROMPT_FILE")"
PROMPT_SHA="$(sha256sum "$PROMPT_FILE" | cut -c1-12)"
mkdir -p runs

for run in 1 2 3; do
  python3 aa_probe.py --k 5 --temp 0 \
  | jq -c --arg sha "$PROMPT_SHA" --arg ts "$(date -u +%FT%TZ)" \
      '. + {ts: $ts, prompt_sha: $sha}' >> runs/aa.jsonl
done

# gate: how many samples fell below 0.9 agreement?
jq -s 'map(select(.modal_share < 0.9)) | length' runs/aa.jsonl

Hashing the prompt file matters more than it looks. When you edit a prompt, you want the fingerprint history to break cleanly at that exact revision, not silently drift underneath it. That's the same discipline as freezing a config hash — the run has to earn its comparison.

This is the kind of job that wants to live somewhere other than your laptop, and it's where a free tier actually fits: MonkeyCode advertises free model access plus a free server option, which is enough to host a probe loop and collect samples while you do other work. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution about the server, though: it's shared, so do not point a latency benchmark at it and report the numbers as a property of the model. Measure latency there and you're measuring the neighborhood.

Reading your own numbers

Here's the triage table I use. The cutoffs are proposed heuristics, not measured constants — calibrate them against your own workload before you quote them at anyone.

`modal_share` at temp 0, k=5	What the harness supports	What it does not
1.0	Exact-text regression diffs	Nothing about semantic equivalence
0.8 – 0.99	Pass@k with k ≥ 3, diffable error classes	Single-run pass/fail gates
0.4 – 0.79	Aggregate scores over many prompts	Any per-prompt verdict
< 0.4	Drafting, brainstorming, triage	Evals, CI gates, anything auditable

A high share is not a quality signal. A model can be perfectly consistent and consistently wrong — that's what your B/B test is for, and you can't run a B/B until the A/A is quiet. Consistency is the floor you build on, not the score you report.

Where this breaks, and who should walk away

Token accounting is the weak seam. If your endpoint returns no usage block, you'll be measuring tokens with a local tokenizer that may not match the server's — and a budget comparison across two mismatched tokenizers is worse than no comparison. Either measure usage on both sides or drop the token column entirely and say so.

The approach also doesn't fit everyone. If you need pinned model versions, uptime commitments, or sub-second p99 latency, a shared free tier is the wrong instrument — not because it's bad, but because those are contract questions, not experiment questions. And if your product is regulated, an eval whose variance you haven't characterized is a liability with a dashboard.

What I'd actually ask you to take away is narrower than "try the free tier." It's that a free model with a known variance profile is more useful than a paid model you never probed, because only one of them lets you tell a regression from a reroll. Run the probe for a week before you commit either way — and if you do, I want to know what your agreement rate was, because that number is the real starting line.

I Audited Ten Million Free Tokens. The Ledger Drifted First.

Jordan Liu — Fri, 16 Oct 2026 05:14:43 +0000

I Audited Ten Million Free Tokens. The Ledger Drifted First.

My free allowance didn't evaporate because a model was slow. It evaporated — inside the simulation I built to check it — because my retry wrapper paid for the same prompt twice while my streaming parser paid for the answer zero times. Two bugs, opposite signs, almost cancelling.

That's the dangerous case, isn't it? A near-cancel looks like "fine." And "fine" is how a token budget turns into a Monday incident instead of a Friday cleanup.

Everyone is wiring agents onto free model access right now; that's the mood of the timeline this week. So the interesting question isn't whether a free model is good enough. It's whether your ledger agrees with theirs, and what it means when it stops agreeing.

Three probes, one table

I built one harness and ran it three ways. Probe A reconciles my own token sum against the total the platform reports, and logs the sign of the drift. Probe B tags every attempt so I can count the duplicate prompt tokens my own retry policy manufactures. Probe C asks what a repair costs when a stream ends without a usage block at all.

Three probes, one SQLite table, standard library only. No SDK, no framework, no dashboard.

I pointed the endpoint variables at MonkeyCode's free model access, since that's the free tier I had a key for. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness

# ledger.py -- stdlib only. Probes A/B/C behind one table.
import hashlib, json, os, sqlite3, time, urllib.request

BUDGET = int(os.environ.get("TOKEN_BUDGET", "10000000"))  # operator-supplied allowance
DB = os.environ.get("LEDGER_DB", "ledger.sqlite")

def conn():
    c = sqlite3.connect(DB)
    c.execute("""CREATE TABLE IF NOT EXISTS calls(
        id INTEGER PRIMARY KEY, ts REAL, attempt INTEGER, phash TEXT,
        ptok INTEGER, ctok INTEGER, estimated INTEGER, status INTEGER, ms INTEGER)""")
    return c

def phash(payload):                      # dedupe key for retry accounting
    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()[:16]

def approx(text):                        # char/4, always flagged, never invoiced
    return max(1, len(text) // 4)

def post(payload):
    url = os.environ["MC_BASE_URL"].rstrip("/") + "/chat/completions"
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode(),
        headers={"Authorization": "Bearer " + os.environ["MC_API_KEY"],
                 "Content-Type": "application/json"},
    )
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        body = json.loads(r.read())
        return body, r.status, int((time.time() - t0) * 1000)

def send(payload, attempt=1):
    body, status, ms = post(payload)
    u = body.get("usage") or {}
    est = 0 if u else 1
    ptok = u.get("prompt_tokens") or approx(json.dumps(payload))
    content = body.get("choices", [{}])[0].get("message", {}).get("content", "") or ""
    ctok = u.get("completion_tokens") or approx(content)
    c = conn()
    c.execute("INSERT INTO calls(ts,attempt,phash,ptok,ctok,estimated,status,ms) "
              "VALUES(?,?,?,?,?,?,?,?)",
              (time.time(), attempt, phash(payload), ptok, ctok, est, status, ms))
    c.commit()
    return body, est

def reconcile(server_total, used):
    c = conn()
    mine = c.execute("SELECT SUM(ptok+ctok) FROM calls").fetchone()[0] or 0
    dup  = c.execute("SELECT SUM(ptok) FROM calls WHERE attempt > 1").fetchone()[0] or 0
    est  = c.execute("SELECT COUNT(*) FROM calls WHERE estimated=1").fetchone()[0]
    drift = mine - server_total
    if abs(drift) > 0.02 * max(used, 1):
        verdict = "unexplained drift -- halt and re-run the fixture"
    elif est:
        verdict = "estimated rows present -- totals are not invoice-grade"
    elif used > 0.7 * BUDGET or used + drift > 0.7 * BUDGET:
        verdict = "close to the line -- reconcile before the next run"
    else:
        verdict = "accounting closed"
    return {"harness": mine, "server": server_total, "drift": drift,
            "retry_duplicate": dup, "estimated_rows": est,
            "pct_of_budget": round(100 * drift / BUDGET, 4), "verdict": verdict}

# arithmetic check -- injects known defects, never touches the network
python ledger.py --fixture 240 --drop-usage 0.03 --retry-rate 0.04

# the live probe, pointed at whatever your own console exposes
TOKEN_BUDGET=10000000 MC_BASE_URL="$BASE" MC_API_KEY="$KEY" python ledger.py --probe all

Note what I deliberately did not do: hardcode a model name. Read the identifier off your own console and put it in the payload, because a blog post is the worst possible source of truth for a string that changes without telling you.

The run, and what I'm willing to publish

Here's where I have to be precise, because the whole value of this account depends on not laundering numbers. The table below comes from the fixture. It is not a measurement of any shared endpoint, and I'd be lying by framing if I presented it as one.

The fixture injects two known defects: dropped usage blocks on 3% of responses, and a retry policy that re-sends full prompts. If the harness arithmetic is right, the two injected causes must sum to the drift exactly. That's the test.

field	value	how it is produced
harness total	9,730,412	SUM(ptok+ctok) across 240 rows plus retries
server counter	9,625,304	the fixture's simulated counter
drift	+105,108	harness minus server
retry duplicate	63,900	SUM(ptok) WHERE attempt > 1
dropped-usage rows	7	rows with estimated=1
dropped-usage tokens	41,208	known value the injector removed
drift check	63,900 + 41,208 = 105,108 ✅	exact, by construction
share of budget	1.05%	drift / 10M

So what did I learn about the free tier? Nothing — and that's the point. I learned the harness closes. If you cannot make your own accounting agree with itself on a fixture where you already know the right answer, you will never correctly interpret a 1% drift against a live counter. You'll just argue about it.

And notice the shape of the failure. One defect under-counted, one over-counted, and the budget looked almost honest. Free tiers don't usually fail loudly. They fail symmetrically.

Where this actually breaks

Retry amplification is the first killer, and it's yours, not theirs. A dropped connection doesn't refund the prompt you already sent. If your wrapper resends the full conversation, you pay twice for identical input, and the second payment buys you nothing but a retry.

Stream truncation is the second, and it fails the other way. When a stream dies before the usage block arrives, a naive wrapper records zero completion tokens for an answer you definitely received. Now your ledger is optimistic, which is worse than pessimistic, because optimism is invisible until the invoice disagrees.

Tool-call loops are the third. An agent that re-reads the same file six times to answer one question isn't broken — it's just expensive, and no per-request counter will tell you which loop did it. You need the phash column for that, and you need to actually look at it.

Concurrency races are the fourth, and they're the one a single-machine harness cannot fix. If something else is using the same key while your cron job runs, your drift will change sign between runs and you'll spend a week blaming the provider. Add a run identifier and compare runs, not instants.

A decision table I'll defend

signal	what it probably means	action
drift under 0.5%, zero estimated rows	your wrapper agrees with the counter	keep the 70% soft alarm and stop optimizing
drift 0.5–2%, retry_duplicate ≈ drift	retries, not the platform	cap attempts at 2, dedupe by phash, re-run fixture then probe
estimated_rows > 0	a stream ended before usage arrived	treat totals as unknown until that probe runs non-streaming
drift flips sign between runs	you're measuring concurrency you don't control	scope the key to the harness, or stop trusting the series
drift above 2% of budget	cause genuinely unknown	treat the allowance as unbounded risk and move the workload

That last row is the important one. "Free" is a price, not a guarantee about accounting. If you cannot reconcile it, you cannot plan on it.

The free server, kept small

My harness needs to run every six hours, write one SQLite file, and keep a key off my laptop. That's a cron job with a file attached — not a platform migration. A free server option is enough for exactly that, and that's the only reason it belongs in this article.

The reason it matters is failure isolation, not compute. If the host disappears tomorrow, my ledger is a file I can move, and the fixture still passes on my machine. If your experiment can't survive losing its host, you built a dependency, not a measurement.

The operator also describes the project as open source. If that matters to you, go read the client before you trust anything above — including its own counter. Verify first, then benchmark.

Who should not do this

Don't use this approach if your reconciliation has to tie to invoiced currency. A free-tier counter is a signal, not an invoice, and no amount of harness engineering changes that.

Don't run the harness on a hosted free server if your payloads carry data with residency or classification constraints. The experiment isn't worth the incident.

Don't bother if the workload is a one-off script you'll delete on Friday. Two hundred lines of ledger code is overkill for a throwaway.

And don't adopt a retry-heavy streaming repair loop on a metered free allowance if you can't tolerate paying twice for the same prompt. Sometimes the honest answer is fewer retries, not a better ledger.

The takeaway

Your free tokens are a budget, not a vibe. Measure the drift, name its cause, and decide before the allowance does it for you.

If you want the harness and somewhere to run it, MonkeyCode is where I pointed mine — bring the fixture first, and trust the arithmetic before you trust anybody's dashboard.

Free Tiers Don't Fail Loudly. Your Harness Should.

Jordan Liu — Thu, 15 Oct 2026 11:24:15 +0000

Cheap model access almost never loses on accuracy. It loses on the tail — the 3% of calls that time out, the one input where the JSON slips, the retry storm that only appears when you fire four requests at once.

So before I trust any free endpoint with real work, I don't write a prompt first. I write a harness that fails loudly, and I write my thresholds down before I run anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm framing it around MonkeyCode's free model access and free server tier because those are the two things I was handed to work with; the operator states a free token allowance (10M at the time of writing) and a free server tier. Treat both as operator claims and check the current terms yourself. The harness below is endpoint-agnostic — anywhere the OpenAI-compatible chat completions shape works, it runs.

I'm not going to quote pass rates. I don't have them, and making them up would defeat the entire exercise. What I can hand you is the measuring rig, the two numbers that actually change my mind, and the rule I apply to whatever the table says.

Start with the grader, not the model

Every model comparison I've seen go wrong went wrong in the same place: the grading. Someone eyeballs twenty outputs, decides the free model "feels about the same," and ships it. Then a week later the extraction job is silently writing null into a column nobody reads.

So the grader comes first, and it has to be deterministic. No LLM-as-judge for the primary metric — a judge adds its own variance, and if you can't explain the judge's failure mode you've just moved the ambiguity one layer down.

Three task families are enough to start: structured extraction with a strict schema, a small code repair graded by a unit test, and a rewrite task graded by assertions (length, banned phrases, required substrings). Thirty cases, spread across easy and nasty. Freeze them in a file. If a case is ambiguous, delete it — an ambiguous case is a coin flip you'll later misread as a model failure.

The metric that ends arguments: tokens per correct answer

Pass rate alone tells you whether the model can do the job. It doesn't tell you what the job costs. A model that passes 80% while burning three retries per success is more expensive than one that passes 70% and lands it first try.

So I log total_tokens per call, and divide the total spent by the number of correct answers. Tokens per correct answer. That single ratio collapses accuracy, retries, and verbosity into one number you can compare across endpoints, and it's the number I actually defend in a review.

There's a recurring genre of post where a model hits near-perfect accuracy and the author still ships the cheaper one. That's not contrarianism. That's someone reading the right column of the table.

The harness

Here's the runner. It's deliberately small — you should be able to read all of it while your coffee cools.

# harness.py — run N trials of a frozen task suite against any
# OpenAI-compatible endpoint. Records only. No conclusions until the table is full.
import asyncio, json, os, random, time
import httpx

BASE  = os.environ["MC_BASE_URL"]   # e.g. https://<host>/v1
KEY   = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL"]

TRIALS     = 3      # variance across trials matters more than the mean
CONCURRENCY = 4     # raise this on purpose, then watch retry_rate move
RETRYABLE  = {"http_429", "http_500", "http_502", "http_503", "http_504"}

async def one_call(client, case, trial):
    t0 = time.perf_counter()
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": case["prompt"]}],
        "temperature": 0,                       # pin it, or you're measuring noise
        "max_tokens": case.get("max_tokens", 512),
    }
    r = await client.post(f"{BASE}/chat/completions", json=payload,
                          headers={"Authorization": f"Bearer {KEY}"}, timeout=60.0)
    wall_ms = (time.perf_counter() - t0) * 1000

    if r.status_code != 200:
        return {"case": case["id"], "trial": trial, "ok": False,
                "wall_ms": wall_ms, "tokens": 0, "error": f"http_{r.status_code}"}

    body = r.json()
    text = body["choices"][0]["message"]["content"]
    ok, why = case["check"](text)               # deterministic, returns (bool, str)
    return {"case": case["id"], "trial": trial, "ok": ok, "why": why,
            "wall_ms": wall_ms, "tokens": body.get("usage", {}).get("total_tokens", 0)}

Retries aren't an implementation detail. They're the signal. Count them.

async def call_with_retries(client, case, trial, attempts=3):
    for n in range(attempts):
        rec = await one_call(client, case, trial)
        if rec["ok"] or rec.get("error") not in RETRYABLE:
            rec["retry"] = n
            return rec
        await asyncio.sleep((2 ** n) * 0.5 + random.random() * 0.25)  # jittered backoff
    rec["retry"] = attempts - 1
    return rec

def summarize(rows):
    passed = [r for r in rows if r["ok"]]
    spent  = sum(r["tokens"] for r in rows)
    lat    = sorted(r["wall_ms"] for r in rows)
    pct    = lambda q: round(lat[min(len(lat) - 1, int(q * len(lat)))])
    return {
        "pass_rate":          round(len(passed) / len(rows), 3),
        "tokens_per_correct": round(spent / max(1, len(passed)), 1),
        "p50_ms":             pct(0.50),
        "p95_ms":             pct(0.95),          # the column free tiers fail in
        "retry_rate":         round(sum(r["retry"] for r in rows) / len(rows), 2),
    }

Drive it with a semaphore so concurrency is a knob you control, write every row to JSONL, and keep the raw outputs. You will need them the first time the grader and your eyes disagree.

Read the table, don't read the demo

One good demo proves a model can do the task once. It proves nothing about next Tuesday. Here's the rule I apply, and I set these thresholds before running, because after running I will rationalize anything.

Signal	Threshold I set in advance	What I do when it's crossed
pass_rate	at or above my current baseline	candidate for the cheap path
tokens_per_correct	within 1.5× of baseline	make it the default for batch work
p95_ms	inside my timeout budget	safe behind a synchronous call
retry_rate	flat as concurrency rises	cap concurrency and queue instead
per-case flips across trials	any case that flips	pin harder, or exclude that task

That last row is the one people skip. If the same case passes on trial one and fails on trial three at temperature 0, you don't have a model problem. You have a task too close to the decision boundary, and no amount of prompt polish fixes it.

Where this breaks

Three trials is not statistics. It's a smoke test that catches gross instability and nothing more. If you need confidence intervals, budget for thirty trials per case and accept the token spend.

The grader is the weak link, always. A regex that's too strict scores a correct answer as a failure, and you'll blame the endpoint for your own bug. Audit false negatives by hand for the first run — read every failure, not just the pass rate.

Endpoints drift. Weights get swapped, quantizations change, rate limits move. Your table is a snapshot with a date on it, not a property of the model. And because the whole point here is a free tier, remember the free tier's terms are someone else's to change: verify the allowance and the server offer before you build a pipeline on top of them.

Who should not do this

If your path is user-facing and sub-second, a free tier is the wrong place to save money — go spend the tokens on latency. If you don't have a labeled task suite and won't build one, this harness will give you numbers that feel like answers and aren't. And if a single wrong output costs more than the tokens you saved, the arithmetic is already decided; skip the experiment.

Everyone else: the rig above is maybe ninety lines, and it runs on the same free server tier you're already evaluating. Park it there on a nightly schedule — a harness is only worth something as a series, not a screenshot. Then look at the p95 column before you look at the pass rate. That's where the honesty is.

I Made the Oracle Fail First

Jordan Liu — Tue, 13 Oct 2026 01:29:26 +0000

A fluent patch is not a pass. A free server that happens to boot is not a pass either. If I cannot make a local oracle fail on a bug I planted myself, I do not get to blame the model, and I do not get to praise it.

That is the conclusion. The rest is a harness I want in the repo before a free token or a free minute gets spent. This draft does not report a live run, because I did not execute one for it.

You know the scene. The assistant nods, the diff looks tidy, and the slice is still wrong. Who gets the lecture, the model or the missing test?

Blame is a warm drink. It does not fail a build. I want the check to fail first, on purpose, with a command I can rerun when I am bored and suspicious. No leaderboard, and no borrowed authority. Just an exit code that was already red before any model was allowed to speak.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and two availability claims here are operator-supplied rather than measurements from 13 October 2026: free model access, and a free server option.

I was also told the free model side includes a grant on the order of 10 million tokens. Confirm that figure, which models it actually covers, and the server terms on the current project page before you plan a week around them. I will not invent model names, hardware, durations, or a win rate.

The call site still has to live in git, next to the plants, or the evaluation is a chat you cannot replay. Free access is a budget for that replay. It is not evidence the patch was right. Delete the product name and the stop-rule still applies to any client you already have.

People keep saying the models got sharper while the bugs stayed put. Maybe they did. Sharpness without a red oracle is a brighter flashlight in a room you never filled with smoke. I am not grading a summer of announcements. I am asking whether my own gate can fail a bug I already understand.

Plant something boring

Clever bugs flatter the person who planted them. Boring bugs audit the gate. I keep one function that returns one element too many, the kind a tired reviewer can spot, because that is the point.

If the oracle cannot see a bug a tired human can see, why would I trust it on the bugs I miss at 1 a.m.? The plant stays small so the failure is about the gate, not about my talent for hiding a defect in a maze.

# harness/plants.py
# Proposal. Not a recorded run.
from dataclasses import dataclass

@dataclass(frozen=True)
class Plant:
    name: str
    buggy: str
    fixed: str

TAKE = Plant(
    name='take-off-by-one',
    buggy='''def take(xs, n):
    return xs[: n + 1]
''',
    fixed='''def take(xs, n):
    return xs[:n]
''',
)

The test is equally dull. Dull is a feature, not a lack of ambition. It names one boundary and refuses to negotiate.

# tests/test_take.py
from take import take

def test_take_does_not_overshoot():
    assert take(['a', 'b', 'c'], 2) == ['a', 'b']

A writer script keeps me from pasting the bug by hand and then forgetting which variant is on disk. Hands are how plants get quietly fixed before the model ever sees them.

# harness/write_plant.py
# Proposal. Overwrites take.py from the plant table.
import argparse
from plants import TAKE

BODIES = {'buggy': TAKE.buggy, 'fixed': TAKE.fixed}

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('--name', required=True)
    parser.add_argument('--variant', choices=sorted(BODIES), required=True)
    args = parser.parse_args()
    if args.name != TAKE.name:
        raise SystemExit('unknown plant')
    with open('take.py', 'w', encoding='utf-8') as fh:
        fh.write(BODIES[args.variant])

if __name__ == '__main__':
    main()

Write the buggy body into place and let pytest speak before you do. If you feel an urge to peek at a completion first, that urge is the experiment trying to cheat.

python harness/write_plant.py --name take-off-by-one --variant buggy
python -m pytest -q tests/test_take.py
echo oracle_exit=$?

If that exit is 0, stop. The plant is invisible. Asking a free model to review an invisible bug is a play, not a measurement. Fix the test, and do not touch the prompt.

A smoke alarm you have never held to smoke can still beep in a drill. The red pytest is the smoke. Without it I am admiring the plastic cover and calling it safety.

Only then may the model look

The prompt stays poorer than the test file. I do not paste the expected list. I do not ask whether the function is good, because that adjective is how a yes gets hired as an engineer.

The instruction I send is plain. One function is attached. A test named test_take_does_not_overshoot fails. Return either a line that starts with REJECT and one sentence, or a unified diff that changes only take. Do not claim the tests were run. Save the reply next to the plant name. Feeling is not a column.

# harness/classify.py
# Proposal. A malformed reply is a miss, not a cue to retry.
def classify(reply: str) -> str:
    text = reply.strip()
    if text.startswith('REJECT:'):
        return 'reject'
    if text.startswith('---') or text.startswith('diff '):
        return 'diff'
    return 'malformed'

Then the mechanical half, still on the laptop, still too boring to argue with:

python harness/extract_diff.py reply.txt > candidate.diff
patch -p0 < candidate.diff
python -m pytest -q tests/test_take.py
echo after_model_exit=$?

Two exits. They do not get to shake hands. The first had to be non-zero or the day is void. The second may be zero, and that is a different sentence.

A REJECT on a function the oracle already hates can be a real hit. The model refused to launder a bad slice. A minimal diff that goes green is a different hit. Would you average those and call the average quality? I will not. Unlike events do not become one fact just because they share a spreadsheet.

If patch refuses the diff, I do not retry inside the scorer. A diff that does not apply is a miss. Retry-until-apply is another study, and it will eat a grant while wearing a lab coat.

The ledger is the third gate

Free tokens disappear in the rephrase. You paste the traceback, you ask again, and you call it iteration. The meter calls it lunch.

This ledger is local, and it should stay proud of that. If the provider meter disagrees, the provider wins. Write the gap down. Do not litigate it in the README and pretend the argument is a measurement.

# harness/ledger.py
# Proposal. Counts what this process tried to spend.
class Ledger:
    def __init__(self, ceiling: int):
        self.ceiling = ceiling
        self.spent = 0

    def charge(self, prompt_tokens: int, completion_tokens: int) -> None:
        self.spent += prompt_tokens + completion_tokens
        if self.spent > self.ceiling:
            raise SystemExit(
                f'stop: spent {self.spent} > ceiling {self.ceiling}'
            )

Set the ceiling to a slice, not to the folklore of the whole grant. A few thousand tokens will tell you whether the loop is bounded. Ten million, if the project page still says that when you look, is headroom. It is not a result I measured, and it is not a promise I can extend.

Quotas move. A method that only works if a published number never changes is not a method. Re-read the page the day you run the card, and write the number you actually saw into the notebook next to the date.

python harness/run_eval.py --plants harness/plants --ceiling 4000 --dry-run

The dry-run flag prints the prompt and charges nothing. I want it the way I want a seatbelt in a parking lot. An evaluation you cannot rehearse will get rehearsed on the grant, by accident, late, when you are sure this one extra call does not count.

The free server witnesses. It does not judge.

I may use the free server only after the local oracle is already red, and only to ask a narrower question. Does the same pytest exit match on that box? That is a witness statement about the room. It does not grade taste, tone, or how confident the completion sounded.

Laptop red and server green means portability, not genius. Both red means the patch missed, or the plant is still in the tree. Both green before the model spoke means the day is invalid, so go back and make smoke.

I am not asking a remote box whether a model is smart. Smart is not a command. Pytest is a command. If the documented runner is different from the placeholder below, keep the exit code and throw my hostname away.

# Placeholder only. Use the documented free-server run command.
# example.test is not a product host.
ssh runner@example.test -- python -m pytest -q tests/test_take.py
echo witness_exit=$?

Keep the exit code. Ignore any story a boot splash tries to sell. A machine that started is a machine that started. That sentence feels obvious until a green banner lands in a screenshot and everybody in the thread relaxes.

Read the card like gauges

When the card comes back I read it the way a mechanic reads gauges, not the way a fan reads a scoreline. Oracle not red? The card is void. No argument, no partial credit, no interesting qualitative notes.

Oracle red and the reply malformed? A miss, and the ledger still charged, which is the lesson hiding in the annoyance. Oracle red, diff applies, pytest green, witness matches? That is the only quiet pass I trust, and only for this plant.

Oracle red, and the model calls the buggy function fine? False praise. That cell matters more to me than a pretty repair. A tool that blesses a known-bad slice will bless the next one when I am tired and late. Where does a free model tend to look strong on a pack like this? On the boring repair, once the oracle has already named the failure. Where does it break? On false praise, on diffs that do not apply, and on any day the local check was blind. The free server breaks the story when its exit disagrees with the laptop, and it proves nothing when you boot it first and call the boot a result.

Four counts sit side by side, and they never get blended. Oracle-fail rate on plants, which must be total or you throw the day out. Repair rate on plants the oracle actually caught. False-praise rate. Witness mismatches between the laptop and the free server.

I am leaving those cells empty on purpose. No timings. No win rate. No claim that one stack beat another. You fill them on your plants, and you keep the raw exits next to the commit. An empty card is stricter than a handy number I could have typed.

Who should walk away

Skip this if you need a procurement bake-off. A toy slice will flatter whoever has seen an off-by-one in training, and it will teach you almost nothing about a 400-file service. Skip it for auth, crypto, or any hazard a unit test does not name.

A green take() will not make a token check safe. Skip it if the free tier forbids scripted calls. A ledger is not a pardon, and I am not suggesting you route around a limit. Skip it if you wanted a leaderboard by Friday. You wanted a mood, and moods do not belong in the README.

Two limits sit in the design, and they will not politely stay small. Leakage is the first. Paste the expected list into the prompt and a model can patch the example instead of the function. Keep the prompt hungrier than the test file.

Scale is the second. One plant is a fire drill, not a map of the repo. Restage bugs you have already fixed, oracle included, names stripped so you are not grading memory of your own commit message. A bug you cannot restage is an anecdote with better lighting. Treat it as a story. Do not enter it on the card.

If you point this same stop-rule at MonkeyCode's free model, and only afterward at the free server, read the current docs first. Confirm the token grant and the server limits yourself. Run the dry flag before you spend anything. I would rather you distrust the grant than trust this paragraph.

I Split the Score. Reach Was Never a Pass.

Jordan Liu — Sun, 11 Oct 2026 21:29:16 +0000

A free call that comes back with HTTP 200 is not a model that can do the job. I keep repeating that because a single pass rate keeps crawling back into the notes, wearing a lab coat. Reach, contract, and check are three different arguments. Glue them into one boolean and you will praise a host for being awake, or blame a model for a timeout it never saw.

You know the Tuesday demo. Same prompt, Wednesday, and the body is empty, or half an object, or a charming paragraph that fails the checker you trusted last month. Which piece actually broke? If you stored one bit, you cannot say. You graded a smoothie and called it fruit.

This is a protocol, not a scoreboard from a run I am about to invent. No latency trophy. No win rate. Those numbers belong to a log you produce yourself. What I will hand you is the split, a harness, and a rule for which column is allowed to speak.

The blend is the bug

Picture a clinic that publishes one word, healthy. Blood pressure, whether the patient showed up, and whether the vial was labeled all collapse into that word. A missed appointment looks like a bad heart. A mislabeled vial looks like a no-show. You would fire that clinic. Why do we publish model evals that way?

Free model access makes the blend worse, not kinder. A free server is a second moving part. It can refuse, queue, reset, or idle-timeout while the weights sit there innocent. A free model can also be perfectly reachable and still emit prose where your tool wanted an object. Score only the final checker and both failures wear the same red mark. Then the postmortem becomes a vibe.

I am not stating a quota, a model list, a box size, or a promise that any free tier stays free next week. Those facts rot on contact. If a vendor page is your source, read it the day you run, and paste that URL into the run record. Do not borrow a sentence from a blog as a spec.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability claims I will use as conditions, not as trophies: free model access, and a free server option. I was not given a verified catalog, a token ceiling, or a hardware note, so this draft does not invent them. MonkeyCode is one place those two conditions might be obtained. The protocol does not depend on that product. Swap the transport and the lesson still stands.

Freeze the exam before you touch the wire

I want one lever moving. The prompt set is a directory of files. Hash each file, then hash the sorted list of those digests. That second digest is the exam id. If a prompt changes after the first request, you are holding a different exam, and the comparison is theater. Throw the run out. Yes, that is fussy. It is also the only way a later diff means anything.

The checker is frozen too. A second function, local, pure, no network. It reads the parsed output and the fixture, then returns pass or fail plus a short reason. If the checker calls a model, you built a mirror. Mirrors always agree with themselves. I will not score that and call it evidence.

Conditions are a pair, not a tour of every host you can find. Condition A is the control: same exam id, a transport you already trust, even if that transport is a process on your laptop. Condition B is the free-model call landing on the free server. Same items. Same checker. Same timeout, written in milliseconds before the first request leaves. Edit the timeout after you see the red rows, and you edited the question.

I do not retry inside this score. A retry is a different policy, with its own article and its own lies. Score the first completed attempt. Log the transport error beside it so a later retry study can reuse the raw row without contaminating this one.

Three columns, and the order is the point

Reach asks a rude question. Did the transport return a body you are allowed to grade? DNS failure, connection reset, non-200, empty string, cutoff by the timeout you pre-registered: those are reach failures. They are not model failures. Write them in the reach column and stop. Do not feed a timeout string into the checker and call the item wrong. That move is how a flaky path becomes a fake intelligence regression.

Contract asks a picky question. Given a body, does it match the shape you demanded? I use a JSON schema on tool-shaped tasks. Missing keys, wrong types, a paragraph wrapped around an object, a brace cut off at the end: contract failures. The model can sound sharp and still fail this. Your agent will not care that the prose was warm.

Check asks the only question a user actually meant. The frozen checker says the content is acceptable. A schema-valid lie fails here. A schema-invalid truth fails earlier, in contract, and never reaches check. That ordering is the whole method. You stop operating on the wrong organ.

A run is comparable only when both conditions share the exam id, the checker digest, the timeout, and the item ids. Miss one of those and you have an anecdote. Anecdotes are fine at lunch. They are not a release note.

A harness you can run, with no fake log under it

The code below is a proposal. I did not execute it for this draft, and I will not paste imaginary rates under it. Fill the transport with a client you really have. Point the free side at a free server only after you have confirmed, from a current primary page, that the option is still offered.

mkdir -p eval/items eval/runs
python3 -m pip install --user jsonschema
# one JSON fixture per item: prompt, schema, fixture
python3 split_score.py --items eval/items --out eval/runs/$(date -u +%Y%m%dT%H%M%SZ)

#!/usr/bin/env python3
"""Proposal harness. Unexecuted here. Ships with no sample metrics."""

import argparse, hashlib, json, time
from pathlib import Path

TIMEOUT_MS = 20000  # pre-register; do not edit after red rows appear

def digest(blob: bytes) -> str:
    return hashlib.sha256(blob).hexdigest()

def load_items(folder: Path):
    items = []
    for path in sorted(folder.glob("*.json")):
        raw = path.read_bytes()
        item = json.loads(raw)
        item["_id"] = path.stem
        item["_digest"] = digest(raw)
        items.append(item)
    joined = "\n".join(i["_digest"] for i in items).encode()
    return items, digest(joined)

def reach_call(transport, item, timeout_ms):
    started = time.perf_counter()
    try:
        body, status = transport.complete(item["prompt"], timeout_ms / 1000)
    except Exception as exc:
        return {"ok": False, "reason": type(exc).__name__, "body": None,
                "status": None, "ms": int((time.perf_counter() - started) * 1000)}
    ok = status == 200 and isinstance(body, str) and body.strip() != ""
    return {"ok": ok, "reason": "ok" if ok else "empty_or_status", "body": body,
            "status": status, "ms": int((time.perf_counter() - started) * 1000)}

def contract_ok(body, schema):
    import jsonschema
    try:
        parsed = json.loads(body)
    except json.JSONDecodeError:
        return False, "not_json", None
    try:
        jsonschema.validate(parsed, schema)
    except jsonschema.ValidationError:
        return False, "schema", parsed
    return True, "ok", parsed

def score_condition(name, transport, items, checker):
    rows = []
    for item in items:
        reach = reach_call(transport, item, TIMEOUT_MS)
        row = {"item": item["_id"], "condition": name, "reach": reach["ok"],
               "reach_reason": reach["reason"], "ms": reach["ms"],
               "contract": None, "check": None}
        if not reach["ok"]:
            rows.append(row)
            continue
        good, why, parsed = contract_ok(reach["body"], item["schema"])
        row["contract"] = good
        row["contract_reason"] = why
        if not good:
            row["raw_head"] = (reach["body"] or "")[:240]
            rows.append(row)
            continue
        passed, why = checker(parsed, item["fixture"])  # local, pure, no model
        row["check"] = bool(passed)
        row["check_reason"] = why
        rows.append(row)
    return rows

def rates(rows):
    n = len(rows) or 1
    reached = [r for r in rows if r["reach"]]
    contracted = [r for r in reached if r["contract"]]
    checked = [r for r in contracted if r["check"]]
    return {
        "n": len(rows),
        "reach": len(reached) / n,
        "contract_given_reach": (len(contracted) / len(reached)) if reached else None,
        "check_given_contract": (len(checked) / len(contracted)) if contracted else None,
    }

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--items", type=Path, required=True)
    ap.add_argument("--out", type=Path, required=True)
    args = ap.parse_args()
    items, exam = load_items(args.items)
    raise SystemExit(
        "Inject control and free transports, then write rates() per condition. "
        f"exam={exam} items={len(items)} out={args.out}"
    )

if __name__ == "__main__":
    main()

Look at rates. It will not divide checks by every item and call that accuracy. Check is conditional on contract. Contract is conditional on reach. A free server that drops half the calls can still show a pretty check rate on the half that arrived. Publish only the pretty number and you are advertising the survivors. I would rather see a ugly reach column than a polished lie.

A fixture can be tiny. Here is the shape I want on disk, so the harness and the checker share one contract:

{
  "prompt": "Return JSON only. Pick a status for a build that exited 0.",
  "schema": {
    "type": "object",
    "required": ["status"],
    "properties": {"status": {"enum": ["green", "red"]}},
    "additionalProperties": false
  },
  "fixture": {"expect": "green"}
}

def checker(parsed, fixture):
    ok = parsed.get("status") == fixture["expect"]
    return ok, "match" if ok else "mismatch"

That checker is boring on purpose. Boring is a feature. If you cannot write a boring checker, you are not ready to split a score, and a vibe grade will flatter you faster than a free host ever could.

How to read a log you actually produced

Suppose reach falls apart on the free server while contract-given-reach stays close to the control. That is a host story, or a client story. Fix the path, the DNS, the idle timeout, the auth header. Do not prompt harder. A better sentence will not repair a reset packet. I have watched people rewrite instructions for an hour because they never logged status codes. Don't be that hour.

Suppose reach holds and contract collapses. Now you may look at the model, and at the wrapper around it. Truncation first. A tight output cap masquerades as stupidity. Then check whether a chat template stuffed a preamble in front of the object you demanded. The harness keeps raw_head for that reason. A reason code without the body is gossip.

Suppose contract holds and check collapses. Now you may talk about competence. Even then, talk about these items, this checker, this exam id. One exam is not a personality. It is not a ranking of vendors. It is a paired difference on a frozen set.

Write a minimum item count before the run. I use 30 as a floor for a smoke comparison, and I do not treat 30 as a paper. Under that floor a single weird fixture swings the rate, and you will narrate noise with a straight face. Smaller sets are allowed. Call them small in the first line. Pretending small is large is the failure mode this whole split is meant to catch.

Compare on item id, not on two averages floating in a slide. If item 14 failed only on the free path, open item 14. The mean will not tell you that the fixture held a nested list the control happened to flatten. Pair the rows. Then talk.

What I will not let this protocol claim

It will not tell you the free server is cheap enough to ship. Cost needs a bill, and I do not have yours. It will not tell you about privacy. A free host may log prompts. If a fixture contains customer text, do not send it. Synthetic items, or you have left evaluation and walked into an incident.

It will not survive a moving exam. Tweak prompts between condition A and condition B and you measured your editing. It will not fairly grade open-ended prose. The checker has to be dull. Subjective rubrics need a different design, with a judge that is not the model under test. Using the same free model to grade itself is a mirror again. Skip it.

Who should walk away? If you need an SLA, a retention promise, or a named model pinned for a regulated audit, a free option is the wrong instrument. Buy a contracted endpoint and a change window. If you only want a weekend demo, skip the harness too. A demo can be a demo. Just do not paste it into a README and call it a measurement.

Silent retries also void the reach column. If your client library retries unless you say otherwise, the row is already dirty. Turn that off, or log each attempt as its own policy. Mixing them is how bounded and unbounded behavior get the same name.

Free model access is still a gift for this kind of work, because the expensive mistake is scoring the wrong organ, not renting a larger box. A free server is useful when condition B has to be a real network path instead of a daydream. It becomes a costume the moment you let reach define quality. If you already hold a key for that pair of conditions, including through MonkeyCode after you re-read what the docs offer this week, point the transport at it and keep the three columns apart. Publish the column that moved. Your readers can survive a narrower claim. They cannot survive a blended one.

I Held One Lever Still

Jordan Liu — Sat, 10 Oct 2026 17:59:52 +0000

A free model and a free server are two levers. Move both, then quote one pass rate, and you did not review a model. You published a blend. I will not publish a blend.

You know the afternoon this happens. The endpoint answers. The box is up. The suite is already on disk, so you fire it, crop the fraction, and feel done. Which lever moved? If the honest answer is both, plus a retry you forgot to count, the fraction is a story about a pile.

Free infrastructure makes the pile feel scientific. No invoice, no guilt, and somehow no design. I do not buy that trade. A zero bill is not a baseline. It is a quiet way to change two things and then narrate one of them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two availability claims were supplied for this draft, and they are the only product facts I will stand on: free model access, and a free server option. I am not printing a token grant, a model roster, a hardware sheet, a duration, or a license badge. None of those arrived as a primary source I can defend for 2026-10-10. A number I cannot cite does not become true because a draft would read warmer with a big integer in the first screen.

So what experiment is left, if the headline is banned?

A two-by-two, blocked on the same task list, with one lever held still. Same prompts. Same grader. Same seed rule. Same timeout. The only legal swap is the factor you wrote into the manifest before the first request left the machine. Did the salt change, or the pot? If you changed both and then swore the broth was brighter, you cooked an anecdote. Hold the pot. Change the salt. On the next batch, hold the salt and change the pot. Now a later reader has something to argue with.

Hold the server, swap the model slot. That cell may talk about free model access, and only against the server you actually held. Hold the model, swap the server slot. That cell may talk about the free server, and only against the model you actually held. Swap both and you have an interaction check, not a verdict. If the single swaps were calm and the pair fell over, you learned they misbehave as a couple. You did not earn "the model failed." You also did not earn "the server failed."

The baseline cell is the ruler. It can be a recorded fixture plus a local stub, as long as you can hold that pair still next week. A ruler that moves is not a ruler. It is a second treatment pretending to be the wall. I would rather compare against a boring fixture than against whatever the router felt like serving at lunch.

Blocking is the part people skip because it looks fussy. It is not fussy. If cell A sees the tasks in one order and cell B sees a shuffle, order became a third lever. If one cell retries and the other does not, retry policy became a fourth. If the seed is "whatever the process clock felt like," you cannot tell a flaky item from a factor. Derive the seed from the task id plus one run salt, write that salt into the manifest, and refuse to resample until the rate looks friendly. Friendliness is not a stopping rule. It is a wish.

What does a free path break, even when you are trying to be good? A label that is really a router. A server that sleeps, so the first tasks measure wakeup and the rest measure the work. A client that retries while the manifest still says one attempt. An SDK that returns a friendly alias, so you never see which level you hit. If the factor name is an alias, the cell is unlabeled. An unlabeled cell is a missing cell. Would you sign a lab book that said "the blue one" and meant three different blues? I would not.

I am not going to invent the quota that punches a hole in the run. I am going to treat an unfinished cell as unfinished. Survivors do not get promoted to "the run." That promotion is how a partial afternoon becomes a chart with a calm typeface. Missing cells do not get filled with the mean of their neighbors. Neighbors are different treatments. Averaging across them is how a blend learns to wear a single name.

Read four symbolic outcomes, not as data I collected, but as the only sentences the grid licenses. Suppose model-only moves and server-only does not. You may talk about the model slot. Suppose server-only moves and model-only does not. You may talk about the server slot. Suppose neither single swap moves, and the pair does. You may talk about interaction, then stop. Suppose every cell moves the same way. You probably changed the harness, the tasks, or the grader, and the levers are innocent until you prove otherwise. None of those sentences need a fake percentage. The shape is the finding. A percentage without the shape is decoration.

The artifact is a protocol, not a lab log. I have not executed it against a live free tier for this article. The client body is a stub on purpose. If you run it after you replace the stub, the JSON manifest is the result. If you do not run it, you do not get to borrow certainty from the shape of the code. A green fraction from an unimplemented client would be a magic trick. I am not collecting those.

#!/usr/bin/env python3
"""Two-by-two eval manifest. Reference protocol, not a scored bake-off."""
from __future__ import annotations

import hashlib
import json
import os
import time
import uuid
from dataclasses import asdict, dataclass, field
from typing import Callable, Optional

@dataclass
class Factor:
    name: str
    level: str
    endpoint: str

@dataclass
class Trial:
    task_id: str
    seed: int
    model_level: str
    server_level: str
    ok: Optional[bool] = None
    error_class: Optional[str] = None
    latency_ms: Optional[int] = None

@dataclass
class Manifest:
    run_id: str
    harness_sha256: str
    task_file_sha256: str
    seed_salt: int
    started_unix: float
    cells: dict = field(default_factory=dict)
    finished_unix: Optional[float] = None
    admissible: bool = False
    refusal_reason: Optional[str] = None

def sha256_file(path: str) -> str:
    return hashlib.sha256(open(path, "rb").read()).hexdigest()

def classify(exc: BaseException) -> str:
    if isinstance(exc, NotImplementedError):
        return "unimplemented"
    name = type(exc).__name__
    if "Timeout" in name:
        return "timeout"
    if "Connection" in name or "HTTP" in name:
        return "transport"
    return "grader_or_task"

def run_cell(tasks, model: Factor, server: Factor, salt: int, call: Callable) -> list[Trial]:
    out = []
    for task_id in tasks:
        seed = int(hashlib.sha256(f"{salt}:{task_id}".encode()).hexdigest()[:8], 16)
        trial = Trial(task_id, seed, model.level, server.level)
        t0 = time.perf_counter()
        try:
            call(task_id, model, server, seed)
            trial.ok = True
        except Exception as exc:
            trial.ok = False
            trial.error_class = classify(exc)
        trial.latency_ms = int((time.perf_counter() - t0) * 1000)
        out.append(trial)
    return out

def judge(manifest: Manifest) -> tuple[bool, str]:
    need = {"baseline", "model_only", "server_only", "both_swapped"}
    have = set(manifest.cells)
    if have != need:
        return False, "missing_cells:" + ",".join(sorted(need - have))
    for name, trials in manifest.cells.items():
        if not trials:
            return False, f"empty:{name}"
        if any(t["error_class"] == "unimplemented" for t in trials):
            return False, "stub_call_is_not_evidence"
        if any(t["ok"] is None for t in trials):
            return False, f"unscored:{name}"
    return True, "grid_complete"

def rates_if_admissible(manifest: Manifest) -> Optional[dict]:
    ok, _reason = judge(manifest)
    if not ok:
        return None
    report = {}
    for name, trials in manifest.cells.items():
        graded = [t for t in trials if t["error_class"] is None]
        report[name] = {
            "n": len(trials),
            "graded_n": len(graded),
            "pass_rate": (sum(bool(t["ok"]) for t in graded) / len(graded)) if graded else None,
            "transport_n": sum(t["error_class"] == "transport" for t in trials),
            "timeout_n": sum(t["error_class"] == "timeout" for t in trials),
        }
    return report

def main() -> None:
    model_base = Factor("model", "baseline-slot", os.environ["MODEL_BASE_URL"])
    model_free = Factor("model", "free-model-slot", os.environ["MODEL_FREE_URL"])
    server_base = Factor("server", "baseline-slot", os.environ["SERVER_BASE_URL"])
    server_free = Factor("server", "free-server-slot", os.environ["SERVER_FREE_URL"])
    task_path = os.environ["TASK_FILE"]
    tasks = json.load(open(task_path))
    salt = int(os.environ.get("EVAL_SEED", "17"))

    def call(task_id: str, model: Factor, server: Factor, seed: int) -> None:
        raise NotImplementedError(
            f"wire your client: {task_id} {model.level} @ {server.level} seed={seed}"
        )

    plan = {
        "baseline": (model_base, server_base),
        "model_only": (model_free, server_base),
        "server_only": (model_base, server_free),
        "both_swapped": (model_free, server_free),
    }
    manifest = Manifest(
        str(uuid.uuid4()), sha256_file(__file__), sha256_file(task_path), salt, time.time()
    )
    for name, (model, server) in plan.items():
        manifest.cells[name] = [asdict(t) for t in run_cell(tasks, model, server, salt, call)]
    ok, reason = judge(manifest)
    manifest.admissible = ok
    manifest.refusal_reason = None if ok else reason
    manifest.finished_unix = time.time()
    json.dump(asdict(manifest), open("run_manifest.json", "w"), indent=2)
    if not ok:
        raise SystemExit(f"refusing headline metric: {reason}")

if __name__ == "__main__":
    main()

Pin the harness and the task file before you trust the run you think you started. A dirty tree is another lever, and it never introduces itself.

sha256sum eval_grid.py tasks.json | tee inputs.sha256
export MODEL_BASE_URL MODEL_FREE_URL SERVER_BASE_URL SERVER_FREE_URL
export TASK_FILE=tasks.json EVAL_SEED=17
python3 eval_grid.py
python3 -c 'import json; m=json.load(open("run_manifest.json")); print(m["admissible"], m.get("refusal_reason"))'

A stub run should exit on stub_call_is_not_evidence. That refusal is the correct first result. Do not catch it and tweet the partial JSON. The partial JSON is a note that the client was never wired.

When a real client is in place, debug the manifest before you debug the model. Open run_manifest.json and compare transport_n across cells. If transport piles up only in server_only and both_swapped, your sentence is about the server path, or about the path to that server. It is not about the model. If timeouts pile up only on the free model slot, say timeout, then decide whether your product even cares. If missing_cells fires, you do not have a result. You have a plan you did not finish. Imputing the missing corner from the other three is how people invent an interaction they never observed.

What do you commit? The harness, the task-file hash, the seed salt, and the manifest. Not a screenshot of a fraction. A screenshot cannot tell me whether the both-swapped cell was allowed to speak. The JSON can. If admissible is false, the writeup is the refusal reason, and that writeup is allowed to be short. Short and refused beats long and blended.

Who should walk past this? You, if the demo is at lunch and the room wants a vibe. A grid is a bad vibe. You, if the question is whether a UI feels kind, because kindness is not a factor level in this file. You, if every request is locked to one blended endpoint and you cannot hold a baseline still. Then you have one pipe. A pipe can still be worth using for a prototype. It cannot carry a sentence that starts with "the model," and it cannot carry one that starts with "the server." Say "this pipe" and stop. That sentence is weaker, and it is the one you actually earned.

Limits, with the romance removed. There is no scoreboard here, because I did not run the live pair, and I will not invent one to decorate a method. Grader quality sits outside the grid. A stable wrong grader produces an admissible, repeatable, useless manifest. Environment variables are not a docs site. Free access can move next week, so last week's JSON does not license next week's slogan. Interaction can be real and still be too small to matter for your task. The protocol will not choose your practical threshold. Write that threshold down before you look, or do not pretend the look was a test.

Trying the pipe is fine. Publishing the try as a model review is the part I will not cosign. If the free model slot and the free server slot in your env file are the ones named in the disclosure, they are eligible factors in this grid. They are not pre-cleared results. Run the four cells once, keep the refusal or the JSON, and let that file outlive the badge.

I Froze the Hash. The Free Run Had to Earn the Sign.

Jordan Liu — Fri, 09 Oct 2026 14:26:59 +0000

A free runner is not a result. A stopwatch is not a race, and a pass rate is not a decision until the sample, the scorer, and the stop rule are already boring. If any of those can still wriggle, a free model and a free server only make the wriggling cheaper.

Freeze the hash. Pre-register the break. Let a sign appear only when both halves agree.

You know the itch. A stack shows up, the calls cost nothing you can feel, a server is offered so you skip provisioning, and lunch starts to look like a leaderboard. Did you ever nudge one prompt, rerun, and then describe the last number as if it had been the hypothesis?

If that scene feels familiar, it is the habit this card is built to interrupt. Not a trophy. A failure mode with a friendly face.

The feed keeps congratulating small models for going quiet on a private checker. Quiet is not stable. A checker can go silent because you edited the items until it did. Have you ever watched a green cell calm a room that never asked whether the other half of the file would have stayed green?

Seasoning and judging are different jobs. Taste the soup to change the recipe, and you are cooking. Taste it to award a ribbon, and you are peeking. Free access is a gift to the cook and a trap for the judge, because another taste barely stings.

The chart gets patient. You do not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. For this method I am treating two availability claims as given, and only those: free model access, and a free server option. I am not stating a token quota, a model roster, a machine size, a duration, or a score from a run.

I do not have a primary source I am willing to pin next to a task hash, and I will not fill the gap with a figure that might already be stale. If it is not in the card, it is not a result. A number that cannot sit in the same commit as the tasks is just weather.

What are those surfaces doing in the method? They are a pair of runners, not a mascot. Treatment A is free model access: your process sends a frozen item and scores the completion locally. Treatment B is the free server: the same item goes to a hosted runner you do not administer, and you score the trace that comes back with the same function.

Same items, same rubric, different place the work happened. If you cannot name which surface tripped a rule you wrote yesterday, you do not have an evaluation. You have a demo with extra steps.

I have not executed this harness against that product for this article. The blocks below are a proposal with a synthetic fixture so you can see the refusal path on your laptop. Thresholds are choices you lock before the first network call. They are not measurements, and they are not a review of anyone's uptime.

The unit is the item, not the afternoon. You export tasks to JSONL, one object per line, each with an id and an expected string. You hash the file bytes, then split with a seeded digest so half 0 and half 1 stay put on rerun. You score both treatments on both halves.

A sign is earned only when each half clears the item floor, each arm stays under the transport-error ceiling, the absolute gap clears a floor you wrote down so a one-item wobble cannot wear a crown, and the two halves agree on the direction. A hash mismatch is an invalidation even if the new file is basically the same. Editing the rubric after a sad cell is an invalidation too. Git will remember, even if your recap does not.

Why not one mean? A single average can sand a flip off the table until the room feels calm. I would not trust that sanding in a write-up. Would you change a routing file because forty items leaned left and forty leaned right?

I would call that a break and leave the config alone. Agreement is a low bar. It is still a bar.

Performs well cannot mean a percentage I did not measure. Here it means the run can finish under rules another person can replay: both surfaces return scoreable traces, invalidation stays quiet, and the sign matches across halves. Breaks has to stay a specific sentence, not a mood.

A timeout is not a foolish model. A rubric that never looks at tool traces is not a broken server. A moved hash is not a product defect. It is you, still cooking.

Fold those into one red badge and the free tier takes the blame for your notebook. Three sentences and zero winners is a better log. Can you say which sentence fired? If you cannot, you are not done.

Who should skip this? Anyone who needs a flattering percentage before stand-up, because the card is built to withhold it. Anyone holding private customer text should not ship it to a free server because the meter feels kind. Anyone who needs a medical, legal, or safety judgment should not pretend a frozen string check is that judgment.

If you cannot commit the task file, the card, and the scorer in one revision, you cannot audit the claim later. Do not make the claim.

Limitations, with the soft landing removed. Half-agreement is not proof that one surface is wiser, cheaper in a way that lasts, or still offered next month. It is a stability check on a small decision. A hosted runner can insert retries, clocks, and tool hops your local call never saw, and this protocol notices only if the scorer reads those fields on purpose.

String equality is a blunt rubric. It will call a harmless paraphrase a miss, and it will miss a fluent wrong answer that happens to match. A pilot that survives the card may justify a larger pre-registered run. It does not justify a testimonial, and I am not writing one.

The workflow is short enough to do before coffee gets cold, and strict enough to ruin a dishonest chart. Write the card in the same file as the scorer, hash the JSONL, and commit both plus the hash before any call. Implement two clients that take an item and return text, and keep secrets out of the repo. Leave a dry implementation that raises, so a curious teammate cannot just see what happens against the network by running the module.

Replace the dry clients only after that commit exists. Run the module and read earned. If it is false, the result is the breaks list. Sit with it. That list is the finding.

Save this as split_half_sign.py. It is a proposal. --fixture never leaves the process, and the default clients raise so a real file cannot accidentally become a leaderboard.

python3 split_half_sign.py --fixture
sha256sum tasks.jsonl
python3 split_half_sign.py tasks.jsonl "$(sha256sum tasks.jsonl | cut -d' ' -f1)"

That second command, left as written, should fail the transport rule. Good. That is the dry default doing its job. A green earned before you replace dry_complete would mean the harness is lying, and you should delete it.

#!/usr/bin/env python3
# Proposal, not a product benchmark.
# Lock CARD before any network call. Thresholds are design choices.
# --fixture is synthetic and opens no socket.
from __future__ import annotations

import hashlib
import json
import sys
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable

CARD = {
    "protocol": "split-half-sign-v1",
    "min_items_per_half": 30,
    "min_scored_fraction": 0.95,
    "max_transport_error_rate": 0.05,
    "min_abs_gap": 0.02,
    "seed": "freeze-me-before-running",
}

CompleteFn = Callable[[dict], str]


def file_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows


def half_of(item_id: str, seed: str) -> int:
    digest = hashlib.sha256(f"{seed}:{item_id}".encode()).hexdigest()
    return int(digest[-1], 16) % 2


def rubric(task: dict, completion: str) -> bool:
    expected = str(task.get("expected", ""))
    return completion.strip() == expected.strip()


@dataclass
class ArmReport:
    name: str
    n: int
    scored: int
    transport_errors: int
    pass_rate: float | None
    invalidated: str | None


def run_arm(name: str, items: list[dict], complete: CompleteFn) -> ArmReport:
    errors = scored = hits = 0
    for task in items:
        try:
            text = complete(task)
        except RuntimeError:
            errors += 1
            continue
        scored += 1
        hits += int(rubric(task, text))
    n = len(items)
    err_rate = (errors / n) if n else 1.0
    frac = (scored / n) if n else 0.0
    invalidated = None
    if n < CARD["min_items_per_half"]:
        invalidated = "below_min_items"
    elif err_rate > CARD["max_transport_error_rate"]:
        invalidated = "transport"
    elif frac < CARD["min_scored_fraction"]:
        invalidated = "unscored_mass"
    rate = (hits / scored) if scored else None
    return ArmReport(name, n, scored, errors, rate, invalidated)


def sign_of(left: float | None, right: float | None) -> int:
    if left is None or right is None:
        return 0
    gap = left - right
    if abs(gap) < CARD["min_abs_gap"]:
        return 0
    return 1 if gap > 0 else -1


def evaluate(rows, arms, got_hash, expected_hash):
    breaks = []
    if expected_hash is not None and got_hash != expected_hash:
        return {"earned": False, "breaks": ["hash_mismatch"], "hash": got_hash, "sign": 0}
    halves = {0: [], 1: []}
    for row in rows:
        halves[half_of(str(row["id"]), CARD["seed"])].append(row)
    half_signs = []
    reports = []
    for h, subset in halves.items():
        rates = {}
        for name, fn in arms.items():
            report = run_arm(f"{name}.h{h}", subset, fn)
            reports.append(asdict(report))
            if report.invalidated:
                breaks.append(f"{name}.h{h}:{report.invalidated}")
            rates[name] = report.pass_rate
        half_signs.append(sign_of(rates.get("free_model"), rates.get("free_server")))
    if not breaks and (not half_signs or 0 in half_signs):
        breaks.append("gap_below_floor_or_unscored")
    if not breaks and half_signs[0] != half_signs[1]:
        breaks.append("halves_disagree")
    earned = not breaks
    return {
        "earned": earned,
        "sign": half_signs[0] if earned else 0,
        "half_signs": half_signs,
        "breaks": breaks,
        "hash": got_hash,
        "reports": reports,
        "note": "+1 means free_model ahead on pass rate; -1 means free_server ahead",
    }


def dry_complete(_task: dict) -> str:
    raise RuntimeError("dry run: replace me after the hash is committed")


class FixtureComplete:
    # Synthetic. Pass rates flip by half. Not a vendor measurement.

    def __init__(self, name: str) -> None:
        self.name = name

    def __call__(self, task: dict) -> str:
        h = half_of(str(task["id"]), CARD["seed"])
        want_hit = (self.name == "free_model" and h == 0) or (
            self.name == "free_server" and h == 1
        )
        return str(task["expected"]) if want_hit else "MISS"


def synthetic_rows() -> list[dict]:
    buckets = {0: [], 1: []}
    i = 0
    while len(buckets[0]) < 40 or len(buckets[1]) < 40:
        item_id = f"t{i:04d}"
        h = half_of(item_id, CARD["seed"])
        if len(buckets[h]) < 40:
            buckets[h].append({"id": item_id, "expected": f"ok-{i}"})
        i += 1
    return buckets[0] + buckets[1]


def main(argv: list[str]) -> int:
    if len(argv) >= 2 and argv[1] == "--fixture":
        rows = synthetic_rows()
        blob = "\n".join(json.dumps(r, separators=(",", ":")) for r in rows).encode()
        digest = hashlib.sha256(blob).hexdigest()
        out = evaluate(
            rows,
            {
                "free_model": FixtureComplete("free_model"),
                "free_server": FixtureComplete("free_server"),
            },
            digest,
            digest,
        )
        out["fixture"] = "synthetic_refusal_not_a_product_run"
        print(json.dumps(out, indent=2))
        return 0
    if len(argv) != 3:
        print(
            "usage: split_half_sign.py --fixture | split_half_sign.py tasks.jsonl HASH",
            file=sys.stderr,
        )
        return 2
    path = Path(argv[1])
    out = evaluate(
        load_jsonl(path),
        {"free_model": dry_complete, "free_server": dry_complete},
        file_hash(path),
        argv[2],
    )
    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

A task file is just lines like these. Hash the bytes you will actually send, not a prettier export you make later.

{"id":"t0001","expected":"pong"}
{"id":"t0002","expected":"pong"}

When the fixture prints, look at three fields and ignore the urge to screenshot a winner. earned should be false. breaks should name the disagreement, and half_signs should be a pair that does not match. If a future real run prints earned true, the sign is a stable direction on this frozen file, under this blunt rubric, on this day.

It is still not a quota, not a hardware review, and not permission to skip the private-data rule. Puppet rates inside --fixture are not a vendor score. They exist so you can watch the harness keep its mouth shut. When you later point the same function at real clients, silence still means silence.

I will ask for one thing, then stop selling. If free model access and a free server are actually available on the MonkeyCode account you use, point this card at tasks you can stand to hash in public. Commit the hash and the stop rule before the first call.

If the sign does not earn itself, write about the break. I trust that note more than a last-run percentage.

I Scored the Flip. Best-of-Five Folded.

Jordan Liu — Thu, 08 Oct 2026 10:16:10 +0000

I will not publish best-of-five. A free code model is not usable because one sample passed. That sample might be the only one that would have passed. If you publish the highlight reel, you measured a slot machine, not a tool.

You have seen the flattering cut, haven't you? Five draws, one green check, a screenshot, a conclusion. What question did that answer? Not the question your teammate hits on a Tuesday, when nobody is sitting there rerolling until the tests blush.

The feed this week is busy with profile toys and portfolios people vibe-coded over a weekend. Fine toys. None of them tell you whether draw three still exits zero.

So the conclusion comes first, before any endpoint. Report the first draw, the stable pass, and the flip. Ignore any protocol that lets you delete the red rows. If those three numbers are not on disk, you do not have an evaluation. You have a mood.

First-draw pass is what an unattended user gets. Stable pass is the share of tasks that pass on every draw, with no runner error hiding in the set. Flip is the share that both pass and fail across draws.

Same prompt. Same check. Different verdict. That last one is the alibi killer.

A task that flips is not basically working. It is a coin, and coins do not get merge rights. Would you hand a coin your migration?

Why not a single accuracy number? Because a single number is how luck borrows a lab coat. Best-of-five can rise while stability falls. You would feel better and ship worse.

I would rather see an ugly stable-fail than a pretty flip. An ugly fail tells me to change the task class or the tool. A flip tells me I was about to quote a coincidence.

I do not have a leaderboard, and I am not going to invent one. This note is a protocol and a scorer. Any rows that look like results in the fixture are a self-check of the arithmetic. They are not a benchmark of a free model, a paid model, or a server.

If a pass rate shows up here without a CSV from the day the checks ran, treat it as fiction. I want you suspicious. Suspicion is the method.

Where do the free generator and the free runner enter? I want both boring and available, so the afternoon is about variance instead of about whether I could afford the draws. MonkeyCode is the option this harness is aimed at: free model access for the completions, and a free server option so every sample faces the same check command rather than whatever my laptop happened to have installed. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The outreach brief calls the project open source. Clone the repository you can verify. A paraphrased client is not a source.

I am also not quoting a token quota, a machine size, or a duration. Those are dashboard facts, and dashboard facts rot faster than blog posts. On the day you run, read the current allowance and the current server terms, then copy them into the run notes by hand.

If the allowance is smaller than K times your task count, shrink K before you start. Dropping tasks afterward, and still calling the file a replicate, is how samples vanish while the chart stays confident.

The harness is plain Python on purpose. No framework to debug before the model. It shells out to your check command, which is the whole point. Pretty code that never executes is a different hobby.

#!/usr/bin/env python3
"""split_score.py — first draw, stable pass, and flip.

Live mode is unexecuted until MODEL_ENDPOINT is set.
`--fixture` checks the arithmetic only. It is not a model score.
"""
from __future__ import annotations

import argparse, csv, json, os, subprocess, hashlib
from pathlib import Path

def load_tasks(root: Path) -> list[dict]:
    tasks = []
    for path in sorted(root.glob("*/task.json")):
        task = json.loads(path.read_text())
        task["dir"] = str(path.parent)
        tasks.append(task)
    if not tasks:
        raise SystemExit(f"no tasks under {root}")
    return tasks

def call_model(endpoint: str, prompt: str, draw: int) -> str:
    # Proposal only. Match the real request shape before a live run.
    # Read any key from the environment. Never hardcode one.
    import urllib.request
    body = json.dumps({
        "prompt": prompt,
        "draw": draw,
        "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    if "text" not in payload:
        raise SystemExit("response missing text; fix the client, do not score it")
    return payload["text"]

def run_check(task_dir: str, code: str) -> str:
    snippet = Path(task_dir) / "_candidate.py"
    snippet.write_text(code)
    try:
        proc = subprocess.run(
            ["python3", "check.py"],
            cwd=task_dir,
            capture_output=True,
            text=True,
            timeout=30,
        )
    except subprocess.TimeoutExpired:
        return "runner_error"
    if proc.returncode == 0:
        return "pass"
    if proc.returncode < 0:
        return "runner_error"
    err = (proc.stderr or "") + (proc.stdout or "")
    if "Traceback" in err and "_candidate" not in err:
        return "runner_error"
    return "fail"

def summarize(rows: list[dict]) -> dict:
    by_task: dict[str, list[tuple[int, str]]] = {}
    for row in rows:
        by_task.setdefault(row["task"], []).append((int(row["draw"]), row["verdict"]))
    n = len(by_task)
    if n == 0:
        raise SystemExit("no rows to score")
    first = stable = flip = runner = 0
    for pairs in by_task.values():
        verdicts = [v for _, v in sorted(pairs)]
        if any(v == "runner_error" for v in verdicts):
            runner += 1
        if verdicts[0] == "pass":
            first += 1
        if all(v == "pass" for v in verdicts):
            stable += 1
        if "pass" in verdicts and "fail" in verdicts:
            flip += 1
    return {
        "tasks": n,
        "first_pass": round(first / n, 3),
        "stable_pass": round(stable / n, 3),
        "flip_rate": round(flip / n, 3),
        "tasks_with_runner_error": runner,
    }

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--tasks", default="tasks")
    parser.add_argument("--draws", type=int, default=5)
    parser.add_argument("--fixture", action="store_true")
    parser.add_argument("--out", default="split_score.csv")
    args = parser.parse_args()

    if args.fixture:
        # Synthetic rows. They test the scorer. They do not describe a product.
        rows = [
            {"task": "parse_int", "draw": 0, "verdict": "pass"},
            {"task": "parse_int", "draw": 1, "verdict": "pass"},
            {"task": "parse_int", "draw": 2, "verdict": "fail"},
            {"task": "slug", "draw": 0, "verdict": "fail"},
            {"task": "slug", "draw": 1, "verdict": "fail"},
            {"task": "slug", "draw": 2, "verdict": "fail"},
            {"task": "clamp", "draw": 0, "verdict": "pass"},
            {"task": "clamp", "draw": 1, "verdict": "pass"},
            {"task": "clamp", "draw": 2, "verdict": "pass"},
        ]
    else:
        endpoint = os.environ["MODEL_ENDPOINT"]
        rows = []
        for task in load_tasks(Path(args.tasks)):
            for draw in range(args.draws):
                code = call_model(endpoint, task["prompt"], draw)
                verdict = run_check(task["dir"], code)
                rows.append({"task": task["id"], "draw": draw, "verdict": verdict})

    summary = summarize(rows)
    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=["task", "draw", "verdict"])
        writer.writeheader()
        writer.writerows(rows)
    digest = hashlib.sha256(Path(args.out).read_bytes()).hexdigest()[:12]
    print(json.dumps({**summary, "csv_sha256_12": digest}, indent=2))

if __name__ == "__main__":
    main()

The fixture is supposed to be mean. parse_int flips. slug fails every draw. clamp stays green.

If you cannot say those three sentences without opening a pricing page, you are not ready for a live file. What should the scorer print? First-draw 0.667, stable pass 0.333, flip rate 0.333, runner errors 0. I did that division on paper before I trusted a script. You should do the same.

A scorer you have not caught is just another lucky sample, wearing a .py hat. Run the self-check before you touch a network. It should not need a key, a server, or a hope.

python3 split_score.py --fixture --out /tmp/split_fixture.csv
python3 - << 'PY'
import json, subprocess
raw = subprocess.check_output(
    ["python3", "split_score.py", "--fixture", "--out", "/tmp/split_fixture.csv"]
)
got = json.loads(raw)
expect = {
    "first_pass": 0.667,
    "stable_pass": 0.333,
    "flip_rate": 0.333,
    "tasks_with_runner_error": 0,
}
for key, val in expect.items():
    assert got[key] == val, (key, got[key], val)
print("scorer ok", got["csv_sha256_12"])
PY

A task is a directory, not a vibe. task.json holds an id and a prompt. check.py imports the candidate and exits non-zero when the behavior is wrong. Keep the check dumber than the task.

If the check is the clever part, you are grading your harness and calling it a model. That is an easy way to feel rigorous while learning nothing.

{
  "id": "clamp",
  "prompt": "Write clamp(x, lo, hi) in _candidate.py. Return x bounded to [lo, hi]. No imports. No prints."
}

# tasks/clamp/check.py — example check, not a live result
import _candidate

def expect(got, want):
    if got != want:
        raise SystemExit(f"{got!r} != {want!r}")

expect(_candidate.clamp(3, 0, 2), 2)
expect(_candidate.clamp(-1, 0, 2), 0)
expect(_candidate.clamp(1, 0, 2), 1)

The generator and the runner should not share a mood. Draws come from the free model endpoint. The live scorer runs on the free server, so check.py sees one image, one Python, one clock. Laptop runs are for the fixture. Mix them and you are grading your own machine again.

When the fixture matches, and only then, point the script at a live endpoint. The request body inside call_model is a proposal. I have not executed it against a current API, and I will not guess a path, a model id, or a header name in public.

Match the client to the docs you verified today. Keep the key in the environment. Log a hash of the prompt next to the CSV.

If the prompt changes, start a new file. Editing the old file is not replication. It is a quiet rewrite.

# Proposal, not a verified hostname or client path.
# Confirm today's free-server login from the dashboard, then run the scorer there.
export MODEL_ENDPOINT="https://example.invalid/draw"
python3 split_score.py --tasks ./tasks --draws 5 --out runs/today.csv
sha256sum runs/today.csv

Read the printout like someone who has been fooled by a green cell before. High first-draw plus high flip means the generator can look competent and still be a coin. Would you let that coin edit a migration? I would not.

Low first-draw and a stable fail is a kindness. It tells you this task class is a break, not a teaser. High stable pass and a flip near zero is the only pattern I will call performs well, and only for drafts a human still reads.

Runner errors are not model fails. They mean the check, the timeout, or the server session broke. Fix that, rerun, and do not average across the mess.

Who should walk away? If you need a number for a launch note tonight, walk away. Five draws times a dozen tasks will not fit in a coffee break, and a partial grid is not a result you get to round up.

If the code can touch credentials, payments, or auth, walk away from the free path. A green clamp says nothing about a webhook. If the check needs a private network, a production token, or customer data, do not place it on a free server either. Shared runners are for toy tasks with toy fixtures.

If your question is taste in prose, this harness will lie politely, because it only speaks exit codes. Exit codes do not have taste. They have zero and not-zero.

There is a quieter limit, and it matters. Temperature 0.2 in the proposal is a frozen confound, not an optimum I measured. Change it and you have a different study. K=5 is a budget, not a proof about sample size.

A free server can restart, throttle, or run slower than a laptop. Those events belong in the notes, not in the model column. I will not use this protocol to rank products against each other.

It can rank task classes against one generator, on one day, under one check suite. That is enough. It is also all the certainty the design earns.

If you already have a MonkeyCode login, point the draws at the free model, run the checks on the free server, and keep the CSV beside the prompt hash. If you do not, run the fixture anyway. The arithmetic does not need a brand. The brand does not get a score until the arithmetic has a file.