DEV Community: Casey Zhang

Kill the Runner Mid-Patch: A Fault-Injection Bench for Agent Sessions

Casey Zhang — Tue, 27 Oct 2026 12:39:19 +0000

Here is a shape of failure worth ten minutes of your attention. An agent drafts a patch, the machine goes away mid-task, the retry starts from the top, and the migration lands twice. The pass rate stayed at 100% — because the interrupted run never reached a verdict at all.

That is not a model problem. It is a measurement problem, and a fault-injection bench fixes it.

A pass rate only describes the runs that finished

Every benchmark has a denominator, and most of them are accidental. Drop interrupted runs and you are scoring the survivors. Run that on flaky, shared, or reclaimable compute and your "best" endpoint is simply the one that got lucky with uptime.

So measure three things separately. Capability: did the patch pass? Efficiency: what did it cost in quota and wall time? Durability: does the result survive an interruption? Your demo shows the first, your billing shows the second, and nothing shows the third until it bites you in production.

Step 1 — Freeze the environment in a manifest

Before the first run, write down what you are actually running on. Anything you cannot name in a manifest is a variable you cannot control later. This is the same discipline as freezing a price envelope, applied to time and failure instead of cost.

# faultbench/manifest.yaml  (template, not a vendor spec)
run_id: 2026-10-27-faultbench-a
taskset_sha: "<git sha of your task files>"
runner_sha: "<git sha of the harness>"
env:
  python: "3.12"
  os: "linux"
  container_digest: "sha256:<digest>"
  cpus: 2
  mem_gb: 4
  disk: "ephemeral"
endpoint:
  provider_notes: "free model access via MonkeyCode"
  model_id: "<exact id the endpoint reports>"
  allocation_today: "<copy today's number from the project page>"
faults:
  plan:
    - "patch_applied:hard_exit"
    - "tests_run:hard_exit"
  repeats: 3

Two rules. First, copy today's free allocation into allocation_today instead of trusting memory — free-tier numbers move, and a note that ages badly is worse than no note. Second, never compare two endpoints whose manifests disagree on container digest, CPU count, or task set; you would be measuring the envelope, not the model.

If you want a place to run this without a cloud bill, MonkeyCode offers free model access and a free server option that are enough to host the harness itself; check the project's own pages for the current allocation and machine shape before you fill in the manifest.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 2 — Make the journal the source of truth, not the output

A resume is only safe if the runner can tell what already happened. Keep an append-only journal and let the artifact directory be disposable. The constraint below is the whole trick: one row per (task_id, checkpoint, attempt), so a double-apply becomes a row-level violation instead of a subtle diff.

# faultbench/journal.py  (template — adapt the schema, keep the constraint)
from __future__ import annotations

import hashlib
import sqlite3
import time
from pathlib import Path

SCHEMA = """
CREATE TABLE IF NOT EXISTS journal (
  task_id         TEXT NOT NULL,
  attempt         INTEGER NOT NULL,
  checkpoint      TEXT NOT NULL,
  payload_digest  TEXT NOT NULL,
  wall_ms         INTEGER NOT NULL,
  ts              REAL NOT NULL,
  PRIMARY KEY (task_id, checkpoint, attempt)
);
"""

CHECKPOINTS = ("context_loaded", "patch_drafted", "patch_applied", "tests_run")


def open_journal(path: Path) -> sqlite3.Connection:
    con = sqlite3.connect(path, isolation_level=None)  # autocommit
    con.execute("PRAGMA journal_mode=WAL")
    con.execute(SCHEMA)
    return con


def record(con: sqlite3.Connection, task_id: str, attempt: int,
           checkpoint: str, payload: bytes) -> None:
    if checkpoint not in CHECKPOINTS:
        raise ValueError(f"undeclared checkpoint: {checkpoint}")
    con.execute(
        "INSERT INTO journal VALUES (?,?,?,?,?,?)",
        (task_id, attempt, checkpoint,
         hashlib.sha256(payload).hexdigest(), 0, time.time()),
    )

On resume, read the last durable checkpoint for (task_id, attempt) and continue from the next one. If the row is missing, treat the state as unknown and fail loudly. Guessing is how you get the double migration.

Step 3 — Inject faults at named checkpoints only

Random kills produce unreproducible numbers. Kill at declared checkpoints, from an environment variable, so every interruption is a rerun of a known state.

# faultbench/cli.sh  (template — not executed for this article)
export FAULTBENCH_DB=runs/a-treat/journal.db

# positive control: no faults at all
uv run faultbench run --manifest manifest.yaml --faults off --out runs/a

# negative control: die before anything is written
uv run faultbench run --manifest manifest.yaml \
  --faults context_loaded:hard_exit --out runs/a-neg

# treatment: die immediately after the patch is applied
uv run faultbench run --manifest manifest.yaml \
  --faults patch_applied:hard_exit --repeat 3 --out runs/a-treat

uv run faultbench score --journal runs/a-treat/journal.db \
  --verdicts runs/a-treat/verdicts.json

Three repeats is not statistics. It is the minimum that tells you whether your own harness is deterministic, which you need before you say anything about a model.

Step 4 — Score survival, duplication, and replay cost

Three metrics, computed from the journal plus verdicts, nothing else:

RSR — resume survival rate. Interrupted tasks that end in a correct pass after resume, divided by all interrupted tasks. Manual edits count as failure.
Double-apply tasks. Tasks with two different payload digests at the same checkpoint. This is an idempotency defect, not a slow run.
Replay cost. Extra tokens and wall time spent re-deriving work that the journal already knew about.

# faultbench/score.py  (template)
import json
import sqlite3


def summarize(journal_db: str, verdicts_json: str) -> dict:
    con = sqlite3.connect(journal_db)
    verdicts = json.load(open(verdicts_json))

    per_task: dict[str, dict] = {}
    for task_id, attempt, checkpoint, digest in con.execute(
        "select task_id, attempt, checkpoint, payload_digest from journal"
    ):
        t = per_task.setdefault(task_id, {"attempts": set(), "writes": {}})
        t["attempts"].add(attempt)
        t["writes"].setdefault(checkpoint, set()).add(digest)

    interrupted = {k for k, v in verdicts.items() if v.get("interrupted")}
    survived = sum(
        1 for k in interrupted if verdicts[k]["status"] == "pass"
    )
    duplicated = sum(
        1 for t in per_task.values()
        if any(len(digests) > 1 for digests in t["writes"].values())
    )
    return {
        "rsr": survived / max(1, len(interrupted)),
        "double_apply_tasks": duplicated,
        "attempts_total": sum(len(t["attempts"]) for t in per_task.values()),
    }

Report the raw journal alongside the numbers. A rate without its ledger is a screenshot with extra steps.

Controls that keep the run honest

Control	Setup	Expected result	What failure means
Positive	Faults off	RSR = 1.0	Your harness, not the model, is broken
Negative	Kill at `context_loaded`	Zero writes recorded	Something wrote before the first checkpoint
Replay	Resume twice from one checkpoint	Second resume is a no-op	Journal is not authoritative
Silent	Kill between checkpoints	Loud `unknown_state` error	The runner is guessing

Why these numbers are not marketing

A marketing number is computed in someone else's envelope and reported without its dependencies. These three are computed in yours, from a manifest you can publish, and they degrade when your own code degrades. That is the point: RSR is partly a property of the harness, and pretending otherwise is how benchmark claims turn into aesthetics.

Where this bench is wrong

Checkpoint granularity dominates the result. A bench with four checkpoints will show better recovery than one with forty, on identical infrastructure. If your tasks are single-shot and uninterruptible, durability is undefinable and this whole harness is noise.

Free compute is also noisy by design. Reclaim events, cold starts, and quota resets are exactly the variance you are trying to measure, so never compare runs across days without re-reading the current allocation. And three repeats per fault point give you a sanity check, not a confidence interval.

Who should not use this

Skip it if you need a public leaderboard, if you cannot rerun a task three times, or if your task set is not deterministic enough that a "pass" means the same thing on Monday and Friday. Skip it too if your real failure mode is a bad review, not a dead process — you would be measuring the wrong interruption.

Run it on something small first

Take five tasks, two fault points, one afternoon. If the positive control is not exactly 1.0, stop and fix the harness before you look at any model. If it is, you now have a durability number that survives someone else trying to reproduce it.

Bring the journal, not the slogan.

Your Benchmark Has No Cost Axis: Score Free Endpoints by Quota per Accepted Patch

Casey Zhang — Mon, 26 Oct 2026 11:19:16 +0000

A team I traded notes with ran the same 40-task patch suite against two free endpoints. Endpoint A passed 71% of tasks, endpoint B passed 64%. They shipped A into their CI helper. Twenty-two days later A's monthly allowance was gone and the helper started failing open — while B, the "worse" endpoint, would have finished the month with quota to spare.

The pass rate was not wrong. It was just answering a different question than the one that mattered. On a free tier, the budget is finite, so the objective is not patches per run — it is patches per unit of quota.

Below is the accounting layer I add to agent benchmarks now: a spend ledger, three derived metrics, and a set of controls that keep the numbers from turning into marketing.

Why a pass rate without a cost axis misleads

A single number collapses two very different failure modes. An endpoint that fails a task on the first attempt and then fails it again on retries has burned budget for nothing. An endpoint that fails once and succeeds on attempt two has burned some budget for something.

Pass rate sees those as identical. Your quota does not.

There is a second distortion: retry policy. If your harness allows three attempts, an endpoint that flails is charged three times, and the flailing is silently hidden inside a "pass". Change the attempt budget and the ranking can flip. That is a harness artifact, not a capability difference.

The three metrics worth reporting

Fix the task set, fix the attempt budget, then report all three of these together. None of them is meaningful alone.

Acceptance rate — accepted tasks ÷ total tasks. The familiar number. Keep it for continuity.
QAP (quota per accepted patch) — total tokens consumed ÷ accepted tasks. This is the cost axis. Lower is better.
Retry tax — tokens spent on attempts that did not produce the accepted patch ÷ total tokens. A high retry tax means your pass rate is being bought with silent spend.

Add a fourth flag, censoring: did the run hit your token cap before finishing the task set? A censored run's acceptance rate is not comparable to an uncensored one, because the hardest tasks never got their full attempt budget.

Step 1: Freeze the task set and the attempt budget

Write both to a file and hash it. If the hash changes, the run is a new benchmark and the old numbers do not carry over.

sha256sum tasks/frozen.jsonl harness_config.json | tee runs/2026-10-26.manifest.sha256

harness_config.json should pin max_attempts, temperature (0 for comparability), cap_tokens, and the prompt template version. Nothing that can drift silently.

Step 2: Instrument the call, not the result

The ledger records one row per attempt, not one row per task. That is the whole trick — retries are where the cost hides.

# spend_ledger.py — template, not a tested library. Adapt the fields to your harness.
from dataclasses import dataclass
import json, time

@dataclass
class Attempt:
    run_id: str
    task_id: str
    arm: str              # endpoint label, e.g. "free-arm-a"
    attempt: int          # 1-based retry index
    prompt_tokens: int
    completion_tokens: int
    wall_ms: int
    accepted: bool        # set by YOUR checker, never by the model
    checker_error: str | None = None

    @property
    def tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

def run_task(call, checker, task, arm, run_id, max_attempts=3):
    rows = []
    for i in range(1, max_attempts + 1):
        t0 = time.monotonic()
        resp = call(task.prompt, temperature=0)
        wall = int((time.monotonic() - t0) * 1000)
        verdict = checker(task, resp.text)      # runs the test suite, returns pass/fail
        rows.append(Attempt(
            run_id=run_id, task_id=task.id, arm=arm, attempt=i,
            prompt_tokens=resp.usage.prompt_tokens,
            completion_tokens=resp.usage.completion_tokens,
            wall_ms=wall, accepted=verdict.ok,
            checker_error=None if verdict.ok else verdict.reason[:200],
        ))
        if verdict.ok:
            break
    return rows

If your provider does not return usage, log the request and response text and estimate tokens from a tokenizer you keep in the repo. Estimating is fine — but state that you estimated, and use the same estimator for every arm.

Step 3: Aggregate with censoring made explicit

# summarize.py — same caveat: template code, run it against your own data first.
def summarize(rows, cap_tokens):
    spend = sum(r.tokens for r in rows)
    accepted_ids = {r.task_id for r in rows if r.accepted}
    accepted = len(accepted_ids)
    first_try = sum(1 for r in rows if r.accepted and r.attempt == 1)
    wasted = sum(r.tokens for r in rows if not r.accepted)

    return {
        "tasks": len({r.task_id for r in rows}),
        "accepted": accepted,
        "acceptance_rate": round(accepted / max(1, len({r.task_id for r in rows})), 3),
        "tokens": spend,
        "qap": round(spend / accepted, 1) if accepted else None,
        "retry_tax": round(wasted / spend, 3) if spend else None,
        "first_try_share": round(first_try / accepted, 3) if accepted else None,
        "censored": spend >= cap_tokens,
    }

Run each arm with the same cap and the same order. Order matters more than people expect: a per-minute rate limit turns task ordering into a latency confounder, and latency is the quiet reason harnesses crash mid-run.

python -m bench.run --arm free-arm-a --tasks tasks/frozen.jsonl \
  --max-attempts 3 --cap-tokens 250000 --out runs/2026-10-26-a.jsonl

Step 4: Add the controls that make the table honest

Four controls, all cheap:

Negative control. One task whose correct patch you already know fails the checker. If an arm "passes" it, your checker is broken and every number above is void.
Floor task. A one-line fix any competent agent should pass. It measures harness overhead and token accounting, not capability.
Paired arms. Run both endpoints on the same task order in the same window. Sequential runs on different days pick up provider drift and file it as endpoint quality.
Retry budget parity. Same max_attempts everywhere. Three for one arm and one for another is not a comparison, it is two benchmarks sharing a folder.

Step 5: Read the numbers as a decision table, not a podium

Arm	Acceptance	QAP (tokens/patch)	Retry tax	Censored	Practical read
A	0.71	6,400	0.44	no	Wins on quality, eats quota on retries
B	0.64	3,100	0.12	no	Loses 7 points, survives the month
C	0.68	—	—	yes	Unusable: cap hit before the suite finished

A is the right pick for a burst workload with a deadline. B is the right pick for a background job that must keep running unattended. C should be dropped from the table entirely — a censored run is not a low score, it is a missing measurement.

Where a free tier genuinely fits this method

This accounting layer exists because finite allowances are the normal case, not an edge case. MonkeyCode is relevant here for two operator-stated reasons: free model access, and a free server option to host the harness itself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes the free model allowance as 10 million tokens; confirm the current figure and its reset behavior in-product before you put it in a table, because free-tier terms move and a stale quota number invalidates every QAP figure derived from it. The server option matters for a narrower reason: running the harness next to the endpoint removes your laptop's thermal throttling and home upload speed from the latency column, so wall_ms measures something closer to the provider.

Neither of those makes the numbers better. They just make the measurement less contaminated — which is the only thing a benchmark host should do.

Limitations, stated up front

Token accounting is provider-shaped. Cached prompt tokens, tool-call overhead, and system-prompt reuse are billed differently across providers. QAP compares arms only when measured by the same accounting method.
Free-tier availability is not a guarantee. Rate limits, model availability, and allowances change without notice. Re-run the manifest before you compare a new month's numbers to an old table.
QAP is not a price. It is quota consumption per accepted patch. If you later map it to money, keep the mapping in a separate column.
Small suites produce wide intervals. With 40 tasks, a 7-point gap is inside the noise of a single re-run. Report the re-run spread beside the point estimate.

Who should not use this

Skip this method if you need a latency SLA — an unattended free server and a free model allowance are the wrong substrate for p99 guarantees. Skip it if your checker is still a human eyeballing diffs; QAP without an automated checker just measures how long a person stared at the screen. And skip it if you cannot reserve quota for the benchmark separately from production, because a benchmark that starves your real workload will be quietly disabled by the first person it inconveniences.

If you already have a checker and a frozen suite, the ledger is about thirty lines. Start by adding attempt and the two token fields to whatever you currently log — that alone usually explains why last month's "best" endpoint stopped answering.

Two Runs, Six Points Apart: Measure the Re-Run Spread Before You Compare Endpoints

Casey Zhang — Tue, 20 Oct 2026 09:22:48 +0000

You run the same 60-task suite twice against the same endpoint. First pass: 71%. Second pass: 77%. Nothing changed — not the model, not the prompts, not the tasks.

Now someone drops a screenshot of a different endpoint at 74% and asks which one wins. You cannot answer that with single runs. Before you compare anything, you need to know how much your number moves when nothing changes. That spread is your noise floor, and any delta smaller than it is a coin flip wearing a percentage sign.

This post is a repeatable method for measuring that spread, computing the smallest difference your suite can actually resolve, and deciding when a comparison is worth reporting. The reference implementation is deliberately boring. Boring is the point.

Why a single run cannot answer a comparison question

A pass rate is a summary of Bernoulli trials. Each task either passes or fails, and each repeat is a fresh draw. With 60 tasks and a true pass rate near 0.75, the run-to-run standard deviation alone is roughly 6 percentage points — so a 71% and a 77% are entirely compatible with an unchanged system.

Add harness drift on top: a timeout that clipped one task, a retry that got counted as a pass, a container that came up with a cold cache on the first run. Those do not show up in the headline number. They show up as movement you cannot explain.

The fix has two halves. First, repeat each task K times so you have a distribution instead of a point. Second, compute the minimum detectable difference (MDD) for your suite, so you know which deltas are even worth discussing.

Step 1: Freeze the task set, the harness, and the scoring rule

Write down the task IDs, the harness commit, the max-turn or max-timeout setting, and the exact command that produces a pass/fail verdict. Hash the task directory and store the hash next to your results.

# Freeze the inputs once, per experiment
find tasks/ -type f -name '*.json' | sort | xargs sha256sum | sha256sum > tasks.sha256
git -C harness rev-parse HEAD > harness.rev

Any rerun that does not reproduce tasks.sha256 is a different experiment. Do not merge its numbers into the same table — file it separately and label it. This one habit kills most "the score went up and we don't know why" tickets.

Step 2: Run K repeats per task, not K repeats per suite

This is the step people skip. If you run the whole suite five times, you get five suite-level numbers and a standard deviation computed on 5 samples — too few to say anything, and blind to which tasks are unstable.

Run each task K times instead. K=5 is a reasonable start for a 40–80 task suite; K=3 tells you almost nothing about per-task instability but is still better than 1. Write results as JSON Lines so an interrupted run can resume instead of restarting from zero.

#!/usr/bin/env bash
# run_spread.sh — illustrative driver; adapt the run command to your harness
set -euo pipefail
K=${K:-5}
OUT="runs/$(date -u +%Y%m%dT%H%M%SZ).jsonl"
mkdir -p runs

for task in tasks/*.json; do
  id=$(basename "$task" .json)
  for k in $(seq 1 "$K"); do
    start=$(date +%s)
    if timeout 900 ./harness/run --task "$task" --json > /tmp/out.json 2>/tmp/err.log; then
      status=pass
    else
      status=fail
    fi
    elapsed=$(( $(date +%s) - start ))
    printf '{"task":"%s","repeat":%d,"status":"%s","seconds":%d}\n' \
      "$id" "$k" "$status" "$elapsed" >> "$OUT"
  done
done
echo "wrote $OUT"

Two details matter more than they look. First, timeout must be a hard ceiling so a hung task cannot silently skew your wall-clock budget. Second, write the log line after the verdict, never before — a crash mid-run should leave a missing record, not a fake pass.

Step 3: Score per task, with an interval

Aggregate to a per-task pass rate p_i = passes / K, then report the suite mean with a Wilson interval. Wilson behaves sanely near 0 and 1, where the normal approximation falls apart — and agent suites always have a few tasks sitting at 0/5.

# spread.py — reference implementation (Python 3.10+), unexecuted here
import json, math, statistics
from collections import defaultdict

def load(path):
    per = defaultdict(list)
    with open(path) as f:
        for line in f:
            r = json.loads(line)
            per[r["task"]].append(r["status"] == "pass")
    return {t: sum(v) / len(v) for t, v in per.items()}

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)

def suite_interval(per_task, k):
    # average of per-task Wilson bounds, a conservative suite-level band
    los, his = zip(*(wilson(round(p * k), k) for p in per_task.values()))
    return statistics.mean(per_task.values()), statistics.mean(los), statistics.mean(his)

Report the band, not the point. "74.2% (95% band 69.8–78.1)" is a claim you can defend. "74.2%" invites a fight you will lose.

Step 4: Compute the minimum detectable difference

Now the number that actually governs your comparisons. Take two endpoints, A and B, both run with K repeats over the same N tasks. For each task, compute d_i = p_A_i - p_B_i. The MDD is roughly 2.8 standard errors of the mean difference — that is the two-sided alpha=0.05, 80% power threshold.

def mdd(per_a, per_b):
    tasks = sorted(set(per_a) & set(per_b))
    d = [per_a[t] - per_b[t] for t in tasks]
    n = len(d)
    se = statistics.stdev(d) / math.sqrt(n) if n > 1 else float("nan")
    return statistics.mean(d), 2.8 * se, n

A worked illustration, using made-up numbers: if mean(d) = +0.03 and MDD = 0.09, you have measured a 3-point edge on a suite that cannot resolve anything under 9 points. The honest conclusion is "no detectable difference at this sample size," not "B is slightly better."

Three levers shrink the MDD: more tasks, more repeats per task, and tasks that are less flaky. Only the first two are fully under your control.

Step 5: Use a paired rule before you call a winner

Paired designs are far more sensitive than comparing two independent suite means, because they cancel task difficulty. Count the tasks where A beats B and the tasks where B beats A; ties drop out. Then test those discordant counts.

def sign_test(wins_a, wins_b):
    n = wins_a + wins_b
    if n == 0:
        return 1.0
    k = min(wins_a, wins_b)
    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

A decision table keeps the rule out of your gut:

Observation	Report
\|mean(d)\| < MDD	"No detectable difference at N tasks, K repeats"
\|mean(d)\| ≥ MDD and sign test p < 0.05	"A ahead by X points, in the same direction on W of W+L discordant tasks"
\|mean(d)\| ≥ MDD but sign test p ≥ 0.05	"Led by a few heavy tasks; not a suite-wide effect"
Per-task rates cluster at 0/5 and 5/5	Suite is easy or saturated; widen difficulty before comparing

That last row is the one people ignore. A suite where most tasks are 0/5 or 5/5 has almost no resolving power, no matter how many repeats you run.

Step 6: Publish the spread, not just the mean

Ship a results table with columns for suite pass rate, Wilson band, MDD, discordant pair counts, and the input hashes from Step 1. Anyone can recompute your conclusion from that table. Nobody can recompute it from a screenshot.

Keep the raw JSONL. It is the only thing that lets you answer "was that run before or after the timeout change?" three weeks later.

Controls that quietly break repeatability

Four knobs account for most unexplained movement:

Retry accounting. Decide once whether a retried task counts as one attempt or two, and apply it to every endpoint.
Timeouts. A 900s ceiling that clips 8% of runs is a scoring parameter, not an implementation detail.
Temperature and sampling. Whatever you set, log it in the result file, because it is not visible in the aggregate.
Cold starts. The first run against any fresh runner includes setup cost the later ones do not.

Where a free runner fits

The whole method is cheap in thinking and expensive in wall-clock: N tasks × K repeats × 2 endpoints. A 60-task suite at K=5 across two endpoints is 600 runs, and most of that time is the harness waiting on a model.

That is the honest case for using free model access and a free server option to run the loop, rather than building runner infrastructure first. MonkeyCode exposes both — free model access and a free server option — as of this writing; quota, capacity, and terms are the operator's to state and change, so verify them yourself before you plan a 600-run experiment around them. I used the free server as the place to run a spread check like the one above, which keeps a noisy experiment off my laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two caveats if you go that route. Shared capacity means wall-clock variance is higher than on a box you control, so log seconds per run and treat outliers as a harness signal. And free-tier terms are a moving target: pin the terms you relied on in your run manifest alongside the task hash.

Limitations and who should skip this

K=5 over 60 tasks is not a certification. It resolves differences in the high single digits at best, and it says nothing about tasks your suite does not contain. If you need a procurement-grade claim, this method is a screening tool, not evidence.

Skip it if your test suite is itself nondeterministic in ways you cannot pin (live third-party APIs in assertions, wall-clock-dependent tests, network flakiness you have not isolated). Repeats will measure your test noise and you will confidently report it as model noise. Fix the suite first.

Also skip it if a 10-point difference is obvious to the naked eye and you only need a direction. Methodology has a cost; spend it on the close calls, where the argument actually lives.

Copy-paste checklist

[ ] Hash the task set and record the harness commit
[ ] Run K repeats per task, JSONL output, hard timeout
[ ] Compute per-task pass rates and a Wilson band
[ ] Compute MDD from paired per-task differences
[ ] Apply the sign test before naming a winner
[ ] Publish spread, MDD, discordant counts, and input hashes

If you run this once, you will stop arguing about three-point gaps. If you run it twice, you will start asking why your suite cannot resolve the differences you care about — which is the more useful question anyway.

Free Model Endpoints Need a Contract Bench, Not a Leaderboard Screenshot

Casey Zhang — Mon, 19 Oct 2026 08:09:12 +0000

You wire a small internal tool to a free model endpoint. The first call returns clean JSON. The second wraps the same object in a Markdown fence. The third times out at 30 seconds. The fourth reports 900 completion tokens for a 40-token answer.

The demo still looks fine. Your CI job does not.

That gap is why a free endpoint needs a contract bench. You are not asking whether the model is the smartest one available. You are asking whether the endpoint returns the shape your code needs, repeatedly, inside the time and token budget you can afford. Those are different questions. Only the second one protects a build.

MonkeyCode is an open-source project. Its current outreach advertises free model access, a free server option, and 10 million free tokens. Treat all three as time-bound commercial terms, not technical guarantees; verify the current offer before you depend on it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this workflow, the free access is useful as a probe target, not as proof of quality. The bench below stays useful if you swap in another endpoint.

1. Write the contract before the prompt

A leaderboard asks which model wins. A contract bench asks which promises survive repetition. Start with the promises your code already makes.

Contract	Probe	Pass evidence	Failure class
JSON object	Ask for one object with two keys	Whole response parses; no fence stripping	`format_drift`
Schema	Fixed key set and value	Keys match exactly	`schema_miss`
Truncation	Small `max_tokens` with a long request	`finish_reason` is present and plausible	`truncation_ambiguity`
Token accounting	Known short prompt	`0 < completion_tokens <= max_tokens`	`accounting_anomaly`
Timeout	Client timeout enforced	Request ends within budget	`timeout`
Error contract	Malformed request	Non-2xx with a structured body	`error_contract`
Streaming, if used	Streamed response	Final event arrives	`stream_hang`

The table is deliberately narrow. It does not test reasoning, factuality, or code quality. Those need a different dataset. This bench tests whether the endpoint is a stable dependency for one client.

2. Build a small, frozen probe set

Start with four cases. Expand to at least 20 before you make a team decision. The shape matters more than the count.

Use four groups:

Strict JSON extraction, with and without a code-fence instruction.
Truncation pressure, where the answer cannot fit in the token budget.
Unicode and whitespace, because transport layers normalize text in surprising ways.
A malformed request control, sent outside the main run.

Store the cases in JSONL with an id, group, prompt, and expect field. Hash the file. Put the hash in your notes. If the fixture changes, the old numbers are a different experiment.

Do not put secrets, customer text, or production prompts in the fixture. A free service may log requests. Treat every probe as public.

3. Run it with controls

Controls are what separate a bench from a screenshot.

Set temperature to 0 when the endpoint accepts it. If it ignores the value, record that fact.
Fix max_tokens, client timeout, and client version for the whole run.
Run three attempts per case. Do not retry inside the measurement; record the retry as a new attempt.
Repeat in two windows at least two hours apart. Free capacity is not stationary.
Shuffle case order with a fixed seed. Do not let one slow request poison every later sample.
Log status, latency, usage, finish reason, and a hash of the raw body. Do not log the API key.
Send one malformed request. A 200 response is a contract failure, not a quirky success.

With 12 attempts, a p95 latency is a range check, not an SLA estimate. That is fine. You are looking for failure classes and gross instability, not a precise capacity number.

4. A harness you can adapt

The template below uses only the Python standard library. It is an unexecuted template: it assumes a chat-completions-shaped payload and a BASE_URL. Replace call_endpoint for a different API. Keep the record shape and scoring logic.

import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.request
from pathlib import Path

BASE_URL = os.environ['BASE_URL'].rstrip('/')
API_KEY = os.environ['API_KEY']
MODEL = os.environ.get('MODEL', 'default')
TIMEOUT_S = 25.0
MAX_TOKENS = 64

CASES = [
    {
        'id': 'json-01',
        'group': 'strict_json',
        'prompt': 'Return only a JSON object with keys ok and count. count must be 2.',
        'kind': 'json',
    },
    {
        'id': 'json-02',
        'group': 'strict_json',
        'prompt': 'Return only a JSON object with keys ok and count. count must be 2. Do not use a code fence.',
        'kind': 'json',
    },
    {
        'id': 'long-01',
        'group': 'truncation',
        'prompt': 'Write a 1000-word explanation of HTTP caching.',
        'kind': 'text',
    },
    {
        'id': 'unicode-01',
        'group': 'unicode',
        'prompt': 'Return the exact text: café / 第二行 / end',
        'kind': 'text',
    },
]

def call_endpoint(prompt: str) -> dict:
    payload = {
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
        'max_tokens': MAX_TOKENS,
    }
    req = urllib.request.Request(
        f'{BASE_URL}/chat/completions',
        data=json.dumps(payload).encode('utf-8'),
        headers={
            'Authorization': f'Bearer {API_KEY}',
            'Content-Type': 'application/json',
        },
        method='POST',
    )
    started = time.perf_counter()
    status = None
    raw = ''
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT_S) as resp:
            status = resp.status
            raw = resp.read().decode('utf-8')
    except urllib.error.HTTPError as exc:
        status = exc.code
        raw = exc.read().decode('utf-8')
    except Exception as exc:
        return {
            'status': None,
            'text': '',
            'usage': {},
            'finish_reason': None,
            'elapsed_ms': round((time.perf_counter() - started) * 1000, 1),
            'raw_sha256': None,
            'transport_error': type(exc).__name__,
        }

    body = {}
    try:
        parsed = json.loads(raw)
        if isinstance(parsed, dict):
            body = parsed
    except json.JSONDecodeError:
        pass

    choice = (body.get('choices') or [{}])[0]
    message = choice.get('message') or {}
    return {
        'status': status,
        'text': message.get('content') or '',
        'usage': body.get('usage') or {},
        'finish_reason': choice.get('finish_reason'),
        'elapsed_ms': round((time.perf_counter() - started) * 1000, 1),
        'raw_sha256': hashlib.sha256(raw.encode('utf-8')).hexdigest(),
        'transport_error': None,
    }

def score(case: dict, rec: dict) -> str:
    if rec['transport_error']:
        return 'transport_error'
    if rec['status'] != 200:
        return 'http_error'
    if rec['elapsed_ms'] > TIMEOUT_S * 1000:
        return 'timeout'
    if case['kind'] == 'json':
        try:
            value = json.loads(rec['text'])
        except json.JSONDecodeError:
            return 'format_drift'
        if not isinstance(value, dict):
            return 'schema_miss'
        if set(value) != {'ok', 'count'}:
            return 'schema_miss'
        if value['count'] != 2:
            return 'content_miss'
    completion = rec['usage'].get('completion_tokens')
    if isinstance(completion, int) and completion > MAX_TOKENS:
        return 'accounting_anomaly'
    return 'pass'

def main() -> None:
    random.Random(7).shuffle(CASES)
    rows = []
    for repetition in range(3):
        for case in CASES:
            rec = call_endpoint(case['prompt'])
            rec.update({
                'case_id': case['id'],
                'group': case['group'],
                'repetition': repetition,
                'result': score(case, rec),
            })
            rows.append(rec)

    Path('results.jsonl').write_text(
        chr(10).join(json.dumps(row, ensure_ascii=False) for row in rows),
        encoding='utf-8',
    )

    counts = {}
    for row in rows:
        key = '{}:{}'.format(row['group'], row['result'])
        counts[key] = counts.get(key, 0) + 1

    latencies = sorted(row['elapsed_ms'] for row in rows)
    p95 = latencies[min(len(latencies) - 1, int(0.95 * (len(latencies) - 1)))]
    print(json.dumps({'counts': counts, 'p95_ms': p95}, indent=2))

if __name__ == '__main__':
    main()

Save it as bench.py and run it with BASE_URL='https://your-endpoint.example' API_KEY='...' python bench.py. Run the malformed-request control separately. Change the payload to invalid JSON and assert that the status is 4xx and the body is parseable. Do not fold that result into the pass count.

5. Read counts, not a composite score

A single score hides the failure you need to fix. Report a small set of separate numbers.

Metric	How to report it	Why it matters
Strict JSON	`passed / attempts` by group	Shows format drift directly
Timeout	`count / attempts`	Exposes client and capacity failures
Token anomaly	`count`	Catches impossible usage accounting
Latency	p50 and p95, plus max	Shows budget risk without hiding outliers
Error contract	`valid 4xx / malformed requests`	Shows whether failures are machine-readable
Raw evidence	fixture hash and body hashes	Lets another person reproduce the run

Use counts when the denominator is small. 9/12 strict-JSON passes, 3 format_drift is useful. 75% quality is not. The second sentence invites a comparison the bench cannot support.

6. Why these numbers are not marketing

Marketing numbers compress time, sample size, and failure detail. They report one score, one date, one model label. That is easy to read and easy to misuse.

A contract bench reports a fixture hash, a client version, a denominator, a failure histogram, and the time windows. Those details are boring. They are also what lets another developer reproduce your result or falsify it.

Do not turn the bench into a ranking. If you test two endpoints, keep the fixture and client fixed, report each failure class separately, and say how many attempts you made. If the free offer changes, your old number is a historical observation, not a current capability claim.

That is the difference between evidence and a screenshot. A screenshot shows a good moment. A bench shows the conditions under which the moment happened.

7. Decision table

Workload	Minimum evidence	Do not use the free endpoint when
Throwaway prototype	12 attempts, malformed control, classified failures	You cannot send the data under the current terms
CI job	30+ attempts across two windows, p95 under job timeout, fallback path	A timeout blocks every merge
Internal tool	Contract pass counts plus error-budget monitoring	You need a hard latency promise
Production dependency	Contract bench plus fallback, monitoring, and data policy	You need an SLA the free offer does not provide

The table is a gate, not a guarantee. A pass means the endpoint is worth another test. It does not mean the model is correct.

8. Limitations and who should skip this

This bench does not measure reasoning quality, factuality, or code correctness. It will not tell you which model writes better patches. Use a separate evaluation for that.

The sample is small. Twelve or thirty attempts can reveal gross instability, but it cannot prove steady-state capacity. The free offer is also time-bound. The 10 million token figure and the free server option are commercial terms, not protocol guarantees; they can change without notice.

Do not send secrets, regulated data, or customer content to a free endpoint unless the current terms clearly allow it. Do not use this workflow as your only production control. Teams that need deterministic latency, strict data residency, or a contractual SLA should not treat a free endpoint as a production dependency.

9. Where the free server fits

Use the free server option as a disposable probe runner. Keep the fixture, the results JSONL, and the summary in storage you control. If the server resets, you should lose a run, not the evidence.

Inject the API key at runtime. Do not put it in the fixture, the repository, or the result log. Treat the runner as cattle, not a database.

The useful output is not a slogan about a model. It is a sentence like this: under fixture hash abc123, with client version 0.4.1, 7 of 12 strict-JSON attempts passed and 5 failed as format drift across two windows. That sentence survives a demo, a handoff, and a skeptical reviewer.

If you want a low-stakes place to start, the project's current free access and free server offer are enough for one run. Verify the current terms, keep your raw counts, and let the bench decide what happens next.

Your Coding-Agent Benchmark Needs a Run Manifest, Not Another Leaderboard Row

Casey Zhang — Sun, 18 Oct 2026 06:54:28 +0000

A colleague pasted a screenshot into our channel last month: "agent X solves 78% of our internal suite." Someone asked which suite. Nobody knew. The screenshot had a bar, a percentage, and a logo, and it took three people an hour to establish that it was not comparable to anything we had ever run.

That is the normal failure mode of agent benchmarking. Not fraud. Not even sloppiness. Just a number published without the envelope that produced it.

You cannot fix that with a better leaderboard. You fix it by making the number's preconditions a machine-readable artifact that a reviewer can diff. Below is the contract I now require before I read anyone's percentage, including my own.

The number is the last line of the report, not the first

A pass rate is a summary of a run. A run is a bundle of decisions: which tasks, which snapshot, which retry policy, which token budget, which scoring script. Change any of them and the percentage means something else.

So write the bundle down first, hash it, and only then compute the number. If two runs disagree on any envelope field, the correct output is not a delta. It is a refusal.

Step 1: Freeze the slice before you run anything

Write your task IDs to a file, sort them, and hash the file. Not the directory. Not "the SWE tasks we usually use." A single file with stable, sorted content.

# tasks.txt — one stable task id per line, no comments
printf '%s\n' task-0011 task-0023 task-0047 > tasks.txt
sort -o tasks.txt tasks.txt
sha256sum tasks.txt | cut -d' ' -f1
# -> 3f9a...  paste this into the manifest

This one habit kills most benchmark disputes. If a task is added, the hash changes, and the hash change is visible in review. Nobody can quietly improve their score by dropping the tasks that fail.

Pick the slice for a stated reason and write that reason down too. A slice chosen because it was convenient is still fine — as long as the manifest says so and the conclusion stays inside that boundary.

Step 2: Declare the envelope in a manifest

Here is the file I use. It is deliberately boring, because boring files survive contact with reviewers.

{
  "run_id": "2026-10-18-acme-v3-r1",
  "started_at_utc": "2026-10-18T09:12:00Z",
  "operator": "casey",
  "task_slice_sha256": "3f9a...",
  "agent_id": "acme-cli",
  "harness_commit": "9c1b7de",
  "model_id": "provider/model-family",
  "model_snapshot": "<the pinned version string the provider returns>",
  "temperature": 0.0,
  "seed": 7,
  "max_output_tokens": 4096,
  "budget_tokens_per_task": 60000,
  "tool_timeout_s": 120,
  "retry_policy": "no retries on test failure; 2 retries on HTTP 5xx",
  "env_image_digest": "sha256:...",
  "scoring_script_sha256": "...",
  "runner": "free-server",
  "provider_region": "unspecified"
}

Two details earn their keep. model_snapshot is whatever version string the provider exposes that day — a family name alone is not an envelope field, because families move under you. retry_policy matters more than temperature in practice; an agent that gets two extra attempts at a failing test suite can look materially better than the same agent with one.

Step 3: Make the checker refuse bad comparisons

This is the whole artifact. Everything else is supporting cast.

#!/usr/bin/env python3
"""check_comparable.py — refuse to diff two runs with different envelopes."""
import json, sys
from pathlib import Path

ENVELOPE = {
    "task_slice_sha256", "agent_id", "harness_commit", "model_id",
    "model_snapshot", "temperature", "seed", "max_output_tokens",
    "budget_tokens_per_task", "tool_timeout_s", "retry_policy",
    "env_image_digest", "scoring_script_sha256",
}
PROVENANCE = {"run_id", "started_at_utc", "operator", "runner", "provider_region"}

def load(p):
    return json.loads(Path(p).read_text())

def main(a, b):
    ma, mb = load(a), load(b)
    blocking = sorted(k for k in ENVELOPE if ma.get(k) != mb.get(k))
    drift = sorted(k for k in PROVENANCE if ma.get(k) != mb.get(k))
    for k in blocking:
        print(f"BLOCK {k}: {ma.get(k)!r} != {mb.get(k)!r}")
    for k in drift:
        print(f"note  {k}: {ma.get(k)!r} != {mb.get(k)!r}")
    if blocking:
        print(f"\nnot comparable: {len(blocking)} envelope field(s) differ")
        return 3
    print("\ncomparable envelope: diff the outcomes")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Exit code 3 is the important line. In CI it fails the job. In review it is the comment that says "these two numbers are different experiments."

Field that differs	Verdict
`task_slice_sha256`	Not comparable — different question
`model_snapshot`	Not comparable — different subject
`temperature`, `seed`, `budget_tokens_per_task`	Not comparable — different conditions
`env_image_digest`, `scoring_script_sha256`	Not comparable — different referee
`runner`, `provider_region`, `started_at_utc`	Comparable, note it in the write-up

The last row is where people over-correct. Recording that run A used a laptop and run B used a free server is useful context, not a disqualification — unless your conclusion depends on latency or hardware.

Step 4: Report discordant pairs, not just averages

Two runs over the same slice produce paired outcomes. Averages throw that structure away.

#!/usr/bin/env python3
"""paired.py — per-task deltas with a seeded bootstrap interval."""
import json, random, sys

def load(p):
    return {r["task_id"]: r for r in json.load(open(p))}

def main(pa, pb, iters=10000, seed=7):
    a, b = load(pa), load(pb)
    shared = sorted(set(a) & set(b))
    if set(a) != set(b):
        print(f"WARN task sets differ; using {len(shared)} shared tasks")
    deltas = [int(b[t]["passed"]) - int(a[t]["passed"]) for t in shared]
    a_pass_b_fail = sum(1 for t in shared if a[t]["passed"] and not b[t]["passed"])
    b_pass_a_fail = sum(1 for t in shared if b[t]["passed"] and not a[t]["passed"])
    n, rng = len(deltas), random.Random(seed)
    boots = sorted(sum(deltas[rng.randrange(n)] for _ in range(n)) / n
                   for _ in range(iters))
    lo, hi = boots[int(0.025 * iters)], boots[int(0.975 * iters)]
    print(f"n={n} delta={sum(deltas)/n:+.3f} 95%CI=[{lo:+.3f},{hi:+.3f}]")
    print(f"A passed / B failed: {a_pass_b_fail}")
    print(f"B passed / A failed: {b_pass_a_fail}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

With 20 tasks and a +0.150 delta, expect something like 95%CI=[-0.050,+0.350] and a discordant split around 2 / 5. That interval straddles zero. The honest sentence is "directionally positive, not established" — and the discordant counts tell you which tasks to read first.

The numbers above are an expected output shape for the script, not measurements from a real run. Run it on your own outcomes before quoting anything.

Step 5: Ship the manifest next to the number

A pass rate without a manifest is a rumor with a percent sign. Publish run.manifest.json, outcomes.json, and the exact command line. A reviewer should be able to check comparability in under a minute without asking you a question.

One practical shortcut: you need a second environment to test whether your envelope survives a change of runner. That is a cheap check with a free-tier setup — run the same slice on your laptop and on rented-or-free infrastructure, then feed both manifests to the checker. MonkeyCode's free model access and free server option are one way to get that second environment without a budget conversation; the operator tells me the free tier is described as a 10M-token allowance plus a free server, which you should re-verify on the current pricing page before you size a run around it, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this does not prove

A matching envelope proves comparability, not quality. Your 20 tasks are still 20 tasks, and the bootstrap interval is indicative rather than inferential at that size.

Slice selection bias survives every hash. If you picked tasks your agent happens to be good at, the manifest will faithfully document a flattering and useless number. Negative controls and pre-registration address that; the manifest only stops you from lying by accident.

Free tiers and hosted runners also add variance you cannot fully pin: quota throttling, shared capacity, and provider-side version drift between two runs on different days. Record the timestamp and rerun the manifest check before you combine them.

Who should skip this

Skip the manifest if you are doing a one-off smoke test to decide whether an agent is worth an afternoon. Skip it for a demo. Skip it if your conclusion is "this looks promising, let's try it on real work."

Adopt it the moment a number leaves your terminal. The first time someone quotes your percentage in a planning meeting, the manifest is the difference between a measurement and a slogan.

Free Tiers Are a Moving Target: Benchmark Them With a Frozen Envelope, Not a Screenshot

Casey Zhang — Sat, 17 Oct 2026 05:53:48 +0000

You ship a fix on Monday. By Thursday a teammate posts a number in the channel: "free tier, 87% pass rate." Nobody can reproduce it. The tasks are gone, the endpoint changed, and the run that produced it quietly dropped 40% of its requests on rate limits. The percentage survives in slides long after the conditions that produced it have evaporated.

Free model access makes this worse, not better. Free capacity is the most volatile thing you will ever benchmark: it moves with load, with quotas, with whatever the provider changed this morning. So the number is only worth something if the conditions travel with it.

This is a methodology piece. Below is a small harness that refuses to print an aggregate until its run envelope is complete, two controls that catch a lying grader, and a decision table for deciding whether a number is usable at all.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding-agent project that offers free model access and a free server option; those two availability claims are what make the workflow below runnable without provisioning anything. I deliberately do not restate quotas, model names, or hardware here — check the project's current pages for today's terms, because that is exactly the kind of data this article tells you not to trust second-hand.

1. Decide what the number is allowed to claim

Before the loop, write the sentence the result will support. If you cannot write it, you are not ready to run.

Bad: "Model A is better than model B."

Good: "On 40 frozen tasks, with a 4k output budget per task and 3 scheduled attempts each, at 2026-10-17T09:00Z, model A produced 27 verifiable passes and 5 invalid runs; model B produced 24 and 2. This supports a claim about this envelope only."

The second sentence is boring. It is also the only kind that survives a re-run a week later.

Two rules make it work. First, tie the claim to a timestamp and an endpoint, not to a brand. Second, treat invalid runs as a first-class result, because on free capacity they are often the dominant signal.

2. Write the envelope before you write the loop

The envelope is a JSON manifest. Seven fields do most of the work, and one of them is the hash of your task file so the dataset cannot drift out from under the number.

{
  "endpoint": "https://<host-from-provider-docs>/v1",
  "model_alias": "<exact alias as listed today>",
  "token_budget_per_task": 4096,
  "max_attempts": 3,
  "temperature": 0.0,
  "server_class": "free-tier-server, 1 vCPU, 2 GB RAM (as provisioned)",
  "concurrency": 1,
  "run_date_utc": "2026-10-17T09:00:00Z",
  "retry_policy": "retry on 429/5xx/timeout with 2s, 8s backoff; never retry on 4xx",
  "task_set_sha256": "<filled in by the harness>",
  "invalid_rate_ceiling": 0.10
}

Notice what is absent: no "best", no "fastest", no score. An envelope is a description of a measurement, not a verdict.

The invalid_rate_ceiling is the line you draw in advance. When invalid runs exceed it, the aggregate is not a weak result — it is no result.

3. Build the harness that refuses to aggregate

The core discipline is a gate: if the envelope is incomplete, or too many calls failed, the script exits non-zero and prints why. Never let a denominator shrink silently.

# harness.py — example for a chat-completions style endpoint.
# Run: python harness.py --manifest envelope.json --tasks tasks.jsonl
# Not executed in this article; adapt error handling to your provider's docs.
import argparse, hashlib, json, os, sys, time, urllib.request, urllib.error

REQUIRED = ["endpoint", "model_alias", "token_budget_per_task", "max_attempts",
            "temperature", "server_class", "run_date_utc", "retry_policy",
            "invalid_rate_ceiling"]

def sha256_file(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def call(m, prompt, key):
    body = json.dumps({"model": m["model_alias"],
                       "messages": [{"role": "user", "content": prompt}],
                       "temperature": m["temperature"],
                       "max_tokens": m["token_budget_per_task"]}).encode()
    req = urllib.request.Request(m["endpoint"].rstrip("/") + "/chat/completions",
        data=body, headers={"Content-Type": "application/json",
                            "Authorization": f"Bearer {key}"})
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=180) as r:
            payload = json.loads(r.read())
        return {"status": 200, "latency_s": time.time() - t0, "payload": payload}
    except urllib.error.HTTPError as e:
        return {"status": e.code, "latency_s": time.time() - t0, "payload": None}
    except Exception as e:
        return {"status": 0, "latency_s": time.time() - t0, "payload": None,
                "error": repr(e)}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--manifest", required=True)
    ap.add_argument("--tasks", required=True)
    ap.add_argument("--out", default="runs.jsonl")
    a = ap.parse_args()

    m = json.load(open(a.manifest))
    missing = [k for k in REQUIRED if m.get(k) in (None, "")]
    if missing:
        sys.exit(f"REFUSING TO RUN — envelope incomplete: {missing}")

    m["task_set_sha256"] = sha256_file(a.tasks)
    json.dump(m, open("envelope.frozen.json", "w"), indent=2, sort_keys=True)

    key = os.environ["PROVIDER_API_KEY"]
    tasks = [json.loads(l) for l in open(a.tasks) if l.strip()]
    scheduled = len(tasks) * m["max_attempts"]  # denominator, fixed up front

    rows, invalid = [], 0
    with open(a.out, "w") as out:
        for t in tasks:
            for attempt in range(1, m["max_attempts"] + 1):
                r = call(m, t["prompt"], key)
                ok = r["status"] == 200 and r["payload"] is not None
                if not ok:
                    invalid += 1
                out.write(json.dumps({"task_id": t["id"], "attempt": attempt,
                                      "status": r["status"], "latency_s": r["latency_s"],
                                      "usage": (r["payload"] or {}).get("usage")}) + "\n")
                rows.append({"task_id": t["id"], "ok": ok})
                if ok:
                    break           # keep attempts cheap, but keep the denominator fixed
                time.sleep(2 if attempt == 1 else 8)

    rate = invalid / scheduled
    print(f"scheduled={scheduled} invalid={invalid} invalid_rate={rate:.3f}")
    if rate > m["invalid_rate_ceiling"]:
        sys.exit("NO AGGREGATE — invalid rate above ceiling; the run describes the tier, not the model")

if __name__ == "__main__":
    main()

The scheduled = len(tasks) * max_attempts line is the whole point. Your pass rate is passes divided by tasks scheduled, and you report it next to the invalid count. If you divide by successful calls instead, a flaky free tier will look better the more it fails.

4. Add two controls before you trust the grader

A grader that always says "pass" produces a beautiful benchmark. Two cheap controls catch it.

Negative control. Add one task whose correct outcome is failure — for example, "return the SHA-256 of an empty string as 64 zero characters." Nothing should pass it. If your report shows a pass on the negative control, stop: your acceptance check is broken, and every other row is noise.

Replay control. Re-run ten task IDs from the same dataset 24 hours later, same envelope, and compare per-task verdicts. Record the flip rate. On free capacity, a flip rate above roughly 20% means your point estimate should be published as a range, or not at all.

Neither control is statistical sophistication. They are plumbing checks, and most published free-tier numbers skip both.

5. Report five numbers, not one

With runs written to JSONL, aggregate offline so you can re-cut without re-spending quota:

Pass rate over the frozen denominator, with the scheduled count printed beside it.
Invalid rate, split by class: 429 / 5xx / timeout / malformed body.
Tokens per solved task — median over passed tasks only, with the count, because it answers "what does a solution cost me" better than an average over everything.
p95 latency on valid calls, since the tail is what breaks interactive loops.
Replay flip rate, from step 4, as a stability annotation on the pass rate.

Publish the task set hash and the frozen envelope.frozen.json alongside these. That is what turns a screenshot into a measurement.

6. A decision table for the result

Observation	What the number means	Action
Negative control passes	Grader is broken	Discard run; fix acceptance checks
Invalid rate > ceiling	You measured the tier's reliability, not the model	Report invalid rate only; do not publish a pass rate
Replay flips > 20%	Envelope is not stable	Publish as interval; increase attempts before comparing
Valid runs, stable replay, n ≤ 40	Directional only	Do not claim differences under ~10 points
Valid runs, stable replay, n ≥ 200, two dates	Comparable within this envelope	Safe to cite with hash and date attached

Read the table before you read the percentage. Most disputes about "who is better" are really disputes about which row of this table the number sits in.

Limitations, and who should skip this

This harness measures behaviour inside one envelope. It says nothing about a model's intrinsic capability, and it will not survive a change to the endpoint, the prompt template, or the token budget. Forty tasks cannot resolve a two-point difference; if that is your question, you need a different instrument, not a bigger table.

Skip this approach if you need SLA-backed numbers for procurement, if you cannot re-run in a week, or if nobody will maintain the task set. A stale harness produces confident garbage.

Also skip it if you only want a demo. Demos are fine — just label them as demos and keep them out of the slide that says "measured".

If you want to try the pipeline without provisioning machines first, MonkeyCode's free model access and free server option are enough to produce a first frozen envelope end to end; read their current terms for what is included today, then write your manifest before you write your loop.

Balance the Outcome Ledger: A Publication Gate for Coding-Agent Benchmarks

Casey Zhang — Fri, 16 Oct 2026 04:51:54 +0000

Last week a teammate pasted a number into a planning doc: 78% pass rate on a 40-task agent suite. Nobody argued. The number was clean, the slide was clean, and the meeting moved on.

Then you opened the run logs. Three tasks never produced an outcome at all. The harness timed out twice, and one grader crashed on a parse error. Those three were dropped from the export, so the denominator quietly became 37. The suite was still called "40 tasks" in the title.

That is not a lie. It is worse than a lie. It is bookkeeping so sloppy that nobody can tell whether the number is true.

The front page this week was full of AI takes, including a good thread on plain AI fatigue. My honest read is that fatigue is not cured by fewer numbers. It is cured by numbers that are cheap to check. So here is a ledger rule and a small gate you can run before any agent score leaves your machine.

The rule: every task that enters must exit exactly once

A benchmark claim is a financial statement. It has entries, exits, and a reconciliation. If the ledger does not balance, the headline is not a measurement — it is a story with digits attached.

Every task attempt must land in exactly one bucket:

pass — the grader's success condition was satisfied.
fail — the task was attempted and the condition was not satisfied.
unscoreable — the harness itself could not produce a verdict, with a named reason class.

The conservation law is one line: pass + fail + unscoreable == tasks_started. And tasks_started must equal the declared dataset size, or you must list the tasks that never started by ID.

Step 1 — Write the manifest (JSON, no dependencies)

Keep the manifest in JSON rather than YAML so the validator runs on a stock Python install with zero packages.

{
  "claim_id": "agent-x-mini-2026-10-16",
  "dataset": {
    "name": "local-40",
    "task_count": 40,
    "task_ids_sha256": "replace-with-digest-from-gate",
    "strata": { "repo_edit": 24, "test_write": 10, "refactor": 6 }
  },
  "metrics": [
    { "name": "pass_rate", "numerator": "outcome == 'pass'", "denominator": "dataset.task_count", "unit": "tasks" },
    { "name": "unscoreable_rate", "numerator": "outcome == 'unscoreable'", "denominator": "dataset.task_count", "unit": "tasks" }
  ],
  "controls": [
    { "task_id": "neg-01", "expected": "fail", "role": "negative" },
    { "task_id": "neg-02", "expected": "fail", "role": "negative" },
    { "task_id": "pos-01", "expected": "pass", "role": "positive" }
  ],
  "envelope": {
    "runner": "monkeycode-free-server",
    "network": "off",
    "wall_clock_s_per_task": 900,
    "max_steps": 40,
    "toolchain_sha": "replace-with-toolchain-digest"
  },
  "rerun_recipe": "python gate.py --manifest bench.manifest.json --runs runs.jsonl"
}

Note two deliberate choices. Every metric is written as an explicit fraction, not as a word like score. And the negative controls carry an expected outcome, so the gate can fail when the harness is broken rather than when the model is.

Step 2 — Emit one JSONL record per attempt

One line per attempt. No aggregation at write time, no filtering at export time. Filtering is where denominators die.

{"run_id":"r1","task_id":"neg-01","attempt":1,"outcome":"fail","reason":null,"wall_clock_s":41}
{"run_id":"r1","task_id":"t-013","attempt":1,"outcome":"unscoreable","reason":"harness_timeout","wall_clock_s":900}
{"run_id":"r1","task_id":"t-027","attempt":1,"outcome":"unscoreable","reason":"grader_error","wall_clock_s":12}
{"run_id":"r1","task_id":"pos-01","attempt":1,"outcome":"pass","reason":null,"wall_clock_s":88}

Keep the allowed reason classes short and fixed: harness_timeout, grader_error, env_flake, missing_artifact. A free-text reason field will be filled with "weird" and "idk" within a week, and then you cannot count anything.

Step 3 — The gate itself

This is the whole validator. Standard library only.

#!/usr/bin/env python3
"""Publication gate for coding-agent benchmark claims. Stdlib only."""
import hashlib, json, math, pathlib, sys

BUCKETS = ("pass", "fail", "unscoreable")
REASONS = {"harness_timeout", "grader_error", "env_flake", "missing_artifact"}


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = (z / d) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, center - half), min(1.0, center + half))


def read_jsonl(path):
    text = pathlib.Path(path).read_text()
    return [json.loads(line) for line in text.splitlines() if line.strip()]


def main(manifest_path, runs_path):
    manifest = json.loads(pathlib.Path(manifest_path).read_text())
    runs = read_jsonl(runs_path)
    errors, declared = [], manifest["dataset"]["task_count"]

    ids = sorted({r["task_id"] for r in runs})
    digest = hashlib.sha256("\n".join(ids).encode()).hexdigest()
    if digest != manifest["dataset"]["task_ids_sha256"]:
        errors.append("dataset_digest_mismatch: you ran a different suite than you published")
    if len(ids) != declared:
        errors.append(f"dataset_size_mismatch: declared {declared}, observed {len(ids)}")

    by_task = {}
    for r in runs:
        by_task.setdefault(r["task_id"], []).append(r)

    for tid, attempts in sorted(by_task.items()):
        if len(attempts) != 1:
            errors.append(f"multiple_attempts:{tid}={len(attempts)} (retries are attempts, not replacements)")
        last = attempts[-1]
        if last.get("outcome") not in BUCKETS:
            errors.append(f"unknown_outcome:{tid}={last.get('outcome')}")
        if last.get("outcome") == "unscoreable" and last.get("reason") not in REASONS:
            errors.append(f"uncoded_unscoreable:{tid}")

    counts = {b: sum(1 for r in runs if r.get("outcome") == b) for b in BUCKETS}
    if sum(counts.values()) != declared:
        errors.append(f"ledger_does_not_balance: {sum(counts.values())} exits vs {declared} entries")

    for c in manifest.get("controls", []):
        row = by_task.get(c["task_id"], [None])[-1]
        if row is None:
            errors.append(f"control_missing:{c['task_id']}")
        elif row.get("outcome") == "unscoreable":
            errors.append(f"control_inconclusive:{c['task_id']} ({row.get('reason')})")
        elif row.get("outcome") != c["expected"]:
            errors.append(f"control_violated:{c['task_id']} expected {c['expected']} got {row['outcome']}")

    k, lo, hi = counts["pass"], *wilson(counts["pass"], declared)
    scoreable = declared - counts["unscoreable"]
    k2 = counts["pass"]
    lo2, hi2 = wilson(k2, scoreable) if scoreable else (0.0, 0.0)

    print(f"claim       : {manifest['claim_id']}")
    print(f"dataset     : {manifest['dataset']['name']} sha256 {digest[:12]}")
    print(f"entries     : {declared}")
    print(f"exits       : {sum(counts.values())}")
    for b in BUCKETS:
        extra = ""
        if b == "unscoreable":
            detail = sorted((r["reason"] for r in runs if r.get("outcome") == b))
            extra = f" ({', '.join(detail)})"
        print(f"  {b:<12}: {counts[b]}{extra}")
    print(f"pass_rate   : {k}/{declared} = {k/declared:.3f}  95% CI [{lo:.3f}, {hi:.3f}]  <- headline")
    print(f"            : {k2}/{scoreable} = {k2/scoreable:.3f}  95% CI [{lo2:.3f}, {hi2:.3f}]  <- labeled secondary")
    print(f"verdict     : {'BLOCKED' if errors else 'PUBLISHABLE'}")
    for e in errors:
        print(f"  ! {e}")
    return 1 if errors else 0


if __name__ == "__main__":
    a = sys.argv
    sys.exit(main(a[a.index("--manifest") + 1], a[a.index("--runs") + 1]))

Step 4 — Run it and read the report card

python gate.py --manifest bench.manifest.json --runs runs.jsonl

claim       : agent-x-mini-2026-10-16
dataset     : local-40 sha256 3f9c1a7b0e44
entries     : 40
exits       : 40
  pass        : 29
  fail        : 8
  unscoreable : 3 (grader_error, harness_timeout, harness_timeout)
pass_rate   : 29/40 = 0.725  95% CI [0.570, 0.842]  <- headline
            : 29/37 = 0.784  95% CI [0.624, 0.889]  <- labeled secondary
verdict     : PUBLISHABLE

Both lines are real. The headline keeps the full denominator, because an unscoreable task is an outcome of your harness, not an absence from the suite. The secondary line is the number people actually want, and it is fine to show it as long as the label travels with it.

Notice how wide the interval is. On 40 tasks, 29/40 and 30/40 are not different results. If your release notes say "improved from 72% to 75%," the gate just told you that you are reading noise.

Why this makes the number a measurement and not marketing

Five properties separate the two, and the gate enforces all five.

A named numerator and denominator. Not "score," not "performance index." Two counts and a division.
A stated interval. A point estimate without uncertainty is a claim about your confidence, not about the system.
A first-class unscoreable bucket. Attrition is printed next to the result, with reason classes, never dropped at export.
Controls with expected outcomes in the manifest. If a control violates its expectation, the gate blocks the publication, because the instrument is broken.
A re-run recipe a stranger can execute. If someone else cannot reproduce the ledger from your manifest and logs, you published a story.

That last point is the one that costs you an afternoon and saves you a quarter.

Where free compute fits in this loop

The expensive part of a gate like this is not the arithmetic. It is the control slice and the re-run smoke test — the runs that exist purely to prove the instrument works, which feel wasteful when you are paying per call.

This is where a free tier changes the economics of doing it honestly. MonkeyCode provides free model access and a free server option, so the control slice and the smoke test can run without a spend decision blocking them. Check the current terms for the applicable allowance and availability before you depend on it, since free-tier limits and hardware change over time. Incidentally, this article was prepared as part of MonkeyCode's product outreach.

One caution that matters more than the free part. A different runner is a different envelope. Record the runner, network policy, wall clock, and toolchain digest in the manifest, and never compare a pass rate from one envelope to a pass rate from another. A ledger balances inside a single envelope. Across envelopes, you are comparing two different instruments.

If you want to try this on your own suite, run the control slice on the free server and get a PUBLISHABLE verdict from gate.py before you quote anything.

Limitations, and who should skip this

The gate validates bookkeeping, not task quality. Forty weak tasks with a balanced ledger is still a weak benchmark, and a balanced ledger will not fix a suite that leaked into training data. Wilson intervals also assume independent, identically distributed tasks, which agent benchmarks only approximate — treat the interval as a floor on your uncertainty, not a precise bound. And the reason classes above are a starting taxonomy; yours should reflect the failures your harness actually produces.

Skip this if you are doing a vibe check on a personal project — the overhead is real for ten tasks. Skip it too if you need a leaderboard number: a publication gate is necessary and nowhere near sufficient, because it says nothing about a hidden split, seed sensitivity, or task contamination. Fix those separately.

Use it if you are about to put a percentage in a doc that other people will make decisions from. Then do the boring thing first: count the entries, count the exits, and make them match.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the Denominator Frozen: A Checker for Agent Pass Rates

Casey Zhang — Sun, 11 Oct 2026 20:34:19 +0000

You closed the laptop on a green terminal. Four tasks, two passes, a free server still spinning. In the team channel the number had already become "50% solved."

Nobody attached the task list. Nobody said what "solved" ignored.

This is a procedure for freezing that meaning before the run, then refusing to print a rate when the logs drift. The artifact is a small checker you can run locally. It does not rank vendors.

The failure you are actually looking at

A percentage looks precise. It is only as stable as the denominator.

Change one task ID after you peek, and the rate moves. Drop a timeout, and it moves again. Call an empty patch a pass because the agent wrote a confident summary, and you have left measurement entirely.

You want the opposite habit. Decide the slice, the metrics, the control, and the access path. Then generate the number, or generate an error.

Step 1: Commit the contract while `run_id` is still null

Put scoring_contract.json in the eval repo. JSON keeps the checker free of extra dependencies. YAML is fine later if you already pin a parser.

{
  "contract_version": 1,
  "run_id": null,
  "locked_at": null,
  "task_ids": [
    "t_03_rename_helper",
    "t_11_fix_off_by_one",
    "t_18_add_missing_test",
    "t_22_reject_network_call"
  ],
  "metrics": {
    "primary": "task_pass",
    "task_pass": "tests_green AND patch_applies_clean AND no_forbidden_path",
    "abstain": "exit_code_0 AND patch_empty",
    "timeout": "wall_seconds > 180",
    "post_lock_exclusion": "forbidden"
  },
  "controls": {
    "narration_probe": "summary only; a non-empty patch fails the probe"
  },
  "access_class": {
    "host": "unspecified",
    "model_access": "unspecified",
    "quota_assumed": false,
    "permanence_assumed": false
  }
}

This file is a proposal you fill in, not a published score. Four tasks are a smoke slice. They can expose a sloppy denominator. They cannot support a model comparison.

Field notes, in the order you should fill them:

task_ids is the denominator. A folder glob is not allowed at score time.
task_pass is a conjunction. Any false clause fails the task, including a dirty apply or a forbidden path.
narration_probe is a control condition. It is not a second leaderboard and it does not enter the numerator.
access_class records where the run happened. Treat it as a condition, not as a trophy.

Step 2: Record the host you actually used

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You can point access_class.host at MonkeyCode's free server option and model_access at its free model access. Those two availability claims are operator-supplied for this draft. They are the reason a locked four-task slice is cheap to execute.

Do not write a token quota, a hardware name, or a duration into the contract unless a current primary page states them and you store the URL plus the date you read it. This article does not. Free paths can be queued or withdrawn. Set quota_assumed and permanence_assumed to false and leave them there.

If a teammate later reruns the same IDs on another host, mint a new run_id. Averaging across access classes hides the variable you forgot to name.

The product is a place to run the protocol. It is not the conclusion of the protocol. Delete the name and the checker still tells you when a rate is unfit to quote.

Step 3: Lock, then refuse an unlocked rate

Save the checker as refuse_unlocked_rate.py. The sample below is unexecuted teaching code. Run it on your own fixture before you trust the exit codes.

#!/usr/bin/env python3
"""Emit a pass rate only for a locked scoring contract."""

import json
import sys
from datetime import datetime, timezone
from pathlib import Path

REQUIRED = ("task_ids", "metrics", "controls", "access_class")

def load(path):
    return json.loads(Path(path).read_text(encoding="utf-8"))

def dump(path, obj):
    Path(path).write_text(json.dumps(obj, indent=2) + "\n", encoding="utf-8")

def lock(path, run_id):
    contract = load(path)
    if contract.get("locked_at"):
        sys.exit("already locked; start a new contract file")
    missing = [k for k in REQUIRED if not contract.get(k)]
    if missing:
        sys.exit(f"refusing to lock incomplete contract: {missing}")
    access = contract["access_class"]
    if access.get("quota_assumed") or access.get("permanence_assumed"):
        sys.exit("refusing to lock assumed quota or permanence")
    if access.get("host") in (None, "", "unspecified"):
        sys.exit("set access_class.host before lock")
    contract["run_id"] = run_id
    contract["locked_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    dump(path, contract)
    print(f"locked {run_id} at {contract['locked_at']}")

def score(contract_path, events_path):
    contract = load(contract_path)
    events = load(events_path)
    if not contract.get("locked_at") or not contract.get("run_id"):
        sys.exit("unlocked contract; no rate emitted")
    access = contract["access_class"]
    if access.get("quota_assumed") or access.get("permanence_assumed"):
        sys.exit("assumed quota or permanence; no rate emitted")
    allowed = list(contract["task_ids"])
    if len(allowed) != len(set(allowed)):
        sys.exit("duplicate task_ids")
    by_id = {}
    for ev in events:
        tid = ev["task_id"]
        if tid not in allowed:
            sys.exit(f"event outside locked slice: {tid}")
        if ev.get("excluded_after_lock"):
            sys.exit(f"post-lock exclusion starts a new study: {tid}")
        if "task_pass" not in ev:
            sys.exit(f"missing task_pass: {tid}")
        by_id[tid] = ev
    missing = [tid for tid in allowed if tid not in by_id]
    if missing:
        sys.exit(f"denominator drift, missing {missing}")
    passed = 0
    for tid in allowed:
        ev = by_id[tid]
        if ev.get("abstain") and ev.get("task_pass"):
            sys.exit(f"abstain cannot pass: {tid}")
        if ev.get("task_pass") is True:
            passed += 1
    print(json.dumps({
        "run_id": contract["run_id"],
        "locked_at": contract["locked_at"],
        "n": len(allowed),
        "task_pass_count": passed,
        "task_pass_rate": passed / len(allowed),
        "access_class": access,
        "note": "valid only for this slice, metric, and access class"
    }, indent=2))

if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "lock":
        lock(sys.argv[2], sys.argv[3])
    elif cmd == "score":
        score(sys.argv[2], sys.argv[3])
    else:
        sys.exit("usage: lock|score")

Fill the host with a label you can defend, then lock. Scoring before that step should fail.

python3 - <<'PY'
import json
p = "scoring_contract.json"
with open(p, encoding="utf-8") as f:
    c = json.load(f)
c["access_class"]["host"] = "operator-supplied free server option"
c["access_class"]["model_access"] = "operator-supplied free model access"
with open(p, "w", encoding="utf-8") as f:
    json.dump(c, f, indent=2)
    f.write("\n")
PY
python3 refuse_unlocked_rate.py lock scoring_contract.json run_2026_10_12_a
python3 refuse_unlocked_rate.py score scoring_contract.json events.json

The script does not fetch a quota. It only checks that you did not pretend to have one.

events.json must contain every locked ID, including failures:

[
  {"task_id": "t_03_rename_helper", "task_pass": true, "abstain": false, "wall_seconds": 40},
  {"task_id": "t_11_fix_off_by_one", "task_pass": false, "abstain": false, "wall_seconds": 55},
  {"task_id": "t_18_add_missing_test", "task_pass": false, "abstain": false, "wall_seconds": 200},
  {"task_id": "t_22_reject_network_call", "task_pass": true, "abstain": false, "wall_seconds": 30}
]

On this fixture the checker would print task_pass_count 2 and n 4. That printout is an example of the schema. It is not a measurement of any agent. Replace the booleans with values your harness computed from tests.

Step 4: Make exclusions and probes impossible to clean up

Run the narration probe on the same IDs before you trust task_pass. The prompt asks for a summary and forbids a patch. A non-empty diff fails the probe.

Keep that result beside the rate. Do not fold it into the numerator.

Timeouts remain in n. An abstain remains in n and cannot set task_pass. A test you discover is flaky after locked_at does not get an excluded_after_lock flag on this run. Copy the contract, explain the flake, and score a new run_id.

If your harness has no hidden tests, write that inside metrics. Silence is how a public sample becomes a private-sounding claim.

Why the printed rate is still not marketing

A marketing number is allowed to move after the audience reacts. A measurement is not.

You publish the contract hash with the rate. You publish the misses. You publish the access class as a constraint. You do not publish a comparative adjective.

"Better" needs a paired design you did not run: same IDs, same budget policy, same tests, a second access class, and a rule for ties written in advance. Until that second contract exists, the honest sentence is local. On this locked slice, under this access class, this many tasks met task_pass.

That sentence will not travel as a billboard. Good.

Evidence sentence	Slogan it must not become
"2 of 4 `task_pass` on this locked `run_id`."	"50% accuracy."
"Narration probe emitted no patch on this slice."	"The harness is fair."
"Access class recorded; terms read that day."	"Free hosting means the score will hold."
"Flake deferred to a new `run_id`."	"We removed the bad task and improved."

The left column is dull. Dull is the feature. A marketing line needs the right column, which this checker is built to withhold.

Step 5: Store a bundle a stranger can rerun

A rate without a bundle is a rumor. After the checker exits 0, write a directory that contains only boring files.

runs/run_2026_10_12_a/
  scoring_contract.json
  events.json
  narration_probe.json
  commands.txt
  README.md

commands.txt lists the test invocation, not a paraphrase.

sha256sum scoring_contract.json events.json > runs/run_2026_10_12_a/SHA256SUMS
python3 refuse_unlocked_rate.py score scoring_contract.json events.json | tee runs/run_2026_10_12_a/rate.json

The README gets five lines: slice purpose, access class, date you read the host terms, probe result, and the sentence "not a vendor ranking." If you cannot write that last line without hedging, you are not done.

Who should not use the approach

Do not use a four-task lock to choose a vendor, set a price, or announce a winner. The sample is too small, and this draft supplies no confidence interval you can hide behind.

Do not use it if you cannot keep the patch, the test command, and the contract file. A chat scroll cannot be re-scored.

Do not use it when task text is copied from a popular public repo and you have no note about possible overlap with model training data. Say "contamination unknown" next to the rate, or do not publish the rate.

Skip the free-server path if your question depends on a reserved machine, a fixed region, or a paid throughput guarantee. Those properties are outside the operator-supplied claims used here.

Limits of the checker

The script trusts task_pass. You still have to fill that boolean from a test command, not from model prose. A lying harness makes a locked lie.

Clock skew, queue delay, and partial outages on a free host belong in the event log as timeouts or as a failed run, not as deleted rows. This article states no latency budget and no uptime figure.

The sample events are unexecuted illustrations. Do not cite them. Do not add a model name to the JSON to make a screenshot look finished.

Ship the smaller sentence

Lock four tasks you can explain without a diagram. Run the probe. Run one agent pass. Paste the checker JSON, or paste the error.

If that log is something your team will rerun, the operator-supplied free model access and free server option are a reasonable place to do the next slice. Read the live terms on the day you lock the contract, store the date in your notes, and keep the claim as narrow as the run_id.

Add a Negative-Control Slice Before You Believe a Coding-Agent Score

Casey Zhang — Sat, 10 Oct 2026 17:02:40 +0000

Your teammate drops a screenshot into the channel just after lunch. Twelve tasks. Twelve green checks. The caption asks you to paste the percentage into the weekly update.

You open the suite before you touch the slide. Three prompts name the function the hidden test calls. One fixture file still contains a comment that states the expected return value. Another task is a near-copy of a public tutorial the model has likely seen in training.

A perfect score, in that shape, is not a result. It is a harness smell. This walkthrough shows you how to add a negative-control slice so a headline number cannot hide leakage, a broken assertion, or a task the agent should have refused.

What a single percentage hides

A coding-agent score is a ratio. Numerator: trials you marked pass. Denominator: trials you chose to count. Change either side and the percentage moves, even if the model did not.

Public write-ups keep circling the same temptation. Someone reports zero mistakes on a small private set, and the number travels faster than the method. You do not need to litigate that post. You need a rule for your own suite: no solvable-slice pass rate gets quoted until the negative-control slice has been scored beside it.

That rule is methodological. It is not a product claim, and it is not a leaderboard.

The three slices you will actually run

Keep the suite small enough to rerun. Eight to sixteen tasks is enough to debug the harness. It is not enough to rank vendors. Label that limit in the report so nobody promotes the figure later.

Use three slices, stored as data, not as a vibe.

Solvable. A real bug or a small feature. The prompt does not contain the patch, the test name, or the expected output.
Negative control. A task the agent should not pass. Examples: the repo already matches the spec, the requested file does not exist, or the instruction contradicts the test. A pass here is a false pass.
Canary. The same shape as a solvable task, plus a unique string that must never appear in a correct patch. If the model emits the canary, treat the trial as contaminated, not as a success.

You score the slices separately. You do not average them into one marketing percentage.

Step 1: Freeze the task card

Write one JSON object per task before you call any model. The card is the dataset. If a field is missing, the checker rejects the task instead of guessing.

{
  "id": "neg-03",
  "slice": "negative",
  "prompt": "Add a discount function. Do not modify billing.py.",
  "forbidden_paths": ["billing.py"],
  "hidden_test": "tests/test_billing.py",
  "canary": null,
  "expected": "fail_or_abstain"
}

A solvable card sets "slice": "solvable" and "expected": "pass". A canary card sets "slice": "canary" and puts a random token in "canary", for example CANARY-7f3a9c. That token must not be a word the solution needs.

Do not generate these cards from the same chat that will later solve them. If you do, you have already leaked the rubric into the context you are grading.

Step 2: Pin the run controls

Controls are the boring part. They are also the part that stops a number from becoming an ad.

Control	What you freeze	Why it matters
Attempt budget	Same max tries per task	Extra retries inflate pass rate
Tool list	Same editor, shell, test command	A richer tool schema is a different agent
Context	Prompt plus repo snapshot only	Hidden tests stay out of the prompt
Clock	One timeout value	A hung run is a fail, not a missing row
Scorer	This checker, versioned	Hand edits after the fact are a new study

Write the freeze into run_envelope.json next to the tasks. If you change the timeout, you start a new run id. You do not edit yesterday's pass rate in place.

Step 3: Run the checker, not a vibe

The script below is a proposed checker. It has not been executed against a live model for this article. Treat it as a starting artifact. Run it yourself before you trust a row.

#!/usr/bin/env python3
"""Slice scorer for a tiny coding-agent suite. Unexecuted example."""
import argparse, json, sys

def load_jsonl(path):
    with open(path, encoding="utf-8") as fh:
        return [json.loads(line) for line in fh if line.strip()]

def false_pass(task, trial):
    if task["slice"] != "negative":
        return False
    return trial.get("verdict") == "pass"

def canary_hit(task, trial):
    token = task.get("canary")
    if not token:
        return False
    blob = trial.get("patch", "") + trial.get("stdout", "")
    return token in blob

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--tasks", required=True)
    parser.add_argument("--trials", required=True)
    args = parser.parse_args()
    tasks = {row["id"]: row for row in load_jsonl(args.tasks)}
    counts = {}
    for trial in load_jsonl(args.trials):
        task = tasks[trial["task_id"]]
        key = task["slice"]
        bucket = counts.setdefault(
            key,
            {"n": 0, "pass": 0, "false_pass": 0, "canary": 0, "abstain": 0},
        )
        bucket["n"] += 1
        if trial.get("verdict") == "pass":
            bucket["pass"] += 1
        if trial.get("verdict") == "abstain":
            bucket["abstain"] += 1
        if false_pass(task, trial):
            bucket["false_pass"] += 1
        if canary_hit(task, trial):
            bucket["canary"] += 1
    solvable = counts.get("solvable", {})
    negative = counts.get("negative", {})
    canary = counts.get("canary", {})
    quote_ok = (
        negative.get("false_pass", 0) == 0
        and canary.get("canary", 0) == 0
        and solvable.get("n", 0) > 0
    )
    report = {"slices": counts, "headline_allowed": quote_ok}
    json.dump(report, sys.stdout, indent=2)
    print()

if __name__ == "__main__":
    main()

Run it like this after your batch finishes:

python3 slice_scorer.py --tasks tasks.jsonl --trials trials.jsonl > report.json
python3 -m json.tool report.json

headline_allowed stays false when any negative-control trial passes, or when any canary token shows up in a patch or log. That is the point. A green solvable slice cannot outvote a dirty control slice.

Step 4: Read the metrics in the right order

Look at four numbers, in this order, and stop if an earlier one is bad.

Canary hit count. Any hit means the prompt, the log, or the scorer leaked a marker into the artifact. Fix the harness. Do not quote a pass rate.
False-pass count on the negative slice. A pass means the tests are weak, the agent ignored a constraint, or you labeled a solvable task as negative. All three are measurement bugs.
Abstain rate. An agent that never abstains on impossible tasks is not more helpful. It is willing to invent a diff. Record abstains as their own outcome. Do not fold them into failures without saying so.
Solvable pass rate. Only now. Report it as passes / solvable_n for this run id. Attach the envelope file. Do not call it accuracy, and do not compare it with a run that used a different retry budget.

If you want a sentence for the weekly update, use this shape: "On this run id, solvable pass rate was X/Y after zero false passes and zero canary hits. N is too small to rank models." Fill X and Y from your report. Leave them blank until the checker prints them.

Where a free run fits

You can execute this loop on a laptop. You can also park the batch on a free server so a dropped SSH session does not eat the run, and call a model through free access so you are not burning a paid quota while you debug the harness.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is the open-source project this draft was asked to cover. The operator supplied two availability claims only: free model access, and a free server option. This article does not name a model, a token allowance, a machine size, or a duration. Those details change. Read the current project docs before you plan a batch, and do not cite a quota from memory or from an older post.

Use the free path for harness debugging, not for a launch number. Generate candidate patches with whatever free model access you currently have. Store each trial as one JSONL row: task_id, verdict, patch, stdout, attempt. Run slice_scorer.py on the server, or copy the JSONL home and run it locally. Keep the raw file. A percentage without the file is a rumor.

If you do not have that access, nothing in the method depends on it. The checker is plain Python. The discipline is the product of the method, not of the host.

Why this number is not marketing

Marketing wants one figure, a rising arrow, and a model name in the title. This method refuses that shape on purpose.

A solvable pass rate with a dirty negative slice is compatible with a broken test, a leaked answer, or a lucky retry. Publishing it as "the agent scores 100%" trains your readers to ignore the denominator. Separating slices makes the failure visible: the agent looked strong because the suite could not say no.

You also refuse cross-run averages. Eight tasks on Tuesday and eight different tasks on Friday are not a sixteen-task study. They are two anecdotes. Say so in the report footer.

Limitations

This slice will not save a bad dataset. If every negative control is obviously impossible, a model can fail them and still be weak on real work. If your canary is a common word, you will flag innocent patches.

Eight tasks cannot support a ranking, a confidence interval you would show a customer, or a claim about coding ability. The checker does not compile code. You still need a real test command in the runner that produces verdict. No model was scored for this draft, so there is no measured pass rate to cite.

Timeouts, flaky tests, and network calls will still move verdict. Pin them in the envelope, or accept that the score jitters.

Who should not use this

Skip the method if you need a statistically powered comparison. You want a larger, pre-registered suite and a frozen hidden set, not a lunch-sized slice.

Skip it if the code is safety-critical. A JSONL checker is not a security review, a license review, or a substitute for reading the diff.

Skip it if you already plan to quote a single percentage in an ad. The checker is designed to withhold that sentence. Fighting it means you wanted a slogan, not a measurement.

What you do next

Pick four solvable tasks from a repo you maintain. Write four negative controls that should not pass. Add two canaries. Freeze the envelope. Run the checker. If headline_allowed is false, fix the suite before you mention a model.

When you want a place to park that first noisy batch, a free MonkeyCode server session is enough to keep the JSONL intact while you iterate. Confirm the current limits yourself, then archive the report next to the tasks. The archive is the result. The percentage is optional.

Name the Failure Class Before You Rank a Coding Agent

Casey Zhang — Fri, 09 Oct 2026 12:57:52 +0000

You are halfway through a review when a teammate drops a screenshot in the channel. The caption is a coding-agent pass rate. The log is a scroll away, and almost nobody opens it.

You do. Compile failures sit beside timeouts. A green row has edited a file the task marked read-only. The caption is still up in the channel. The evidence is not in the caption.

This piece is a method for that moment. You will keep a failure-class ledger beside every coding-agent score you intend to cite. The ledger names how each task ended, and it includes a negative-control slice that a correct run must not pass. If you cannot publish the ledger, you do not publish the percentage.

What a blended pass rate conceals

A coding-agent run is not a coin flip. The agent can fail to compile, fail a test, time out, return no patch, or wander into files it was told to leave alone. Those endings are different events.

A single green rate treats a timeout like a near miss and a forbidden edit like a success. You already know a score needs a frozen run envelope before the percentage is worth quoting. This workflow adds the next artifact: a class for every row, counted before anyone ranks a model.

The classes belong in the dataset contract. They are not a story you attach after the chart looks good. Public write-ups still open on one figure. You can notice that habit without borrowing their numbers, their tasks, or their captions.

The ledger you are going to build

You need three files, and you need them before the call.

A task file that marks each item positive or negative.
A run log in JSONL, one object per finished task, appended by the harness.
A classifier that refuses a citeable rate when the controls fail.

No model name belongs on the cite line until those three exist. The script below is a proposed harness. It is not a measured leaderboard. The sample rows are illustrative, so you can see the rule fire. They are not results from a named model, and they are not a claim about any release this week.

Dataset, metrics, and controls

Dataset. Split the set before you call any model. Positive tasks have a known acceptable patch and a test command that can fail. Negative-control tasks have no acceptable patch. That can be a planted failure in a file the agent must not touch, a test that must stay red, or a prompt that asks for a change outside the allowed paths.

Keep a real negative slice, not a single toy row. A set of only solvable tasks cannot detect a harness that marks everything pass. Date the task file. If you later suspect a row has leaked into public training text, retire it in the notes and say so. Do not swap in an easier task and keep the old chart.

Metrics. Report counts, then rates. Use the positive slice only for the pass rate. Use the negative slice only for the false-pass rate. Across the whole run, also report compile-fail share, test-fail share, timeout share, over-edit share, and unresolved share. Unresolved means a missing log row, a crash, or an outcome the schema does not name.

Controls. Freeze these before the first call, and write them into a header the classifier can see.

Harness version and prompt-template hash.
Timeout, retry cap, and token budget, identical for every row you will compare.
Allowed path list per task.
Seed list. One seed is a demo. Three seeds is the minimum if you intend to cite a rate.
A cite rule, written in advance, for when the rate may be quoted at all.

Do not drop a class because it makes the chart uglier. That drop is how a method turns into a slogan. A number you can only publish after deleting rows is marketing, even if the arithmetic is correct.

How to build a negative control that is not a trick

A negative control is a task the agent must not solve. You are not trying to embarrass a model. You are checking whether the harness awards a pass for the wrong reason.

Three shapes work on ordinary application code you own.

A read-only sentinel. The prompt asks for a feature under src/. A known bad line lives in a file the task marks forbidden, and the grader fails if that file's hash changes. If the agent edits it and tests go green, the outcome is over_edit, not pass.
An unsatisfiable local test. The task asks for a function to return two incompatible results for the same input, and you actually run the checker. A pass means the grader did not run, or someone weakened the test.
A planted red test outside the allowed diff. The agent may edit billing.py only. A second test is already failing and must stay failing. If the run deletes that test, log over_edit.

Write the expected class in the task file before the call. If you invent the expected class after you see the patch, you are narrating. You are not measuring. Keep the control inside a repository you can explain to a reviewer. This is a grader check, not a security exercise.

Step by step: build the classifier

Work on a small box you can wipe. The ledger itself is a file and a short script. A free server can hold both. Model calls are a separate budget. Do not assume a tier, a quota, or a machine shape you have not read on the day you run.

1. Fix the outcome schema

Create a short schema note in the run directory and treat it as code. Allowed outcome values are pass, compile_fail, test_fail, timeout, no_patch, over_edit, and unresolved.

If the agent edits a forbidden path, the outcome is over_edit even when tests are green. A green test on a negative-control task stays pass in the raw log. The classifier, not the agent, turns that row into a false pass. You want the raw outcome honest. You want the judgment in the script, where a reviewer can see it.

2. Append one JSON object per task

Have the harness append, not overwrite. A crash should leave a partial file you can still score. Do not backfill missing rows from memory.

mkdir -p runs/ledger-demo
cat > runs/ledger-demo/header.json << 'EOF'
{
  "harness_version": "0.3.1",
  "prompt_hash": "sha256:replace-me",
  "timeout_s": 120,
  "max_retries": 0,
  "token_budget": 8000,
  "seeds": [7, 19, 23]
}
EOF

The hash and the budget in that header are placeholders for the envelope you actually freeze. Replace them. Do not cite a run whose header is still the example. 8000 is a stand-in so the field exists. It is not a recommended product limit.

Illustrative rows, not an evaluation:

{"task_id":"pos-01","slice":"positive","outcome":"pass","seed":7}
{"task_id":"pos-02","slice":"positive","outcome":"compile_fail","seed":7}
{"task_id":"pos-03","slice":"positive","outcome":"timeout","seed":7}
{"task_id":"neg-01","slice":"negative","outcome":"test_fail","seed":7}
{"task_id":"neg-02","slice":"negative","outcome":"pass","seed":7}

neg-02 is the row that should stop you. The log recorded a pass on a task with no acceptable fix. A caption would have buried that row inside a friendly rate.

3. Classify, then apply the cite rule

Save the following as failure_ledger.py. Run it on your own logs. The demo file exists so you can watch the rule reject a flattering mix. It is not evidence about a product.

#!/usr/bin/env python3
"""Failure-class ledger. Proposed harness, not a published score."""
import json
import sys
from collections import Counter

CLASSES = (
    "pass",
    "compile_fail",
    "test_fail",
    "timeout",
    "no_patch",
    "over_edit",
    "unresolved",
)

def load_jsonl(path):
    rows = []
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows

def citeable(n_pos, n_neg, unresolved, false_passes):
    # Pre-register this rule before you look at model identity.
    if n_pos == 0 or n_neg == 0:
        return False, "missing positive or negative slice"
    if unresolved:
        return False, "unresolved rows present"
    if false_passes:
        return False, "negative control recorded a pass"
    return True, "class ledger complete; quote counts beside the rate"

def main(path):
    rows = load_jsonl(path)
    by_class = Counter(r.get("outcome", "unresolved") for r in rows)
    n_pos = sum(1 for r in rows if r.get("slice") == "positive")
    n_neg = sum(1 for r in rows if r.get("slice") == "negative")
    pos_pass = sum(
        1
        for r in rows
        if r.get("slice") == "positive" and r.get("outcome") == "pass"
    )
    false_pass = sum(
        1
        for r in rows
        if r.get("slice") == "negative" and r.get("outcome") == "pass"
    )
    unresolved = sum(1 for r in rows if r.get("outcome") not in CLASSES)
    ok, reason = citeable(n_pos, n_neg, unresolved, false_pass)
    pos_rate = (pos_pass / n_pos) if n_pos else None
    print("rows", len(rows))
    print("positive_pass", pos_pass, "/", n_pos, "rate", pos_rate)
    print("false_pass", false_pass, "/", n_neg)
    for name in CLASSES:
        print("class." + name, by_class.get(name, 0))
    print("citeable", ok)
    print("reason", reason)

if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.stderr.write("usage: python3 failure_ledger.py runs/seed.jsonl\n")
        sys.exit(2)
    main(sys.argv[1])

Run the illustrative file first:

python3 failure_ledger.py runs/ledger-demo/seed-7.jsonl

On that file, citeable is false because a negative control passed. That is the method working. A caption-style summary would have counted green rows and moved on.

After you have real logs, score every seed the same way:

for seed in 7 19 23; do
  python3 failure_ledger.py "runs/ledger-demo/seed-${seed}.jsonl"
done

Compare class counts across seeds before you compare models. If one seed is mostly timeouts and the next is mostly test failures, you do not have a stable rate yet. You have noise. Wait.

4. Read the decision table before you write the post

Condition	What you may say	What you must not say
Missing negative slice	Harness smoke test only	Any pass rate as a quality claim
Any false pass	Negative control failed; do not rank	Solved the set
Unresolved rows dropped from the denominator	Nothing comparative	A higher rate than a run that kept those rows
Classes published, false passes at zero, envelope frozen	Positive pass rate with class counts and seeds	A bare percentage
Budgets or timeouts differ across endpoints	Separate logs, not a ranking	One endpoint beat the other

The right column is the marketing failure mode. If your draft has the number and none of the conditions, delete the number. Readers can trust a smaller, qualified rate more than a clean one they cannot recompute.

5. Point the driver at the endpoint you actually have

The ledger does not care which vendor answered. It cares that every row shares the header you froze.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option can carry this loop when you do not already have a box and an endpoint: the server stores the JSONL and runs the classifier, and free model access is only the call path. This article does not state a quota, a hardware shape, a model list, a duration, or a promise that the free tier will stay as it is. Read the current plan before you depend on it. If that plan changes, keep the ledger and move the call path.

One check after the first seed: open the header and confirm the prompt hash, timeout, and token budget match the run you think you launched. A drifted template is a different experiment. Score it separately, or do not score it.

What to do when the first ledger is red

Your first real run will probably print citeable false. Leave that line in the notes. It means the rule survived contact with a log.

Change one control, not three.

If the reason is a false pass, inspect the grader before you blame the agent. A test that always exits zero is a harness bug.
If the reason is unresolved rows, fix the append path. A lost row is not a timeout. Do not reconstruct it.
If timeouts dominate, you may raise the timeout. That is a new envelope. Keep the old log. Do not overwrite it and pretend both runs shared a clock.

Rerun the same seed. Compare class counts, not impressions. Spend a second seed only after you can name the largest class without looking at the caption. A single flattering seed is how the screenshot in the opening gets made.

Comparing two endpoints without inventing a winner

You can point the same task file at two endpoints. Freeze the header, swap only the base URL, and write separate JSONL files. Rank them only when both ledgers are citeable and both headers match on timeout, retry cap, and token budget.

If one call path stops early because a limit cut the run, mark the missing rows unresolved. Do not impute passes. A shorter bill is not a better agent. Write the stop reason into the header so a reader can see the run ended for a budget reason, not a quality reason.

Keep the two logs in separate directories. Mixing them into one average is how unequal budgets sneak back into the chart. If you cannot show both headers, you cannot show a winner.

Why the ledger is not a slogan

A class ledger is a poor advertisement. It shows timeouts, compile failures, and false passes that one rate can hide. That is why you keep it.

You pre-register the cite rule in code, then you run. You do not edit the rule after you see which endpoint looks kinder. Publish the counts even when the positive pass rate is flattering. Readers can recompute.

A figure they cannot recompute is a claim about your editing, not about the agent. Link the harness commit and the task-file commit next to any rate. Without those two hashes, the number cannot be reproduced, so it should not be cited.

Limits, and who should skip this

This method does not measure whether a patch is maintainable, secure, or what a user wanted. It measures whether your harness can tell a real pass from a miss, and whether you will show the miss.

Skip it when any of these are true.

You cannot inspect the tasks, so you cannot build a negative slice you trust.
You need a ranking today and will quote the pass rate even if the script prints citeable false.
Your real question is human preference or production incident rate. Use a different instrument.

A tiny or obvious negative slice can lie in the other direction. A false-pass rate of zero means little if every control is trivial. Rotate controls when they become familiar in your prompt archive. This is also the wrong tool if you wanted a security study. Do not turn the negative slice into an attack exercise. Keep it inside code you own, with tests you can explain to someone who was not in the room.

Hand-built sets overfit the failures you remembered to name. Twenty tasks can teach you whether the harness is honest. They cannot crown a general coding agent. Say the sample size in the same sentence as the rate, or do not say the rate.

What you should ship with the score

Ship four files, not a screenshot.

header.json with the frozen envelope.
The JSONL log for every seed.
The classifier output, including citeable and the reason.
A short note naming who wrote the tasks and whether any row was removed after the run.

If those files are missing, keep the percentage in the draft and out of the post. The ledger is the result. The rate is a summary of one column, and only after the other columns are clean.

If you already have a free model endpoint and a small free server, run one seed of your own tasks through this script before you paste a percentage into a review thread. Keep the ledger next to the figure. That pairing is the part worth sharing.

Reproduce the Score Before You Cite It: A 20-Task Harness You Can Run on Free Infrastructure

Casey Zhang — Thu, 08 Oct 2026 09:22:06 +0000

A leaderboard screenshot lands in your team channel. One coding agent solves 74% of real issues, says the chart. Nobody in the channel has the task list, the commit SHA, the token budget, or the stop condition behind that number. Two weeks later it is in a slide deck; a quarter later it is the reason you picked a vendor.

The fix is smaller than most people expect: a task list, one harness file, and a ledger you can hand to someone else. This article builds that, then runs it on free infrastructure.

Why a published score is a manufacturing process

A benchmark number is not a lie. It is the output of a process where someone chose the tasks, the budget per task, the retry policy, and the fail condition. Those choices are all legitimate. Together they mean the score answers one narrow question: what does this agent do under conditions we picked?

When you cite it, you are implicitly claiming those conditions match yours. Usually they do not. So before you argue about scores, reproduce one.

Step 1 — Fingerprint the claim before you run anything

You cannot reproduce a number you have not described. Write down six fields. If any field is missing from the source, mark it unknown and treat the score as directional only.

Field	Why it changes the score
Task set + version	Different repos, different difficulty
Budget per task	More tokens, more retries, higher pass rate
Attempts per task	pass@1 vs pass@5 are different metrics
Harness commit	Prompt, tools, and stop logic all move the number
Model + endpoint	A hosted alias can change under you
Verify command	The definition of "solved"

Do this on paper first. It takes ten minutes and saves a week.

Step 2 — Build the smallest harness that can be re-run

Two files. Everything that can move the score lives in config, not in code.

{
  "model_id": "REPLACE_ME",
  "base_url": "REPLACE_ME",
  "task_dir": "tasks",
  "max_tokens_per_task": 6000,
  "max_wall_clock_seconds": 180,
  "retries": 0,
  "seed": 20261008,
  "tasks": ["issue-001", "issue-002"]
}

Now the runner. Note the two design rules in the docstring: the harness records its own hash, and it never decides pass/fail — the task's own verify command does.

#!/usr/bin/env python3
"""reproduce.py - rerun a fixed task list, one ledger row per attempt.

Design rules:
  * every knob that can change the score lives in eval.config.json
  * the harness records its own hash, so an edited harness is visible
  * the harness never decides pass/fail; the task's verify command does
"""
from __future__ import annotations

import hashlib
import json
import os
import pathlib
import platform
import subprocess
import sys
import time

ROOT = pathlib.Path(__file__).resolve().parent
CFG = json.loads((ROOT / "eval.config.json").read_text())
RUN_ID = os.environ.get("RUN_ID") or time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
LEDGER = ROOT / "runs" / f"{RUN_ID}.jsonl"


def fingerprint() -> dict:
    return {
        "run_id": RUN_ID,
        "harness_sha256": hashlib.sha256(
            pathlib.Path(__file__).read_bytes()).hexdigest()[:16],
        "python": sys.version.split()[0],
        "platform": f"{platform.system()}/{platform.machine()}",
        "model_id": CFG["model_id"],
        "base_url_host": CFG["base_url"].split("/")[2],
        "max_tokens_per_task": CFG["max_tokens_per_task"],
        "max_wall_clock_seconds": CFG["max_wall_clock_seconds"],
        "retries": CFG["retries"],
        "seed": CFG["seed"],
    }


def patch_hash(workdir: pathlib.Path):
    diff = subprocess.run(["git", "diff", "--binary"], cwd=workdir,
                          capture_output=True)
    return hashlib.sha256(diff.stdout).hexdigest()[:16] if diff.stdout else None


def run_one(task_id: str) -> dict:
    workdir = ROOT / CFG["task_dir"] / task_id
    spec = json.loads((workdir / "task.json").read_text())
    env = {
        **os.environ,
        "MODEL_ID": CFG["model_id"],
        "BASE_URL": CFG["base_url"],
        "MAX_TOKENS": str(CFG["max_tokens_per_task"]),
        "RETRIES": str(CFG["retries"]),
        "SEED": str(CFG["seed"]),
    }

    started = time.monotonic()
    agent = subprocess.run(
        [sys.executable, str(ROOT / "agent.py"), "--task", str(workdir)],
        capture_output=True, text=True,
        timeout=CFG["max_wall_clock_seconds"], env=env,
    )
    verify = subprocess.run(
        spec["verify"], shell=isinstance(spec["verify"], str),
        cwd=workdir, capture_output=True, text=True,
        timeout=spec.get("verify_timeout", 300),
    )

    usage = {}
    usage_file = workdir / "usage.json"   # your agent wrapper writes this
    if usage_file.exists():
        usage = json.loads(usage_file.read_text())

    return {
        "task_id": task_id,
        "passed": verify.returncode == 0,
        "verify_cmd": spec["verify"],
        "verify_stderr_tail": verify.stderr[-400:],
        "agent_exit": agent.returncode,
        "wall_clock_s": round(time.monotonic() - started, 2),
        "tokens_in": usage.get("tokens_in"),
        "tokens_out": usage.get("tokens_out"),
        "patch_sha256": patch_hash(workdir),
    }


def main() -> int:
    LEDGER.parent.mkdir(exist_ok=True)
    with LEDGER.open("a") as fh:
        fh.write(json.dumps({"_fingerprint": fingerprint()}) + "\n")
        for task_id in CFG["tasks"]:
            row = run_one(task_id)
            fh.write(json.dumps(row) + "\n")
            print(f"{task_id:>12}  {'PASS' if row['passed'] else 'FAIL'}  "
                  f"{row['wall_clock_s']:>7.2f}s  "
                  f"in={row['tokens_in']} out={row['tokens_out']}")
    print(f"\nledger: {LEDGER.relative_to(ROOT)}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

One field is deliberately fragile: tokens_in / tokens_out come from a usage.json your agent wrapper writes. Providers report usage differently. Do not scrape stdout for token counts — that is how silent zeros enter a ledger.

The harness needs Python 3.10+ for the union type on patch_hash. Drop the annotation if you are on 3.9.

Step 3 — Pick a substrate you do not have to pay for

A reproduction run is bursty: idle most of the week, then 20 tasks in an hour. That is the worst shape for a monthly cloud bill.

This is where MonkeyCode fits the method rather than the marketing. MonkeyCode is an open-source project that offers free model access and a free server option. Both of those availability claims come from the project operator; this article did not test them, and nothing below depends on them being permanent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because the claims are operator-supplied, treat them as configuration to verify, not facts to assume. Before your first run, check the project's current docs for: which models the free access currently covers, what the free server currently provides, and what limits or expiry apply. This article asserts none of those specifics, on purpose — quotas, hardware, and duration are exactly the fields that change, and an article is a terrible place to freeze them.

Then keep key hygiene boring:

# Never in the repo. Never in the ledger. Never in a screenshot.
export MONKEYCODE_API_KEY="..."
export RUN_ID=$(date -u +%Y%m%dT%H%M%SZ)
python reproduce.py

The fingerprint logs base_url_host, not the key. That is intentional: you want a teammate to see which endpoint produced the run, and you never want the ledger to be a secret store.

Step 4 — Turn the ledger into one honest summary

Raw JSONL is not a score. Reduce it once, in code, so nobody reduces it by hand in a spreadsheet.

#!/usr/bin/env python3
"""score.py - one ledger in, one honest summary out."""
import json
import pathlib
import sys

rows = [json.loads(l) for l in pathlib.Path(sys.argv[1]).read_text().splitlines()]
fp, attempts = rows[0]["_fingerprint"], rows[1:]
passed = sum(a["passed"] for a in attempts)
tokens = sum((a["tokens_in"] or 0) + (a["tokens_out"] or 0) for a in attempts)

print(json.dumps({
    "run_id": fp["run_id"],
    "harness": fp["harness_sha256"],
    "tasks": len(attempts),
    "pass_rate": round(passed / len(attempts), 3),
    "tokens_per_solved_task": round(tokens / passed, 1) if passed else None,
    "p50_wall_clock_s": sorted(a["wall_clock_s"] for a in attempts)[len(attempts) // 2],
}, indent=2))

Now the part that keeps you honest. With 20 tasks, a pass rate near 50% has a standard error around 0.11, so a 95% interval spans roughly plus or minus 22 points. If your number differs from the published one by five points, you have measured nothing. You have re-measured noise with extra steps.

Step 5 — Diff the fingerprint, not the headline

When your result does diverge, compare fingerprints field by field before you compare conclusions.

Your observation	Most likely cause	What to do next
Delta under ~10 points at n=20	Sampling noise	Add tasks; do not publish a claim
Delta 25+ points	Different conditions	Diff fingerprint fields one at a time
Budget or retries differ	Not the same benchmark	Re-run both under one config
Task list unavailable	Unverifiable source	Keep it as directional context only
Verify command fails on the clean checkout	Flaky tests, not agent failure	Run verify 3x pre-patch, then exclude

That last row matters more than it looks. Before blaming an agent, run each task's verify command on the unmodified checkout three times. A test suite that fails 1 in 3 times on clean code will manufacture agent failures out of nothing, and you will spend a weekend debugging a model that was fine.

Limitations, and who should not do this

A free tier is a moving target. If the terms change mid-study, your run is invalid, not merely stale. Pin what you verified and re-run after a change.
Wall-clock numbers from free, shared infrastructure are not latency benchmarks. Use them for timeout tuning only.
20 tasks cannot resolve a two-point difference. Do not pretend otherwise.
Token accounting is only as good as the provider's usage reporting. If a field is null, the summary shows null.

Skip this approach if you need procurement-grade or regulatory-grade evidence, if you are comparing models that differ by under a few points, or if your organization cannot run untrusted agent-generated patches in an isolated worktree. That last constraint is real: an eval harness executes code a model wrote.

The short version

Fingerprint the claim. Pin the config. Let the verify command, not the harness, decide pass or fail. Record a ledger. Reduce it in code. Check the noise floor before you check your ego. Then ship the harness alongside the number, so the next person can re-derive it instead of trusting it.

If you want the shortest path to a first run, start from the MonkeyCode docs to confirm what free access currently includes, then paste the config above and swap in your own tasks — the docs are the only current source for the availability claims, and the config is the checklist to verify them against.

Refuse the Percentage Until the Run Envelope Is Frozen

Casey Zhang — Thu, 08 Oct 2026 06:39:27 +0000

You opened the morning export and almost pasted it into a team note. Twelve coding tasks. Nine green. The sentence in your head was short: 75 percent, overnight, on a free server.

Then the raw log spoiled it. Task 4 had a cold start and a second attempt. Task 7 died on a timeout before any patch existed. Task 11 looked green, but the token-budget cell was empty. The percentage is a caption. It is not a result.

This walkthrough shows you how to freeze a run envelope, log host confounders, and block a headline rate when the envelope is incomplete. You get a dataset card, three metrics, a control table, and a small Python gate. Nothing here ranks a model. The script was not executed against a live host for this draft.

Name the question before the run

A citeable score answers one sentence you wrote first. Example: on this frozen task list, with this hidden-test hash, this timeout, and this token budget, how many tasks passed on the first completed attempt?

That sentence does not mean a product is better. It does not mean a free host is production-ready. If you cannot say the question without adjectives, stop. You are about to publish a demo.

Write the question into the same directory as the logs. Future you will try to widen it. The file is the brake.

1. Freeze the dataset card

Rerunability beats size. Eight to twenty tasks you can repeat will teach you more than a hundred you run once.

Give every task the same fields, and pin repo_revision to a commit id rather than a branch. Hash the prompt and the hidden tests. Do not commit those tests beside the agent.

Required card fields:

task_id
repo_revision
prompt_sha256
hidden_tests_sha256
allowed_files
timeout_s
token_budget

This row is a schema example. It is not a measured outcome.

task_id: calc-parse-017
repo_revision: 9f3c1ab
prompt_sha256: 6d2ec1aa
hidden_tests_sha256: a91b0440
allowed_files: ["src/parse.py"]
timeout_s: 180
token_budget: 12000

If two runs differ on any field, they are different experiments. Keep them in different files. Do not pool them later because the pass rates look similar.

2. Keep three metrics, not one

You will be tempted to publish a single rate. Don't. Store three, and print them together.

first_complete_pass_rate is the share of tasks whose first completed attempt passed the hidden tests.
retry_rate is the share of tasks with a second attempt, a cold start, or a transport error.
truncation_rate is the share of first attempts that stopped on timeout or token budget, not on a finished patch.

Add patch_apply_fail_rate when the harness cannot apply the diff. That failure belongs to you, not the agent. A task with an apply error must not sit in the denominator as a model fail unless you label the loss as harness loss.

Unequal budgets still must not be averaged. That older rule still holds. What changes here is the host: a retry measures the path, so it stays next to the pass rate.

3. Treat the server as a treatment

Shared compute changes the experiment even when the prompt file does not change. Queue wait, a cold start, a concurrency cap, or a silent retry can move the green count. Leave those out, and readers will blame the agent for a scheduler.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two availability claims are operator-supplied for this draft: free model access, and a free server option. You may use that pair as the place where you execute this method. You may not turn the product name into a metric.

This draft states no token quota, hardware shape, free-window length, or permanence promise. Those details were not checked against a vendor status page here. Write the class you actually used into access_class and server_class.

Field	What it holds still	If you skip it
access_class	Free and paid paths may queue differently	Do not compare the runs
server_class	Shared CPU is part of the treatment	Say hosted, not model-only
concurrency	Parallel jobs share one cap	Headline rate is invalid
timeout_s	Slow correct patches get cut	Split truncation out
token_budget	Budget is part of the task	Refuse any average
attempt_policy	Retries hide transport noise	Keep retry_rate in the sentence

4. Run a citeability gate

You want a script that refuses a percentage when the envelope is thin. The program below is a proposal, and it was not run on a live agent for this article. It does not open a socket. Point it at JSONL you already trust.

#!/usr/bin/env python3
"""Citeability gate. Proposal only. No network calls."""
import json
import sys
from pathlib import Path

REQUIRED = [
    "task_id", "prompt_sha256", "hidden_tests_sha256",
    "access_class", "server_class", "timeout_s", "token_budget",
    "attempt", "cold_start", "queue_wait_ms", "stop_reason",
    "patch_sha256",
]

def load_jsonl(path):
    rows = []
    for line in Path(path).read_text().splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows

def gate(rows):
    missing = []
    for i, row in enumerate(rows):
        for key in REQUIRED:
            if row.get(key) in (None, ""):
                missing.append({"row": i, "task_id": row.get("task_id"), "field": key})
    if missing:
        return {"citeable": False, "reason": "incomplete_envelope", "missing": missing}

    first = {}
    for row in sorted(rows, key=lambda r: (r["task_id"], r["attempt"])):
        first.setdefault(row["task_id"], row)

    n = len(first)
    passed = sum(1 for r in first.values() if r["stop_reason"] == "pass")
    retried = {
        r["task_id"] for r in rows
        if r["attempt"] > 1 or r["cold_start"] or r["stop_reason"] == "transport"
    }
    truncated = sum(1 for r in first.values() if r["stop_reason"] in {"timeout", "budget"})
    return {
        "citeable": True,
        "n_tasks": n,
        "first_complete_pass_rate": round(passed / n, 4),
        "retry_rate": round(len(retried) / n, 4),
        "truncation_rate": round(truncated / n, 4),
        "note": "Single envelope only. Not a product ranking.",
    }

if __name__ == "__main__":
    report = gate(load_jsonl(sys.argv[1]))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    sys.exit(0 if report["citeable"] else 2)

Save the next object as ok.jsonl. Every required field is present, and cold_start is false. The row is synthetic.

{"task_id":"calc-parse-017","prompt_sha256":"6d2ec1aa","hidden_tests_sha256":"a91b0440","access_class":"free_model","server_class":"free_server","timeout_s":180,"token_budget":12000,"attempt":1,"cold_start":false,"queue_wait_ms":840,"stop_reason":"pass","patch_sha256":"bb10"}

Save the following object as gap.jsonl. Queue wait is absent. That hole is the failure you want the gate to catch.

{"task_id":"calc-parse-017","prompt_sha256":"6d2ec1aa","hidden_tests_sha256":"a91b0440","access_class":"free_model","server_class":"free_server","timeout_s":180,"token_budget":12000,"attempt":1,"cold_start":false,"stop_reason":"pass","patch_sha256":"bb10"}

Run both commands. The second should exit 2.

python3 cite_gate.py ok.jsonl
echo "ok_exit=$?"
python3 cite_gate.py gap.jsonl
echo "gap_exit=$?"

If your local run of ok.jsonl matches the proposal, the shape looks like this. Treat it as the expected shape of the function, not as a captured host measurement.

citeable: true
n_tasks: 1
first_complete_pass_rate: 1.0
retry_rate: 0.0
truncation_rate: 0.0

Read the exit code before the pretty JSON. Exit 2 means notes, not a rate. A blank queue_wait_ms is enough. A dashboard screenshot is not a substitute.

5. Decide what sentence you may ship

Gate result	You may write	You may not write
citeable false	Envelope incomplete.	Any percentage
citeable and retry_rate above 0.2	Pass rate on a retry-heavy path.	The agent scores P.
citeable and truncation_rate above 0	Some tasks stopped on budget or timeout.	Those fails were bad patches.
one access class and one server class	On this frozen envelope, first-complete pass rate was P (n=N), retry rate R.	Better than another product.
mixed server classes in one file	Split the file and rerun.	A pooled average

The allowed sentence is dull on purpose. Dull is how you keep it out of a launch post. Drop n, drop the retry rate, or add a ranking verb, and the same arithmetic becomes marketing.

What the number still cannot prove

You hashed one prompt and one hidden suite. You did not sample the world's repositories. A second night on the same free server can change queue wait. Until you repeat the envelope, you have one draw.

Hidden tests stop you from grading prose. They do not detect training leakage. This gate does not check that. If the tasks are public puzzles, say so next to the rate, or do not cite the rate as generalization.

Do not treat a free server as an SLO oracle. Shared capacity is a fair place to debug the method. It is a bad place to promise latency.

Who should not use this gate

Skip it when you are showing a UI and you will not quote a rate. A tour does not need a denominator.

Skip it when you have no hidden tests. A model agreeing with its own summary is not an oracle. Build the tests first, then the gate.

Do not aim the agent at machines, accounts, or repos you are not allowed to modify. Keep the harness on fixtures you own. The method measures patches you can revert. It is not a scanning workflow.

If your decision is a paid capacity purchase, this design is the wrong instrument. It will not tell you how the same agent behaves once the queue disappears.

Put the gate in the commit

Drop cite_gate.py next to the task card. Run it on last week's JSONL before a percentage reaches a README or a slide. If the gate exits 2, publish the missing-field list instead. That list is the useful artifact.

If free model access plus a free server is where you can afford to repeat the same card, use that access to rerun the envelope. Keep the product name out of the title. One lab note with n, a retry rate, and a frozen hash will outlive a louder score.

DEV Community: Casey Zhang

Kill the Runner Mid-Patch: A Fault-Injection Bench for Agent Sessions

A pass rate only describes the runs that finished

Step 1 — Freeze the environment in a manifest

Step 2 — Make the journal the source of truth, not the output

Step 3 — Inject faults at named checkpoints only

Step 4 — Score survival, duplication, and replay cost

Controls that keep the run honest

Why these numbers are not marketing

Where this bench is wrong

Who should not use this

Run it on something small first

Your Benchmark Has No Cost Axis: Score Free Endpoints by Quota per Accepted Patch

Why a pass rate without a cost axis misleads

The three metrics worth reporting

Step 1: Freeze the task set and the attempt budget

Step 2: Instrument the call, not the result

Step 3: Aggregate with censoring made explicit

Step 4: Add the controls that make the table honest

Step 5: Read the numbers as a decision table, not a podium

Where a free tier genuinely fits this method

Limitations, stated up front

Who should not use this

Two Runs, Six Points Apart: Measure the Re-Run Spread Before You Compare Endpoints

Why a single run cannot answer a comparison question

Step 1: Freeze the task set, the harness, and the scoring rule

Step 2: Run K repeats per task, not K repeats per suite

Step 3: Score per task, with an interval

Step 4: Compute the minimum detectable difference

Step 5: Use a paired rule before you call a winner

Step 6: Publish the spread, not just the mean

Controls that quietly break repeatability

Where a free runner fits

Limitations and who should skip this

Copy-paste checklist

Free Model Endpoints Need a Contract Bench, Not a Leaderboard Screenshot

1. Write the contract before the prompt

2. Build a small, frozen probe set

3. Run it with controls

4. A harness you can adapt

5. Read counts, not a composite score

6. Why these numbers are not marketing

7. Decision table

8. Limitations and who should skip this

9. Where the free server fits

Your Coding-Agent Benchmark Needs a Run Manifest, Not Another Leaderboard Row

The number is the last line of the report, not the first

Step 1: Freeze the slice before you run anything

Step 2: Declare the envelope in a manifest

Step 3: Make the checker refuse bad comparisons

Step 4: Report discordant pairs, not just averages

Step 5: Ship the manifest next to the number

What this does not prove

Who should skip this

Free Tiers Are a Moving Target: Benchmark Them With a Frozen Envelope, Not a Screenshot

1. Decide what the number is allowed to claim

2. Write the envelope before you write the loop

3. Build the harness that refuses to aggregate

4. Add two controls before you trust the grader

5. Report five numbers, not one

6. A decision table for the result

Limitations, and who should skip this

Balance the Outcome Ledger: A Publication Gate for Coding-Agent Benchmarks

The rule: every task that enters must exit exactly once

Step 1 — Write the manifest (JSON, no dependencies)

Step 2 — Emit one JSONL record per attempt

Step 3 — The gate itself

Step 4 — Run it and read the report card

Why this makes the number a measurement and not marketing

Where free compute fits in this loop

Limitations, and who should skip this

Keep the Denominator Frozen: A Checker for Agent Pass Rates

The failure you are actually looking at

Step 1: Commit the contract while run_id is still null

Step 2: Record the host you actually used

Step 3: Lock, then refuse an unlocked rate

Step 4: Make exclusions and probes impossible to clean up

Why the printed rate is still not marketing

Step 5: Store a bundle a stranger can rerun

Who should not use the approach

Step 1: Commit the contract while `run_id` is still null