DEV Community: Blake Yang

Your First Reproduction Is Usually Wrong: A Receipt-Before-Patch Workflow for OSS Fixes

Blake Yang — Tue, 27 Oct 2026 12:42:10 +0000

A contributor opens a pull request against a parser library with a failing test, a two-line fix, and a confident description of the bug. The test goes red on their machine, green after the patch, and the maintainer merges it after a quick skim. Six weeks later the same crash reappears in a downstream project, because the original test never actually depended on the code that was changed. Patches drafted with an assistant are common enough now that maintainers triage them like any other contribution, which makes the reproduction step more important than the fix itself.

This article describes a small workflow that biases the whole contribution toward evidence: pin the failure, emit a machine-readable receipt, and prove the assertion is tied to the fix by removing the fix again. Free-tier models can help with parts of that loop, and a free server option can host the reproducer so a maintainer does not need to rebuild the environment on their laptop.

Three ways a reproduction lies

A reproduction is a claim about causality, and most bad reproductions fail in one of three predictable ways.

Hidden local state. The failure depends on a virtualenv, a cached build directory, a .env file, or a fixture that exists only on the reporter's disk.
Wrong revision. The report was filed against a tag, but the script runs against a moving branch, so the observed behavior is from different code.
Assertion drift. The test was written against the patched output, so it cannot fail before the patch and proves nothing about the fix.

The first two are environment problems and are usually solved by a clean clone at an explicit commit. The third is a logic problem, and it survives clean environments, green CI runs, and polite review comments, which is why it needs a mechanical check.

Step 1: pin the failure on a clean checkout

Start from a throwaway clone at the exact commit the report describes, and record the exit code of the command that is supposed to fail. The following script assumes only git, bash, jq, and sha256sum, and it treats every unexpected outcome as an error rather than a warning.

#!/usr/bin/env bash
# repro-receipt.sh — prove a bug fails before the patch and passes after it.
set -euo pipefail

REPO_URL="${REPO_URL:?set REPO_URL}"
BASE_SHA="${BASE_SHA:?set BASE_SHA}"        # commit the bug was reported against
TEST_CMD="${TEST_CMD:?set TEST_CMD}"        # command that must fail on BASE_SHA
PATCH="${PATCH:?set PATCH}"                 # path to the proposed .patch
WORKDIR="$(mktemp -d)"

git clone --quiet "$REPO_URL" "$WORKDIR/repo"
cd "$WORKDIR/repo"
git checkout --quiet "$BASE_SHA"

run () {  # prints: <exit code> <digest of combined output>
  local out code digest
  set +e
  out="$($TEST_CMD 2>&1)"; code=$?
  set -e
  digest="$(printf '%s' "$out" | sha256sum | cut -c1-16)"
  printf '%s %s\n' "$code" "$digest"
}

read -r BEFORE_CODE BEFORE_DIGEST <<<"$(run)"
if [ "$BEFORE_CODE" -eq 0 ]; then
  echo "FAIL: test passes before the patch — this is not a reproduction" >&2
  exit 2
fi

git apply --check "$PATCH"
git apply "$PATCH"
read -r AFTER_CODE AFTER_DIGEST <<<"$(run)"
if [ "$AFTER_CODE" -ne 0 ]; then
  echo "FAIL: patch does not turn the test green" >&2
  exit 3
fi

git apply -R "$PATCH"
read -r REVERT_CODE REVERT_DIGEST <<<"$(run)"
if [ "$REVERT_CODE" -eq 0 ]; then
  echo "FAIL: test stays green with the patch removed — assertion is not tied to the fix" >&2
  exit 4
fi

git apply "$PATCH"
jq -n \
  --arg base "$BASE_SHA" --arg cmd "$TEST_CMD" --arg url "$REPO_URL" \
  --argjson b "$BEFORE_CODE" --arg bd "$BEFORE_DIGEST" \
  --argjson a "$AFTER_CODE"  --arg ad "$AFTER_DIGEST" \
  --argjson r "$REVERT_CODE" --arg rd "$REVERT_DIGEST" \
  '{repo:$url, base_sha:$base, test_cmd:$cmd,
    before:{exit:$b, digest:$bd},
    after:{exit:$a, digest:$ad},
    reverted:{exit:$r, digest:$rd}}' \
  > "$WORKDIR/receipt.json"

cat "$WORKDIR/receipt.json"

A plausible invocation for a Python project looks like this, and it takes about as long as reading the contributor's description carefully.

REPO_URL=https://github.com/example/parser \
BASE_SHA=3f9c1ab \
TEST_CMD='pytest -q tests/test_parser.py::test_empty_input' \
PATCH=./fix-1394.patch \
./repro-receipt.sh

The exit-code contract matters more than the output itself. Codes 2, 3, and 4 name the three failure modes above, so a reviewer can tell from one number whether the problem is the report, the patch, or the test.

Step 2: emit a receipt the reviewer can check in seconds

The JSON block produced at the end is the artifact worth attaching to the pull request. It records the repository URL, the pinned commit, the exact command, and the three exit codes with short digests of the captured output.

{
  "repo": "https://github.com/example/parser",
  "base_sha": "3f9c1ab",
  "test_cmd": "pytest -q tests/test_parser.py::test_empty_input",
  "before":   { "exit": 1, "digest": "9c1f4a0e77b2d315" },
  "after":    { "exit": 0, "digest": "0a77c2b1e5d90f43" },
  "reverted": { "exit": 1, "digest": "9c1f4a0e77b2d315" }
}

Two properties make this useful rather than decorative. First, before.digest and reverted.digest should match, because the same failure should recur when the fix is removed; a mismatch usually means the environment is not deterministic. Second, the receipt is small enough to paste into a PR comment, which keeps the evidence next to the discussion instead of in a linked log that expires.

Step 3: let a free model review the receipt, not the patch

This is where a free model tier earns its place in the loop, and where it should be fenced in. Summarizing a diff, listing branches the test does not exercise, and checking a receipt for internal contradictions are narrow, checkable tasks. Deciding whether a parser change breaks a public contract is not.

Review task	Reasonable for a free model tier	Reason	Human still required
Summarize the diff and the reported symptom	Yes	Text-to-text comparison against the issue	Confirm the summary matches intent
Enumerate untested input branches	Yes	Static reading of conditionals and fixtures	Decide which gaps block the merge
Validate receipt consistency (exit codes, digest equality)	Yes	Mechanical checks with a fixed schema	Nothing, if the script produced it
Judge API or ABI compatibility	No	Requires knowledge of downstream consumers	Always
Security and input-sanitization review	No	Failure modes are adversarial, not textual	Always
License, CLA, and provenance checks	No	Policy, not code	Always

MonkeyCode, an open-source coding assistant project, is one place to run those first three passes: the operator describes free model access and a free server option, and the harness above runs unchanged in either setting. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability, model choices, and any usage limits can change, so read the project's own pages before wiring the workflow into a team process instead of relying on a blog post.

A practical guardrail is to make the model's output an input to the review, never the gate. Ask for a numbered list of missing test cases, then add the ones that matter to the contributor's own test file and re-run the script. The receipt ends up in the PR, the model's prose does not.

Step 4: hand over a reproducer, not a description

Maintainers triage on small screens and short windows, and rebuilding a contributor's environment is the most expensive part of the job. A free server option changes the handoff: run the receipt script there and publish only the receipt, so the reviewer fetches evidence without installing anything.

# on the server instance, with the same variables as before
./repro-receipt.sh > receipt.json
python3 -m http.server 8080 --directory .   # reviewer fetches /receipt.json

Two cautions apply. Do not expose an unpatched checkout on a public interface, because the bug may be exploitable, and do not treat a hosted run as proof for other platforms, since a Linux container says nothing about a Windows-only failure. Publish the receipt, keep the working tree private, and delete the instance when the discussion closes.

Limits and who should skip this

The workflow assumes deterministic failures, and it degrades badly elsewhere. Flaky tests, timing-sensitive races, and bugs that need proprietary services or specific hardware will produce digests that never match, which turns the receipt into noise.

Nondeterministic bugs: use repetition counts and report a failure rate instead of a single exit code.
Security patches: keep the reproduction private until a coordinated disclosure date.
Cross-platform claims: one hosted run proves one platform; say so in the PR.
Very large builds: a fresh clone per run can cost more than the review saves, so cache the dependency layer and pin it.
Projects with strict provenance rules: if the contribution policy requires signed artifacts or a specific build toolchain, a free server run is supplementary evidence rather than a substitute.

Teams that already require a failing test in CI will find most of this redundant, and that is fine; the value here is for external contributors who cannot see the maintainer's pipeline. The point is not to add ceremony to healthy projects, but to make an unfamiliar fix checkable by someone who has never seen the codebase.

A checklist to paste into the pull request

Clean clone at an explicit commit hash, not a branch name.
Command that fails before the patch, with its exit code recorded.
Patch applied with git apply --check first, then the same command passing.
Patch removed, and the same failure returning with a matching digest.
Receipt pasted into the PR body, with a note on the platform it ran on.
Test gaps listed, with the blocking ones added to the contributor's test file.

If you want to run the model-assisted passes on a free tier, the harness above is a reasonable first script to try, and it is worth checking the project's current documentation for what the free options actually include today.

Clean Merge, Wrong Patch: Re-Validating an OSS Contribution After Upstream Moves

Blake Yang — Mon, 26 Oct 2026 11:22:54 +0000

A contributor opens a parser fix, waits eleven days for review, and watches upstream refactor the very function the patch touches. The rebase produces no conflict, the test suite stays green, and the second review round takes four minutes. The merged code is still wrong, because the assertion was pinned to a string literal that the refactor had moved into a different module.

That scenario is a composite of a well-known failure mode rather than a report from a specific repository, and every path below is a placeholder. The interesting part is not the bug; it is the review evidence that silently expired between the first approval and the merge. A patch can be textually current and semantically stale at the same time, and Git will not tell the difference.

Why a clean rebase proves nothing

git rebase answers a question about text, not about intent. When two changes touch disjoint lines, the tool applies both and reports success, even if the second change assumed a contract the first one removed. Reviewers then re-read the diff, recognise the shape they approved last week, and skim.

The practical consequence is that review effort must be re-allocated after upstream moves. Instead of re-reading the patch, re-capture the evidence the patch was supposed to produce: the failing reproduction, the passing assertion, and the reason the assertion still targets the original defect. Only the third item is expensive, and it is the only one a rebase can quietly invalidate.

Store the base commit before the first review

Evidence expires because nobody recorded what it was measured against. A short brief file, committed on the contributor branch, fixes that with three fields and no tooling investment.

# .pr-brief.yml — committed on the contributor branch
issue: "#4121 parser drops trailing empty field"
base_commit: 9f3c1ab          # origin/main at the time evidence was captured
repro: "python -m pytest tests/test_parser.py::test_trailing_empty -q"
assertion_target: "parse() return value, not the log line"

The base_commit field is the whole trick. It converts "the tests pass" into "the tests pass against this tree", which is a claim that can be re-checked later. Keep the file small; a brief that needs maintenance gets deleted in the first review round.

Re-apply the patch and re-capture evidence

The script below rebases onto the current upstream head using the recorded base, then re-runs exactly the command from the brief. It refuses to continue when the rebase conflicts, because a conflict in the touched file means the reproduction must be re-derived first.

#!/usr/bin/env bash
# rebase-evidence.sh — re-apply a reviewed patch onto a moved upstream
set -euo pipefail

BRANCH="${BRANCH:?set BRANCH}"        # e.g. review/parser-trailing-field
TEST_CMD="${TEST_CMD:?set TEST_CMD}"

git fetch origin main --quiet
NEW_BASE="$(git rev-parse origin/main)"
OLD_BASE="$(sed -n 's/^base_commit: *//p' .pr-brief.yml)"

echo "old base: $OLD_BASE"
echo "new base: $NEW_BASE"

git switch -c drift/retest "$BRANCH"
if ! git rebase --onto "$NEW_BASE" "$OLD_BASE" drift/retest; then
  echo "CONFLICT: re-derive the reproduction before re-running tests" >&2
  exit 2
fi

$TEST_CMD 2>&1 | tee "drift-after-$(git rev-parse --short "$NEW_BASE").log"

A green log from this script means less than it appears to. The next step decides whether the evidence is still about the same defect.

Classify the drift instead of re-reading the diff

Four outcomes cover almost every re-application, and each one maps to a different amount of human attention. The table is deliberately ordered by risk, not by frequency.

Signal after re-application	Interpretation	Action
Conflict inside a patch hunk	Upstream rewrote the same lines	Re-derive the reproduction; do not resolve mechanically
Clean rebase, test now fails	Test rot or a genuine upstream regression	Bisect before touching the patch
Clean rebase, upstream hunks overlap patch hunks, test passes	Highest risk: intent may have moved	Read the upstream hunk, not the patch
Clean rebase, no hunk overlap, coverage on touched lines unchanged	Low risk	Re-approve with the new log attached

The third row is the one reviewers miss, and it is mechanical to detect. Extract the hunk ranges the patch changed and the hunk ranges upstream changed since the recorded base, then compare them numerically.

# hunks the patch owns
 git diff -U0 "$OLD_BASE"..."$BRANCH" -- src/parser.py \
  | grep -E '^@@' | sed -E 's/^@@ -([0-9]+)(,([0-9]+))? .*/\1 \3/' > patch.hunks

# hunks upstream changed since the last review
 git diff -U0 "$OLD_BASE".."$NEW_BASE" -- src/parser.py \
  | grep -E '^@@' | sed -E 's/^@@ -([0-9]+)(,([0-9]+))? .*/\1 \3/' > upstream.hunks

# report any upstream hunk whose range intersects a patch hunk
awk 'NR==FNR{s[NR]=$1; c[NR]=($2==""?1:$2); n=NR; next}
     {for(i=1;i<=n;i++){e=s[i]+c[i]-1;
       if($1<=e && $1+($2==""?1:$2)-1>=s[i]){print "overlap:",$1,$2; break}}}' \
    patch.hunks upstream.hunks

The output is not a verdict; it is a pointer to the two or three files where a human must read the upstream change before re-approving. Everything else can be re-approved with the fresh log attached to the brief.

Where narrow free-model tasks fit in the loop

Re-validation is repetitive and low-judgement for most files, which makes it a reasonable place to use free model access rather than paid capacity. The operator of MonkeyCode describes both free model access and a free server option in the product; only the first is relevant to this section.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three tasks survive contact with real review, and each one keeps the context window small on purpose:

Summarise a single upstream hunk. Paste the hunk and the issue text only, and ask what behaviour changed. A model that cannot see the rest of the repository cannot invent call sites, which is the failure mode that makes whole-repo prompts unreliable.
Check whether the assertion still names the defect. Paste the test function and the issue description, then ask which observable behaviour is asserted. The answer is cheap to verify by reading two paragraphs.
Propose three edge cases the test omits. Treat the output as a list of candidates for a human to convert into tests, never as tests to append directly.

Two rules keep this honest. Every model claim must quote text that was pasted into the prompt, and every proposal must be labelled as a proposal until a human has run it. A model is useful here as a fast reader of a fifty-line hunk; it is not a substitute for reading the upstream change yourself.

Run the matrix in a disposable environment

Re-validation is cheap to describe and annoying to run, because it needs a clean checkout, the recorded base commit, and a test run per scenario. Running it on a laptop leaves evidence scattered across shell history and stale branches.

The disposable alternative is to script the whole re-application and run it in a fresh environment per patch, so the log files become the artefact rather than a memory of what happened. A free server option is enough for this class of job, because the workload is a clone, a rebase, and one targeted test command per patch. Do not set expectations about runtime or concurrency for a shared free tier; size the matrix to a handful of patches and check the logs in afterwards.

# one environment per patch, same script, log archived with the base commit
for pr in review/*; do
  [ -d "$pr" ] || continue
  BRANCH="$pr" TEST_CMD="python -m pytest tests/test_parser.py -q" \
    ./rebase-evidence.sh || echo "manual: $pr"
done

Limitations and who should skip this

The loop assumes a patch with a stable, recorded base commit and a test command that finishes quickly. It does not catch intent drift in untouched callers, so a refactor that changes a shared helper will still slip through unless the touched-file check is widened deliberately.

Anyone merging within hours of a review can ignore most of this, because the window for drift is too small to justify the bookkeeping. The same applies to patches over generated code, where hunk ranges carry little meaning. Finally, a free tier is a convenience, not a contract: if re-validation becomes a release gate, give it an owner and a paid fallback rather than depending on a shared free resource.

If you already keep a brief per patch, the only new habit is re-running rebase-evidence.sh after any upstream fetch and attaching the new log before re-approving. That single step converts a skimmed re-approval into a checkable claim about a specific tree.

Two Reviewers, One Patch: Treating Model Disagreement as a Triage Signal

Blake Yang — Tue, 20 Oct 2026 09:26:28 +0000

A maintainer opens a pull request that touches eleven files under src/net/, then asks two free coding models for a review. Reviewer A reports a null dereference at line 214; reviewer B reports the same function, a different line, and a leaked file handle. The patch is small enough to read in twenty minutes, but the two reports share no anchors, and the maintainer now has to decide which report deserves the next hour.

That situation is common in open-source review, and the usual response is to trust whichever model sounds more confident. A better response is to measure where the two reviewers disagree, because disagreement is cheap to compute and it concentrates human attention on the parts of a patch that are genuinely ambiguous. This article describes a four-step loop that produces that measurement, along with two small scripts that do the mechanical work.

Why a single reviewer inflates confidence

One model producing twelve findings looks like thorough coverage, but the output rarely distinguishes between a verified defect and a restatement of the diff. A reviewer that walks a hunk and narrates it will always produce findings, and the count says nothing about correctness. The failure modes repeat across providers:

Findings anchored to line numbers that do not exist in the post-image, which makes verification slower than reading the file.
Style observations mixed with defect claims in one flat list, so a missing error check ranks beside a naming preference.
Silent truncation on large diffs, where later files receive no attention at all but the output does not say so.
Confident restatement, where the review describes what the patch does and calls that description a risk.

A second reviewer does not eliminate those failures, but the intersection of two independent runs filters a useful share of them. Phantom line numbers rarely agree by accident, and restatement noise rarely lands on the same hunk twice.

The four-step loop

Build a bounded, line-anchored packet from the patch so both reviewers see identical material.
Restrict findings to a small, fixed vocabulary so that matching is meaningful.
Bucket the two finding sets into consensus, contested, and singleton anchors.
Spend human reading time by bucket, with a time box attached to each.

Steps one and three are mechanical, and the two scripts below cover them. Step two is a convention that has to be agreed before the first run, and step four is the part that actually saves hours.

Step 1: a bounded, line-anchored packet

Feeding a raw diff to a reviewer invites anchor drift, because hunk headers are hard to map back to source. The packet builder below reads a unified diff, extracts the changed line ranges per file, and emits the full patch plus numbered excerpts around each range. A character budget keeps the packet small enough to stay inside whatever context the reviewer accepts.

#!/usr/bin/env python3
# review_packet.py - build a bounded, line-anchored review packet from a unified diff
import pathlib
import re
import sys

HUNK = re.compile(r'^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@')
CHAR_BUDGET = 24000
CONTEXT = 6


def changed_ranges(diff_text):
    files, current = {}, None
    for line in diff_text.splitlines():
        if line.startswith('+++ b/'):
            current = line[6:]
            files.setdefault(current, [])
        elif line.startswith('@@') and current:
            match = HUNK.match(line)
            if match:
                start = int(match.group(1))
                count = int(match.group(2) or 1)
                files[current].append((start, start + max(count - 1, 0)))
    return files


def excerpt(root, path, ranges):
    target = root / path
    if not target.exists():
        return ['(new file: no post-image to excerpt)']
    lines = target.read_text(encoding='utf-8', errors='replace').splitlines()
    keep = set()
    for start, end in ranges:
        keep.update(range(max(1, start - CONTEXT), min(len(lines), end + CONTEXT) + 1))
    out, previous = [], None
    for number in sorted(keep):
        if previous is not None and number != previous + 1:
            out.append('...')
        out.append(f'{number:>5} | {lines[number - 1]}')
        previous = number
    return out


def main(diff_path, repo_root):
    diff_text = pathlib.Path(diff_path).read_text(encoding='utf-8')
    root = pathlib.Path(repo_root)
    parts = ['## Patch (verbatim)', '```

diff', diff_text.strip(), '

```', '']
    budget = CHAR_BUDGET - sum(len(p) for p in parts)
    for path, ranges in changed_ranges(diff_text).items():
        block = '\n'.join(
            [f'## {path} (changed lines: {ranges})', '```

', *excerpt(root, path, ranges), '

```', '']
        )
        if len(block) > budget:
            parts.append(f'## {path}\n(omitted: exceeds remaining packet budget)')
            continue
        budget -= len(block)
        parts.append(block)
    sys.stdout.write('\n'.join(parts))


if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

The invocation is deliberately boring, because it has to be repeatable on a branch that changes hourly:

git fetch origin main
git diff origin/main...HEAD > /tmp/patch.diff
python3 review_packet.py /tmp/patch.diff . > /tmp/packet.md
wc -c /tmp/packet.md

The omitted-file marker matters more than it looks. When a packet is truncated, the reviewer should be told which files it never saw, so that a thin review of file nine is not read as a clean bill of health. Verify the packet on a patch with a renamed file before trusting it, since rename headers do not always produce a post-image path the script can open.

Step 2: a fixed finding vocabulary

Matching two reviews requires a shared label set, and a short one is enough. The taxonomy below covers most defects in small patches while keeping style out of the comparison:

bug for wrong behaviour on a reachable path.
error-handling for ignored or narrowed failures.
resource for leaks, missing cleanup, unbounded growth.
concurrency for shared state and ordering assumptions.
test-gap for a changed branch no test exercises.
perf for work that is quadratic or repeated unnecessarily.

Style and naming comments are excluded on purpose. They rarely agree across runs, they inflate singleton counts, and a maintainer reading a style singleton learns nothing about the patch.

Step 3: bucket the disagreement

The bucket script reads two JSON arrays of findings with file, line, and category fields, then matches anchors on the same file and category within a three-line window. Everything unmatched becomes a singleton attributed to one side, which is the set worth reading with fresh eyes.

#!/usr/bin/env python3
# bucket.py - split two reviewers' findings into consensus, contested, and singletons
import json
import sys

WINDOW = 3


def load(path):
    with open(path, encoding='utf-8') as handle:
        return {(row['file'], int(row['line']), row['category'].strip().lower())
                for row in json.load(handle)}


def bucket(left, right):
    pairs, taken = [], set()
    ordered = sorted(right)
    for finding in sorted(left):
        for index, candidate in enumerate(ordered):
            if index in taken:
                continue
            same_anchor = finding[0] == candidate[0] and finding[2] == candidate[2]
            if same_anchor and abs(finding[1] - candidate[1]) <= WINDOW:
                taken.add(index)
                pairs.append((finding, candidate))
                break
    left_only = [f for f in sorted(left) if all(f != pair[0] for pair in pairs)]
    right_only = [f for i, f in enumerate(ordered) if i not in taken]
    return pairs, left_only, right_only


def main(left_path, right_path):
    pairs, left_only, right_only = bucket(load(left_path), load(right_path))
    for first, second in pairs:
        print(f'CONSENSUS    {first[0]}:{first[1]} ~ {second[0]}:{second[1]}  [{first[2]}]')
    for finding in left_only:
        print(f'SINGLETON-A  {finding[0]}:{finding[1]}  [{finding[2]}]')
    for finding in right_only:
        print(f'SINGLETON-B  {finding[0]}:{finding[1]}  [{finding[2]}]')
    total = len(pairs) + len(left_only) + len(right_only)
    print(f'\n{len(pairs)} consensus of {total} anchors')


if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

For two finding sets where both reviewers flagged src/net/conn.c as error-handling six lines apart, plus one private finding each, the output takes this shape:

CONSENSUS    src/net/conn.c:214 ~ src/net/conn.c:220  [error-handling]
SINGLETON-A  src/net/pool.c:88  [resource]
SINGLETON-B  src/net/conn.c:301  [test-gap]

1 consensus of 3 anchors

The example above is illustrative rather than a production run, so compare the printed anchors against the packet before acting on them. A consensus count of one out of three is the useful signal here, not the individual findings.

Step 4: spend attention by bucket

Bucket	Reading	Action	Time box
Consensus	Two independent runs landed on the same file, category, and neighbourhood	Open the code, write a failing test, then decide whether the finding survives it	10 minutes
Contested	Same file, different category, or anchors more than three lines apart	Treat it as a specification question and read the surrounding function before the finding	15 minutes
Singleton	Only one side raised it, often outside a changed hunk	Check the anchor against the packet; drop anything outside a changed range immediately	5 minutes
Empty bucket	Neither reviewer flagged a changed hunk with a test	Add the test yourself before approving	15 minutes

The time boxes are the point of the exercise. A bucket report that does not change how long a maintainer spends on a patch has produced nothing but extra output.

Where free access changes the economics

Running two reviewers per patch only works if the second run costs nothing to authorize, which is where free model access matters. MonkeyCode offers free model access and a free server option, which makes the two-reviewer pass a default rather than a budget conversation. Pick two models available on the free tier and keep the pairing stable across patches, because rotating models turns the bucket counts into noise. The disclosure for this article is that it was prepared as part of MonkeyCode's product outreach, and the scripts above are provider-neutral and run without it.

The free server option is a reasonable home for the harness if a team wants each packet to be reproducible by the next reviewer. A checkout, one diff, and two JSON files need very little, and keeping the run off a laptop makes the packet addressable later. Confirm the current limits before scheduling anything against it, and keep the packet and bucket scripts in the repository either way, since they are the durable part.

Limitations and who should skip this

The loop is a triage tool, not evidence. Neither bucket report proves a defect exists, and consensus between two models that share training data is weaker than consensus between a model and a human. Patches that touch authentication, cryptography, or memory ownership need a human reviewer with domain context regardless of what the buckets say. The scripts assume a unified diff with standard +++ b/ headers, so patches from unusual diff drivers or binary-heavy changes will produce a thin packet.

Teams without a fixed finding vocabulary should skip step three entirely and run one reviewer with a careful reading pass. Anyone who cannot read the patched code well enough to judge a finding should also skip the loop, because a bucket report consumed without code reading is a more organized form of guessing. Small patches under roughly fifty changed lines usually do not repay the setup, and a single careful review is faster.

Closing

The reusable asset here is the bucket report, which turns two opinions into a short list of places where the patch is actually contested. Teams that already review patches this way can add the second reviewer through free model access and keep the packet and bucket scripts under version control. The loop only pays off when the report changes the order in which a maintainer reads the diff.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Mutant Run: Catching Weakened Tests in an OSS Patch With Three Git Worktrees

Blake Yang — Mon, 19 Oct 2026 08:10:10 +0000

A patch lands on a small utility library. It adds the missing test for a paging helper, fixes a ceiling-division bug, and, in the same commit, rewrites two older assertions into looser >= comparisons. CI is green, the diff reads cleanly, and the review takes ninety seconds. Six weeks later, a pagination regression slips through the exact code path those softened assertions used to guard.

That scenario is a composite, not a specific incident, and it can be rebuilt locally in about ten minutes. Everything below runs against a four-file fixture repository that any reader can construct with the commands shown. The goal is a mechanical check that a patch's tests actually observe the patch's behavior, instead of merely coexisting with it.

Why a green suite proves less than it appears

A passing suite answers one narrow question: the assertions that exist did not fail. It says nothing about whether those assertions are strict enough, or whether the new test would also pass on the unfixed code. Reviewers usually compensate with attention, which is exactly the resource that runs out on a Friday afternoon.

Three cheap signals restore most of the missing information. The first is whether the new test fails on the base commit. The second is whether the suite fails when the fix is removed but the tests are kept. The third is how many assertion lines the patch deletes or relaxes.

The three states every patch should visit

Each state is a detached Git worktree, so no branch juggling and no stash risk is involved:

Base state — upstream tip plus the contributor's test changes only. The new test must fail here, and it must fail for the intended reason.
Patched state — the contributor's full diff applied. The suite must pass here.
Mutant state — the patched tree with src/ reverted to base and the tests left alone. The suite must fail here, or the tests do not observe the fix at all.

State three is the one most review workflows skip, and it is the only state that detects a test written to describe the fix rather than to constrain it.

Step 1: build the fixture repository

The fixture is deliberately small: one module, one test file, one bug. The bug is a missing ceiling division that returns zero pages for any total smaller than the page size.

mkdir tamper-demo && cd tamper-demo
git init -q
mkdir -p src tests bin

src/paging.py on the base commit:

def pages(total, size):
    return total // size

tests/test_paging.py on the base commit:

from src.paging import pages


def test_exact_multiple():
    assert pages(10, 5) == 2


def test_large_total():
    assert pages(11, 5) == 2


def test_zero_total():
    assert pages(0, 5) == 0

Commit that as main, then create a patch branch that fixes the division, adds a fourth test, and weakens two existing assertions. The added test uses pages(10, 5) == 2, which passes on the buggy code, so it captures nothing.

Step 2: the three-state script

The script below assumes pytest is installed and that src/ is the only directory carrying the fix. Both assumptions are adjustable, and the limitations section returns to them.

#!/usr/bin/env bash
# bin/three-state-check.sh <base-ref> <patch-ref>
set -euo pipefail

BASE_REF="${1:-origin/main}"
PATCH_REF="${2:-HEAD}"
PY="${PY:-python3}"
WORK="$(mktemp -d)"
LOG="$WORK/report.txt"
status=0

cleanup() {
  for d in base patched mutant; do
    git worktree remove --force "$WORK/$d" 2>/dev/null || true
  done
}
trap cleanup EXIT

note() { printf '%s\n' "$*" | tee -a "$LOG"; }

suite() { ( cd "$1" && "$PY" -m pytest -q 2>&1 | tail -n 25 ); }

git worktree add -q --detach "$WORK/base"    "$BASE_REF"
git worktree add -q --detach "$WORK/patched" "$PATCH_REF"
git worktree add -q --detach "$WORK/mutant"  "$PATCH_REF"
git -C "$WORK/mutant" checkout -q "$BASE_REF" -- src/

note "== state 1: base ($BASE_REF) =="
base_out="$(suite "$WORK/base")"
printf '%s\n' "$base_out" | tee -a "$LOG"
if printf '%s' "$base_out" | grep -qE '[0-9]+ passed' && \
   ! printf '%s' "$base_out" | grep -qE '[0-9]+ failed'; then
  note "WARN base is green; the failing reproduction is missing"
fi

note "== state 2: patched ($PATCH_REF) =="
patched_out="$(suite "$WORK/patched")"
printf '%s\n' "$patched_out" | tee -a "$LOG"
if printf '%s' "$patched_out" | grep -qE '[0-9]+ failed'; then
  note "FAIL the patch does not make the suite pass"
  status=1
fi

note "== state 3: mutant (fix reverted, tests kept) =="
mutant_out="$(suite "$WORK/mutant")"
printf '%s\n' "$mutant_out" | tee -a "$LOG"
if ! printf '%s' "$mutant_out" | grep -qE '[0-9]+ failed'; then
  note "FAIL the suite passes without the fix; the tests observe nothing"
  status=1
fi

note "== assertion churn =="
added="$(git diff -U0 "$BASE_REF" "$PATCH_REF" -- 'tests/' \
  | grep -E '^\+[^+]' | grep -c 'assert' || true)"
removed="$(git diff -U0 "$BASE_REF" "$PATCH_REF" -- 'tests/' \
  | grep -E '^-[^-]' | grep -c 'assert' || true)"
note "assertions added=$added removed=$removed"
if [ "${removed:-0}" -gt 0 ]; then
  note "REVIEW the test diff first: $removed assertion line(s) removed"
fi

exit "$status"

Step 3: read the output like a reviewer

Running bin/three-state-check.sh main patch on the fixture produces a report close to the following. The exact pytest counts vary with the fixture, and the verdict lines do not.

== state 1: base (main) ==
1 failed, 3 passed in 0.11s
== state 2: patched (patch) ==
4 passed in 0.09s
== state 3: mutant (fix reverted, tests kept) ==
4 passed in 0.09s
FAIL the suite passes without the fix; the tests observe nothing
== assertion churn ==
assertions added=1 removed=2
REVIEW the test diff first: 2 assertion line(s) removed

Two independent signals point at the same weakness. The mutant run proves the new test cannot distinguish fixed from broken code, and the churn count shows that two earlier guarantees were softened. Neither signal is conclusive on its own, but together they justify a request for changes instead of an approval.

The failure message matters as much as the failure itself. A mutant run that fails on an import error has proved nothing about the assertion, so the log should be checked for AssertionError or the named test before the result counts.

Where free models genuinely help, and where they do not

A free model is useful for the parts of this loop that are text-shaped and low-risk, and it is a poor substitute for the maintainer's judgment. The division of labor that has held up in practice looks like this:

Model-assisted: rewriting a failing mutant log into a two-sentence PR comment, drafting the missing test from the diff, naming the observable behavior a patch changes, and proposing which existing assertions a refactor legitimately touches.
Maintainer-only: deciding whether a weakened assertion is justified, judging whether a suite is flaky rather than wrong, and approving the merge.

A prompt skeleton that keeps the model inside those bounds:

You are given: a unified diff, the base test log, the patched test log,
and the mutant test log. Do not propose a merge decision.
1. List the observable behaviors the diff changes.
2. For each new test, state whether it would fail without the diff.
3. Propose one additional test that would fail in the mutant state.
4. Mark anything you cannot tell from the logs as unknown.

Free model access through MonkeyCode covers the drafting steps above without a paid key, and the free server option gives the three worktrees a disposable host instead of the reviewer's laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Capacity and terms for both the free model access and the free server option are operator-supplied and can change, so check the currently published terms before depending on them in a team workflow.

The disposable host is a real convenience rather than a marketing point. Three worktrees plus a virtualenv plus a test cache accumulate quickly, and a host that can be destroyed after the report is written makes "start from a clean state" the default rather than a chore. Running the script there and copying only the report log back keeps the review artifact small and auditable.

git clone --quiet https://example.invalid/your-fork.git verify
cd verify && git fetch --quiet origin pull/1234/head:pr-1234
PY=python3 bin/three-state-check.sh origin/main pr-1234 > report.txt 2>&1 || true

A decision table for patch triage

The table converts the script output into a repeatable first response, which matters more than any single verdict.

Signal	Merge	Ask the author	Block
New test fails at base, passes patched, fails mutant	yes	—	—
Mutant state passes	—	—	yes
Assertion lines removed	—	yes, with a written reason	—
Test files exceed 30% of the diff	—	yes	—
Base state already green	—	yes, reproduction missing	—

Limitations and failure modes

The mutant run is a heuristic, and it degrades in predictable ways that reviewers should anticipate:

Reverting src/ can break imports, so the mutant failure may originate outside the assertion under test.
Suites that assert on mocks or log strings often stay green in the mutant state even when behavior changed.
Flaky tests make state three noisy, and a retry policy is needed before the signal is trusted.
Large monorepos need affected-test selection, which reduces the mutant run's coverage to the selected subset.
Patches that legitimately rename or restructure tests will trip the churn counter and require manual reading.

Who should skip this workflow

Repositories without a working test command should not start here, because the script only amplifies a suite that already exists. Teams that treat tests as documentation rather than constraints will find state three permanently red and will disable it within a week. Maintainers reviewing one small patch a month will get more value from reading the test diff carefully than from maintaining three worktrees and a shell script.

The workflow earns its keep on repositories where patch volume outpaces reviewer attention, and where a weakened assertion is expensive to discover late. Three worktrees, one script, and a log file are enough to make the check routine, and readers who want a throwaway host for the run can trial MonkeyCode's free server option on a scratch fork and compare the report against their local output.

chmod +x bin/three-state-check.sh
./bin/three-state-check.sh main patch; echo "exit=$?"

Anchor-Checked Review: A Mechanical Gate for Model Findings on OSS Patches

Blake Yang — Sun, 18 Oct 2026 06:56:16 +0000

A maintainer opens a pull request that replaces a hand-rolled retry loop with a small helper module, then asks a free model for a review before reading the diff closely. The model returns eleven findings: an off-by-one index, two race conditions, a missing null check, a leaked file handle, and a remark about a function that does not exist anywhere in the repository. Six of the findings point at lines the patch never touched, because the off-by-one is reported three hundred lines away from the only hunk in the file.

The uncomfortable part is not that the model was wrong. The uncomfortable part is that nothing in the loop could separate a claim about the patch from a claim about the file's general reputation. The maintainer spent forty minutes on findings that had no anchor, and the two real defects in the patch were found later by a human reviewer on a second pass.

Anchor drift is the actual cost of cheap review

Low-cost or free model access changes the economics of review, not the epistemics of it. When a review costs nothing, contributors run it on every patch, and the volume of unanchored prose grows faster than human attention. Three failure modes dominate in practice.

Line drift: findings cite a file and a line that exist, but sit outside every changed hunk, so the comment is about pre-existing code presented as a patch defect.
Phantom symbols: the model names a helper or config key that was never in the diff, usually reconstructed from the function name rather than the packet.
Evidence-free severity: a finding arrives labelled high with no command output, no quoted line, and no reproduction, which makes it impossible to accept or reject on technical grounds.

A workable loop needs one mechanical rule: every finding must resolve to a real line inside the diff, and must carry evidence that a script can check without judgement. Everything else is commentary.

The artifact: a review packet with verifiable anchors

The artifact below is deliberately small. It consists of a contract file, a packet built from git diff, a JSON output schema, and a checker that rejects findings whose anchors do not resolve. The examples are illustrative and were not executed against a specific upstream repository.

Step 1 — Freeze what the patch must prove

Start with a contract, committed next to the patch or kept in the review branch. It states the claim, the command that demonstrates it, and the invariants the reviewer may reason about.

{
  "claim": "the retry helper preserves a 3-attempt cap and closes the body exactly once",
  "reproduce": ["python -m pytest tests/test_retry.py -q"],
  "invariants": ["at most 3 attempts", "body closed exactly once on failure"],
  "out_of_scope": ["logging format", "metrics naming"]
}

The out_of_scope field matters more than it looks. It gives the reviewer a licensed place to put style opinions, and it gives the checker a category for findings that are legitimate but not patch defects.

Step 2 — Build a packet small enough to stay anchored

set -eu
mkdir -p review
git fetch origin pull/1234/head:pr-1234
git diff --unified=6 origin/main...pr-1234 -- '*.py' > review/diff.patch
git log --format='%h %s' origin/main..pr-1234 > review/commits.txt
wc -l review/diff.patch

The unified context is set to six lines on purpose. That window is wide enough for a reviewer to see surrounding control flow, and narrow enough that anything outside it is visibly a separate concern. Sending the whole repository makes phantom findings more likely, because the model can mine unrelated files for plausible defects.

Step 3 — Contract the answer as JSON with mandatory anchors

You are reviewing a patch. Use only review/diff.patch and review/contract.json.
Return JSON only, no prose before or after:
{
  "findings": [
    {
      "file": "<path exactly as it appears in the diff>",
      "line": <integer, new-side line number inside a hunk>,
      "severity": "high" | "medium" | "low",
      "claim": "<one sentence about this patch only>",
      "evidence": "<quoted diff line or command output>",
      "confidence": "low" | "medium" | "high"
    }
  ]
}
Rules:
1. file must appear in the diff; line must be inside a hunk.
2. evidence must quote text that appears in the packet.
3. If a claim cannot be anchored, omit it and do not guess.
4. Do not restate the contract as a finding.

This is where free model access becomes useful rather than noisy. MonkeyCode is described by its maintainers as an open-source project offering free model access and a free server option, and the operator states that the free tier provides 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. No model names, rate limits, or hardware details are assumed here; verify current terms yourself before relying on them.

Step 4 — Reject findings that do not resolve to the diff

The checker is the whole point. It uses only the standard library, reads the packet diff, and rewrites the model's output into an accepted list and a rejected list with a reason per finding.

#!/usr/bin/env python3
"""anchor_check.py — reject model findings that do not resolve to the reviewed diff.

Usage: anchor_check.py findings.json [allow_context]
Reads: review/diff.patch (built in step 2)
"""
import json, re, sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")


def anchored_lines(diff_text):
    """Map path -> set of new-side line numbers visible in the diff (context + added)."""
    touched, path, new_line = {}, None, None
    for raw in diff_text.splitlines():
        if raw.startswith("diff --git "):
            path, new_line = None, None          # reset per file
            continue
        if new_line is None and raw.startswith("+++ b/"):
            path = raw[6:].split("\t")[0]
            touched.setdefault(path, set())
            continue
        m = HUNK.match(raw)
        if m:
            new_line = int(m.group(1))
            continue
        if path is None or new_line is None:
            continue
        if raw.startswith("+") or raw.startswith(" "):
            touched[path].add(new_line)
            new_line += 1
        elif raw.startswith("-"):
            pass                                 # deletion: no new-side line
    return touched


def verdict(items, touched, allow_context=0):
    accepted, rejected = [], []
    for item in items:
        path, line = item.get("file"), item.get("line")
        if path not in touched:
            reason = "file not in diff"
        elif not isinstance(line, int):
            reason = "line is not an integer"
        elif allow_context == 0 and line not in touched[path]:
            reason = "line outside visible hunk lines"
        elif allow_context and not any(abs(line - t) <= allow_context for t in touched[path]):
            reason = "line beyond allowed context window"
        elif not str(item.get("evidence", "")).strip():
            reason = "no evidence field"
        else:
            reason = None
        (accepted if reason is None else rejected).append(
            item if reason is None else {**item, "reject_reason": reason}
        )
    return accepted, rejected


def main():
    findings = json.load(open(sys.argv[1], encoding="utf-8"))["findings"]
    allow = int(sys.argv[2]) if len(sys.argv) > 2 else 0
    touched = anchored_lines(open("review/diff.patch", encoding="utf-8").read())
    accepted, rejected = verdict(findings, touched, allow)
    report = {
        "accepted": accepted,
        "rejected": rejected,
        "reject_ratio": round(len(rejected) / max(1, len(findings)), 3),
    }
    print(json.dumps(report, indent=2, ensure_ascii=False))
    sys.exit(0 if accepted else 1)               # non-zero when nothing survives


if __name__ == "__main__":
    main()

python review/anchor_check.py review/findings.json 0 | tee review/verdict.json
jq '.rejected[] | {file, line, reject_reason}' review/verdict.json

The exit code is the enforcement mechanism. A CI job that runs this after the model call turns review into a gate rather than a suggestion, and the jq line gives the maintainer a one-screen list of what the model tried to assert without support.

Read the reject ratio, not the finding count

One number is worth tracking per patch: the share of findings that survive the anchor check. A ratio near zero means the packet is too small to review meaningfully, or the prompt is inviting speculation. A ratio near one across many patches usually means the checker is too permissive and the context window has swallowed the whole file.

Ratio below 0.3: shrink the question set, add the contract's out_of_scope list, and stop sending unrelated files.
Ratio above 0.9 with high severity everywhere: enforce a cap on findings per file and require a quoted evidence line for each one.
Accepted findings that a human later dismisses: record the dismissal reason next to the anchor, so the prompt contract learns the repository's actual conventions.

None of these metrics measure model quality in the abstract. They measure whether the review output can be acted on, which is the only property a maintainer can verify cheaply.

Where a free server option fits in the loop

Anchor checking, pytest, and lint runs are deterministic and cheap, which makes them reasonable candidates for a small always-on environment rather than a laptop. If the free server option can host the repository's toolchain and reach the git remote, the sequence fetch → diff → test → anchor_check can run there on a schedule, and the maintainer only reads verdict.json plus a green test log.

Verify the runtime, persistence, and network reachability yourself before depending on it, because a resolver failure inside that environment looks exactly like a legitimate anchor rejection from the outside. A one-line sanity check at the start of the job prevents that confusion.

python review/anchor_check.py review/selftest_findings.json 0 >/dev/null \
  && echo "checker ok" || { echo "checker misconfigured"; exit 2; }

Planning a 10-million-token allowance

The allowance is rarely the bottleneck once packets are small. The table below is a planning model, not a product fact: every token figure is an assumption to be replaced with measured numbers from your own runs.

Item	Assumption	Monthly total
Model calls per patch	3 (triage, review, re-check of rejected)	24 calls at 8 patches
Input tokens per call	~6,000 (contract + diff + rules)	~144,000
Output tokens per call	~1,500 (JSON findings)	~36,000
Total per month	—	~180,000

At that scale a 10-million-token allowance covers many months of eight-patch cadence, which suggests the real constraint is not tokens but the human time spent on accepted findings. Teams that burn through allowances quickly are usually sending whole files instead of packets, or asking for exploration instead of anchored review.

Limitations and who should skip this

The anchor check proves that a finding points at the patch. It proves nothing about whether the claim is correct, and a model can attach a real line number to a wrong diagnosis with no difficulty. The checker also rejects legitimate design-level comments about untouched code, which is why out_of_scope and the review comments should stay in a separate channel.

Renames, deletions, and rebases shift line numbers, so the packet must be rebuilt after any rebase and the verdict regenerated from the new diff. Security review is out of scope here: absence of anchored findings is not evidence of absence of vulnerabilities, and the loop should not be presented as an audit.

Skip this approach if the repository has no runnable reproduction command, if the patch is dominated by moves and deletions, or if policy forbids sending diff content to an external model. In those cases the same contract and anchor rules still help a human reviewer, but the automated gate adds little.

The practical takeaway is narrow: put the evidence requirement in the prompt, verify the anchors with a script, and let the reject ratio decide how much a model contributes to review. A team that wants to try the loop before committing budget can start on the free tier, keep anchor_check.py local, and treat the reject ratio as the deciding signal for whether model review earns a permanent slot in the patch path.

Reproduce, Minimize, Then Review: A Three-Pass Loop for Open-Source Patches

Blake Yang — Sat, 17 Oct 2026 05:57:18 +0000

A pull request arrives with a confident description, three changed files, and a passing CI badge that turns green only on the author's branch. The failing test that motivated the change is nowhere in the diff, and the reviewer is asked to judge correctness from prose alone. This loop is a composite scenario, not a report of one specific repository, but it is common enough that the review order matters more than the review tools.

The loop below reverses the usual habit. Reproduce first, minimize second, and only then ask a model for review, with the model's output treated as a list of claims that must be verified against files and logs. Free model access helps at the last pass, not the first, and that ordering is the whole point.

Why the review order beats the review tool

Most review failures are not reasoning failures; they are context failures. A reviewer who has not reproduced the bug cannot tell which hunks are load-bearing and which are decorative refactors. A model reading the same diff inherits that ambiguity and amplifies it with fluent guesses.

Reproduction produces two artifacts that constrain everything after it: a deterministic command with one exit code, and a captured log. Minimization produces a third: a patch whose hunks can each be tied to a reason. A review request built from those artifacts is short, checkable, and hard to bluff.

Pass 0: pin the reproduction before reading the diff

The goal is a command that fails on the base commit for the stated reason, on a clean clone, with a pinned toolchain. Versions inherited from a workstation make failures unrepeatable, so the script records rather than assumes them.

#!/usr/bin/env bash
# repro.sh - deterministic reproduction of the reported failure
set -euo pipefail

REPO="${REPO:-https://github.com/example/project}"
BASE_SHA="${BASE_SHA:?set BASE_SHA to the commit under test}"
WORK="$(mktemp -d)"

git clone --quiet --no-checkout "$REPO" "$WORK/src"
cd "$WORK/src"
git checkout --quiet "$BASE_SHA"

# Pin everything that can drift between machines.
export PYTHONHASHSEED=0
export SOURCE_DATE_EPOCH=0
python3 -m pip install --quiet --require-hashes -r requirements-test.txt

set +e
python3 -m pytest -x -q tests/test_reported_case.py 2>&1 | tee "$WORK/test.log"
STATUS=${PIPESTATUS[0]}
set -e

echo "exit=${STATUS} base=${BASE_SHA}" | tee "$WORK/repro.result"
exit "$STATUS"

The --require-hashes flag matters because it turns a floating dependency into a recorded one. If the project cannot produce hashed requirements, pin exact versions in a lockfile and commit it alongside the script. The captured test.log becomes the only acceptable evidence for claims made later in the review.

A reproduction is accepted only when all four conditions hold:

It runs from a fresh clone with no local state.
It fails on BASE_SHA with the message quoted in the issue.
It passes on the patched commit, using the same command.
Reverting the patch restores the failure, byte-identical in the assertion line.

Pass 1: minimize the patch and prove each hunk earns its place

A large patch hides its own bugs. Two cheap gates catch most of them before any model sees the diff: a diffstat ceiling agreed with the maintainer, and a revert test.

BASE_SHA=origin/main
git fetch --quiet origin "$BASE_SHA"

git diff --numstat "$BASE_SHA"...HEAD \
  | awk '{add+=$1; del+=$2; printf "%s +%s -%s\n", $3, $1, $2} END {print "total +", add, " -", del}'

# Revert test: the patch must be the reason the test passes.
git diff --binary "$BASE_SHA"...HEAD > /tmp/patch.diff
git checkout --quiet "$BASE_SHA"
git apply -R /tmp/patch.diff
BASE_SHA="$BASE_SHA" ./repro.sh || echo "revert restored the failure, as expected"

When the revert does not restore the failure, one of three things is true: the test never exercised the changed path, the fix lives in an untracked file, or the environment drifted. Each of those is a blocker, and none of them is visible from reading the diff.

Refactors mixed into a fix belong in a separate commit, because they change what the reviewer must hold in mind. Splitting them is not style policing; it is a way to make the revert test meaningful.

Pass 2: build a scoped review brief instead of pasting a diff

A raw diff invites a raw summary. A brief that pairs the diff with the failure lines and the changed-file inventory invites a bounded answer. The script below emits that brief as JSON, which keeps the prompt small and the claims enumerable.

#!/usr/bin/env python3
"""build_brief.py - emit a review brief from patch stats and the test log."""
import json, subprocess, sys
from pathlib import Path

base, log_path = sys.argv[1], Path(sys.argv[2])
test_log = log_path.read_text(errors="replace")

numstat = subprocess.run(
    ["git", "diff", "--numstat", f"{base}...HEAD"],
    capture_output=True, text=True, check=True,
).stdout

files = []
for row in numstat.splitlines():
    added, deleted, path = row.split("\t")
    files.append({"path": path, "added": int(added), "deleted": int(deleted)})

failure_lines = [
    line for line in test_log.splitlines()
    if "FAILED" in line or line.startswith("E ")
][:20]

print(json.dumps({
    "base": base,
    "files_touched": files,
    "failure_lines": failure_lines,
    "questions": [
        "Which hunks are required for the failing assertion to pass?",
        "Which hunks are behaviour-neutral and belong in another commit?",
        "Does any hunk change behaviour outside the failing test?",
    ],
}, indent=2))

The accompanying prompt should impose a citation format, because an answer without locations cannot be audited. A workable instruction set uses four rules: cite path:line for every claim, quote the failing assertion the hunk addresses, mark unsupported statements as UNVERIFIED, and output one row per hunk rather than one paragraph per file.

Auditing the claims that come back

Model output is a hypothesis list. Each row is copied into a claim table and closed only with evidence from the repository or the log.

Claim	Cited location	Evidence	Verdict
Guard clause prevents the null deref	`src/parse.py:212`	`test.log` assertion line matches	accepted
Extra branch preserves old behaviour	`src/parse.py:240`	no test covers the branch	needs test
Helper rename is safe	`src/util.py:8`	grep finds third-party import	rejected

Rows marked UNVERIFIED are deleted rather than debated. A confident sentence with no location costs more reviewer time than the original bug report.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free model access and a free server option, per the operator. In this loop, the free server is the natural host for repro.sh and the review job, because a disposable machine removes the local-state variable that makes reproductions unrepeatable. The free models serve the Pass 2 role: bounded questions over a small brief, with citations required.

Confirm the current terms, limits, and availability before depending on either offer, since free-tier conditions change and this article makes no claim about quotas, hardware, or duration. The workflow above stays valid if the free options disappear tomorrow, which is the test any tooling recommendation should pass. Anyone working through the same loop may find the free model access and free server worth trying against their own repository's reproduction script.

When model review is adequate, and when it is not

Patch type	Free-model review	Human gate
Mechanical rename, formatting	useful	diffstat skim
Test-only change	useful	confirm failure on base
Bug fix with a new assertion	useful with brief	revert test plus one maintainer read
Dependency bump	limited	changelog and lockfile diff
Concurrency or scheduling	not reliable	stress test under load
Auth, crypto, licence, CLA	not reliable	specialist and legal review

The pattern is consistent: the more a patch depends on behaviour that no test captures, the less a text-only reviewer can contribute.

Limitations and who should skip this

Text-only review never executes the patch, so a model cannot confirm that the suite passes or that a race is absent. Long diffs get truncated, and truncation silently removes the hunks most likely to be wrong. Line numbers can drift between the model's reading and the current branch, so citations need re-checking after any rebase.

Teams working on security-sensitive code, unreviewed production changes, or repositories whose contribution rules restrict AI-assisted review should use the reproduction and revert gates while skipping the model pass entirely. Some projects require disclosure when AI assistance is used in preparing a contribution, and that rule overrides convenience.

A checklist to run before the pull request opens

repro.sh fails on the base commit and passes on the patch, same command.
Reverting the patch restores the exact failing assertion.
The diffstat matches the scope agreed in the issue, with refactors split out.
The review brief contains failure lines, file inventory, and three questions.
Every model claim was accepted or rejected through the claim table.
No UNVERIFIED row survives into the final description.

The order is the contribution: a patch that has been reproduced and minimized is already most of the way to reviewable, and a model then works on evidence instead of adjectives.

Route an Incoming OSS Patch by Blast Radius Before a Model Drafts the Review

Blake Yang — Fri, 16 Oct 2026 04:54:22 +0000

Route an Incoming OSS Patch by Blast Radius Before a Model Drafts the Review

A review queue that looks calm and is not

On a Monday morning, the maintainer of a small Python library opens the pull request list and finds eleven items waiting. Four are dependency bumps, three fix docstring typos, two add small features, one only rewrites tests, and one rewrites the argument parser from scratch. The natural instinct is to open the parser rewrite first, because it is the largest and most intimidating diff in the queue.

That instinct is expensive and frequently wrong. A three-line change to a lockfile can break every downstream install, while a two-hundred-line refactor inside a well-covered module may be exactly what the existing tests already describe. Review attention is finite, and it should follow risk rather than diff size.

The workflow below is a composite illustration rather than a report from one named repository. It defines a small routing procedure that assigns each incoming patch a tier, then decides which tiers may receive model-drafted review notes and which must not. Every command in it is ordinary git, so it runs against any clone without extra services.

Three signals that predict review cost

Diffs carry a few cheap, language-agnostic signals that correlate with how much human attention a patch will consume. None of them is a correctness signal, and none of them replaces reading the code.

Signal	Why it raises review cost	Cheap check
Build or dependency files	Changes install resolution, CI images, and lockfile integrity for everyone	Basename match against a known build-file set
Public surface paths	Alters behaviour callers depend on, including undocumented callers	Path prefix match (`src/`, `lib/`, `pkg/`)
No test delta	Removes the fastest way to falsify the claimed behaviour	Marker match (`tests/`, `spec/`, `test_`)
Wide file spread	Suggests a cross-cutting change with more integration edges	Count of files in `--numstat`
Large net change	Longer to read, more places for a silent regression to hide	Sum of added and removed lines

The first three signals are properties of what the patch touches, and the last two are properties of how much it touches. Combining them produces a coarse but useful ordering, and the ordering is what the routing script encodes.

A routing script for a fresh clone

The script below reads a diff between a base ref and a head ref, scores it, and prints a tier. It has no dependencies outside the standard library, and it is written as a starting point rather than a finished product, so pinning your own path globs and build-file set is expected.

#!/usr/bin/env python3
"""Route an incoming OSS pull request by blast radius.

Usage:
    python review_route.py --base origin/main --head pr-4821
"""
from __future__ import annotations

import argparse
import subprocess

PUBLIC_PREFIXES = ("src/", "lib/", "pkg/", "internal/api/")
BUILD_FILES = {
    "pyproject.toml", "setup.py", "setup.cfg", "requirements.txt",
    "package.json", "package-lock.json", "Cargo.toml", "Cargo.lock",
    "go.mod", "go.sum", "Makefile", "Dockerfile",
}
TEST_MARKERS = ("/tests/", "/test/", "spec/", "test_", "_test.")


def git(*args: str) -> str:
    proc = subprocess.run(["git", *args], check=True, capture_output=True, text=True)
    return proc.stdout


def numstat(base: str, head: str) -> list[tuple[int, int, str]]:
    rows = []
    for line in git("diff", "--numstat", f"{base}...{head}").splitlines():
        added, removed, path = line.split("\t", 2)
        rows.append((int(added) if added.isdigit() else 0,
                     int(removed) if removed.isdigit() else 0,
                     path))
    return rows


def classify(rows):
    added = sum(r[0] for r in rows)
    removed = sum(r[1] for r in rows)
    paths = [r[2] for r in rows]
    public = any(p.startswith(PUBLIC_PREFIXES) for p in paths)
    build = any(p.split("/")[-1] in BUILD_FILES for p in paths)
    tests = any(any(m in p for m in TEST_MARKERS) for p in paths)
    return added, removed, public, build, tests, len(paths)


def tier_for(files, added, removed, public, build, tests) -> tuple[int, str]:
    score = 0
    score += 3 if build else 0
    score += 2 if public else 0
    score -= 2 if tests else 0
    score += 1 if files > 5 else 0
    score += 1 if (added + removed) > 200 else 0
    if score >= 4:
        return score, "T0 human-only review"
    if score >= 2:
        return score, "T1 human review; model may draft notes"
    return score, "T2 model draft first; human sign-off"


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--head", default="HEAD")
    args = ap.parse_args()

    rows = numstat(args.base, args.head)
    added, removed, public, build, tests, files = classify(rows)
    score, tier = tier_for(files, added, removed, public, build, tests)
    print(f"base={args.base} head={args.head}")
    print(f"files={files} +{added} -{removed}")
    print(f"public_surface={public} build_files={build} tests_touched={tests}")
    print(f"score={score} tier={tier}")


if __name__ == "__main__":
    main()

Sample output

$ python review_route.py --base origin/main --head pr-4821
base=origin/main head=pr-4821
files=7 +412 -96
public_surface=True build_files=False tests_touched=True
score=2 tier=T1 human review; model may draft notes

A one-line lockfile bump from the same queue scores differently, because build=True adds three points and tests=False adds two more. That inversion is the point of the exercise: the smallest diff in the queue is not the cheapest one to review.

How to read the tiers in practice

T0 — human-only review. Dependency and lockfile edits, authentication or authorization paths, CI workflow files, and any patch that deletes or weakens tests. No model-drafted note should precede a human read here.
T1 — human review, model notes allowed. Public surface changes that ship with tests. A model may summarize the diff or list untested branches, but the review comment itself is written by a person.
T2 — model draft first. Test-only changes, internal helpers, docstring fixes, and small additions behind private names. A draft is useful here because the acceptance criteria are usually mechanical.

Two rules keep the tier list honest. First, a maintainer's knowledge of their own codebase always overrides the score, because the score has never read the code. Second, the tier decides who drafts the first comment, never whether the patch gets merged.

Where free model access fits in the queue

Drafting review notes for a T2 patch is a low-stakes use of a hosted model, since the reviewer still reads the diff before responding. MonkeyCode is an open-source project whose operator states it offers free model access and a free server option, which matters mainly for contributors who do not want a paid API key attached to routine review work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two boundaries are worth stating plainly. Model access is used only for T1 and T2 drafting, and T0 patches are read by a human before anything else happens. Availability, model selection, and any usage limits are operator-stated and change over time, so confirm the current terms on the project page rather than trusting a figure quoted in an older post.

Two runs, then mechanical verification

The weakest part of model-drafted review notes is that they cite symbols and files that may not exist. A cheap protocol removes most of that noise, and it has three steps.

Run the drafting prompt twice with different phrasing or sampling settings, and collect both answer sets as JSONL.
Keep only claims that appear in both runs with the same file and the same symbol, which discards most one-off hallucinations.
Verify every surviving claim mechanically against the patch before it reaches the contributor.

The drafting prompt should require one JSON object per line, with a file, a symbol, and a short note, and it should forbid claims that cannot name a symbol. Claims without a symbol are opinions, and opinions belong in the human review, not in an automated pass.

The verification script

The script below keeps only claims that survive two mechanical checks: the file appears in the patch, and the symbol resolves at the head revision. It is deliberately narrow, and it proves existence rather than correctness.

#!/usr/bin/env python3
"""Keep only model review claims that resolve mechanically against the patch.

Input JSONL, one claim per line:
    {"file": "src/parser.py", "symbol": "parse_args", "note": "..."}
"""
from __future__ import annotations

import argparse
import json
import subprocess
import sys


def git(*args: str):
    proc = subprocess.run(["git", *args], capture_output=True, text=True)
    return proc.returncode, proc.stdout


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("claims")
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--head", default="HEAD")
    args = ap.parse_args()

    _, changed_raw = git("diff", "--name-only", f"{args.base}...{args.head}")
    changed = set(changed_raw.split())

    kept, dropped = [], []
    with open(args.claims, encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if not line:
                continue
            claim = json.loads(line)
            path, symbol = claim["file"], claim["symbol"]
            reasons = []
            if path not in changed:
                reasons.append("file not in patch")
            rc, hits = git("grep", "-n", "-F", symbol, args.head, "--", path)
            if rc != 0 or not hits.strip():
                reasons.append("symbol not found at head")
            record = {**claim, "rejected_because": reasons}
            (dropped if reasons else kept).append(record)

    print(json.dumps({"kept": kept, "dropped": dropped}, indent=2))
    print(f"kept={len(kept)} dropped={len(dropped)}", file=sys.stderr)


if __name__ == "__main__":
    main()

What the filter does not catch

A symbol that exists can still be described incorrectly, and the filter says nothing about whether a suggested change is right. Claims that survive both checks are candidates for human reading, not conclusions, and the rejection list is often the more informative output because it shows which class of claims the model tends to fabricate.

Limitations and who should not adopt this

Very small repositories. The script overhead exceeds the review time saved, and a maintainer who already knows every file gains nothing from a tier column.
Repositories without tests. The tests=False penalty has no meaning when no test suite exists, so the score degenerates into a diff-size rank.
Monorepos with sparse paths. Public prefix matching needs per-package configuration, and a single global glob set will misclassify most of the tree.
Security-sensitive patches. A tier score is the wrong instrument for vulnerability fixes, and those should bypass the routing pipeline entirely.
Projects that forbid hosted tooling. If repository policy excludes third-party model services, run the routing script alone and stop before the drafting step.

The scripts above are a starting point rather than a benchmarked tool, and they have not been validated across every language layout. Treat the build-file set and prefix list as configuration to maintain, not as constants.

A twenty-minute checklist

Add review_route.py to a scripts directory and adjust PUBLIC_PREFIXES and BUILD_FILES for your language.
Run it against the three oldest open pull requests and compare the tiers with your own intuition.
Fix the scoring only where your intuition is consistently right and the script is consistently wrong.
Draft notes for one T2 patch twice, and diff the two answer sets before verifying anything.
Run verify_claims.py and read the rejected list first, because it reveals the failure mode fastest.
Record which tiers produced useful notes, and drop model drafting from any tier that did not earn its place.

The procedure is deliberately unglamorous, and its value comes from being repeatable on a Monday morning when eleven patches are waiting. If you want to try the drafting step without wiring up local inference, the free model access and free server option described by MonkeyCode are a reasonable place to start, beginning with the T2 queue and a small rejected-claims list.

Score the Issue Before You Claim It: A Reproducibility Ledger for OSS Work

Blake Yang — Thu, 15 Oct 2026 11:11:56 +0000

Liquid syntax error: Unknown tag 'endraw'

A Citation Check for OSS Patches Starts With a Minimized Public Fixture

Blake Yang — Mon, 12 Oct 2026 23:58:48 +0000

A maintainer on a small library closed a pull request after the patch looked tidy and the model review sounded confident. The contributor had pasted a long log into chat, received praise, and opened the request without a public failing fixture. On a clean clone of the default branch, the reported command exited zero, so the patch had nothing to prove. That scene is a composite of common review failures, not a claim about one named project.

The same pattern appears when a model reviews a diff before anyone proves the bug on the upstream default branch. Agreement then becomes a style comment about the patch, rather than evidence that the failure existed at all. A useful workflow freezes a public bundle first, then invites a model to comment only on fields inside that bundle. The bundle, not the model's tone, decides whether the patch is ready for a human maintainer.

Freeze a public bundle the model is allowed to see

A review bundle is a directory committed on a scratch branch, or attached to the issue, before any model sees the patch. It holds the failing command, the expected exit status, a fixture file, and a short note about the default-branch commit. It also holds a license check result and a test command copied from the repository's own documentation. Secrets, tokens, and private hostnames stay out of that directory so the packet can be shared with a model.

Fields that make the packet checkable

The file base_sha.txt records the upstream commit where the failure was observed, and later reruns must detach at that commit.
The file exit_status.txt stores the status of the failing command, and a clean default-branch run must reproduce that status.
The file fixture.sha256 names the public input, so a review cannot swap in a quieter file after the model has spoken.
The file license_gate.txt records whether an SPDX identifier search succeeded, which is a gate and not a full legal review.

Build the bundle with a script that can be rerun

The following shell script is an unexecuted proposal, not a recorded run from this account, and paths must be adapted. It writes the base commit, copies one fixture, captures status streams, probes for an SPDX line, and hashes the packet. A contributor should replace the sample CLI invocation with the command already documented by the upstream project. The hash file is the handle a later review uses to prove the packet did not drift between machines.

#!/usr/bin/env bash
# Proposal: unexecuted example. Adapt paths before use.
set -euo pipefail

ROOT="$(git rev-parse --show-toplevel)"
SHA="$(git rev-parse HEAD)"
OUT="${ROOT}/.review-bundle"
mkdir -p "${OUT}/fixture"

echo "${SHA}" > "${OUT}/base_sha.txt"
cp "${ROOT}/repro/input.txt" "${OUT}/fixture/input.txt"
sha256sum "${OUT}/fixture/input.txt" > "${OUT}/fixture.sha256"

set +e
python3 -m project.cli --input "${OUT}/fixture/input.txt" \
  > "${OUT}/stdout.txt" 2> "${OUT}/stderr.txt"
STATUS=$?
set -e
echo "${STATUS}" > "${OUT}/exit_status.txt"

if git grep -n "SPDX-License-Identifier" -- '*.py' > "${OUT}/spdx.txt"; then
  echo "spdx-present" > "${OUT}/license_gate.txt"
else
  echo "spdx-missing" > "${OUT}/license_gate.txt"
fi

find "${OUT}" -type f ! -name 'packet.sha256' -print0 \
  | sort -z | xargs -0 sha256sum > "${OUT}/packet.sha256"

Running the script on a dirty worktree will mix unrelated edits into the story the packet tells the reviewer. A clean detached checkout of the recorded base commit is the honest place to capture the failing status. If that status is already zero, the contributor stops and does not ask a model to invent a bug. That stop is the point of the bundle, and it saves the maintainer from reviewing a speculative patch.

Climb three rungs before any model comment

Rung one runs the failing command on a clean checkout of the recorded base commit and expects the frozen exit status. Rung two applies the patch on that same commit and expects the command to exit zero. Rung three runs the repository test command, not a new test invented for the patch alone. A model should not be asked to invent those commands after the fact, because that invites a moving target.

Repeat the rungs on a second checkout

git fetch origin
git switch --detach "$(cat .review-bundle/base_sha.txt)"
bash scripts/oss-review-bundle.sh
test "$(cat .review-bundle/exit_status.txt)" = "1"
git apply --check patches/fix.diff
git apply patches/fix.diff
# Proposal: after the patch, rerun the documented command and expect status 0.

The first test line fails the workflow when the recorded status is not reproduced, which is the desired alarm. The apply check catches a patch that does not fit the recorded base commit, before anyone discusses coding style. After the patch applies, the same command should exit zero, and the repository test command should still pass. A model review that arrives before these three results exist is commentary, not evidence a maintainer can trust.

Minimize the fixture without deleting the failure

A long log invites a model to summarize noise and then miss the line that actually fails the command. The contributor should cut the fixture in halves, rerun the failing command, and keep the half that still fails. Each cut is recorded by appending the new fixture hash to a short minimization log stored inside the bundle. When a cut makes the status become zero, that cut is discarded and the previous fixture remains the public input.

Steps that keep the failure alive

Copy the original fixture into the bundle and record its hash, its line count, and the failing exit status.
Split that fixture into two halves and rerun the documented failing command against each half on the recorded commit.
Keep the smallest half that still returns the frozen status, and append that hash to the minimization log.
Stop when neither half fails, restore the previous fixture, and treat that file as the public input for review.

The log format below is a proposal with placeholder hashes, not measured output from a repository. A real run replaces those tokens with sha256sum output and the exit status just observed. The discarded row stays in the log so a reviewer can see which cut destroyed the failure. That trail matters more than a shorter file that no longer fails.

step=0 lines=48 sha256=placeholder-original status=1 kept
step=1 lines=24 sha256=placeholder-half status=1 kept
step=2 lines=12 sha256=placeholder-quarter status=0 discarded

Score review comments by citation

A model review earns trust only when each remark names a file that already sits inside the frozen bundle. A comment that calls the patch correct without naming exit_status.txt or fixture.sha256 scores zero for evidence. A comment that cites a packet file and quotes a line can be checked by a human in one pass. This scoring habit resists fluent agreement that replaces a failed reproduction on the recorded base commit itself.

The table below is a proposal for sorting comments, not a measured study from this account's history. Strong rows still require a human to open the cited file before the patch is allowed to move. Weak rows are corrections to the note, not reasons to merge the patch into the default branch. Empty rows are discarded even when the wording sounds confident, polite, and ready for a maintainer.

Model comment pattern	Packet citation	Human action
Fluent praise with no filename	None	Discard the comment
Cites `exit_status.txt` but quotes the wrong digit	Weak	Open the file and correct the note
Cites `fixture.sha256` and the minimization log	Strong	Compare hashes, then read the diff
Suggests a test command absent from the repo docs	Out of packet	Ignore until a human adds that command

# Proposal: unexecuted. Scores whether review lines cite packet files.
from pathlib import Path

PACKET = {
    "base_sha.txt",
    "exit_status.txt",
    "fixture.sha256",
    "license_gate.txt",
    "packet.sha256",
}
text = Path("model-review.md").read_text(encoding="utf-8")
cited = {name for name in PACKET if name in text}
missing = PACKET - cited
print(f"cited={sorted(cited)}")
print(f"missing={sorted(missing)}")
if not cited:
    raise SystemExit("review cited no packet file")

The checker is also an unexecuted proposal, and it only tests whether filenames appear in the review text. It cannot tell whether the model understood the exit status, so a human still opens every cited file. If the checker exits because no packet file was named, the review is rejected without a second pleading prompt. A second prompt that begs for agreement would only recreate the original failure mode this bundle is meant to stop.

Use free model access only after the hash is frozen

MonkeyCode is relevant here only as an optional place to run the citation pass and to repeat the bundle script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied claim used here is that free model access and a free server option currently exist. The operator also describes the project as open source, which readers should confirm in the repository before they depend on it.

This article does not state model names, token quotas, hardware sizes, or how long that access lasts. Readers should confirm current terms on the project page before relying on either option for a real review. A missing or changed offer does not break the ladder, because the packet can be reviewed by a person alone. Hosted help is a convenience after the hash exists, not a requirement for the reproduction or the patch.

A free model pass is useful after the packet hash is frozen, because the model then has a closed set of files to cite. A free server rerun is useful when the contributor's laptop has extra packages that the upstream image does not install. Both steps stay optional, and the ladder still works with a local model or with a second container on the same desk. The packet hash, not the model's tone, remains the acceptance criterion for moving the patch toward a human reviewer.

The prompt should ask for missing packet fields, mismatched hashes, and a license gate that printed spdx-missing. It should not ask for a verdict that the patch is ready to merge, because that verdict is a human decision. Output belongs in model-review.md beside the packet, so the filename checker can read it without extra parsing. The contributor then keeps only sentences that name a packet file before the maintainer sees the note.

The contributor deletes any sentence that does not name a packet file before the maintainer sees the note. That deletion is a human edit, and the pull request should say the model text was filtered by the checker. A maintainer can then review the patch, the packet, and a short list of cited gaps without rereading praise. The model does not appear as an author of the commit, because authorship stays with the person who can answer questions.

Limitations that should stay visible

This workflow does not prove that a patch is correct, and it does not replace the maintainer's review or the project's legal process. A minimized fixture can drop the real bug if the contributor cuts the input until the failure disappears. License header searches miss files that use a different SPDX line, or a license that lives only in a top-level file. A model can still cite the right filename and misread the exit status, so a human must open the cited file.

The free server option, where it is available, does not guarantee the same packages as the project's continuous integration image. A rerun that passes on a generic machine can still fail inside the upstream workflow file on the next push. Contributors should treat a green second machine as a portability hint, not as a substitute for the project's own checks. Quotas, model names, and how long free access lasts sit outside this article and must be read from current terms.

Who should skip the bundle

Skip this bundle when the repository forbids attached fixtures, or when the bug depends on private data that cannot be reduced. Skip it when the change is a documentation typo with no failing command, because a fake exit status would only add noise. Skip it when the contributor cannot read the test output well enough to reject a confident but wrong model comment. In those cases a short issue comment and a human review remain the honest path, and a model pass would not help.

What the maintainer should be able to replay

A maintainer should be able to detach at the recorded commit, rerun the bundle script, and see the same fixture hash. The patch should flip the frozen status, and the citation checker should show at least one packet file in the review note. If either hash differs, the conversation stops until the contributor publishes a new packet rather than explaining the drift in chat. That replay is the whole method, and it remains useful even if no hosted model or free server is used.

Readers who still want a clean second machine for that replay can check whether MonkeyCode's free server option is open. They should require the same packet hash on that machine before they trust the rerun at all. The free model access, if currently offered, fits only the citation pass after the hash file already exists. Neither option is a merge approval, and the project page is the place to confirm what is available that day.

Separate the Evidence Commit From the OSS Fix Before Model Output Runs

Blake Yang — Sat, 10 Oct 2026 17:06:52 +0000

A maintainer declined a parser pull request after the diff rewrote three files and never preserved the original failure. The contributor had pasted a repository link into a coding model, accepted the generated patch, and pushed one squashed commit. Local checks passed on that laptop, yet reviewers could not replay the reported bug from any commit on the branch. The project wanted a failing case stored in history before anyone discussed the shape of the repair.

The scene above is a composite illustration, not a report from this account or a named maintainer. The rejection pattern still points to a missing contribution gate rather than a simple score of model quality. A careful open-source change records the failure in its own commit, then limits what a model may read and where its patch may run. The workflow below uses a red commit, a narrow packet, and a disposable runner, and it still works with no hosted model.

What reviewers can replay

Maintainers usually gain confidence when a failure replays on a clean tree that does not depend on the author's shell profile. The first commit stores the command, the fixture, and a captured non-zero result before production code changes. The second commit may repair the defect, but it should not rewrite, weaken, or delete that recorded evidence. A reviewer can move between those commits and watch the same command change from failing to passing.

This pattern fits a public issue, a license that permits sharing the selected files, and a command that fails for a known reason. It does not require write access on the upstream project, and it does not treat a model suggestion as already correct. Patch text stays untrusted until a clean runner executes it and a human compares the result with the red commit.

Land the evidence commit first

The contributor clones the upstream project, opens a topic branch, and adds only reproduction assets. Those assets may be a focused test, a tiny fixture, or a shell script kept under a clearly named directory. Formatting sweeps, lockfile refreshes, and drive-by refactors remain outside the evidence commit on purpose. If the command cannot fail on a clean upstream checkout, work stops until the fixture is honest.

The shell below is a proposed example, not a transcript from a real repository run:

git clone --depth 1 https://example.invalid/org/cli.git
cd cli
git checkout -b issue-1842-quoted-flag
mkdir -p repro
cat > repro/quoted-flag.sh << 'EOF'
#!/usr/bin/env bash
set -eu
# Proposed stand-in. Point this at the real project binary and fixture.
./bin/cli --config repro/quoted.toml run
EOF
chmod +x repro/quoted-flag.sh
set +e
./repro/quoted-flag.sh > repro/actual.txt 2> repro/stderr.txt
status=$?
set -e
printf 'exit=%s\nstderr_bytes=%s\n' "$status" "$(wc -c < repro/stderr.txt)" > repro/status.txt
git add repro
git commit -m "test: reproduce quoted flag drop from issue 1842"

The status note should record the exit code and the stderr size, not a dump of the local environment. Home-directory paths, access tokens, and internal hostnames do not belong in files that may later leave the laptop. A case that fails only inside one customized shell profile is not ready for a model or a maintainer.

Choose a packet, then choose a runner

A free model can read the reproduction and one suspected function, then propose an edit that leaves the evidence files untouched. That help is optional, and the boundary around execution matters more than the vendor behind the suggestion. Generated patch text should run first on a disposable runner, not beside developer credentials or private clones. A full-tree upload is unnecessary when the evidence commit already names the command and the fixture.

The reproduction script and fixture may be sent when the license allows those files to be redistributed.
A short status note may be sent after paths, tokens, and hostnames have been removed from the captured text.
One suspected function may be sent, while an archive of the whole working tree should stay on the laptop.
Environment files, keys, cookies, private logs, and unpublished vulnerability detail stay out of the packet.
Unseen patch text should not run on a laptop that holds SSH keys, release tokens, or private clones.

Decision	Allowed default	Stop if this is false
Evidence is its own commit	Yes	The failure exists only in chat history
Model sees a narrow packet	Yes	The only prompt is a full-tree paste
First execution is disposable	Yes	The only runner holds personal credentials
Reproduction files stay unchanged	Yes	The patch edits the evidence to force green
Human edits the pull-request text	Yes	A model summary would be posted unchanged

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant only as one way to reach free model access and a free server for this bounded review step. The same checks remain valid in another editor, with another model, or with no hosted service in the loop. No model names, token quotas, hardware sizes, time limits, or permanent terms are stated here, because those details were not verified for this draft.

The operator-supplied claim is limited to free model access for the review note and a free server option for the first patch execution. The contributor still owns redaction, the evidence commit, and the later decision to open a pull request. Readers who skip the product name still have the commit split, the runner sequence, and the rejection rules.

Apply the suggestion away from the laptop

After the evidence commit exists, the contributor may ask for a minimal production change that makes the reproduction return zero. The reply is saved as patch text and is not executed on the laptop that holds SSH keys or browser sessions. A disposable free server, or any other clean virtual machine, receives a fresh clone and that patch file. The runner has no maintainer credentials, no personal tokens, and no extra repositories beyond the public topic branch.

# Proposed sequence for a clean runner. Do not target a production host.
git clone --depth 1 --branch issue-1842-quoted-flag https://example.invalid/you/cli.git
cd cli
git apply --check /tmp/model-fix.patch
git apply /tmp/model-fix.patch
set +e
./repro/quoted-flag.sh
echo "post-patch:$?"
set -e
git diff --exit-code -- repro

Stop conditions

Three observable results are enough to stop the trial before a pull request is opened at all. Each result is a reason to keep the evidence commit and to discard the current suggestion entirely. None of these results should be patched over by sending a larger private context to the model.

The patch fails git apply --check, so the next request stays narrow instead of adding private files for context.
The reproduction command still fails, so the suggestion is discarded and the evidence commit remains authoritative.
A non-empty diff under the reproduction path means the patch mixed repair with evidence and should be rejected.

When the same command passes and the reproduction diff is empty, a second commit may hold only the production change. The branch history then shows the miss first and the attempted repair second, which is easier to review than a squash. A model may draft the pull-request summary from those two diffs, but a human must edit every factual claim. The maintainer should be able to replay both commits without trusting the tool that wrote the second one.

Check the branch shape before upload

Proposed checker

The script below is a proposal for the fix commit, and it has not been executed against a real upstream project here. Path names, branch names, and the reproduction directory must be adapted before anyone relies on the result. A passing shape check still does not replace the reproduction command or the project's own test target.

#!/usr/bin/env bash
# Proposed shape check. Run while HEAD is the fix commit.
set -eu
base="${1:-HEAD~1}"
git diff --exit-code "$base" -- repro
if git diff --quiet "$base" -- . ':!repro'; then
  echo "fix commit has no production change" >&2
  exit 1
fi
if git diff --name-only "$base" | grep -E '(^|/)\.env$|id_rsa|credentials\.json'; then
  echo "sensitive-looking path in the fix diff" >&2
  exit 1
fi
echo "commit shape accepted; still rerun the reproduction command"

Manual replay

The numbered checks below are the human gate between a green runner and a public pull request. These steps remain proposed checks for the contributor, not measured results from an executed upstream trial. A skipped check leaves the branch unready, even when the shape script prints an acceptance line.

The contributor checks out the evidence commit on a clean tree and confirms a non-zero status.
The contributor checks out the fix commit and confirms a zero status from the same command.
The contributor confirms that the reproduction path did not change between the evidence commit and the fix.
The contributor runs the repository's own targeted test and records that exact command in the pull request.
The contributor searches the diff for home paths, tokens, and private hostnames before any upload.
If a model remains in the loop, it may only hint at callers outside the suspected function, and project tests must confirm that hint.

Limits, and who should skip this

This gate suits small, replayable defects in projects that accept external pull requests and can tolerate a failing test beside a fix. It is a poor fit for races that vanish on a clean runner and for failures that require private production data. Security reports that belong in a private disclosure channel should not be turned into a public reproduction commit. A free server is also the wrong place for customer logs, licensed corpora, or any tree the contributor may not upload.

Model assistance stays optional here and is easy to overrate once the reproduction already looks clean. A free model can suggest a narrow edit, yet it cannot certify license fit, maintainer preference, or timing stability. If the failing command cannot be published, sending a richer private log to a hosted model is not an acceptable workaround. The honest alternatives are a purely local fix, a question to the maintainers, or no patch until the fixture is shareable.

Teams that already require a paired regression test can adopt the commit split without subscribing to any hosted service. The disposable runner matters most when the patch text came from a model and has not been read line by line. Contributors who cannot isolate a public fixture should pause rather than widen the packet to force progress.

A single open issue is a better trial than a batch of generated pull requests across unrelated repositories. The contributor can land the evidence commit locally, then use free model access only on the redacted packet and a free server only for the first apply. The resulting pull request should stay small enough that replay, not the drafting tool, carries the review.

Freeze a Failing Oracle Before Any Model Touches an OSS Patch

Blake Yang — Fri, 09 Oct 2026 13:17:09 +0000

A weekend contributor clones a small library, reproduces a parser bug, and asks a coding model to repair it before dinner. The model returns a tidy diff, a confident summary, and a test that already passes on the patched tree. The maintainer later discovers that the same test also passed on the unpatched tag, so the pull request never proved a fix. That pattern wastes review time, and a frozen oracle stops it before the branch leaves the fork.

Why a passing test is not evidence

An open-source review needs a failure that exists before the patch and disappears after the patch, under the same command. A generated test written after the edit can encode the new behavior without ever having failed. Reviewers then argue about intended behavior instead of comparing two captured runs from the same command. The workflow below treats those two runs as the contract that every later comment must respect.

Run the loop in order

Write the manifest with an argument list, both expected statuses, and two markers before any production edit.
Capture the before side on the untouched checkout and stop if the status is already the success code.
Apply the smallest patch that can clear the failure marker, whether a person or a model drafted the diff.
Capture the after side, run the checker, and refuse the pull request text when the checker returns an error.
Ask the model to compare the diff with those files, and leave the merge decision with a human reviewer.

Lay out the evidence before editing

The contributor creates an evidence directory before touching production code, then records the failing command in a small manifest. That manifest names the command, the expected nonzero status, and a short marker that must appear in the failure output. A later checker refuses to bless a patch when either capture is missing or when the command string has changed. The point is narrow: prove this bug, on this command, before any model rewrites the story.

Directory contract

evidence/
  manifest.json
  before/stdout.txt
  before/stderr.txt
  before/status.txt
  after/stdout.txt
  after/stderr.txt
  after/status.txt

Proposed manifest

The sample below is a proposal for one pytest node, and it is not a result from a named upstream repository. Replace the argument list with the project's real test command before capturing either side of the oracle. Keep markers short and specific so a later reviewer can see them in the saved output without guessing. Do not treat the sample statuses as defaults for every language or every test runner on the project.

{
  "command": ["python", "-m", "pytest", "-q", "tests/test_parser.py::test_trailing_comma"],
  "before_status": 1,
  "after_status": 0,
  "failure_marker": "trailing comma",
  "success_marker": "1 passed"
}

Capture both sides with one script

The capture script reads the manifest, runs the listed command, and writes status plus both streams into the requested side directory. It does not interpret the bug, and it does not accept a free-form shell string, because argument lists avoid accidental quoting surprises. The contributor runs it once on the untouched checkout, applies the human or model patch, then runs it again with the after side. If the before status already matches the after status, the oracle is invalid and the patch should not be opened.

#!/usr/bin/env python3
"""Proposed local helper. Not a recorded run against an upstream repository."""
import json
import subprocess
import sys
from pathlib import Path

def capture(side: str) -> int:
    manifest = json.loads(Path("evidence/manifest.json").read_text())
    out = Path("evidence") / side
    out.mkdir(parents=True, exist_ok=True)
    proc = subprocess.run(
        manifest["command"],
        text=True,
        capture_output=True,
        check=False,
    )
    (out / "stdout.txt").write_text(proc.stdout)
    (out / "stderr.txt").write_text(proc.stderr)
    (out / "status.txt").write_text(str(proc.returncode))
    return proc.returncode

if __name__ == "__main__":
    raise SystemExit(capture(sys.argv[1]))

python capture_oracle.py before
python capture_oracle.py after
python check_oracle.py

Check the oracle before review comments exist

The checker loads the manifest and both status files, then confirms the failure marker and the success marker landed on the correct side. It also rejects a success marker that already appears in the before output, which catches tests that never failed. A nonzero checker result means the pull request description must not claim that the captured bug is fixed. The script below is a proposal for a fork-local gate, and it should be adapted to the project's real test entry point.

#!/usr/bin/env python3
"""Proposed checker. Sample markers are not universal pass criteria."""
import json
import sys
from pathlib import Path

def main() -> int:
    root = Path("evidence")
    manifest = json.loads((root / "manifest.json").read_text())
    before_status = int((root / "before" / "status.txt").read_text().strip())
    after_status = int((root / "after" / "status.txt").read_text().strip())
    before_text = (
        (root / "before" / "stdout.txt").read_text()
        + (root / "before" / "stderr.txt").read_text()
    )
    after_text = (
        (root / "after" / "stdout.txt").read_text()
        + (root / "after" / "stderr.txt").read_text()
    )
    errors = []
    if before_status != manifest["before_status"]:
        errors.append("before status does not match the manifest")
    if after_status != manifest["after_status"]:
        errors.append("after status does not match the manifest")
    if manifest["failure_marker"] not in before_text:
        errors.append("failure marker missing from before output")
    if manifest["success_marker"] not in after_text:
        errors.append("success marker missing from after output")
    if manifest["success_marker"] in before_text:
        errors.append("success marker already present before the patch")
    if errors:
        print("\n".join(errors))
        return 1
    print("oracle holds: failure captured, then cleared, under one command")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Let a free model review the contract, not invent it

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can sit in the review lane after the oracle already exists on disk. The useful task is a bounded reading of the diff against the evidence files, not a fresh verdict on the bug. A careful prompt asks the model to list mismatches, untested branches, and claims that the captured files do not support. The free server option helps when a clean remote run is needed, provided the resulting evidence files still pass the local checker.

Review only evidence/manifest.json, the before and after captures, and the diff.
Do not claim a bug is fixed unless the checker would pass on those files.
List pull request claims that the captured output does not support.
Do not widen the patch, and do not suggest new dependencies.

Split model comments from maintainer decisions

Question	Model may draft	Human must decide
Did the before status match the manifest?	Quote the status file	Reject the request if it does not
Did the marker move from failure to success?	Point at the two captures	Confirm the marker is specific enough
Does the diff touch paths outside the fix and the test?	List the extra paths	Accept the scope or split the series
Are license, API, or release notes correct?	Flag uncertainty only	Own the final answer
Should this merge?	No draft of a merge vote	Yes, after reading the files

The table is the review boundary for this workflow, and it stays useful even if the model name changes next month. Current quotas, model lists, and server limits are product facts that expire, so this article does not quote them. Contributors should read the current MonkeyCode documentation before relying on a free run for a time-boxed contribution. A changed limit is a reason to run the same scripts locally, not a reason to skip the oracle.

Keep the patch inside the oracle

A frozen oracle does not prove that every edited line was necessary, so the contributor still reads the diff before opening the request. Extra refactors, renamed helpers, and drive-by formatting changes should move to a separate commit or leave the branch. The model may list those extra paths, but it should not be asked to clean up the module after the evidence is green. A small patch plus a failing-then-passing command is easier to revert than a broad rewrite with a flattering summary.

Commands a reviewer can rerun

The pull request body should contain the manifest command and the checker result, copied from the terminal rather than paraphrased. A reviewer on another machine can then run the same argument list, compare status files, and stop if the marker is too vague. A documented setup target belongs in the manifest notes, and this article does not invent a universal container name. The evidence files stay in the fork until the maintainer asks for them, because some projects prefer not to commit bulky logs.

When the marker lies

A marker that appears in ordinary passing output will make the checker reject a real fix, and that failure is useful. The contributor should choose a string that shows up only in the broken run, such as a specific exception text or a fixture name. If no such string exists, the test itself is the missing artifact, and editing production code first would hide that gap. Flaky commands are out of scope, because a status that changes between two before captures means the oracle is not ready for model review.

Who should skip this gate

This approach is a poor fit for patches that cannot be reduced to one command with a stable marker. Documentation-only edits, visual changes, and rare race conditions need a different proof, because a single captured status can look green while the bug remains. Projects that already require a recorded replay file should follow that project rule instead of adding a second evidence format. The checker also does not replace secret scanning, license review, or a blast-radius look at shared helpers.

Close the branch only after the files agree

A contributor who wants a second reading can paste the checker output and the diff into a free model session. The only requested output should be unsupported claims, checked against files already stored in the evidence directory. Those scripts remain unexecuted proposals, so each fork should adjust markers to its own test runner before trusting a green line. When the two captures and the checker agree, the branch is ready for a human reviewer rather than for another generated summary.

Convert CONTRIBUTING.md Into an Executable Preflight Gate Before an OSS Patch Leaves Your Fork

Blake Yang — Thu, 08 Oct 2026 09:26:47 +0000

A contributor spends a weekend on a null-dereference fix, opens the pull request on Monday morning, and watches CI fail on a rule that CONTRIBUTING.md stated in paragraph nine. The patch logic is correct, but the branch name lacks the expected prefix and the changelog entry is missing. Three review rounds later, the maintainer merges a change whose packaging cost more than its implementation.

That failure mode is rarely about code quality, and it is almost always about packaging. Contribution guidelines are written as prose for humans and enforced as machines somewhere else, so the gap gets paid for in review latency. The workflow below closes that gap with an executable preflight gate, a reproducible test step, and a narrow review role for free-tier models.

Why prose rules and CI checks drift apart

Most repositories accumulate contribution rules in three separate places, and none of them is authoritative. CONTRIBUTING.md explains intent, CI enforces a subset, and maintainers enforce the remainder from memory during review.

The document drifts after a tooling change, because nobody re-reads paragraph nine.
CI cannot check intent, so reviewers absorb the difference as unpaid coordination work.
Contributors learn the real rules only after a failed pull request, which is the most expensive classroom.

The practical fix is not stricter CI. It is a local gate that runs the rules the repository already claims to have, before a patch becomes other people's problem.

The snippets below are illustrative and were not run against a specific public repository; adapt paths and tool names to your target project before relying on them.

Step 1: Extract the rules into a reviewed YAML file

Convert prose into a small machine-readable file that a human approves once and CI can diff later. Keep every rule traceable to the sentence it came from, so a disagreement can be settled by reading the source document.

A free model tier handles the mechanical extraction well, because the task is reading and structuring, not deciding. The prompt below deliberately forbids inference, which is the failure mode that turns an extraction pass into invented policy.

You are extracting contribution rules from a repository document.
Return YAML only, with this schema:
  version: 1
  branch_pattern: <regex or null>
  required_when:
    - path: <file that must change>
      when_changed: [<glob>, ...]
Return a separate key `unresolved` for any sentence that is advisory rather than checkable.
For every rule, include `source_line`, the line number in the document.
Do not infer rules that are not written. Mark ambiguous rules with `needs_human: true`.

The resulting file stays small on purpose. A gate with forty checks gets disabled within a week, while one with five honest checks survives contact with a real release cycle.

# .contrib-rules.yml — extracted from CONTRIBUTING.md, human-reviewed
version: 1
branch_pattern: '^(fix|feat|docs|chore)/[a-z0-9._-]+$'
required_when:
  - path: CHANGELOG.md
    when_changed: ['src/**/*.py', 'lib/**/*.js']
  - path: docs/
    when_changed: ['**/public_api.py']
unresolved:
  - "Prefer small pull requests."

Step 2: Build the preflight gate

The gate has two jobs: reject packaging mistakes locally, and run the project's own test dialect exactly as the repository defines it. A shell wrapper handles ordering, while a small Python checker evaluates the file-level rules.

#!/usr/bin/env bash
# scripts/preflight.sh — run the repo's stated rules before pushing a patch
set -euo pipefail

BASE="${1:-origin/main}"
RULES=".contrib-rules.yml"

branch="$(git rev-parse --abbrev-ref HEAD)"
pattern="$(python3 -c "import yaml;print(yaml.safe_load(open('$RULES'))['branch_pattern'])")"

if [[ ! "$branch" =~ $pattern ]]; then
  echo "preflight: branch '$branch' does not match $pattern" >&2
  exit 1
fi

python3 scripts/preflight_rules.py "$BASE"
make fmt
make lint
make test

#!/usr/bin/env python3
"""preflight_rules.py — verify file-level contribution rules locally."""
from __future__ import annotations

import fnmatch
import pathlib
import subprocess
import sys

import yaml

RULES = yaml.safe_load(pathlib.Path(".contrib-rules.yml").read_text())


def changed_files(base: str) -> list[str]:
    out = subprocess.run(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        capture_output=True,
        text=True,
        check=True,
    )
    return [line for line in out.stdout.splitlines() if line.strip()]


def check_required_files(changed: list[str]) -> list[str]:
    problems: list[str] = []
    for rule in RULES.get("required_when", []):
        triggers = rule.get("when_changed", [])
        touched = any(fnmatch.fnmatch(f, p) for f in changed for p in triggers)
        if touched and rule["path"] not in changed:
            problems.append(f"required file not updated: {rule['path']}")
    return problems


if __name__ == "__main__":
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    issues = check_required_files(changed_files(base))
    for issue in issues:
        print(f"preflight: {issue}", file=sys.stderr)
    sys.exit(1 if issues else 0)

Reproducing the failure in a clean checkout

A gate that only passes on a dirty working tree teaches nothing, so reproduce the failure once in a throwaway worktree. This isolates the patch from editor state, local build artifacts, and half-installed dependencies.

git worktree add ../oss-repro -b preflight-check origin/main
cd ../oss-repro
git cherry-pick <your-commit-sha>
bash scripts/preflight.sh

If the gate passes in the clean worktree and fails in CI, the difference is environmental rather than logical, and the useful artifact becomes a pinned environment file instead of another code change.

Step 3: Use a free model tier as a second reviewer

Preflight catches mechanical rules, but it cannot notice that a diff touches a code path the rule file never anticipated. That is where a free model tier adds value, provided the review is constrained to the same rule file that the gate uses.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, and the workflow below uses the free model access tier for the two reading-heavy steps.

Input A: RULES (the .contrib-rules.yml content)
Input B: DIFF (unified diff, no unrelated files)

List every rule from RULES that the diff may violate.
For each candidate, output: rule path, evidence line from the diff, confidence (low/medium/high),
and the exact question a human must answer.
Do not suggest preferences that are absent from RULES.
If no rule applies, return the single line: no rule-based findings.

The prompt is intentionally boring, because an unconstrained reviewer produces stylistic noise that costs more to triage than it saves. Every finding is a question for a human, never an automatic change request.

Where the free server option fits

Some projects cannot run their test suite on a contributor laptop, because the suite needs containers, services, or a long wall-clock budget. In those cases the free server option can host the preflight and test run, so the gate executes in a consistent environment instead of a laptop that happens to have the right toolchain.

Availability limits, provisioning details, and whether the free tier covers a specific workload should be confirmed against the project's current documentation before a team depends on it. Treat the free path as a convenience for review and reproduction steps, not as a substitute for the project's own CI.

Decision table: what to automate, what to keep human

Task	Delegate to a model?	Human check required
Extract rules from CONTRIBUTING.md	Yes, free tier is sufficient	Every rule and its source line
Branch and file-naming checks	No, run deterministically in preflight	None beyond approving the rules
Test failure triage	Yes, as a summarizer	Reproduce the failure in a clean worktree
Writing the patch itself	Optional, with a scoped prompt	Full diff review plus test evidence
Changelog and docs wording	Yes, as a first draft	Accuracy of version and behavior claims
Security-sensitive changes	No	Maintainer review and threat modeling
Final merge decision	Never	Maintainer, always

Limitations and failure modes

Rule extraction can misread a conditional sentence, so needs_human markers matter more than tidy output.
Model review produces confident false positives, and each one consumes maintainer attention.
The rules file drifts from CONTRIBUTING.md unless a scheduled CI job re-extracts and diffs it.
A preflight gate hides failures only if contributors remember to run it, so wire it into a pre-push hook rather than a wiki page.
Free-tier availability is not a service guarantee, and long-running suites still need a fallback path.

Who should skip this workflow

Repositories that already expose one authoritative make check target do not need a parallel gate, because the rules are already executable. Projects with strict supply-chain requirements should keep model-assisted review out of the loop entirely, since prompts can leak unreleased code. Single-maintainer projects with low contribution volume usually gain more from a shorter CONTRIBUTING.md than from automation.

A short checklist before push

Confirm which sentence in CONTRIBUTING.md each rule in the YAML file came from.
Run the preflight script in a clean worktree, not in your editor's dirty tree.
Restrict model review to rules that already exist, and convert every finding into a question.
Re-extract the rules file when the underlying document changes, and review the diff.
Revert any change that the gate cannot explain in one sentence.

The goal is not to automate contribution judgment; it is to stop paying review latency for rules that were written down months ago. Teams that want to try the free model access and free server path described above can evaluate it against their own repository rules before adopting any of it.