DEV Community: Emery Lin

A Model-Backed Test That Passed Once Is a Sample Size of One

Emery Lin — Tue, 27 Oct 2026 13:24:38 +0000

A single green run of a model-backed test tells you almost nothing. It is one draw from a distribution you have never measured. Until you know that distribution, your merge button is a coin flip with extra steps.

Here is the fix in one sentence: sample each model-backed fixture N times, write a ledger, and let the merge rule read the ledger instead of the last exit code.

This post gives you the fixture contract, a reference sampler, a decision table, and two hooks — a cheap one locally, a strict one in CI.

Why reruns make the problem worse

Most teams handle a red model test by rerunning it. It goes green. Everyone moves on. Repeat that three times and you have trained the team to ignore the check.

There are two failure modes and they look identical in the UI. A real regression that failed once and passed on retry. Or a genuinely unstable case that passed this time and will fail after lunch. Reruns cannot separate them, because a rerun only gives you a second sample you throw away.

So stop rerunning to green. Start recording samples, then decide.

Step 1: Make the fixture assert on invariants, not prose

Most false flakes are self-inflicted. If your assertion is "output equals this paragraph," every run is a coin flip. Assert on structure, ranges, and required fields instead.

# tests/model/cases/summarize_changelog.yaml
id: summarize_changelog
input: fixtures/changelog.txt
expect:
  format: json
  required_keys: [breaking, added, removed]
  max_items_per_key: 5
  forbid_substrings: ["I cannot", "As an AI"]
  max_output_chars: 1200

Four rules keep this honest. Assert JSON shape, not wording. Assert counts and bounds, not exact strings. Never assert on whitespace or markdown formatting. And keep one "golden" case per feature, not per fixture — golden cases are for humans to read, not for the gate.

Now write each case outcome to a log so a sampler can read it.

# tests/model/conftest.py
import json, os
from pathlib import Path
import pytest

LOG = Path(os.environ.get("MODEL_RUN_LOG", "artifacts/run.jsonl"))
RUN_ID = os.environ.get("MODEL_RUN_ID", "local")

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when != "call":
        return
    LOG.parent.mkdir(parents=True, exist_ok=True)
    with LOG.open("a") as fh:
        fh.write(json.dumps({"run": RUN_ID, "case": item.nodeid, "passed": report.passed}) + "\n")

Step 2: Sample, tally, and write a ledger

The sampler below is a reference implementation. Trim it, rename it, and run it against your own suite — the numbers it prints will be yours, which is the only kind worth having.

#!/usr/bin/env python3
# tools/sample_model_tests.py
import hashlib, json, os, subprocess, sys
from collections import defaultdict
from pathlib import Path

RUNS = int(os.environ.get("SAMPLE_RUNS", "5"))
FLAKY_BUDGET = int(os.environ.get("FLAKY_BUDGET", "0"))
CASES = Path("tests/model/cases")
LEDGER = Path("artifacts/model-ledger.json")


def fixture_digest() -> str:
    h = hashlib.sha256()
    for p in sorted(CASES.glob("*.yaml")):
        h.update(p.name.encode())
        h.update(p.read_bytes())
    return h.hexdigest()[:12]


def main() -> int:
    digest = fixture_digest()
    log = Path("artifacts/run.jsonl")
    log.unlink(missing_ok=True)

    for i in range(RUNS):
        env = {**os.environ, "MODEL_RUN_ID": f"{digest}-{i}", "MODEL_RUN_LOG": str(log)}
        subprocess.run([sys.executable, "-m", "pytest", "tests/model", "-q"], env=env, check=False)

    tally = defaultdict(lambda: [0, 0])  # case -> [passes, runs]
    for line in log.read_text().splitlines():
        row = json.loads(line)
        tally[row["case"]][1] += 1
        tally[row["case"]][0] += int(row["passed"])

    cases, flaky = {}, []
    for case, (passes, runs) in sorted(tally.items()):
        rate = passes / runs
        state = "stable-pass" if rate == 1 else "stable-fail" if rate == 0 else "flaky"
        if state == "flaky":
            flaky.append(case)
        cases[case] = {"passes": passes, "runs": runs, "pass_rate": round(rate, 2), "state": state}

    ledger = {"fixture_digest": digest, "runs_per_case": RUNS, "flaky": flaky, "cases": cases}
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    LEDGER.write_text(json.dumps(ledger, indent=2) + "\n")
    print(json.dumps({"fixture_digest": digest, "runs_per_case": RUNS, "flaky": flaky}, indent=2))

    if len(flaky) > FLAKY_BUDGET:
        print(f"BLOCK: {len(flaky)} flaky case(s) exceed budget {FLAKY_BUDGET}", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Two details matter more than they look. The fixture_digest ties a ledger to an exact set of fixtures, so a stale ledger cannot silently bless new cases. And the budget is a number you set deliberately, not a vibe.

Step 3: Turn the ledger into a merge rule

A pass rate is only useful if it maps to an action. This table is the whole policy — copy it into your PR template so reviewers see the same rule the gate uses.

Ledger state	Pass rate	Merge decision
stable-pass	1.00 over 5 runs	Merge. Keep it as a required check.
stable-fail	0.00 over 5 runs	Block. This is a regression, not noise.
flaky, mostly passing	≥ 0.60	Quarantine to nightly, open a ticket, block only if the case guards a high-blast-radius path.
flaky, mostly failing	< 0.60	Block. Treat it as failing until proven otherwise.
fixtures changed	digest mismatch vs. baseline	Block pending an explicit re-baseline in the PR body.

The "mostly passing" row is where teams cheat. A case at 0.6 passes four times out of five, so it will merge bad code eventually. Quarantine is a debt ticket, not a fix.

Step 4: Two hooks, two different jobs

Run a cheap sample before you push, and a strict one in CI. The local hook should be fast enough that nobody disables it.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
SAMPLE_RUNS=3 FLAKY_BUDGET=0 python tools/sample_model_tests.py

Install it once per clone:

git config core.hooksPath .githooks
chmod +x .githooks/pre-push

The CI job runs more samples and uploads the ledger as an artifact, so a reviewer can read the evidence instead of trusting the checkmark.

model-flake-gate:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - name: Sample model-backed fixtures
      env:
        MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
        MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
        SAMPLE_RUNS: "5"
        FLAKY_BUDGET: "0"
      run: python tools/sample_model_tests.py
    - uses: actions/upload-artifact@v4
      if: always()
      with:
        name: model-ledger
        path: artifacts/model-ledger.json

Pin action versions to commit SHAs in a real repository, and keep the API key out of logs — the sampler never needs to echo it.

Where MonkeyCode fits in this workflow

Sampling multiplies your run count by five, which is exactly the workload you do not want to pay per call for while you are still calibrating fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source project offering free model access — currently stated as 10 million free tokens — plus a free server option. For this workflow that combination is a reasonable place to run the sampler: point MODEL_BASE_URL at the free endpoint, and use the free server to host the sampler job so five-times runs do not eat your own CI minutes. Check the project's current terms yourself before you build a budget or a contract around them.

One caveat that is not a caveat but a requirement: temperature 0 reduces variance, it does not remove it. Ledgers are endpoint-scoped. If you switch providers or models, re-baseline, or you will compare two different distributions and call it a regression.

Who should not use this approach

If you need contractual latency or availability guarantees, data residency, or audited per-request logs, a free tier is the wrong foundation for your gate — use a provider you can hold to terms. If your model tests are a demo rather than a required check, the sampler is overhead; one run is fine. And if your fixtures assert on exact prose, fix the fixtures first, because sampling will only tell you what you already know.

The useful habit is smaller than the tooling: stop asking whether the test passed, and start asking how often. If you want to try the sampler before wiring a billing account, the free endpoint and free server are enough to get your first ledger written.

Your Review Prompt Is a Dependency — Pin It Before It Reaches the Merge Queue

Emery Lin — Mon, 26 Oct 2026 11:55:09 +0000

Most model-backed review jobs do not fail because the model is bad. They fail because someone edited prompts/review.md at 4pm on a Friday, and nothing in the pipeline noticed.

AI-authored diffs keep arriving faster than human review capacity grows, so more teams are wiring a model call into the PR path. That is a reasonable move. The problem is that the prompt, the decoding settings, and the fixture set now behave like production code, while being versioned like a scratch note.

This article gives you a small, reproducible artifact for that gap: a review lockfile plus a checker that fails CI when the prompt drifts without an explicit lock bump. It is boring on purpose. Boring is what you want sitting between a git push and a merge queue.

Three failure classes, three different fixes

When a model-backed review job goes red, people reach for the same reflex: rerun it. That reflex is only correct for one of the three failure classes.

Symptom	Class	Wrong fix	Right fix
`429`, timeout, DNS blip	Transport	none needed — rerun it	Backoff, request budget, alerting
Same diff, different verdict	Nondeterminism	Retry until green	k-of-n agreement, quarantine on split
Verdict flips after an unreviewed prompt edit	Drift	Usually invisible, so nothing	Lockfile + lock-bump PR

The third row is the expensive one. It does not produce a flaky red. It produces a confident green on a rule nobody approved.

So the goal is not "make the model deterministic." The goal is to make the inputs to the model reviewable, and to make disagreement loud instead of flaky.

Step 1 — Separate the inputs you can pin

Pin exactly three things. Anything else is configuration you should not silently inherit.

The prompt text — one file, no inlined strings scattered across the codebase.
The decoding settings — temperature, max tokens, top-p, stop sequences, in a committed JSON file.
The fixture set — a handful of diffs with the verdict label you expect.

The model handle itself belongs in settings too, but treat it as a named slot your operator resolves, not a hardcoded identifier you scatter across jobs. When the slot's resolution changes, your lock should change with it — loudly.

prompts/review.md              # the instruction text
review.settings.json           # decoding params + model slot name
fixtures/review/*.json         # diff + expected label
review.lock.json               # generated hashes, committed

Step 2 — Generate the lock with a canonical hash

Hashing raw file bytes is a trap. A line-ending change or a trailing space from your editor will produce a spurious drift signal, and your team will start ignoring the check within a week.

Canonicalize first. Normalize line endings, strip trailing whitespace per line, strip the trailing blank lines, then hash. Sort settings keys before serializing. This is the whole trick, and it is the difference between a check people trust and a check people --no-verify around.

#!/usr/bin/env python3
# tools/lock_review.py
import argparse, hashlib, json, pathlib, sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
LOCK = ROOT / "review.lock.json"

def canonical_text(path: pathlib.Path) -> str:
    raw = path.read_text(encoding="utf-8").replace("\r\n", "\n")
    lines = [line.rstrip() for line in raw.split("\n")]
    return "\n".join(lines).strip() + "\n"

def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def digest_prompt() -> str:
    return sha(canonical_text(ROOT / "prompts/review.md"))

def digest_settings() -> str:
    settings = json.loads(canonical_text(ROOT / "review.settings.json"))
    blob = json.dumps(settings, sort_keys=True, separators=(",", ":"))
    return sha(blob)

def digest_fixtures() -> str:
    files = sorted((ROOT / "fixtures/review").glob("*.json"))
    blob = "".join(f"{f.name}:{sha(canonical_text(f))}\n" for f in files)
    return sha(blob)

def current() -> dict:
    return {
        "schema": 1,
        "prompt_sha256": digest_prompt(),
        "settings_sha256": digest_settings(),
        "fixtures_sha256": digest_fixtures(),
    }

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--check", action="store_true")
    ap.add_argument("--update", action="store_true")
    args = ap.parse_args()

    now = current()
    if args.update:
        LOCK.write_text(json.dumps(now, indent=2) + "\n", encoding="utf-8")
        print("lock written — open a lock-bump PR, do not push this alone")
        return 2

    if not LOCK.exists():
        print("missing review.lock.json; run --update once and commit it")
        return 1

    locked = json.loads(LOCK.read_text(encoding="utf-8"))
    drift = {k: (locked.get(k), v) for k, v in now.items() if locked.get(k) != v}
    if not drift:
        print("review lock OK")
        return 0
    for key, (was, is_) in drift.items():
        print(f"DRIFT {key}: locked={was} current={is_}")
    return 1

if __name__ == "__main__":
    sys.exit(main())

Run it once locally to create the baseline, then treat the output as read-only:

python tools/lock_review.py --update && git add review.lock.json
python tools/lock_review.py --check
echo $?   # expect 0

Now edit one word in prompts/review.md and rerun --check. You want exit code 1 and a printed prompt_sha256 drift line. If you get a clean pass after editing the prompt, stop and fix the canonicalization before continuing.

Step 3 — Catch drift before it leaves a laptop

The CI check is the backstop. The hook is what saves your reviewers from a red build they did not cause.

#!/usr/bin/env bash
# .git/hooks/pre-push
set -euo pipefail
if ! python tools/lock_review.py --check; then
  echo "review.lock.json is stale."
  echo "Fix: python tools/lock_review.py --update, then open a lock-bump PR."
  exit 1
fi

Keep the CI side narrow. You only need it to run when the pinned inputs change:

name: review-lock
on:
  pull_request:
    paths:
      - "prompts/**"
      - "review.settings.json"
      - "fixtures/review/**"
      - "tools/lock_review.py"
      - "review.lock.json"
jobs:
  lock:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python tools/lock_review.py --check

The lock-bump PR is the whole point. A prompt change now shows up as a reviewable diff of hashes plus the prompt body, owned by a human, instead of a silent behavioral change buried in a squash merge.

Step 4 — Make disagreement quarantine, not fail

Here is where most teams get flaky. They call the model once, get approve on one run and needs_human on the next, and the job flaps. Retrying until green is not a fix; it is a coin flip with extra billing.

Run the verdict k times per fixture and require agreement. On a split, emit quarantine — a distinct state that never masquerades as pass or fail.

# tools/fixture_check.py  (sketch — call_model is your own adapter)
import json, pathlib

K = 3
LABELS = {"approve", "block", "needs_human"}

def verdict_for(diff: str) -> str:
    # your transport, your auth, your timeout handling
    return call_model(prompt=PINNED_PROMPT, diff=diff, settings=SETTINGS)

rows = []
for fx in sorted(pathlib.Path("fixtures/review").glob("*.json")):
    data = json.loads(fx.read_text())
    runs = [verdict_for(data["diff"]) for _ in range(K)]
    bad = [r for r in runs if r not in LABELS]
    if bad:
        state = "quarantine"          # unparseable output is not a verdict
    elif len(set(runs)) > 1:
        state = "quarantine"          # split decision, human decides
    else:
        state = "pass" if runs[0] == data["expected_label"] else "fail"
    rows.append((fx.name, state, runs))

for name, state, runs in rows:
    print(f"{state:10s} {name:32s} {runs}")

Two rules keep this honest. First, quarantine must be a first-class state in your reporting, not a relabeled failure. Second, never let the fixture runner write to review.lock.json — generating the lock and validating behavior are separate jobs with separate permissions.

Step 5 — The green-to-merge decision table

Write this down and put it next to the merge queue config. Ambiguity is where broken gates live.

Lock state	Fixture state	Merge queue action
drift	not run	Block. Needs a lock-bump PR.
OK	any `fail`	Block. The prompt no longer matches your expected labels.
OK	any `quarantine`	Hold. Route to a human, keep the diff out of the queue.
OK	all `pass`, verdict `block`	Block on content, not on infrastructure.
OK	all `pass`, verdict `approve`	Admit to queue.
OK	all `pass`, verdict `needs_human`	Route to a human. Do not treat as green.

That last row matters more than it looks. A model that says "I cannot judge this" is not a passing check, and a pipeline that treats it as one has quietly converted uncertainty into approval.

Running the whole loop without a budget fight

All five steps are runnable end to end before you ask anyone for spend, because none of them require a large sample. You are checking lock stability and label agreement on a dozen fixtures, not benchmarking quality.

For that bootstrap phase I used MonkeyCode's free model access and its free server option to host the fixture runner, which meant the gate existed before the procurement conversation started. Check the current terms on the official MonkeyCode project page before you plan around them — free-tier scope and availability are exactly the kind of thing that changes, and I am not going to quote you numbers I cannot verify today.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If your team already has a paid endpoint and a runner, skip the product entirely. Nothing above depends on it. The lockfile is the artifact; the vendor is a convenience.

Limitations and who should not use this

The lock proves provenance, not quality. A pinned, reviewed, thoroughly wrong prompt passes the lock check every time.
k = 3 is a starting point, not a measurement. I have not measured disagreement rates on your diffs. Log the run counts for two weeks and pick k from your own data.
Skip this if your review is fully deterministic (linters, type checkers, policy engines). You already have this property for free; a hash lock adds nothing.
Skip this if you cannot version prompts in the same repo as the fixtures. A lockfile that hashes files living in three different systems is a maintenance tax with no payoff.
Do not use quarantine as a soft pass. If humans never triage the quarantine queue, you have built an approval machine with extra steps.

The honest version of the pitch: this adds a small amount of friction to prompt edits and removes a class of silent behavior change. If your prompt changes weekly and nobody has ever been surprised by a verdict flip, you may not need it yet. When you do get surprised, come back — the checker is forty lines and it will still be here.

A 90-Minute Context-Budget Workshop Your Team Can Run This Week

Emery Lin — Tue, 20 Oct 2026 10:03:34 +0000

Most prompt bugs are arithmetic bugs wearing a personality. A team that cannot state what currently sits inside its context window cannot debug what comes out of it. So before anyone rewrites another instruction, run a short internal workshop that makes the arithmetic visible, repeatable, and shared.

What follows is a plan for a ninety-minute session with five numbered steps, one small runnable calculator, three exercises with checkable answers, and a decision table for the moment the numbers go negative. The calculator is ordinary Python that I have run locally; the exercises use illustrative constants rather than measured production figures, so verify them against your own tokenizer before quoting them in a design review.

The failure this session is designed to prevent

The recurring shape is unglamorous. Retrieved chunks grow as the corpus grows, conversation history accumulates because deleting it feels rude, and the system prompt never shrinks because it was carefully written. Eventually the oldest instructions fall silently out of the window, and the model keeps answering with confidence from a context you never intended.

Nobody notices, because nothing errors. The output stays fluent, the latency stays flat, and the only symptom is a slow drift in behaviour that gets blamed on the model.

A budget written on one page turns that drift into a diff. The workshop exists to produce that page and to make sure five people, not one, can read it.

Materials and room setup

You need a prompt repository people can actually open, one machine per pair with Python 3.9 or newer, and a tokenizer library installed. For the third exercise you also need a place to run a handful of behaviour checks, and that is the only part that touches a model.

That last part is the usual reason internal workshops never happen, because twenty participants poking a paid endpoint is a budget conversation. MonkeyCode's outreach includes free models and a free server option, which is enough to run the behaviour checks in this session without opening a procurement ticket. I have not benchmarked either, so size the room conservatively and read the current terms on the project page yourself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Book a room with a shared screen, and ask each pair to bring one real prompt file. Abstract examples produce abstract arguments.

Step 1 — Agree on one measuring stick

Tokens are the only unit your provider bills and truncates in, so characters and words are not substitutes. Pick the tokenizer your provider publishes, and if none is published, pick one library and admit in writing that it is an approximation.

python -m pip install tiktoken

# count.py — prints the token length of a file, per encoding
import os, sys, tiktoken

name = os.environ.get('TOKENIZER', 'cl100k_base')
enc = tiktoken.get_encoding(name)
text = open(sys.argv[1], encoding='utf-8').read()
print(f'{name}\t{len(enc.encode(text))}\t{sys.argv[1]}')

The teaching point here is not the number, it is the disagreement. Ask two pairs to count the same file with different encodings and put both results on the board, because a three to eight percent gap across families is normal and worth seeing once.

Record the choice in the repository, next to the prompts. A budget measured with an undocumented tokenizer is not a budget, it is a rumour.

Step 2 — Do the subtraction before the writing

Hand out the calculator and make every pair run it against their own prompt file. Deliberately keep it boring: no network, no SDK, plain arithmetic that anyone in the room can audit line by line.

#!/usr/bin/env python3
# context_budget.py — offline arithmetic, exit 1 when over budget
import json, sys
from pathlib import Path

def compute(window, reserve, parts):
    used = sum(parts.values())
    return {'window': window, 'reserved': reserve, 'used': used,
            'headroom': window - reserve - used, 'parts': parts}

cfg = json.loads(Path(sys.argv[1]).read_text(encoding='utf-8'))
result = compute(cfg['window'], cfg['reserve'], cfg['parts'])
print(json.dumps(result, indent=2, sort_keys=True))
if result['headroom'] < 0:
    print(f"OVER BUDGET by {abs(result['headroom'])} tokens", file=sys.stderr)
    sys.exit(1)

{"window": 32000, "reserve": 2000,
 "parts": {"system": 1400, "tools": 2600, "retrieved": 18000,
           "history": 9000, "scratch": 600}}

Run it and the session gets its first useful moment. This configuration is over by 1600 tokens, which the script reports on stderr with a non-zero exit code, and the room immediately starts negotiating instead of writing prose.

That negotiation is the workshop. Every suggestion is now a number someone can accept or reject on the record.

Step 3 — Three exercises, ordered by how badly they fail

Exercise one checks agreement rather than correctness. Each pair counts the same prompt file twice, on two machines, and the pass condition is that both counts match exactly; if they do not, the encodings differ and the room has found a real inconsistency.

Exercise two is the JSON above. Pairs must restore positive headroom with the smallest single change, and the expected answer is to drop roughly two retrieved chunks of about 800 tokens each, or trim history by an equivalent amount. Any pair that trims the system prompt first should explain why, because that is usually the wrong lever.

Exercise three is a behaviour check, and it needs a model. Pairs rewrite a six-sentence system prompt into three sentences without losing a constraint, then run five fixed inputs through both versions and diff the outputs. The check is not whether the shorter prompt is better, it is whether the constraints survived, and five inputs are enough to catch the obvious losses.

Step 4 — The over-budget decision table

Agreement matters more than precision here, so give the room one table and ask them to argue about the ranking rather than the wording.

Lever	Tokens recovered	Quality risk	Latency effect	Owner
Fewer retrieved chunks	High	Medium, can drop evidence	Lower	Retrieval owner
Trim oldest history turns	High	Low for chat, real for tasks	Lower	Product
Compress history into a summary	Medium	Medium, lossy	Slight increase	Prompt author
Shorten the system prompt	Low	Low if constraints kept	Negligible	Prompt author
Drop tool schemas not used	Medium	Low	Lower	Platform
Split into two sequential calls	Large per call	Low, adds complexity	Higher total	Architect

The single most common mistake in the room is treating the system prompt as the big lever. In the configuration above it is 1400 tokens out of 31600, so cutting it in half recovers less than the reserve margin.

Put that sentence on the slide. It saves a week of pointless rewriting.

Step 5 — Ship one page, not a slide deck

The deliverable is a file committed next to the prompts, roughly fifteen lines, listing the window, the reserve, the parts, and the tokenizer name. It should fail loudly in the repository's checks when the parts stop summing to something sane.

The page also names the person who owns each line, because a budget without an owner is a comment. Finish the session by having each pair read their page aloud for thirty seconds; anything confusing gets fixed while the author is still in the room.

What this workshop will not fix

Arithmetic will not tell you whether your retrieval is any good, and a perfectly budgeted context full of irrelevant chunks is still irrelevant. It also will not survive a provider change on its own, since tokenizers differ and a budget written for one family drifts for another.

Finally, it will not make anyone write better instructions. It only removes one class of invisible failure so that instruction quality becomes the visible problem.

Skip the session entirely if your prompts live in scattered chat threads, because there is nothing to budget yet. Skip it as well if your context problems are really retrieval-quality problems, and skip it if you are a solo developer who already knows what is in the window.

Assignment for next week

Each pair leaves with one task: add the budget page for their own prompt file and open a pull request that touches nothing else. Review those pull requests together at the next team meeting, and accept them only when the numbers are reproducible by a second person.

Teams that want the behaviour check from exercise three in their own workflow can point it at the free models and free server option while the exercise is still an experiment. Keep the arithmetic local, keep the page in the repository, and let the model be the least interesting dependency in the room.

Put a Token Bucket Between Your CI Jobs and the Model Endpoint

Emery Lin — Mon, 19 Oct 2026 08:45:33 +0000

If two or three CI jobs share one model endpoint, the failure you notice first is not a bad answer. It is a 429 that your retry loop converts into three slow jobs, one starved job, and a queue that reads like flakiness. Admission control fixes that. More retries make it worse.

This post builds a small gate you can run in front of a shared endpoint, a decision table for what to retry, and a local harness that reproduces the collision without spending a single token in CI.

The shape of the failure

A retry loop assumes the bottleneck is transient. With a shared endpoint, the bottleneck is your own concurrency. Job A retries, so it holds its slot longer; job B retries, so it overlaps A's second attempt; job C waits on a runner that is now billed for idle time. Nobody's prompt changed, but the pipeline got slower and less predictable.

Free tiers make this shape common because capacity is shared rather than reserved. That is a fair trade for prototyping, and a bad one for an unguarded merge path.

MonkeyCode is an open-source project that offers free model access and a free server option, which is exactly the situation where a gate earns its keep. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free allowance as a budget to be spent deliberately, not as an unmetered pipe. Check the project page for the current allowance and terms before you size anything below — free-tier numbers change, and a stale figure in a comment block is worse than no figure.

What to measure before you build anything

You need two numbers: how many model calls a pipeline run makes, and how many of those came back rate-limited. Both are already in your logs.

# calls per run
gh run view "$RUN_ID" --log | grep -c 'model_call_ok'

# rate-limited responses in the same run
gh run view "$RUN_ID" --log | grep -c 'status=429'

# how long jobs waited on the endpoint
grep 'model_wait_seconds' ci.log | awk -F= '{s+=$2} END {print s"s total wait"}'

Write the three numbers into the repository, next to the gate. If rate-limited responses are zero across a week of runs, stop here — you do not need this. If they are above zero, the ratio of wait time to wall-clock time tells you whether to raise capacity or lower concurrency.

Step 1 — One bucket, one lock file

One bucket for the whole pipeline, not one per job. A per-job bucket multiplies your concurrency by the number of jobs, which is the bug you were trying to fix.

# ci_gate/bucket.py
import fcntl, json, os, random, time
from dataclasses import dataclass

@dataclass
class Bucket:
    path: str
    capacity: int = 6          # in-flight calls you allow
    refill_per_sec: float = 0.2  # recovered calls per second

    def _load(self):
        try:
            with open(self.path) as f:
                return json.load(f)
        except FileNotFoundError:
            return {"tokens": float(self.capacity), "ts": time.time()}

    def _write(self, state):
        tmp = self.path + ".tmp"
        with open(tmp, "w") as f:
            json.dump(state, f)
        os.replace(tmp, self.path)

    def acquire(self, timeout=90.0):
        deadline = time.time() + timeout
        while True:
            with open(self.path + ".lock", "w") as lk:
                fcntl.flock(lk, fcntl.LOCK_EX)
                state = self._load()
                now = time.time()
                tokens = min(
                    self.capacity,
                    state["tokens"] + (now - state["ts"]) * self.refill_per_sec,
                )
                if tokens >= 1:
                    self._write({"tokens": tokens - 1, "ts": now})
                    return True
                self._write({"tokens": tokens, "ts": now})
                wait = (1 - tokens) / self.refill_per_sec
            if time.time() > deadline:
                return False
            time.sleep(min(wait, 5.0) * (1 + random.random() * 0.3))

Two properties matter here. The lock makes the read-modify-write atomic, and the timeout means a job fails loudly instead of hanging until the workflow timeout kills it. The default numbers are placeholders — set capacity from your measured concurrency, not from optimism.

Step 2 — Separate admission from retry

Admission asks may I call now. Retry asks should I call again. Mixing them produces the loop that starves your other jobs.

# ci_gate/call.py
import time
from .bucket import Bucket

BUCKET = Bucket(path="/var/ci/model_bucket.json", capacity=6, refill_per_sec=0.2)

def call_model(payload, attempts=3):
    for i in range(attempts):
        if not BUCKET.acquire(timeout=90):
            raise SystemExit("admission timeout: not calling, not retrying")
        resp = post(payload)
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code != 429:
            raise RuntimeError(f"non-retryable status={resp.status_code}")
        time.sleep(min(2 ** i, 20) * (1 + random.random()))
    raise RuntimeError("exhausted attempts after admission")

Every attempt re-enters the bucket, so a retry costs a token like any other call. If you skip that, retries bypass the only mechanism you built.

Step 3 — Classify before you retry

The gate stops the pile-up. The table stops you from retrying things that cannot succeed.

Signal	Retry?	Where the fix actually belongs
429 with `Retry-After`	Yes, after admission	Honour the header; lower `capacity`
429 with no header	Yes, capped at 3, jittered	Your `capacity` is above real capacity
500 / 502 / 503	Once	Endpoint health, not your prompt
400 / 422	Never	Prompt or fixture is wrong
Timeout after admission	Once, and log the duration	Your latency budget, not the bucket
Output differs from fixture	Never	This is a test failure, not infra

That last row is the one teams get wrong. A model answer that no longer matches a recorded fixture is a signal about the model or your prompt, and retrying it politely until it agrees is how a green check stops meaning anything.

Step 4 — Reproduce the collision locally, not in CI

You cannot debug concurrency in a pipeline that takes eleven minutes to start. Use a fake endpoint that rate-limits the first few requests and run the real gate against it.

# tests/fake_endpoint.py
import itertools, json
from http.server import BaseHTTPRequestHandler, HTTPServer

seen = itertools.count()

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = next(seen)
        code = 429 if n < 4 else 200
        body = {"ok": True} if code == 200 else {"error": "rate_limited"}
        raw = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Retry-After", "1")
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(raw)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8787), Handler).serve_forever()

Run three workers against it and assert on behaviour, not on elapsed time:

python tests/fake_endpoint.py &
python -m pytest tests/test_gate.py -k parallel -q
# assert: requests_in_flight <= capacity at all times
# assert: total calls stayed within the attempt budget
# assert: exactly one job exited on admission timeout

Keep this test deterministic: no network, no sleeps longer than the retry cap, no wall-clock assertions. A gate whose own tests are flaky will be deleted by the second person who sees it go red.

Step 5 — Where the gate process lives

The lock file needs a filesystem shared by every job that calls the endpoint. On ephemeral hosted runners, /var/ci/ disappears between jobs, and your bucket silently resets to full every time — the worst kind of green.

Two honest options. Run the gate as a small service on one always-on host, and have jobs ask it for a slot over HTTP. Or run the whole pipeline, gate included, on a single machine you control. MonkeyCode's free server option is one place to put that process while you are evaluating the workflow; a lint-checked systemd unit beside the repo keeps it reproducible either way.

The pre-commit hook that keeps the gate honest

A hook cannot test the endpoint, but it can test the gate.

# .git/hooks/pre-commit
#!/usr/bin/env bash
set -euo pipefail
python -m pytest tests/test_gate.py -q --no-header
python -m ruff check ci_gate/

If the hook and CI run the same two commands, a broken bucket fails in seconds instead of after a queue. That is the whole green-to-merge path: same gate locally, same gate in CI, one shared budget.

Limitations and who should skip this

fcntl.flock is POSIX-only; on Windows runners you need a different primitive or the service model. Token buckets also cannot discover the endpoint's real capacity — you supply the number, and a wrong number just fails more quietly. And if your model call is not on the merge path at all, this is infrastructure for a problem you do not have.

Skip it if you have a dedicated quota, if a single job calls the endpoint once per run, or if your team has no owner for the gate's configuration. An unowned limit is a limit that gets raised on the first red build.

Merge checklist

Logged calls-per-run and 429s-per-run for a full week.
Set capacity from the measured peak, not from the highest number that once went green.
Verified retries re-enter the bucket.
Confirmed fixture mismatches are classified as test failures, never retried.
Ran the fake-endpoint test in the pre-commit hook and in CI.
Documented which host owns the lock file, and what happens when it moves.

If you try the workflow, run the fake-endpoint suite against MonkeyCode's free model access first and compare your measured 429 rate with the table above — that is the cheapest way to find out whether your concurrency was ever the problem.

Your CI Gate Should Not Call a Hosted Model: Record, Replay, Then Merge

Emery Lin — Sun, 18 Oct 2026 07:23:27 +0000

A model call inside a required check is an uncontrolled dependency. The lane that blocks merge should read fixtures from disk. Only the lane that cannot block merge should touch the network.

That single split removes most of the flakiness people blame on "AI in CI". It also makes review cheaper, because every model answer your gate saw is a file you can diff.

Below is the adapter, the contract test, the two-job workflow, and the triage order I use when the gate goes red.

Why the live call breaks the gate

Three failure shapes show up almost immediately once a model call sits behind a required check.

Latency variance. A 900 ms call and a 14 s call in the same job produce different timeouts on different days. Your retry logic then hides a real prompt regression behind a slow afternoon.

Sampling drift. Even at low temperature, output shape changes between runs. A gate that asserts on prose will pass on Monday and fail on Tuesday with no code change in between.

Outage equals red. When the provider is down, your PR shows as failing. The author sees a red check and starts editing code that was never broken.

There is a fourth problem that gets less attention: secrets. If the blocking job holds a model key, that key is now in the same process as pull-request code you do not control. Moving live calls out of the blocking lane is a security decision, not only a stability one.

The anti-pattern looks like this:

# do not do this in a required check
try:
    return call_model(prompt)
except TimeoutError:
    return load_cassette(prompt)   # silent fallback hides drift

A silent fallback is worse than no fallback. You cannot tell a replayed answer from a live one in the logs, so nobody knows which runs actually tested anything.

Two lanes, one adapter

Before writing code, agree on which lane does what.

Lane	Trigger	Network	Blocks merge	Purpose
`gate`	every PR	none	yes	prove your code handles the recorded contract
`live-drift`	nightly schedule	yes	no	detect provider or model drift, open an issue
`record`	manual, local	yes	n/a	write or refresh one cassette, reviewed in the PR

If a lane cannot block merge, it cannot hold a secret that PR code can reach. Put the model key on live-drift only, and keep the gate job at contents: read.

Build the adapter in five steps

1. Make the call hashable

Hash every input that can change the answer, including the template identity. If you hash only the prompt text, an edit to the system instruction can silently reuse a stale cassette.

# ci/model_gate.py
from __future__ import annotations

import hashlib
import json
import os
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable

ADAPTER_VERSION = "3"  # bump when hashing or request shape changes
CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "ci/cassettes"))
MODE = os.environ.get("MODEL_MODE", "replay")  # replay | record | live


class MissingCassette(RuntimeError):
    pass


@dataclass(frozen=True)
class Call:
    template_id: str
    template_version: int
    prompt: str
    temperature: float = 0.0
    max_tokens: int = 512


def cassette_key(call: Call) -> str:
    blob = json.dumps(
        {"adapter": ADAPTER_VERSION, **asdict(call)},
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:20]

2. Fail closed in replay mode

Missing fixture means missing fixture. Raise, do not call out.

Transport = Callable[[Call], dict]


def run(call: Call, transport: Transport | None = None) -> dict:
    path = CASSETTE_DIR / f"{cassette_key(call)}.json"

    if MODE == "replay":
        if not path.exists():
            raise MissingCassette(
                f"no cassette for {call.template_id} v{call.template_version} "
                f"({path.name}); record it with MODEL_MODE=record"
            )
        return json.loads(path.read_text())["response"]

    if MODE == "record":
        if path.exists():
            return json.loads(path.read_text())["response"]
        if transport is None:
            raise ValueError("record mode needs a transport")
        response = transport(call)
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(
            json.dumps(
                {"request": asdict(call), "response": response},
                indent=2,
                sort_keys=True,
            )
            + "\n"
        )
        return response

    if transport is None:
        raise ValueError("live mode needs a transport")
    return transport(call)

Store the request inside the cassette. Reviewers need to see what was asked, not only what came back.

3. Assert contracts, not prose

Your gate exists to prove the shape your code consumes. Validate structure and bounds, then stop.

# ci/contracts.py
PATCH_PLAN = {
    "type": "object",
    "required": ["files", "reason"],
    "properties": {
        "files": {"type": "array", "items": {"type": "string"}, "maxItems": 20},
        "reason": {"type": "string", "minLength": 10},
    },
    "additionalProperties": False,
}

# tests/test_patch_plan_contract.py
import pytest
from jsonschema import ValidationError, validate

from ci.contracts import PATCH_PLAN
from ci.model_gate import Call, run


def test_plan_is_mergeable(cassette_prompt):
    call = Call(template_id="patch_plan", template_version=4, prompt=cassette_prompt)
    result = run(call)
    try:
        validate(result, PATCH_PLAN)
    except ValidationError as exc:
        pytest.fail(f"recorded output breaks the contract: {exc.message}")

Do not assert on wording. The moment you test prose, you have a test that fails for reasons you cannot reproduce.

4. Split the workflow

The gate job gets no key. The nightly job gets the key and is allowed to fail.

# .github/workflows/model-gate.yml
name: model-gate
on:
  pull_request:
  schedule:
    - cron: "17 3 * * *"

jobs:
  gate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      contents: read
    env:
      MODEL_MODE: replay
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r ci/requirements.txt
      - run: pytest tests/ -q

  live-drift:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    continue-on-error: true
    env:
      MODEL_MODE: live
      MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
      MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r ci/requirements.txt
      - run: pytest tests/ -q -m live

This is also where a free endpoint earns its place. The nightly lane makes a small, bounded number of real calls, so it is the right consumer of a free model allowance. MonkeyCode's free model access and free server option fit that lane: you can run the drift job without opening a paid budget line first, and the free server option gives the schedule somewhere to live.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms for free offerings change. Check the current terms on the project page before you let a nightly job depend on them, and keep a continue-on-error job non-blocking until you have watched it for a week.

5. Keep the fixtures small

One cassette per template version. When you bump template_version, the old file stops matching and you should delete it in the same commit. If ci/cassettes/ grows without bound, nobody reviews it, and an unreviewed fixture is a test with a comment that lies.

When the gate goes red, triage in this order

Read the exception type. MissingCassette is a fixture problem. ValidationError is a contract problem. A KeyError on response means a hand-edited or truncated cassette.
MissingCassette: did you change the prompt? Record once with MODEL_MODE=record, then read the cassette diff. If the new answer surprises you, that is the review you wanted.
ValidationError: fix the fixture, not the assertion. The recorded output never satisfied the contract. Either the prompt is underspecified or the contract is wrong. Decide which, in the PR.
Contract passes but production breaks. Your schema is looser than your consumer. Tighten maxItems, add additionalProperties: false, and add a bound you actually enforce downstream.

Track one number: cassette miss rate per week. Above roughly one in ten PR runs means your templates are churning, and the gate is measuring your edit speed rather than your code.

Limits worth stating out loud

Replay proves your code handles a known answer set. It cannot prove the model still behaves. That is the nightly lane's job, and it needs a human to read the failure.

Cassettes go stale when a provider swaps a model behind an endpoint name. Pin the model identifier in the request and re-record deliberately, not on a schedule that erases the diff.

A single recorded answer is not an evaluation. If you need quality numbers, you need multiple samples and a scoring rule, which is a different pipeline with different costs.

And the drift job is a smoke alarm, not a gate. Once it blocks merges, you have rebuilt the problem this article removes.

Who should not do this

Skip the replay lane if your prompts are assembled at runtime from unbounded user input, because the cassette space explodes and you will spend more time pruning than testing.

Skip it if you need a per-PR quality score on model output. Replay gives you a contract check, and a contract check is not a quality signal.

And skip it if nobody on the team owns the prompts. Record/replay makes prompt changes visible, which is only useful if someone is accountable for them.

Start with one template. Record it, wire the gate, and watch the miss rate for a week before you add the second. If you want to stand up the nightly lane before you commit a budget, the free model access and free server option in the MonkeyCode project are a reasonable place to try the loop, terms permitting.

A Green Pipeline That Ran Three of Five Shards Is Not Green

Emery Lin — Sat, 17 Oct 2026 06:25:04 +0000

Your merge queue trusts a color. It should trust a coverage claim.

A sharded CI run where shard 3 died during setup, shard 5 collected zero tests, and shard 1 passed on its third attempt produces exactly the same green check as a clean sweep of the whole matrix. Nothing in the check name tells you which one you got. The fix is not a smarter retry policy — it is a per-shard manifest plus a gate that refuses to merge anything the manifest cannot prove.

Four ways a shard quietly disappears

The matrix shrinks. --splits is computed from a test count that changed between the plan job and the test job. Fewer jobs get scheduled, needs is satisfied, the workflow is green.
A partition is empty. Shard 4 is assigned no tests. pytest exits with code 5, and someone already appended || true to keep the pipeline moving.
A step is allowed to fail. continue-on-error: true left over from a migration week means a crashed shard reports success at the job level.
A retry hides the first attempt. pytest-rerunfailures turns a flake into a pass, and the report contains no field that says "this took three tries."

All four are coverage bugs, not flake bugs. Treat them that way.

The contract: one manifest per shard

Stop reading the job summary. Have every shard write a small JSON file next to its report, and let the gate read only those files.

The manifest needs six fields and nothing more: index, commit, collected, node_ids, status, attempts. Add quarantined if you run a quarantine lane. If a field is missing, the gate blocks — absence is not consent.

Step 1 — Discover once, split deterministically

Collect node IDs in a single job, persist them, and split from that file. Never recompute the split inside the test job.

python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
wc -l < nodes.txt   # the number every later step must agree with

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      shards: ${{ steps.plan.outputs.shards }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - id: plan
        run: |
          python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
          n=$(wc -l < nodes.txt)
          echo 'shards='$(( (n + 199) / 200 )) >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with: { name: nodes, path: nodes.txt }

Step 2 — Emit the manifest from a hook

Do not parse the terminal output. Use pytest_sessionfinish and derive everything from session.items.

# conftest.py
import json, os, pathlib

def pytest_sessionfinish(session, exitstatus):
    out = pathlib.Path(os.environ.get('SHARD_DIR', 'shard-reports'))
    out.mkdir(parents=True, exist_ok=True)
    nodes = sorted({item.nodeid for item in session.items})
    payload = {
        'index': int(os.environ['SHARD_INDEX']),
        'commit': os.environ['GITHUB_SHA'],
        'collected': len(nodes),
        'node_ids': nodes,
        'status': 'passed' if exitstatus == 0 else 'failed',
        'attempts': int(os.environ.get('SHARD_ATTEMPT', '1')),
        'quarantined': os.environ.get('QUARANTINE') == '1',
    }
    (out / f'shard-{payload["index"]}.json').write_text(json.dumps(payload))

Expose SHARD_INDEX from the matrix, not from a counter inside the process. Counter drift is how two shards end up writing shard-0.json and the gate sees four files where five were expected.

Step 3 — Gate on the manifest, not the color

Stdlib only, no dependencies, no network. It exits non-zero on the first violation it can explain.

# gate_shards.py
import json, os, pathlib, sys

head = os.environ['GITHUB_SHA']
expected = int(os.environ['SHARD_COUNT'])
files = sorted(pathlib.Path(sys.argv[1]).glob('shard-*.json'))

def block(msg):
    print(f'::error::{msg}')
    sys.exit(1)

if len(files) != expected:
    block(f'found {len(files)} manifests, expected {expected}')

indexes, owner, retried = set(), {}, []
for path in files:
    m = json.loads(path.read_text())
    commit, index, status = m['commit'], m['index'], m['status']
    if commit != head:
        block(f'{path.name} tested {commit[:8]}, head is {head[:8]}')
    if index in indexes:
        block(f'shard {index} reported twice')
    indexes.add(index)
    if status != 'passed':
        block(f'shard {index} finished as {status}')
    if m['collected'] == 0:
        block(f'shard {index} collected zero tests')
    if m['attempts'] > 1 and not m['quarantined']:
        retried.append(index)
    for node in m['node_ids']:
        if node in owner:
            block(f'{node} ran in shards {owner[node]} and {index}')
        owner[node] = index

missing = sorted(set(range(expected)) - indexes)
if missing:
    block(f'no result for shard(s) {missing}')
if retried:
    block(f'green only after a retry in shard(s) {retried}')
print(f'OK: {len(owner)} node ids across {expected} shards on {head[:8]}')

Note what the duplicate check buys you: a split bug that sends the same 40 tests to two shards now fails the gate instead of inflating your pass count.

Step 4 — Wire it as its own required check

  gate:
    needs: [plan, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: shard-*
          path: shard-reports
          merge-multiple: true
      - env:
          SHARD_COUNT: ${{ needs.plan.outputs.shards }}
          GITHUB_SHA: ${{ github.event.pull_request.head.sha }}
        run: python3 gate_shards.py shard-reports

Two details earn their keep. if: always() stops the gate from being skipped when test fails, and a skipped gate reads like a passed one in most UIs. Overriding GITHUB_SHA with the PR head SHA means the manifest is compared against the code you reviewed, not the merge commit.

Step 5 — Make the retry ledger boring

The attempts > 1 rule is the flake policy, and it is deliberately blunt: a test that only goes green on the second attempt blocks the merge until someone either fixes it or moves it to the quarantine lane with an owner and an expiry date. Rerunning until green is not a decision, it is a coin flip with a paper trail.

Track the top five quarantined files weekly. If the list does not shrink, your quarantine is where tests go to be forgotten.

Where a free model actually fits

Not in the gate. The gate must stay deterministic, offline, and explainable in one line of output.

Triage is a different job: turning twenty scattered failure snippets into three probable root causes is exactly the kind of summarization that saves a reviewer ten minutes. MonkeyCode is an open-source project that offers free model access and a free server option, which makes it a low-friction place to run that advisory step before you decide to pay for anything. The project advertises a 10M-token allowance and a free server tier — treat both as vendor claims and check the current numbers in its console before you build a dependency on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# triage.py — advisory only; never writes status
import json, os, urllib.request, pathlib

fails = []
for path in pathlib.Path('shard-reports').glob('shard-*.json'):
    fails.extend(json.loads(path.read_text()).get('failures', []))
if not fails:
    raise SystemExit('nothing to triage')

prompt = 'Group by shared root cause. Markdown, max 200 words.\n\n' + '\n'.join(fails[:20])
body = json.dumps({'messages': [{'role': 'user', 'content': prompt}], 'temperature': 0}).encode()
req = urllib.request.Request(
    os.environ['MONKEYCODE_BASE_URL'] + '/chat/completions',
    data=body,
    headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_KEY'],
             'Content-Type': 'application/json'},
)
print(urllib.request.urlopen(req, timeout=60).read().decode())

Read the base URL and model ID from the project's own docs; endpoints move, and a hardcoded one in a tutorial rots the week after publication. Scrub secrets and customer identifiers from fails before it leaves the runner — failure logs are the most common accidental leak in this pattern.

Hard rule: triage output lands in a PR comment. It never sets status, never flips quarantined, never authorizes a merge.

Limits, and who should skip this

Small suites. Under roughly 200 tests, sharding overhead exceeds the wall-clock win, and one manifest beats five.
Non-hermetic runners. If your runner image digest can change mid-run, a passed shard is not reproducible and the manifest proves less than you think.
Regulated audit trails. Don't put a compliance gate behind a free tier whose quota and retention you have not verified in writing.
Anyone tempted to parse model prose. If you cannot keep the machine summary out of the decision path, don't add the summary.

The checker above is stdlib Python 3.12 and untested against your report format — adapt the status mapping to your runner before you make the gate required. Once it is required, the interesting metric stops being pass rate and becomes coverage: how many node IDs, from how many shards, on which commit.

If you want to try the triage step before wiring a billing account, the project's free model access and free server option are the shortest path to finding out whether it earns a slot in your pipeline.

Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

Emery Lin — Fri, 16 Oct 2026 05:25:39 +0000

Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

A retry counts as evidence only when it replays from a cold cache and leaves a record a later job can verify. A warm rerun proves that your cache works. It does not prove that your branch builds.

Most advice about flaky CI ends at "just retry the job". That advice quietly hands your merge queue a pass that was never produced under the same conditions as a fresh clone, and it leaves nothing behind for the merge job to inspect. This post fixes the boring half of the problem: make one replay emit a machine-checkable admission record, then make the merge path refuse anything else.

The workflow below is a proposal you can adapt, not a transcript of a run I am handing you as a benchmark. Label your own numbers when you measure them.

The failure mode, stated precisely

A pull request fails on a cold runner because a dependency resolves differently without the warm module cache. Someone clicks retry. The job runs on the same runner, reuses the toolchain, and goes green.

The green check is real, but its subject is different. It tested the tree plus the warm cache, and your merge queue only knows about the tree. When main is built from scratch later, the original failure returns, now attributed to a merge that "passed CI".

So the rule is short: a replay resets the environment and writes down what the environment was. Everything else is a cache hit wearing a check mark.

Write the admission record first

Define the record before you write the workflow. You want fields that the merge job can compare against current reality, not a prose summary. Keep it to one small JSON artifact.

{
  "schema": "admission/v1",
  "tree_sha": "9f1c4e2",
  "base_sha": "3ab07d1",
  "runner_image": "ghcr.io/acme/ci-runner@sha256:91a0...",
  "hook_version": 7,
  "cache": "cold",
  "seed": 4211,
  "toolchain": { "node": "20.11.1", "python": "3.12.3" },
  "result": "pass",
  "started_at": "2026-10-16T09:12:04Z"
}

Four fields do the heavy lifting. cache must equal cold, or the record is void by definition. base_sha catches the case where main moved after your replay. runner_image catches a silent runner rotation. hook_version lets you invalidate every old record in one commit when you change the test harness itself.

If you have already learned not to let a cancelled run count as a pass, the same logic applies here: a cancelled or timed-out replay writes no record, and a missing record is a failure, never a neutral state.

Step 1: purge, then replay

The workflow has one job that runs on a clean runner and deliberately destroys every cache it can reach. Note that the retry path and the first attempt use the same job; there is no separate "fast" path for reruns.

name: replay
on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  checks: write

jobs:
  cold-replay:
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0

      - name: Record the base commit
        run: echo BASE_SHA=${{ github.event.pull_request.base.sha }} >> $GITHUB_ENV

      - name: Record the tree commit
        run: echo TREE_SHA=$(git rev-parse HEAD^{tree}) >> $GITHUB_ENV

      - name: Purge every warm artifact
        run: rm -rf node_modules .venv ~/.npm ~/.cache/pip ~/.cargo/registry

      - name: Read the runner image digest
        run: echo IMAGE_DIGEST=$(jq -r .image /etc/runner-manifest.json) >> $GITHUB_ENV

      - name: Install pinned toolchain
        run: |
          corepack enable
          npm ci --no-audit --no-fund

      - name: Run the suite with a fixed seed
        run: npm test -- --seed=${{ github.run_number }}4211 --reporter=json > test-report.json

      - name: Emit the admission record
        if: always()
        run: python3 scripts/emit_record.py

      - uses: actions/upload-artifact@v4
        with:
          name: admission-record
          path: admission.json
          retention-days: 14

Two details matter more than the rest. The seed is derived from the run number so a retry is a genuinely different ordering, not a repeat of the same lucky interleaving. And the record upload uses if: always(), so a failing replay still produces an artifact that says "result: fail" — invisible failures are how teams end up re-running until green.

Step 2: gate the merge on the record

The merge job never runs the test suite. It downloads the record and checks it against the repository as it exists right now. If any input moved, the record is stale and the merge is refused.

#!/usr/bin/env python3
# scripts/admit.py
import json, subprocess, sys

REQUIRED = {
    'schema', 'tree_sha', 'base_sha', 'runner_image',
    'hook_version', 'cache', 'seed', 'result',
}

def git(*args):
    return subprocess.check_output(['git', *args], text=True).strip()

def admit(record, head_ref, base_sha, image_digest, hook_version):
    missing = REQUIRED - record.keys()
    if missing:
        return False, f'missing fields: {sorted(missing)}'
    if record['schema'] != 'admission/v1':
        return False, 'unknown record schema'
    if record['cache'] != 'cold':
        return False, 'record came from a warm cache'
    if record['result'] != 'pass':
        return False, 'record is not a pass'
    if record['tree_sha'] != git('rev-parse', head_ref + '^{tree}'):
        return False, 'tree changed after the replay'
    if record['base_sha'] != base_sha:
        return False, 'base moved; rebase and replay'
    if record['runner_image'] != image_digest:
        return False, 'runner image rotated; replay on the new image'
    if str(record['hook_version']) != str(hook_version):
        return False, 'hook version changed'
    return True, 'admitted'

if __name__ == '__main__':
    record = json.load(open(sys.argv[1]))
    ok, reason = admit(
        record,
        head_ref=sys.argv[2],
        base_sha=sys.argv[3],
        image_digest=sys.argv[4],
        hook_version=sys.argv[5],
    )
    print(reason)
    sys.exit(0 if ok else 1)

Keep this script in the default branch and call it from a required check that the pull request author cannot edit. A gate that lives in the same diff it approves is decoration.

Step 3: classify the flake before you spend a retry

Not every red run deserves a replay. Split failures into two buckets before you rerun anything, because a product failure replayed three times is three times the noise.

Infrastructure failures look like exit code 143, OOM kills, DNS resolution errors, or a registry 5xx in the log. These are the only ones eligible for a replay, and each replay voids the previous record. Product failures — an assertion, a snapshot diff, a type error — block the branch and go to a human.

Budget the replays per pull request, not per job. Two replays is a common ceiling; if a third is needed, the branch has a real ordering dependency and you should fix it instead of paying for it.

Step 4: the green-to-merge decision table

Situation	Action
Record passes; tree, base, image and hook version all match	Auto-merge
Any infrastructure replay happened	Require a fresh cold replay before merge
Failure is a product assertion	Block; human triage
Base branch moved after the replay	Rebase, then cold replay
Runner image rotated since the replay	Cold replay on the new image
No record (cancelled, timeout, OOM)	Treat as failing, never neutral
Hook version changed in this diff	Reject the diff; harness changes land separately

That last row is the one people skip, and it is the one that keeps the gate honest. If a pull request can change the harness and the code it tests at the same time, your record describes a moving target.

Where a free model and a free server fit

If your patches come from an agent, the same record shape still applies — the agent is just another producer of a tree hash. In that setup the operator states that MonkeyCode offers free model access plus a free server option, which is enough to try this loop before committing paid CI minutes to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the part you can control: generate the patch with the free model access, run the cold replay on the free server option, and keep the admission record identical to the one your paid runners emit. Do not hard-code quota or hardware numbers into your workflow — free-tier terms change, so read the current page before you depend on a specific allowance, and never treat a free runner as a compliance or isolation boundary.

Limitations, and who should skip this

Cold replay costs time. A suite that takes twelve minutes warm can take twenty-five or more cold, and monorepos with large native builds will feel it first. If your build already requires a licensed artifact you cannot download on a clean machine, this approach will block you until you solve caching deliberately — which is a different project.

Skip it if your team has no required check that pull request authors cannot edit, because the record will not be enforceable. Skip it if every failure is already a product failure and your flake rate is near zero. The overhead only pays for itself when a warm cache is a plausible explanation for a green run.

A five-line checklist for the next pull request

Does the failing job emit an admission record even when it fails?
Does the record include the tree, the base, the runner image, and the hook version?
Is the merge gate reading that record from a path the author cannot modify?
Is the replay budget per pull request, not per job?
When the base moves, does the old record become invalid automatically?

If you can answer yes to all five, a green check means something you can defend after the outage.

A Cancelled Review Run Is Not a Passing Review

Emery Lin — Thu, 15 Oct 2026 11:30:30 +0000

The rule, in one line. A cancelled run is missing evidence, and a missing verdict must never be read as a pass.

That sentence is the article. The rest is the machinery that enforces it in a pipeline where one non-deterministic lane — an AI review step, a fuzz run, an end-to-end suite — sits next to the checks that actually block a merge.

Why the cancelled lane is the expensive one

Most teams bolt on an AI review step because it is cheap and advisory. Then they do two reasonable things that combine badly.

They set concurrency: cancel-in-progress: true, so a fresh push cancels the review of the previous commit. That is the correct cost decision.
They either require the review check in branch protection, or they treat the bot's comment as the verdict. Both make a cancelled run load-bearing.

Put those together and you get the worst kind of bug: a lane that has never once emitted "no evidence", merged into a system where nobody has tested what "no evidence" means. A check whose failure path has never run is not a check. It is a habit.

The same shape appears in browser suites cancelled on the third retry, scanners that time out, and any job with a 10-minute timeout sitting next to a 6-minute model call.

Bind the verdict to fields, not to a comment

A comment is not evidence. Evidence is a small artifact with fields the gate can verify without trusting the producer. Four fields carry the weight:

head_sha — the commit the verdict is about. Without it, a review of commit A authorizes a merge of commit B.
input_digest — a hash of exactly what the lane read: tracked file modes and contents plus lockfiles. Without it, a review of last week's lockfile authorizes today's dependency bump.
run_attempt — which attempt produced this. A verdict from attempt 1 must not authorize attempt 3.
status: complete — set only on normal completion of the lane. A cancelled or killed job writes nothing.

That last point is the design: a cancelled lane produces no artifact. Absence is the signal. You never have to teach the gate to interpret cancelled, because there is nothing to interpret.

Step 1 — the advisory workflow

Reference implementation, not something I have run against your repository. Treat it as a starting point, and verify the cancellation behaviour on your own runner first, because step-level conditions during cancellation have changed between runner versions.

name: advisory-review
on:
  pull_request:
    types: [opened, synchronize, reopened]

concurrency:
  group: advisory-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}

      - name: Compute input digest
        id: digest
        run: |
          DIGEST="$( { git ls-files -s | sort; sha256sum package-lock.json 2>/dev/null || true; } | sha256sum | cut -d' ' -f1 )"
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Run review lane
        id: lane
        env:
          MONKEYCODE_API_KEY: ${{ secrets.MONKEYCODE_API_KEY }}
        run: ./ci/run-review.sh > review.md

      - name: Write verdict
        if: success() || failure()
        run: |
          STATUS="fail"
          [ "${{ steps.lane.outcome }}" = "success" ] && STATUS="pass"
          jq -n \
            --arg sha "${{ github.event.pull_request.head.sha }}" \
            --arg digest "${{ steps.digest.outputs.digest }}" \
            --arg attempt "${{ github.run_attempt }}" \
            --arg status "$STATUS" \
            '{status: "complete", conclusion: $status, head_sha: $sha,
              input_digest: $digest, run_attempt: $attempt}' > verdict.json

      - name: Upload verdict
        if: success() || failure()
        uses: actions/upload-artifact@v4
        with:
          name: verdict-${{ github.event.pull_request.number }}-${{ github.run_attempt }}
          retention-days: 7
          path: verdict.json

Note the asymmetry: the artifact is uploaded on both success and failure of the lane, but not on cancellation. A red verdict is a verdict; a missing verdict is not green.

Step 2 — the required gate

Keep the gate in a separate workflow triggered by workflow_run, or as a job that downloads the artifact. It reads the repository and nothing else. No secrets, no tokens, no write access.

#!/usr/bin/env bash
# ci/gate.sh — required check. Fails closed on any missing or stale evidence.
set -euo pipefail

VERDICT="${1:-verdict.json}"
[[ -f "$VERDICT" ]] || { echo "No verdict artifact: fail closed." >&2; exit 1; }

HEAD_SHA="$(git rev-parse HEAD)"
DIGEST="$( { git ls-files -s | sort; sha256sum package-lock.json 2>/dev/null || true; } | sha256sum | cut -d' ' -f1 )"

jq -e --arg sha "$HEAD_SHA" --arg digest "$DIGEST" '
  .status            == "complete"
  and .conclusion    == "pass"
  and .head_sha      == $sha
  and .input_digest  == $digest
' "$VERDICT" >/dev/null \
  || { echo "Verdict rejected: stale, partial, or mismatched input." >&2; exit 1; }

echo "Evidence accepted for $HEAD_SHA"

The digest formula has to be identical in both places. Copy it from one file into the other, or export it from a shared script — the moment the two commands drift by one sort flag, every verdict is silently rejected, and someone "fixes" it by loosening the gate.

Step 3 — a pre-push hook that keeps the deterministic subset honest

Hooks do not replace CI. They keep the cheap, deterministic checks off the critical path, so the required lanes stay short enough that cancelling them is never tempting.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
BASE="${BASE_REF:-origin/main}"

CHANGED="$(git diff --name-only "$BASE"...HEAD || true)"

if echo "$CHANGED" | grep -qE '^(ci/|\.github/workflows/)'; then
  echo "Pipeline config changed — running the gate self-test locally." >&2
  ./ci/gate.sh --self-test
fi

./ci/test.sh --subset=hermetic --no-retry

Enable it with git config core.hooksPath .githooks, and add one CI job that asserts the hook path is still set — otherwise the hook is decoration on everyone's machine but yours.

Step 4 — quarantine flakes with an expiry date, not a retry count

A retry nobody records is a false green with extra steps. Keep a small ledger, and make the gate read it.

# ci/flakes.yaml
- id: "tests/test_upload.py::test_resume_after_timeout"
  first_seen: 2026-09-02
  expires: 2026-10-02
  owner: "@emery"
  reason: "S3 mock intermittently returns 500"

Two rules make the ledger work:

The gate fails if an entry is expired. Not warns. Fails. An expiry you can ignore is a permanent exemption with a nice date on it.
Any retry increments retry_count in the verdict, and a non-zero retry_count on a required lane opens a tracking issue automatically. The retry stays visible instead of dissolving into a green check.

Choosing what is allowed to block

Lane	Required?	Can write to the repo?	Verdict artifact?
Advisory AI review	No	No (comments only)	Yes, complete-only
Hermetic unit tests	Yes	No	No
Integration suite	Yes, after quarantine review	No	No
Gate / evidence check	Yes	No	Reads only
Deploy preview	No	Preview environment only	No

The rule behind the table: anything that can produce a verdict must not be able to produce the input it is judging, and anything that can write to the repository must not be required.

Limitations, and who should skip this

This costs roughly a day of setup and one more artifact to keep alive. If you merge a handful of PRs a month, or your tests are not hermetic yet, the quarantine ledger becomes a place where failures go to be forgotten. Fix flakiness first.

Three more cases where this is the wrong trade:

Repositories where the model cannot legally see the code. If your data policy forbids sending source to a hosted model, the advisory lane does not exist and neither does most of this article.
Teams without artifact retention. If artifacts expire in a day and PRs live for three, the gate fails on old PRs for the wrong reason.
Pipelines where the expensive lane is genuinely required. A required non-deterministic lane is a queue you cannot merge through. Make it deterministic or make it advisory; do not make it required and cancellable.

Where a free lane actually fits

The advisory review lane is the right place to experiment, because it is the one lane in this design that cannot authorize a merge, cannot write to the repository, and can be cancelled without losing evidence.

The operator states that the MonkeyCode open-source project offers free model access and a free server option. Those are the two claims I am passing along. I am deliberately not quoting a token figure, a model name, or a duration here, because free-tier quotas and terms are exactly the kind of data that goes stale between writing and reading. Check the project's current terms before you size a lane around them.

If you run the lane on a free server, treat the box as ephemeral: no durable caches, no long-lived secrets, and rotate anything you put there. The gate above assumes the advisory lane is untrusted, which is the correct assumption for a free host.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One way to use it: point the advisory lane at the free server on a scratch fork, then push a second commit while the first review is still running and watch what your gate does. If it accepts the cancelled verdict, you have found a real bug — and the bug was never in the model.

Do Not Let the Explain Job Authorize the Retry

Emery Lin — Tue, 13 Oct 2026 02:29:21 +0000

A model can explain a red job. It must not authorize the retry, and it must not mint the merge. You freeze a local decision record before any assistant runs, and you point branch protection at that record alone.

This is a practical split you can add beside checks you already trust. The scripts are a proposed example. They were not executed for this draft. Run them on a branch, read the diff, and only then consider requiring the new check.

Stop treating a summary as an exit code

Logs are easier to summarize than they were a year ago. Your merge rule did not get stricter by itself. A paragraph that says "this looks flaky" invites a rerun. The rerun sometimes goes green for a reason the paragraph never named, and the paragraph becomes the story the team remembers.

You break that habit by refusing to parse prose in the gate. The classifier sees an exit class, a fixture digest, and a remaining flake budget. The explain job sees a redacted log. Those readers do not share a pen.

Give each artifact one writer

You want three files, and you want three different write permissions.

ci/decision.json is the merge evidence. The trusted test job writes it once per attempt, then the file becomes read-only.

ci/notes/failure.md is a comment. An explain job may create it. No required check scores its wording.

ci/retry-ledger.jsonl is an append-only list of rerun reasons. The allowed reasons are a short local vocabulary. "The note sounded flaky" is not in that vocabulary.

If one token can write all three, you do not have a gate. You have a notebook with permission to go green. Split the tokens before you split the workflow file.

Seal the record before any network call

Run the classifier on the trusted runner, immediately after the suite, with no model call and no socket. Exit code 75 is a convention you define in the harness for a known flake signature. It is not a platform default. If that signature does not exist in code, set retry_allowed to false and leave it there.

GITHUB_SHA below is the GitHub Actions name for the commit under test. Map it to your forge's own SHA variable. The rule is the seal, not the vendor.

#!/usr/bin/env bash
set -euo pipefail
# Proposed post-test hook. Unexecuted. No network.
rec=ci/decision.json
digest="$(sha256sum fixtures/manifest.json | awk '{print $1}')"
budget="$(jq -r '.flake_budget' policy/ci.json)"
used="$(wc -l < ci/retry-ledger.jsonl || true)"
used="${used:-0}"
remain=$((budget - used))
class="fail"
retry=false
if [[ "${TEST_EXIT}" == "0" ]]; then
  class="pass"
elif [[ "${TEST_EXIT}" == "75" && "${remain}" -gt 0 ]]; then
  class="flake_candidate"
  retry=true
fi
umask 022
jq -n \
  --arg head "${GITHUB_SHA}" \
  --arg digest "${digest}" \
  --arg class "${class}" \
  --argjson remain "${remain}" \
  --argjson retry "${retry}" \
  '{schema:1,job:"trusted-test",head:$head,fixture_sha256:$digest,exit_class:$class,flake_budget_remaining:$remain,retry_allowed:$retry,note_status:"absent"}' \
  > "${rec}"
sha256sum "${rec}" > ci/decision.json.sha256
chmod a-w "${rec}" ci/decision.json.sha256

note_status starts as absent and is not a slot for the model to fill in later. A rewrite of exit_class or retry_allowed must fail the hash check. You are freezing a small tuple, not collecting a richer story.

Keep policy/ci.json in git, with flake_budget as a plain integer. Review a budget change the way you review a production flag. A budget edited in the same pull request that is currently red is a smell. It is not a fix.

Walk the red job in five steps

Run the suite on the trusted runner. Save the raw exit code. Do not hide a retry inside the test process, and do not let a wrapper convert a timeout into success.
Run the hook above. It writes and freezes ci/decision.json for this attempt. If the hook fails, the attempt fails. You do not continue to an explain job without a seal.
If retry_allowed is true, append one ledger line with reason=local_flake_signature, then rerun once under a new attempt id. Write a new sealed record. Do not edit the first file in place.
Start the explain job only after a terminal failure. It may draft ci/notes/failure.md. It must not call the status API, and it must not push to the branch.
Require a pinned check that runs the verifier in the next section. Do not require the explain job. A missing note leaves the merge decision unchanged.

The ledger line is supposed to look boring. One reason, one attempt, no prose:

printf '%s attempt=%s reason=local_flake_signature head=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${ATTEMPT}" "${GITHUB_SHA}" >> ci/retry-ledger.jsonl

A second automatic retry means you do not have a budget. You have a loop. If you want another attempt, a person changes the policy in git and reviews that change in the open.

Host the note where it cannot set status

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The explain job is the only step where a hosted assistant belongs. MonkeyCode's free model access and free server option can host that optional job when you want a drafted note and you do not want to stand up another build machine. Treat both as current product terms you must read, not as facts this article measured. No token quota, machine size, time limit, or permanence is claimed here. If the docs you can cite disagree with the button you clicked, skip the remote job until they match.

Put a read-only redacted log on that runner. Limit its write path to ci/notes/. Do not mount the decision record read-write. Do not inject a credential that can create check runs. A free server is still someone else's CPU. You would not paste production secrets into a chat box. Do not upload them in a log either.

Stamp the note before any prose:

---
role: advisory
writer: explain-job
may_affect_status: false
---

Redact first. Remove Authorization headers, echoed environment files, signed URLs, and customer identifiers. If you cannot redact with confidence, write a local note that says the log was withheld, and do not call the model. The gate does not need a guess. It already has an exit class.

When the explain call times out, returns empty text, or never starts, you leave the sealed record alone. A missing comment is not a missing test. People get this backwards when a red X from the note job sits next to a real failure. Fix the job so it cannot publish a commit status. Do not "fix" the X by teaching reviewers to ignore red.

Verify the seal and ignore the tone

This verifier is the required check. It is a proposed, unexecuted script. It does not grade sentences. It assumes a Unix mode bit, which matches the chmod line above.

#!/usr/bin/env python3
"""Merge gate. Reads the sealed record. Does not score note prose."""
import hashlib, json, os, sys
from pathlib import Path

rec = Path("ci/decision.json")
seal = Path("ci/decision.json.sha256").read_text().split()[0]
body = rec.read_bytes()
if hashlib.sha256(body).hexdigest() != seal:
    sys.exit("decision record hash mismatch")
if rec.stat().st_mode & 0o222:
    sys.exit("decision record still has a write bit")
data = json.loads(body)
if data.get("schema") != 1:
    sys.exit("unexpected schema")
expected = os.environ.get("EXPECTED_SHA", "")
if not expected or data.get("head") != expected:
    sys.exit("head mismatch")
if data.get("job") != "trusted-test":
    sys.exit("unexpected writer")
if data.get("exit_class") != "pass":
    sys.exit("not green: %s" % data.get("exit_class"))
if data.get("retry_allowed") is True:
    sys.exit("final record still allows retry")
note = Path("ci/notes/failure.md")
if note.exists():
    text = note.read_text(encoding="utf-8")
    if "may_affect_status: false" not in text or "role: advisory" not in text:
        sys.exit("note missing advisory stamp")
print("merge-gate ok")

Give this check a stable name and protect that name. Do not let the explain workflow reuse it. The success line is there so you can grep the log. The script does not print the note. Once the note is in the required log, someone will screenshot it into the merge thread and treat the screenshot as evidence.

An empty EXPECTED_SHA fails closed. A verifier on the wrong checkout should not pass. Set that variable from the same commit the trusted job sealed.

Use a table when you want to rerun

Local signal	Note	Retry	Merge
exit 0, digest matches, record sealed	ignored	no	only through this required check
exit 75, budget left, first attempt	optional, after the seal	one local retry	no, until a new sealed pass exists
exit 75, budget already spent	optional	no	no
exit 1, unknown failure	optional, advisory	no	no
note says flake, exit is 1	present	no	no
explain job down or empty	absent	unchanged	unchanged
hash mismatch, or record still writable	any	no	no

The sixth row is the one to rehearse. Model downtime must not block a sealed pass. It also must not turn a fail into a pass. If your forge fails the whole commit whenever any job fails, the explain job is publishing a status it should not own. Narrow the token. Do not widen the retry rule to compensate.

The last row is the corruption case. A writable record, or a hash that does not match, means you stop. You do not ask the model whether the mismatch looks harmless. Harmless-looking mismatches are how a second writer learns it can edit the gate.

Limits, and who should skip this

This workflow does not rank models, and it does not show that a free server is faster or cheaper than a runner you already operate. It does not replace a fixture manifest check, a clean worktree assertion after tests, or a lock on replay artifacts. Those answer different questions. If you lack them, add them. Do not trade them for a clearer paragraph.

You should not use this approach when branch protection is off, when every green check is treated as equal, or when "flake" is a feeling instead of an exit code in the harness. You should not use the remote explain lane for release signing or for a production deploy. A diagnostic box is not a promotion environment. Skip the remote call entirely if you cannot deny status scope on that job's token, or if the log holds data you would refuse to paste into a third-party prompt.

Turn the explain job on only after the verifier already passes with no note file present. Until that rehearsal is done, a model in the pipeline is extra surface, not extra certainty. If the rehearsal holds and you still want a drafted note without another machine of your own, MonkeyCode's free model access and free server option are enough to try the explain job. Stop if that job can write the check your branch protection requires, or if the current docs no longer match the workflow you merged.

Refuse the Rerun Until the Fixture Manifest Still Matches

Emery Lin — Sun, 11 Oct 2026 21:55:56 +0000

A rerun is only a rerun if the files under test did not move. If the same workspace can edit a fixture and then go green, the required check is grading a different contract than the one that failed.

You do not need a smarter retry button for that. You need a frozen manifest, a job that cannot write it, and a verdict small enough to reject.

What this path actually proves

This workflow separates three facts that teams often collapse into one green square.

The fixture bytes at failure time.
The advisory opinion about whether a retry is even allowed.
The required status check that a merge queue is permitted to trust.

The first and third stay on a runner with no model credential. The second may call a model. It may not push, commit, or rewrite the check name.

None of this measures flake rate. It only stops a class of false greens: a suggestion lane that repairs the evidence and then reports success.

1. Pin the fixture manifest in the required job

Start from an explicit list. Do not hash the whole repository. A lockfile bump or a changelog edit is not a fixture change, and treating it as one will train people to ignore the check.

# Proposal: run this inside the required job, before tests.
# It has not been executed against your repository.
mkdir -p .ci
find tests/fixtures -type f | sort > .ci/fixture-list.txt
sha256sum $(cat .ci/fixture-list.txt) > .ci/fixture-manifest.txt
cp .ci/fixture-manifest.txt "$RUNNER_TEMP/fixture-manifest.txt"
test -s .ci/fixture-list.txt

Keep the committed list under review. The generated manifest is an artifact, not a file the test job may commit back.

If tests/fixtures is empty, fail closed. A missing contract is not a passing contract. Add the directory to the job's checkout only. Do not generate replacement files in this step just to make the hash command succeed.

2. Upload the red log, then leave the workspace

On failure, publish the log and the manifest. Do not publish the worktree. The worktree is how a later step smuggles a rewrite back in.

# Illustrative fragment, not a live workflow from this account.
# Pin the action version your organization already trusts.
- name: Preserve failure evidence
  if: failure()
  uses: actions/upload-artifact
  with:
    name: red-evidence-${{ github.run_id }}
    path: |
      .ci/fixture-manifest.txt
      test-output.log
    if-no-files-found: error

The required job ends here when tests fail. It does not call a model. It does not apply a patch. It does not rerun itself.

That split matters more than the model you pick later. A credential on the required runner is a write path, even if today's prompt says only summarize.

3. Let the advisory job read, not repair

Use a second job. Give it contents: read and the downloaded artifact. Deny it pull-requests: write and any deploy key.

This is the one place a hosted model is useful. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied availability claim used here is limited: free model access, plus a free server option, can host this advisory step. This draft does not assert a token quota, a model name, a hardware shape, a time limit, or that the free tier will stay as it is.

The advisory job may return only this JSON shape:

{
  "schema": "retry-verdict/v1",
  "action": "stop",
  "failure_class": "assertion",
  "reason": "expected status drifted; fixture bytes unchanged"
}

Allowed action values are retry, quarantine, and stop. There is no patch field. If a field appears that the schema does not list, the verdict is invalid.

# Unexecuted proposal. Adapt paths before you rely on it.
import hashlib, json, sys
from pathlib import Path

ALLOWED = {"retry", "quarantine", "stop"}
REQUIRED = {"schema", "action", "failure_class", "reason"}

def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def main() -> int:
    verdict = json.loads(Path(sys.argv[1]).read_text())
    if set(verdict) != REQUIRED or verdict["schema"] != "retry-verdict/v1":
        print("reject: schema")
        return 2
    if verdict["action"] not in ALLOWED:
        print("reject: action")
        return 2
    if "\n" in verdict["reason"] or len(verdict["reason"]) > 240:
        print("reject: reason")
        return 2
    pinned = Path(sys.argv[2]).read_text().splitlines()
    for line in pinned:
        digest, rel = line.split(None, 1)
        rel = rel.lstrip("*").strip()
        if sha256_file(Path(rel)) != digest:
            print("reject: fixture moved: " + rel)
            return 3
    print("accept")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Call it with the verdict file and the pinned manifest. A nonzero exit blocks the rerun. It does not open a pull request.

Feed the model the log excerpt and the manifest hash, not the repository remote. Ask it to fill the schema. Then throw the prose away and keep the JSON only if the checker accepts it.

4. Decide with a table, not with a vibe

Use the failure class only after the hash check passes. If the hash check fails, the table does not apply. The bytes already told you to stop.

Failure class	Fixture hash	Verdict action	What the required lane does
timeout	unchanged	retry	one rerun, same job name
assertion	unchanged	stop	stay red, no rerun
missing fixture	changed or missing	stop	stay red, no rerun
network	unchanged	quarantine	rerun only if a human label already allows it
unknown	any	stop	stay red

retry is not a merge. It is permission to schedule the same required job again, against the same manifest. The second green is still just that job. Your branch protection rule should name it, and it should not name the advisory job.

Quarantine is the narrow middle. Use it for failures you have already labeled as environmental, in a file a human reviews. Do not let the advisory job add labels to that file. Otherwise the model can grant itself the next retry.

Cap the retry at one. A second red is a stop, even if the model would say retry again. Record the attempt id in the job output so a later reader can see the budget was spent.

5. Re-hash immediately before the rerun

Download the original manifest into the new run. Recompute hashes before the test command. If they differ, exit 3 and skip the suite.

# Proposal: first step of the rerun job.
python3 scripts/check_verdict.py "$VERDICT_PATH" "$MANIFEST_PATH"
status=$?
if [ "$status" -ne 0 ]; then
  echo "rerun refused"
  exit "$status"
fi

Do not fix up a mismatch in this step. A mismatch is the finding. Fix the fixture in a normal commit, with review, and let the required job hash the new bytes on a clean run.

6. Leave the advisory name out of branch protection

Name the required check something stable, such as tests (manifest). Name the advisory job retry-verdict (advisory). Protect the branch with the first name only.

You will still see advisory failures. That is fine. They are not the merge signal. If you add the advisory name to the required list, you have glued the suggestion lane back onto the gate this workflow just removed.

Also skip the rerun when the verdict file is missing. Absence is not retry. A down advisory host should leave the original red check in place, not invent a pass.

Inspect a refused rerun

Three commands are enough. You are looking for which gate fired, not for a narrative from the model.

# Proposal commands. Replace the artifact path with the run you are debugging.
sha256sum -c .ci/fixture-manifest.txt
python3 scripts/check_verdict.py verdict.json .ci/fixture-manifest.txt; echo "exit=$?"
git status --short

sha256sum -c failing means the workspace is not the failed commit's fixture set. Exit 2 from the checker means the verdict itself is the problem. A dirty git status means something wrote the tree after checkout. Stop there. Do not rerun tests until the tree matches the manifest.

If all three are clean and the action is stop, read the test log. The workflow has done its job. The bug is in the change, not in the gate.

Where to hook this

Put the manifest step in pre-test, not in post-test. A post-test hash tells you the suite already ran against whatever the workspace had become.

If you use a local hook runner, call the same script from a pre-push hook only as a convenience. The server-side required job is the one that counts. A laptop hook can be skipped. Branch protection cannot, unless an admin bypasses it, which you should treat as an incident rather than a path.

Keep model credentials in the advisory job's secret store. Do not copy them into the required job to make debugging easier. That convenience recreates the write path you just removed.

Limitations

This catches byte-level fixture edits. It does not catch a test that reads an unlisted file, a clock, or a live service. Those need their own pins.

The verdict can be wrong and still be schema-valid. retry on a real assertion bug wastes a run. It does not, by itself, hide the bug, provided the rerun uses the same assertions.

Free model access and a free server option are availability claims, not a capacity plan. If the advisory host is down, the required job must still fail closed without it. Do not block merges on the advisory job, and do not let a missing verdict become an implicit retry.

The sample code is a proposal. It is not a benchmark, and it is not a result from this account's repository. Paths, action versions, and permission keys will need to match your forge before you trust them.

Who should skip this

Skip it if you have no required status check. The manifest then has nothing to protect.

Skip it if fixtures are generated inside the test job on purpose. Pin the generator and its inputs instead of hashing outputs that are supposed to change.

Skip it if you want a model to open the fixing pull request automatically. That is a different risk, and this path refuses it on purpose.

If your required check already cannot write fixtures, the remaining step is small. Put the summarizer on a read-only job, and keep its token off the merge runner. MonkeyCode's free model access and free server option are enough for that advisory role when the suggestion does not need to deploy, commit, or rename the check.

Keep the Draft Lane From Applying Its Own Patch

Emery Lin — Sat, 10 Oct 2026 18:34:08 +0000

A remote green is not merge authority. If a free runner can both write the change and certify it, the required check is theater. Freeze the proposal, apply it on a trusted job, and fail closed when the receipt is missing, cancelled, or quota-stopped.

That is the gate. The rest of this piece is how you wire it so a noisy draft lane cannot bless itself.

The failure you are actually blocking

The badge goes green and the log is long, so the pull request looks finished. You merge. Later the fixture no longer matches the assertion, because the same job edited both.

A second failure mode is quieter. The inner test process exits non-zero, the wrapper still exits zero, and branch protection records success. Cancellation and quota stops get laundered the same way when a skip is treated as a pass.

Generated workflow files make this easier to miss. The diff is large, the job name sounds official, and the first green check arrives before anyone reads the runner label. You do not fix that with a longer prompt. You fix it by splitting lanes.

Where a free draft lane fits

Use the free lane to draft. Do not use it as the merge gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project with free model access and a free server option. Those two availability claims are the only product facts used here.

This article does not state a token quota, a hardware size, a time limit, a benchmark, or a promise that the free option stays fixed. Read the current project docs before you depend on either lane.

Point free model access at a patch draft. Point the free server option at a non-required smoke job. You get a cheap place to explore a change. The required job still has to live on a runner you control, with a name you pin in branch protection.

Decide the lanes before you write YAML

Put the rule in a table, then make the hook enforce the table. A comment in the workflow will not stop a later edit.

Lane	May write	May execute fixtures	May satisfy the required check
Draft: free model plus free server	`proposals/<id>/` only	short smoke, non-required	no
Trusted CI runner	apply the frozen diff	full fixture set, once	yes, only when the apply receipt matches
Laptop pre-push hook	nothing, unless you are the author	same fixture digest, local only	no; it only blocks your push

If one job can edit workflow files and also set the required check name, stop. You have collapsed the table into a single lane, and a single lane cannot police itself.

Receipt contract

The draft lane writes proposals/<id>/receipt.json next to change.diff. The trusted job writes artifacts/apply-receipt.json after it applies that diff and reruns fixtures. Your merge hook compares the overlapping fields. It does not compare log tone.

Require all of these:

proposal_digest: sha256: plus the digest of change.diff
fixture_digest: digest of the fixture tree recorded before apply
runner_class: draft on the proposal, trusted on the apply artifact
exit_class: passed, failed, cancelled, or quota_stopped
inner_exit: exit code of the test process, not the wrapper
draft_attempts: integer, used only to stop noisy draft reruns

A missing field is a failure. An absent file is a failure. Do not default either one to pass.

Do not skip the trusted fixture suite just because the draft receipt says passed. That field means the draft lane thinks it finished. It is not merge evidence.

Numbered path from draft to merge

Freeze fixtures first. Compute fixture_digest in the workflow environment, and do not let the draft patch rewrite that value.
Restrict the draft lane to proposals/<id>/. If the diff touches any other path, the guard exits non-zero and you do not apply.
Run smoke tests on the free server job. Leave that job off branch protection. continue-on-error is acceptable only because this job is not the merge gate.
On the trusted runner, recompute the patch digest. Refuse apply when it differs from proposal_digest.
Confirm the worktree fixture digest still equals the frozen value. Then apply the diff and run the fixture suite once.
Write the apply receipt with runner_class set to trusted. Set exit_class to passed only when inner_exit is 0.
Name the required check after that trusted job. Keep the draft job name off the protection rule, even if the draft name looks cleaner in the UI.
If exit_class is cancelled or quota_stopped, leave the proposal unapplied. Do not convert that status into a skipped success.

Guard script

The following script is an unexecuted example. Review it before you add it to a repository. It is not a measured benchmark, and it was not run against a live server for this article.

#!/usr/bin/env python3
"""Refuse a draft proposal that is incomplete or out of bounds."""
import hashlib
import json
import sys
from pathlib import Path

REQUIRED = (
    "proposal_digest",
    "fixture_digest",
    "runner_class",
    "exit_class",
    "inner_exit",
    "draft_attempts",
)
BLOCKED = {"cancelled", "quota_stopped", "failed"}

def added_paths(text):
    paths = []
    for line in text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            if path != "/dev/null":
                paths.append(path)
    return paths

def main():
    proposal = Path(sys.argv[1])
    receipt = json.loads((proposal / "receipt.json").read_text())
    patch = (proposal / "change.diff").read_bytes()
    missing = [key for key in REQUIRED if key not in receipt]
    if missing:
        print("missing fields: " + ", ".join(missing), file=sys.stderr)
        return 2
    if receipt["runner_class"] != "draft":
        print("runner_class must be draft", file=sys.stderr)
        return 2
    if receipt["exit_class"] in BLOCKED:
        print("refuse apply: " + receipt["exit_class"], file=sys.stderr)
        return 3
    if int(receipt["draft_attempts"]) > 2:
        print("draft attempts exceeded; stop and review", file=sys.stderr)
        return 3
    digest = "sha256:" + hashlib.sha256(patch).hexdigest()
    if receipt["proposal_digest"] != digest:
        print("proposal_digest does not match change.diff", file=sys.stderr)
        return 4
    bad = [p for p in added_paths(patch.decode()) if not p.startswith("proposals/")]
    if bad:
        print("patch leaves proposals/: " + ", ".join(bad), file=sys.stderr)
        return 5
    print("draft proposal is eligible for trusted apply")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Call the guard from the draft job, then again at the start of the trusted job. These commands are part of the same unexecuted example.

python3 scripts/check_proposal.py "proposals/${PROPOSAL_ID}"
sha256sum "proposals/${PROPOSAL_ID}/change.diff"

After trusted apply, assert the artifact. set -euo pipefail matters here. A missing file must fail the job, not print a warning and continue.

set -euo pipefail
test "$(jq -r .runner_class artifacts/apply-receipt.json)" = "trusted"
test "$(jq -r .exit_class artifacts/apply-receipt.json)" = "passed"
test "$(jq -r .inner_exit artifacts/apply-receipt.json)" = "0"
test "$(jq -r .fixture_digest artifacts/apply-receipt.json)" = "${FIXTURE_DIGEST}"

Wire the jobs so the required name is the trusted job. if: always() is deliberate. You want the trusted job to run and fail when the draft receipt is blocked. A skipped required check is how a cancelled free-server run sneaks toward merge.

# Unexecuted example. Job names are illustrative.
jobs:
  draft_on_free_server:
    continue-on-error: true
    steps:
      - run: python3 scripts/check_proposal.py "proposals/${PROPOSAL_ID}"
  trusted_apply:
    needs: draft_on_free_server
    if: always()
    steps:
      - run: python3 scripts/trusted_apply.py "proposals/${PROPOSAL_ID}"
      - run: bash scripts/assert_apply_receipt.sh

Do not gate trusted_apply on needs.draft_on_free_server.result == 'success' alone. A wrapper can report success after a quota stop. The receipt check, plus a fresh fixture run, is the control. The draft result is only a hint.

Flake control that cannot borrow a draft green

You may rerun the draft lane when smoke is noisy. Record the count in draft_attempts. Above 2, stop and ask a person to look at the proposal.

Those reruns never satisfy the required check. The trusted job runs the fixture suite once per apply receipt. If you need another trusted run, write a new receipt with a new digest comparison. Do not reuse an older green artifact from the draft lane.

This is narrower than a repository-wide flake budget. You are blocking one specific wash: a red inner exit that later becomes a zero from a wrapper on a free server.

Debug a merge that should not have happened

When a pull request lands and the fixture looks wrong, reconstruct the lane before you rerun anything.

Read the required check name on the merge commit. If it equals the free-server job, the gate was never split.
Download artifacts/apply-receipt.json. If the artifact is absent, the job skipped or never started. Treat that as a failed gate, not as an empty pass.
Hash the diff that landed and compare it with proposal_digest. A mismatch means apply did not use the frozen proposal.
Recompute fixture_digest at the parent commit. Drift means something moved the fixture between freeze and apply.
Read inner_exit and any wrapper status separately. Keep inner_exit. Discard the wrapper.

Use a stable file order when you recompute the fixture digest. Directory iteration order is not a contract.

# Unexecuted illustration of a stable fixture digest.
find fixtures -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum

If step 1 already shows the draft job as the required check, stop digging through model logs. Fix the protection rule first. The rest of the receipt work will not help while the wrong job name can still go green.

Limitations

Matching digests does not prove the patch is good. It proves the trusted job applied the same bytes, against the fixture digest you froze, and recorded the inner exit you checked.

A compromised trusted runner can forge the apply receipt. Treat that runner like a release credential. Do not place it on the same account or the same machine as the free server lane.

The +++ b/ scan misses some rename and copy headers. Extend the parser before you treat this guard as a security boundary. Schema-check fixture files after apply as well. A draft model can still invent a field inside a path you allowed.

Free-server availability is not something this article can promise. Queue loss, image drift, and quota stops belong on the exit_class list. None of them is a pass, and none of them should delete the proposal you still need to inspect.

Who should not use this flow

Do not use it if you have only one runner and you plan to label it both draft and trusted. The split is the control. One machine wearing both labels removes the control.

Do not auto-apply patches that edit workflow files, branch protection, or signing configuration. Those edits need a human review path with a different required check.

Do not treat a proposal receipt as a release signature. It is a CI comparison. It is not a key ceremony, and it does not replace code review.

Try the split on one non-production branch

If a free model lane and a free server option are already available to you, aim them at proposals/ only. Confirm current access terms in the project docs, add the receipt asserts, and only then consider naming trusted_apply as the required check.

Pin the Merge Check Name, Then Keep the Noisy Lane Off It

Emery Lin — Fri, 09 Oct 2026 15:01:05 +0000

A merge rule should wait on one check name you pinned before the pull request opened. Other jobs may hunt noise, draft notes, or retry a shaky test. They do not get to publish that pinned name, rewrite the fixture behind it, or cancel the run the rule is watching.

That split is the whole gate. Rename the check, and the badge is a new label. Keep the name, and you still have to prove the job is read-only and bound to a fixture hash.

Why the badge string is the control

Required status checks match text. GitHub does not match your intention, and it does not match the story in the pull request body. Actions usually shows a check as a workflow name plus a job name, but you should copy the live string from a dry run into the ruleset instead of guessing. Confirm the current behavior in GitHub's workflow syntax docs and status-check docs, then trust the string your repository actually emits.

An agent that can edit .github/workflows can publish a friendly green under a new job name. If your ruleset still lists the old name, the new badge is decoration. If someone updates the ruleset to follow the new name without reviewing the job, the decoration becomes the gate. You close that gap by pinning the name in a policy file and failing the pull request when the workflow and the policy disagree.

This is not a claim about any model family. It is a claim about strings, permissions, and hashes. Coding agents make the gap easier to hit, because a generated workflow edit and a generated test can land in the same pull request. The ruleset still only sees names.

Set the policy before the workflow

Write the contract first. The table is the decision. The files only encode it.

Decision	Pinned merge check	Noisy lane
Check name	Exact string copied from a dry run	Any other name
Fixture directory	Read a hashed bundle	Scratch only, never uploaded as the bundle
`cancel-in-progress`	`false`	`true` is allowed
Token scope	`contents: read`	No write scope either
Model note	Ignored	Advisory artifact only
Listed in the ruleset	Yes	No

If the noisy lane must be green before merge, it is not a noisy lane. Move that test into the pinned check, or delete it. Do not keep a second required context "temporarily." Temporary required checks are how old names survive.

1. Store the pinned name outside the job

Create .github/merge-policy.json. Review it like production code. A pull request that changes it needs an owner review, even when the rest of the diff is a one-line test.

{
  "required_context": "gate/merge-fixture",
  "required_job_id": "merge-fixture",
  "quarantine_job_id": "flake-explore",
  "quarantine_context_prefix": "explore/flake-",
  "fixture_bundle": "fixtures/locked"
}

The context value above is an example string, not a name GitHub will invent for you. After a dry run on a non-default branch, compare it with the check name you actually see. If the live name includes the workflow title, put that full string in the ruleset, and keep the job name: field aligned with the portion you control.

2. Give the two jobs different lifetimes

This workflow is a proposal. It has not been run for this article. Pin action versions your organization has already approved before you adopt it.

name: merge-gate
on:
  pull_request:
permissions:
  contents: read
jobs:
  merge-fixture:
    name: gate/merge-fixture
    runs-on: ubuntu-latest
    permissions:
      contents: read
    concurrency:
      group: merge-fixture-${{ github.event.pull_request.number }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v4
      - name: Check policy, permissions, and fixture hash
        run: python3 scripts/check_merge_gate.py --policy .github/merge-policy.json
      - name: Run tests bound to the locked bundle
        run: python3 -m unittest discover -s tests/locked -v
  flake-explore:
    name: explore/flake-note
    runs-on: ubuntu-latest
    permissions:
      contents: read
    concurrency:
      group: flake-note-${{ github.event.pull_request.number }}
      cancel-in-progress: true
    steps:
      - uses: actions/checkout@v4
      - name: Rerun the noisy suite without failing the pin
        run: python3 -m unittest discover -s tests/noisy -v
        continue-on-error: true
      - name: Write an advisory note
        if: always()
        run: python3 scripts/flake_note.py --log noisy.log --out flake-note.md

The pinned job cannot be cancelled by a newer push on that pull request. The noisy job can. That difference is intentional. A follow-up commit may abandon an exploration. It must not abandon the run your ruleset is waiting for.

Both jobs stay read-only. A required job that can push is a job that can repair the fixture it is about to trust. Keep contents: write off both lanes even if a later step "only" wants to commit a regenerated cassette.

3. Fail the pull request when the lanes collapse

Save the checker as scripts/check_merge_gate.py. It is a narrow proposal, not a full Actions language parser. It extracts two job blocks by indentation, then refuses the collisions that matter.

#!/usr/bin/env python3
"""Proposal checker. This article does not report an execution result."""

import argparse
import hashlib
import json
import pathlib
import sys

def sha256_tree(root: pathlib.Path) -> str:
    files = sorted(p for p in root.rglob("*") if p.is_file())
    if not files:
        sys.exit(f"empty fixture bundle: {root}")
    digest = hashlib.sha256()
    for path in files:
        rel = path.relative_to(root).as_posix().encode()
        digest.update(rel)
        digest.update(b"\0")
        digest.update(path.read_bytes())
        digest.update(b"\0")
    return digest.hexdigest()

def job_block(workflow: str, job_id: str) -> str:
    lines = workflow.splitlines()
    start = None
    for index, line in enumerate(lines):
        if line == f"  {job_id}:":
            start = index
            break
    if start is None:
        sys.exit(f"missing job id: {job_id}")
    block = [lines[start]]
    for line in lines[start + 1 :]:
        if line.startswith("  ") and not line.startswith("   ") and line.endswith(":"):
            break
        block.append(line)
    return "\n".join(block)

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--policy", required=True)
    parser.add_argument("--workflow", default=".github/workflows/merge-gate.yml")
    args = parser.parse_args()
    policy = json.loads(pathlib.Path(args.policy).read_text(encoding="utf-8"))
    workflow_path = pathlib.Path(args.workflow)
    workflow = workflow_path.read_text(encoding="utf-8")
    required = policy["required_context"]
    prefix = policy["quarantine_context_prefix"]
    if required.startswith(prefix):
        sys.exit("required context uses the quarantine prefix")
    required_job = job_block(workflow, policy["required_job_id"])
    quarantine_job = job_block(workflow, policy["quarantine_job_id"])
    if required_job.count(f"name: {required}") != 1:
        sys.exit("pinned job must publish the required context once")
    if required in quarantine_job:
        sys.exit("noisy lane reuses the pinned context")
    if "cancel-in-progress: false" not in required_job:
        sys.exit("pinned job must refuse cancel-in-progress")
    if "contents: read" not in required_job:
        sys.exit("pinned job must declare contents: read")
    if "contents: write" in required_job:
        sys.exit("pinned job must not request contents: write")
    digest = sha256_tree(pathlib.Path(policy["fixture_bundle"]))
    receipt = {
        "required_context": required,
        "fixture_sha256": digest,
        "workflow_file": workflow_path.as_posix(),
        "advisory_note": "not-a-status",
    }
    pathlib.Path("merge-receipt.json").write_text(
        json.dumps(receipt, indent=2) + "\n", encoding="utf-8"
    )
    print(digest)

if __name__ == "__main__":
    main()

Run it before you enable the ruleset entry:

mkdir -p fixtures/locked tests/locked tests/noisy
printf 'id: sample\n' > fixtures/locked/case.json
python3 scripts/check_merge_gate.py --policy .github/merge-policy.json
git status --short

You want a receipt whose fixture_sha256 matches the locked tree, and a worktree diff you can explain. If the checker rewrites merge-receipt.json on every run, commit that file only when the hash changed for a reason you reviewed. An unexpected hash change means the bundle moved. Do not shrug that into a green badge.

The indentation parser will miss a job written with a different layout. That is a reason to keep the workflow boring, not a reason to trust an unparsed file. If your organization already calls a reusable workflow from a locked repository, prefer that pin, and use this script only as a local tripwire.

4. Put model text on the noisy lane only

The advisory writer does not decide merge. It copies a tail of the noisy log so a reviewer can see why the exploration failed. The sample does not call a network.

#!/usr/bin/env python3
"""Advisory note writer. A failing suite must not change this exit code."""

import argparse
import pathlib

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--log", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    source = pathlib.Path(args.log)
    text = source.read_text(encoding="utf-8", errors="replace")[-4000:] if source.exists() else ""
    pathlib.Path(args.out).write_text(
        "Advisory only. Do not add this file's job to the ruleset.\n\n" + text,
        encoding="utf-8",
    )

if __name__ == "__main__":
    main()

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a shorter note than a raw log tail, MonkeyCode's free model access can sit behind this writer on the noisy lane. Keep that call out of merge-fixture. The free server option matters for the same boundary: run the draft there so a slow summary cannot occupy, cancel, or impersonate the pinned runner. This article does not name a model, a token quota, a machine size, or a period of free availability. Those details change, and they are not the gate. Read the current product documentation before you treat either option as present.

One use is enough. Try the free model access and the free server option for that advisory note if you want the draft off the required job, then leave the note out of the ruleset.

5. Turn the pin on in this order

Do not enable auto-merge in the same change that introduces the workflow.

Land the policy, the checker, and the workflow with the ruleset still pointing at your existing required checks.
Open a dry-run pull request and copy the live check name from the checks list. gh api repos/OWNER/REPO/commits/SHA/check-runs --jq '.check_runs[].name' is the command shape. Substitute your owner, repository, and commit.
If the live name differs from required_context, fix the policy in a reviewed change. Do not pick whichever badge is already green in the ruleset UI.
Add only that verified string as a required check. Do not add explore/flake-note.
On the next pull request, compare merge-receipt.json with a fresh hash of fixtures/locked/ at the merge commit.

A reviewer still reads the diff. The receipt is a breadcrumb, not an approval. If step 2 shows two names that both look official, stop and delete the extra job before you touch branch protection.

Who should skip this

Skip the pattern when you have no required checks at all. A local script cannot invent that platform feature. Skip it when one unreviewed credential can both edit workflows and approve merges. The pin is theater in that setup. Skip it when you need a model verdict to be the merge authority. This design refuses that role on purpose.

Also skip the in-repo tripwire if a locked reusable workflow already publishes the required check from a ref the pull request cannot move. Adding a second, weaker parser will only drift. Do not use the noisy lane to hide a product bug you should fix. Isolation is for tests you have already decided are non-blocking. It is not a pardon.

Limits of the artifact

The checker does not evaluate GitHub expressions. A job that concatenates the context name at runtime can slip past a string search. The receipt hash covers file bytes under fixtures/locked/, not the Actions cache, not container images, and not the runner image. continue-on-error: true on the noisy suite means a red exploration can still look quiet in the pull request conversation. That quiet is acceptable only because the job is absent from the ruleset.

Free model access and a free server option are availability claims supplied for this draft. They are not a capacity plan, an SLA, or a measured speedup. No pass rate is reported here because none was measured.

Pin the name you verified. Hash the bundle the pinned job reads. Let every other lane stay optional, including a model note written somewhere else.