DEV Community: Finley Sun

Properties Outlive the Patch

Finley Sun — Tue, 27 Oct 2026 13:00:12 +0000

A patch lands at 16:50 on a Friday. The diff is small. It rewrites the discount path in billing. Every example test passes, so the suite says ship.

Example tests record one walk through the code. An agent can satisfy them and still break the shape around them. The agent optimizes for the assertions it can actually see.

So move the verdict somewhere else. Let invariants carry the decision, and keep examples for readability. That is the whole idea below: one contract file, three fields, one runner.

The contract file

The contract is a JSON file in the repo root. It travels with the code and gets reviewed like code. Nothing in it is generated at run time.

{
  "today": "2026-10-27",
  "fixtures": {
    "tests/fixtures/rates.csv": "9f2c1a77d0b3e481",
    "tests/fixtures/ledger.json": "3b7d40c2aa915f06"
  },
  "properties": [
    {"test": "tests/props/test_billing.py::test_total_never_negative", "seed": 20261027},
    {"test": "tests/props/test_billing.py::test_order_invariance", "seed": 20261027}
  ],
  "flaky": [
    {"test": "tests/test_checkout_flow.py::test_retry", "expires": "2026-11-10"}
  ]
}

Three fields, three jobs. Hashes pin the inputs. Properties carry the verdict. Quarantine entries are debts with a due date.

The today field looks odd, and it is deliberate. It makes expiry a reviewed value instead of a clock reading. A patch cannot quietly move the deadline by running on a different day.

The runner

Thirty lines of standard library, no framework. It fails closed and prints machine-readable output.

#!/usr/bin/env python3
"""check_contract.py - judge a patch against patch-contract.json"""
import hashlib, json, pathlib, subprocess, sys

CONTRACT = pathlib.Path("patch-contract.json")

def digest(path: str) -> str:
    return hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()[:16]

def main() -> int:
    c = json.loads(CONTRACT.read_text())
    failures = []

    for path, expected in c["fixtures"].items():
        actual = digest(path)
        if actual != expected:
            failures.append(f"fixture drift: {path} {expected} -> {actual}")

    for entry in c["flaky"]:
        if entry["expires"] < c["today"]:
            failures.append(f"stale quarantine: {entry['test']} expired {entry['expires']}")

    for prop in c["properties"]:
        cmd = ["pytest", prop["test"], "-q", "-p", "no:randomly",
               f"--hypothesis-seed={prop['seed']}"]
        run = subprocess.run(cmd, capture_output=True, text=True)
        if run.returncode != 0:
            failures.append(f"property failed: {prop['test']} seed={prop['seed']}")

    print(json.dumps({"ok": not failures, "failures": failures}, indent=2))
    return 0 if not failures else 1

if __name__ == "__main__":
    sys.exit(main())

Seed the hashes once. Then watch what an agent does to them.

sha256sum tests/fixtures/*.csv | cut -c1-16

A re-recorded fixture is a silent spec change. The hash makes it loud.

Properties that survive a refactor

This is the part that does the work. Three invariants on a quote function, written with Hypothesis. They hold for every input, not for three hand-picked numbers.

from hypothesis import given, settings, strategies as st
from billing import quote

@given(st.lists(st.integers(min_value=0, max_value=10_000), max_size=20),
       st.integers(min_value=0, max_value=90))
@settings(derandomize=True, max_examples=200)
def test_total_never_negative(lines, pct):
    q = quote(lines, discount_pct=pct)
    assert q.total >= 0
    assert q.total == sum(lines) - q.discount

@given(st.lists(st.integers(min_value=0, max_value=999), min_size=1))
@settings(derandomize=True, max_examples=200)
def test_order_invariance(lines):
    assert quote(lines).total == quote(list(reversed(lines))).total

An agent refactoring the discount path usually preserves the happy example. It rarely preserves order invariance. Line-item order is invisible in a single assertion, so it gets lost in the rewrite.

That is the asymmetry to exploit. Examples describe yesterday. Properties describe the space the code must live in.

The freeze, with teeth

A quarantined test is a promise, not a pass. It cannot count as evidence for a patch. It only stops blocking, and only until its date.

When the date passes, the runner fails the contract, not the patch. Someone must fix, delete, or renew. Renewal gets its own commit with a written reason.

Keep that commit separate from the patch under review. An agent that can renew its own exemption will do so, cheerfully, in the same diff.

Run the baseline before judging anything. Same commit, same command, twice, and compare. If the base commit already fails a property, the patch inherits nothing.

git stash && python check_contract.py > base.json
git stash pop && python check_contract.py > patch.json
diff <(jq -S . base.json) <(jq -S . patch.json)

Differences in that diff are the only claims worth discussing.

Where the model fits

Keep the roles clean. The contract is the judge. The model is a candidate generator, and candidates are cheap.

MonkeyCode offers free model access and a free server option, so you can generate a few competing patches and run the seed sweep off your laptop. The operator also states a generous free token allotment; check the current terms on the project page before you budget around it, because quotas move.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A practical split that has worked in review discussions: hash checks stay local, because they are pure file reads. Property sweeps go to the free server, because they are the noisy, multi-seed part. The candidate patch comes back as a diff you apply yourself.

That last rule matters. Never let the generator write directly to the branch the contract reads from.

git diff > candidate.patch
python check_contract.py

Limitations, stated plainly

This approach needs invariants you can actually name. If the domain has no statable rule, the contract becomes theater with extra steps.

It also assumes a stable base. During heavy refactors the fixture hashes churn, and a red contract may just mean someone legitimately re-recorded a fixture. Expect noise on those days.

It is not a security review. Malicious code can be perfectly consistent with your properties, and then do exactly what it was told.

And it does not read the diff for you. The contract narrows the surface a human must inspect. It does not replace that inspection.

Who should skip it

Skip this if your tests are still finding bugs in the code rather than in the spec. You need a baseline before you need a contract.

Skip it if nobody owns the quarantine list. An expiry date nobody enforces is a comment with extra syntax.

Skip it if the team cannot agree on one invariant in an afternoon. The argument is the deliverable, and you have not had it yet.

A small first run

Start with one property on the function your last agent patch touched. Add the hash for one fixture. Quarantine one known-flaky test with a real date.

Then run the contract against a patch you already trust and expect green. Learn the shape of the output before you need it on a Friday evening.

Green means the patch kept the promises you wrote down. That is a smaller claim than correctness, and a much stronger one than a green example suite.

If the workflow earns a place in your process, MonkeyCode's free server is an easy place to run the sweep. Bring your own invariants; those are the valuable part.

Three Passes Before You Trust an Agent Patch

Finley Sun — Mon, 26 Oct 2026 11:32:46 +0000

A patch arrives from an agent at 02:14. CI turns green in ninety seconds. By 02:40 refund totals are wrong in production.

The agent did not cheat. It optimised to the only signal it could see: the example tests sitting in the repo. Example tests are a memory of past bugs, not a specification.

So the question changes. Not "did the tests pass" but "which tests was the agent allowed to read". Order matters more than the individual checks.

Pass one: properties the agent never sees

A property holds for every input. "Dedupe never grows the list" survives cases nobody wrote down. An example test defends exactly one typed case and then stops.

Hold the property file out of the agent's context. Not in the prompt, not in the working tree it edits, not in the diff it reads back. An agent that can read your invariants will satisfy the invariants and nothing else. That is the same failure mode, moved one level up.

Properties are cheap to write and expensive to fake. Three invariants catch more than thirty examples, most of the time.

Pass two: fixtures with a contract

A fixture is only a control if you can prove it did not move. Hash the fixture files and pin the digest in a manifest. If the digest drifts, the run is void, not failed.

Void and fail are different verdicts. A failure accuses the patch. Drift accuses your environment. Mixing them trains reviewers to ignore both.

Pin the seed too. A property pass with an unpinned seed is a coin you cannot re-flip.

Pass three: freeze flakes by runs, not by hours

A flake is a test whose result you cannot reproduce on demand. Freezing it by calendar time is guesswork. Count consecutive stable runs instead.

Each quarantined test carries a repeat budget. It needs twenty clean runs before it regains the right to block a patch. Until then it reports, it does not gate.

The ledger is a file in the repo, reviewed like code. Flake status is a claim someone signed.

The harness

Below is the gate I would run before any agent patch touches a shared branch. Standard library only, no framework, one JSON report out. It is illustrative rather than benchmarked; I have not measured it on your repo.

# gate.py — stdlib only. Three passes, one verdict.
import hashlib, json, random, subprocess, sys
from pathlib import Path

ROOT = Path(".")
MANIFEST = ROOT / "fixtures" / "manifest.json"
LEDGER = ROOT / "flake_ledger.json"
PROPS = "held_out_props"          # never shown to the agent
SEED = 20261026
RUNS = 200


def fixture_digest(manifest):
    h = hashlib.sha256()
    for rel in sorted(manifest["files"]):
        h.update((ROOT / rel).read_bytes())
    return h.hexdigest()[:12]


def run_properties():
    sys.path.insert(0, str(ROOT))
    props = __import__(PROPS)
    rng = random.Random(SEED)
    for i in range(RUNS):
        case = props.generate(rng)
        ok, reason = props.check(case)
        if not ok:
            return [{"run": i, "case": repr(case)[:120], "reason": reason}]
    return []


def run_examples():
    p = subprocess.run(["pytest", "-q", "--maxfail=1"],
                       capture_output=True, text=True)
    tail = p.stdout.strip().splitlines()[-1:] or ["no output"]
    return p.returncode, tail


def main():
    manifest = json.loads(MANIFEST.read_text())
    ledger = json.loads(LEDGER.read_text()) if LEDGER.exists() else {}

    actual = fixture_digest(manifest)
    prop_failures = run_properties()
    code, tail = run_examples()

    if actual != manifest["digest"]:
        verdict = "BLOCK:fixture-drift"
    elif prop_failures:
        verdict = "BLOCK:property"
    elif code != 0:
        verdict = "BLOCK:example"
    else:
        verdict = "PASS"

    print(json.dumps({"verdict": verdict, "fixture": actual,
                      "quarantined": sorted(ledger),
                      "props": prop_failures, "examples": tail}, indent=2))
    return 0 if verdict == "PASS" else 1


if __name__ == "__main__":
    sys.exit(main())

One held-out property file does the real work. This one tests a dedupe helper without naming a single input.

# held_out_props.py — keep this file away from the agent's context.
def generate(rng):
    return [rng.randrange(0, 50) for _ in range(rng.randrange(0, 40))]


def check(items):
    out = dedupe(items)                       # function under test
    if len(out) > len(items):
        return False, "output grew"
    if not set(out) <= set(items):
        return False, "invented values"
    if out != list(dict.fromkeys(items)):
        return False, "order or dedupe shape changed"
    if dedupe(out) != out:
        return False, "not idempotent"
    return True, ""

The ledger stays small and boring. Quarantine is a fact with a number attached.

{
  "tests/test_billing.py::test_refund_rounding": {
    "stable_runs": 3,
    "repeat_required": 20,
    "reason": "depends on local date"
  }
}

Reading the verdict

Three booleans arrange into four outcomes. The table is the whole policy.

Property	Fixture digest	New flake	Verdict
fail	any	any	BLOCK — patch guilty
pass	drift	any	VOID — environment guilty
pass	match	yes	QUARANTINE — report, do not gate
pass	match	no	PASS — human review next

Most teams collapse rows two and three into "red". That collapse is why flaky suites get ignored. A verdict that never means anything specific stops being read.

Where the free tier fits

The gate needs two things to be useful: candidate patches and a place to run them. MonkeyCode's free model access supplies the first, and its free server option hosts the second, so the harness does not fight your laptop for CPU while you keep reviewing. The operator advertises a free token allowance; check the current terms for the exact number, since allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A loop that fits the three passes looks like this. Generate two or three candidate patches against the issue. Run python gate.py on each in the hosted environment. Keep only the candidates that reach PASS, then read those diffs yourself.

Nothing here needs a large model. It needs a model that produces varied diffs, and a gate strict enough to throw most of them away.

The free tier is a starting point, not a guarantee. Anything you would not run on a stranger's VM stays off it.

Limits, and who should skip this

Properties are hard to write when the correct output is itself contested. If nobody on the team can state the invariant, pass one will encode a guess, and a confident guess is worse than no gate.

Skip this order if your baseline is already red. Freeze the suite first, get one clean run, then add passes. A gate built on a broken baseline measures noise.

Also skip it for changes that need formal review or regulatory sign-off. A PASS here means the patch survived your invariants, not that it is safe.

The harness is roughly ninety lines and will save you the 02:40 page. Start with one property on the helper your agents keep rewriting. Then tell me which invariant finally caught a bad patch.

Freeze Flaky Tests Before You Judge a Patch

Finley Sun — Tue, 20 Oct 2026 09:37:56 +0000

A reviewer opens an agent patch at 2am. CI is red. The diff touches a cache layer and a date formatter.

Nothing in the diff explains the failure. So the reviewer reruns the job. It goes green.

That single rerun is the problem. An agent patch has no author to ask. Its only defense is the verdict, and a flaky suite makes every verdict negotiable.

This is a plan for three layers: property checks, pinned fixtures, and a hard freeze on flaky tests. The freeze is the part most teams skip, and it is the part that decides whether the other two matter.

Why the freeze comes first

Think of a flaky test as a smoke alarm wired to a toaster. You cannot remove it, because it might be real. You cannot trust it, because it fires on nothing.

Every rerun is a human deciding which alarm to ignore. That decision is invisible in the log. Two reviewers can reach opposite verdicts on the same patch and both look reasonable.

Agents make this worse in a specific way. A model that sees a red run will often patch around the failure, not the cause. You get a green build and a diff that now depends on ordering luck.

So before the agent runs, the suite needs a declared set of tests whose signal is known to be unreliable. Not deleted. Not quarantined silently. Frozen, dated, and owned.

Step one: measure flakiness, do not guess

Run each suspect test many times in isolation. Record the pass rate. Anything between all-pass and all-fail is flaky by definition.

# tools/flake_sweep.py
import collections
import subprocess
import sys

RUNS = int(sys.argv[1]) if len(sys.argv) > 1 else 7
NODES = [l.strip() for l in open("tests/suspects.txt") if l.strip()]

results = collections.defaultdict(list)

for node in NODES:
    for _ in range(RUNS):
        code = subprocess.run(
            [sys.executable, "-m", "pytest", node, "-q", "-p", "no:cacheprovider"],
            capture_output=True,
        ).returncode
        results[node].append("pass" if code == 0 else "fail")

for node, outcomes in sorted(results.items()):
    rate = outcomes.count("pass") / len(outcomes)
    verdict = "STABLE" if rate in (0.0, 1.0) else "FLAKY"
    print(f"{verdict:6} {rate:4.2f} {RUNS:3} {node}")

Seven runs is a starting budget, not a proof. A test with a 90% pass rate survives seven runs about half the time. Raise the run count for anything near the boundary and treat the number as a lower bound.

Isolation matters more than volume. A single-process sweep with shared state will tag honest tests as flaky and hide the real offenders.

Step two: make the freeze expire

A freeze with no deadline becomes a permanent mute button. Write it with an owner and a review date, then let the gate fail when the date passes.

# tests/flaky-freeze.toml
[freeze]
owner = "platform"
review_by = "2026-11-30"
reason = "ordering-dependent cache eviction; ticket PLAT-4412"

# tools/freeze_gate.py
import datetime
import pathlib
import sys
import tomllib  # Python 3.11+

path = pathlib.Path("tests/flaky-freeze.toml")
if not path.exists():
    sys.exit("no freeze manifest; run the sweep first")

manifest = tomllib.loads(path.read_text())
review_by = datetime.date.fromisoformat(manifest["freeze"]["review_by"])

if datetime.date.today() > review_by:
    sys.exit(f"flaky freeze expired on {review_by}: re-sweep or delete entries")
print("freeze valid")

The expiry turns an excuse into a task. The owner sees the date in the manifest long before the gate turns red.

Step three: property checks before examples

Agents break invariants far more often than they break specific examples. A test named test_refund_1250 tells you almost nothing about a rewritten refund path.

A property check states the rule that must hold for every input. Balance never goes negative. Ordering never duplicates an event id. Serialization round-trips.

# tests/test_ledger_properties.py
from hypothesis import given, settings, strategies as st

@given(st.lists(st.integers(min_value=-10**6, max_value=10**6), max_size=200))
@settings(max_examples=200, deadline=None, derandomize=True)
def test_ledger_total_is_order_independent(entries):
    forward = sum(entries)
    backward = sum(reversed(entries))
    assert forward == backward

derandomize=True keeps the failure reproducible. Without it, a red property run is one more thing nobody can reproduce.

Drafting property candidates from a diff is tedious and models are reasonable at it. That is where I lean on MonkeyCode's free model access: I paste the changed function and ask for invariants, then keep only the ones I can state in one sentence.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step four: pin the environment, not just the data

Fixtures drift in three places at once: records, clock, and hash seed. Pin all three or the sweep measures your machine, not the code.

PYTHONHASHSEED=0 pytest -q --tb=short -k "not frozen"

Layer	Runs on	Budget	Failure means
Freeze manifest check	every push	< 2s	freeze expired, fix the freeze
Pinned fixtures + examples	every push	minutes	real regression
Property checks	every push	minutes	real regression or bad generator
Flake sweep	nightly	tens of minutes	suite is lying to you

The sweep is the only CPU-heavy row. It is also the only row where a bad run costs nothing but time.

Running it on a laptop means it gets skipped. The free server option is a better home for a nightly sweep, and it keeps the freeze manifest honest without a dedicated runner.

What this plan does not fix

A freeze hides real failures in the frozen tests. That is the trade, and it is why the expiry date is not optional. If a frozen test guards a payment path, freeze with a failing assertion replaced by a named ticket, not a silent skip.

Small samples cannot separate flakiness from a real bug. If a test fails once in ten runs, you have one data point and a suspicion. Escalate to more runs before you freeze it.

Property checks need decent generators. A generator that only emits zeros will pass forever and prove nothing. Inspect the inputs, not just the assertions.

And this whole approach costs CI minutes. If your suite runs in under a minute and has never flaked, you do not need it. Adding a sweep to a deterministic suite is ceremony, not safety.

Who should skip it

Skip this if your test suite is small and deterministic. Skip it if your CI budget is already tight and cannot afford a nightly job. Skip it if there is no one willing to own the freeze manifest, because an unowned freeze is just an ignored test with extra steps.

If you do adopt one piece, adopt the manifest and its expiry date. Measured flakiness, with a deadline attached, is the cheapest thing on this list. You can run the first sweep tonight, and the free server option is the least disruptive place to try it.

Two Runs, One Verdict

Finley Sun — Mon, 19 Oct 2026 08:21:31 +0000

The patch touched one cache helper. CI came back green on the first run. Two days later, main went red on a test nobody had edited.

That test had been flipping for weeks. The patch did not break it. The patch landed on the lucky side of a coin.

A single suite run cannot tell those two cases apart. Agents produce patches fast enough that luck becomes the main source of noise. The verdict needs more samples than the patch itself.

The aggregate exit code hides the flip

pytest returns a zero or a one. That number collapses hundreds of test outcomes into a single bit. A patch that breaks three tests and fixes three others can still exit zero.

Compare per-test outcomes instead. Run the suite on the parent commit and on the patched commit, twice each. Four runs total. The extra runs pay for themselves the first time you skip a false green.

The harness

The example below is illustrative and unbenchmarked. It shells out to pytest inside two git worktrees and reads the JUnit XML. Keep the reporting boring. Keep the classification explicit.

# paired_run.py — illustrative example, not benchmarked
import json, subprocess, sys
import xml.etree.ElementTree as ET
from pathlib import Path

def run(tree, tag):
    xml = Path(f'/tmp/{tag}.xml')
    subprocess.run(
        [sys.executable, '-m', 'pytest', '-q',
         '-p', 'no:cacheprovider', f'--junitxml={xml}', 'tests/'],
        cwd=tree, check=False,            # exit code is not the verdict
    )
    out = {}
    for case in ET.parse(xml).getroot().iter('testcase'):
        key = case.get('classname') + '::' + case.get('name')
        if case.find('skipped') is not None:
            out[key] = 'skip'
        elif case.find('failure') is not None or case.find('error') is not None:
            out[key] = 'fail'
        else:
            out[key] = 'pass'
    return out

def classify(parent, patch):
    verdict = {}
    for key in set(parent[0]) | set(patch[0]):
        before = {r.get(key, 'missing') for r in parent}
        after = {r.get(key, 'missing') for r in patch}
        if len(before) > 1:
            verdict[key] = 'frozen'        # unstable before the patch
        elif before == after:
            verdict[key] = 'unchanged'
        elif before == {'fail'} and after == {'pass'}:
            verdict[key] = 'repaired'
        elif before == {'pass'} and after == {'fail'}:
            verdict[key] = 'broken'
        else:
            verdict[key] = 'inconclusive'  # missing test or collection error
    print(json.dumps(verdict, indent=2, sort_keys=True))
    return verdict

if __name__ == '__main__':
    parent_tree, patch_tree = Path(sys.argv[1]), Path(sys.argv[2])
    verdict = classify(
        [run(parent_tree, 'parent-1'), run(parent_tree, 'parent-2')],
        [run(patch_tree, 'patch-1'), run(patch_tree, 'patch-2')],
    )
    sys.exit(1 if 'broken' in verdict.values() else 0)

The worktrees come from three commands. No stashing. No branch juggling.

git worktree add /tmp/parent HEAD
git worktree add /tmp/candidate HEAD
git -C /tmp/candidate apply /tmp/candidate.diff
python paired_run.py /tmp/parent /tmp/candidate

Note what the exit code does here. It fires only on broken. Every other label is text for a reviewer to read, not a signal to merge.

The decision table

Verdict	Meaning	Action
`broken`	passed twice before, fails twice after	reject, no debate
`repaired`	failed twice before, passes twice after	allow, cite the test
`frozen`	parent disagreed with itself	exclude from vote, log an owner
`unchanged`	same outcome four times	no signal, no credit
`inconclusive`	missing or uncollected	block until collection is fixed

unchanged deserves emphasis. A patch that keeps 400 tests green has not earned trust. It has produced zero evidence about the diff. Only broken and repaired move information from the patch into your decision.

Fixtures need a hash, not a name

Property checks inherit whatever the fixture hands them. A fixture that drifted silently turns a property test into decoration. Pin the inputs by content.

# conftest.py — illustrative example, run first in every paired run
import hashlib, json
from pathlib import Path

PINS = json.loads(Path('tests/fixtures/pins.json').read_text())

def digest(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

def test_fixtures_are_pinned():
    for name, expected in PINS.items():
        assert digest(Path('tests/fixtures') / name) == expected, name

If that test fails, throw the classification away. A changed fixture invalidates the comparison, and the verdict table is not worth reading after it.

Where the four runs live

Four runs of a fast suite fit on a small box. Four runs of a slow suite do not. MonkeyCode offers free model access and a free server option, which is enough to host the harness above without renting a runner on day one. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product does not change the method. A model proposes a patch. The paired run classifies it. A human reads the table. Keep those three steps separate and the tooling stays swappable, whatever generates the diff.

Limits, and who should skip this

Double running costs double compute. A suite that takes 20 minutes now takes 40, per patch, and that is a real bill on a busy repository.

Two runs also cannot freeze a flake with a period longer than two runs. A test that flips once in fifty will still slip through. Add a third run when the verdict sits close to your release gate.

Skip this approach if your suite has no deterministic fixtures, or if your CI cannot check out two commits at once. Skip it if collection alone costs more than a human review. And skip it if you intend the table to replace a person reading the diff.

Start with one repo and one flaky file. Run both sides twice, print the table, and count the unchanged rows. That count tells you how much of your green build was ever evidence.

If you want to try the loop without provisioning a box first, the free model access and free server on MonkeyCode are a reasonable place to start.

A Gate Order for Agent Patches

Finley Sun — Sun, 18 Oct 2026 07:03:46 +0000

A patch lands Friday at 16:40. Two files change, tests go green, review takes nine minutes. Monday the billing cron charges twice for 3% of accounts. Nobody wrote a bad test. The order in which the tests ran was wrong.

Agent-generated patches make this failure mode common. The agent iterates until the suite passes, not until the code is right. Green means "the checks I could see agreed", nothing more. So the gate has to be ordered, and each stage has to answer a different question.

Three questions, three stages

Property checks ask whether the logic survives inputs you did not write. Fixtures ask whether the world the test ran against is the world you shipped to. The flake freeze asks whether a passing test is allowed to count at all.

Run them in that order. Stop at the first failure. A patch that cannot hold an invariant has no business consuming fixture review time.

Here is a single pytest switch that enforces the order. It is small enough to read in one sitting.

# conftest.py — gate order for agent-generated patches
import hashlib, json, pathlib, time
import pytest
from hypothesis import settings as hyp_settings

ROOT = pathlib.Path(__file__).parent
GATE = json.loads((ROOT / "agent_gate.json").read_text())


def pytest_addoption(parser):
    parser.addoption("--gate", default="props",
                     choices=["props", "fixtures", "freeze"])


def pytest_configure(config):
    if config.getoption("--gate") == "props":
        hyp_settings.register_profile("gate", derandomize=True, max_examples=200)
        hyp_settings.load_profile("gate")


def pytest_sessionstart(session):
    if session.config.getoption("--gate") != "fixtures":
        return
    drift = []
    for path, want in GATE["fixtures"].items():
        p = ROOT / path
        if not p.exists() or want is None:
            drift.append(f"{path} (missing or unpinned)")
        elif hashlib.sha256(p.read_bytes()).hexdigest() != want:
            drift.append(path)
    if drift:
        pytest.exit(f"fixture gate failed: {drift}", returncode=2)


def pytest_collection_modifyitems(config, items):
    if config.getoption("--gate") != "freeze":
        return
    frozen = {f["test"]: f for f in GATE.get("frozen", [])}
    today = time.strftime("%Y-%m-%d")
    for item in items:
        entry = frozen.get(item.nodeid)
        if entry is None:
            continue
        if entry["until"] < today:
            pytest.exit(f"freeze expired on {item.nodeid}: fix it or re-file", returncode=3)
        item.add_marker(pytest.mark.skip(reason=f"frozen: {entry['reason']}"))

The config file stays plain and reviewable.

{
  "fixtures": {"fixtures/invoice_oct.json": null, "fixtures/plan_table.csv": null},
  "frozen": [
    {"test": "tests/test_sync.py::test_retry_wall_clock", "reason": "clock skew in CI", "until": "2026-11-01", "owner": "payments"}
  ]
}

null means unpinned, and the gate treats that as drift. You cannot quietly inherit a fixture hash from somebody else's laptop. Regenerate hashes in a commit that a human signs off on, never inside the agent's diff.

pytest -q --gate=props -x && \
pytest -q --gate=fixtures && \
pytest -q --gate=freeze -m "not live"

One more habit earns its cost: sweep the hash seed. Order dependence hides in dict iteration, and a single run will never show it.

for seed in 1 7 42 1009; do PYTHONHASHSEED=$seed pytest -q --gate=props || exit 1; done

The freeze is a rule, not a folder

Skipping a flaky test is cheap. Forgetting why is expensive. Every freeze row needs an owner and a date, and the date must fail loudly, not silently extend.

Here is how I would triage a red test inside an agent patch. Read the row top to bottom; the first match wins.

Symptom	Action
Fails on rerun with same seed	Block the patch. Not a flake.
Passes alone, fails in suite	Freeze 7 days, file order bug
Touches network or wall clock	Freeze 14 days, inject a fake
Only fails under parallel workers	Freeze 7 days, mark serial
Test file edited in this patch	Block. Re-review the test first
Already frozen twice	Block. Delete or fix it

The last row matters most with agents. A model will happily "fix" a flaky test by relaxing the assertion. Freeze count per test is the counter that catches that move.

Two honest limits. Demanding derandomize=True trades bug-finding breadth for reproducibility, so keep a nightly unseeded run. And hashes prove the fixture did not change, not that it was ever correct. Wrong data, pinned forever, is still wrong.

What this costs

Stage one is the slowest. Property tests at 200 examples per invariant add real minutes. If your invariants are vague, you pay that cost and learn nothing, so write the invariant first and let the agent patch second.

The freeze stage is nearly free. It runs at collection, before a single test executes. Expired rows fail in under a second, which is the right moment to interrupt a merge.

Where free capacity fits

Running four seeds across a growing suite is exactly the workload people abandon on a laptop. MonkeyCode offers free model access and a free server option, so the patch-generating side and the seed sweep can live on the same machine you did not have to provision. Current limits are documented by the product itself, and I would check them before planning a nightly sweep around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The order still matters more than the host. Models change weekly, and the gate order does not.

Who should skip this

If you are one engineer with a five-minute suite, three stages is ceremony. Run property checks, keep fixtures in git, and move on. Skip the freeze entirely when your suite has no known flakes; an empty list is honest.

Skip it too if your agent patches only touch documentation or generated assets. Hash gates on files nobody asserts about are pure noise. And if your team cannot staff an expiry owner, do not freeze at all. An unowned freeze list is a graveyard with a cron job.

Start with one invariant and one pinned fixture. Add the freeze the first time a test lies to you. Then keep the order fixed and let the suite grow around it.

Pin the Fixture, Budget the Run

Finley Sun — Sat, 17 Oct 2026 06:04:47 +0000

The patch was twelve lines. It touched one adapter and one test file. CI went green on the fourth attempt, and nobody asked why.

That green is not evidence. It is three reruns and a silent skip, stacked until the runner agreed. Agent patches make this worse, because volume hides the pattern.

I have not run the incident above. Treat it as an illustrative shape, not a log. The mechanism is common enough to design against.

Three leaks, one cause

A rerun is an unrecorded vote. Each retry quietly deletes one failure from the record. Two failures out of five looks identical to zero out of one.

Fixtures drift without anyone editing a test. A regenerated snapshot or a rewritten JSON file changes the input, not the code. The suite still passes, but it now measures a different world.

Skips are the third leak. A patch that adds @pytest.mark.skip to a failing test has improved the build without improving the software. Every skip in a patch is a claim someone forgot to defend.

These three leaks share a cause. The patch decides what counts as done. The budget has to come from outside the patch.

Fix the inputs before the oracle

Start with the fixtures. Hash them, commit the hash, and fail closed on any mismatch. This is cheap and it removes an entire class of "it passed locally" arguments.

# tools/pin_fixtures.py
import hashlib, json, pathlib

FIXTURES = pathlib.Path("tests/fixtures")
LOCK = pathlib.Path("tests/fixtures.lock.json")

def digest(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

lock = {p.name: digest(p) for p in sorted(FIXTURES.rglob("*")) if p.is_file()}
LOCK.write_text(json.dumps(lock, indent=2, sort_keys=True) + "\n")
print(f"pinned {len(lock)} fixtures")

Run it once, commit the lock file, and check it from a session fixture. A drifted fixture should stop collection, not fail a random assertion three modules later.

# tests/conftest.py
import json, pathlib, pytest
from tools.pin_fixtures import FIXTURES, digest

LOCK = json.loads(pathlib.Path("tests/fixtures.lock.json").read_text())

@pytest.fixture(scope="session", autouse=True)
def fixtures_pinned():
    seen = {p.name: digest(p) for p in FIXTURES.rglob("*") if p.is_file()}
    drift = [n for n, h in seen.items() if LOCK.get(n) != h]
    if drift:
        pytest.exit(f"fixture drift: {drift}", returncode=3)

The exit code matters. A distinct code lets CI say "inputs changed" instead of "code broke". Those two sentences route to different people.

A freeze list that expires

Flaky tests need to leave the gate, but they must not vanish. A frozen test stays in the repository, out of the run, and on a clock.

{
  "tests/test_payments_retry.py::test_backoff_jitter": {
    "frozen_on": "2026-10-02",
    "expires": "2026-11-01",
    "owner": "payments"
  }
}

Check the expiry during collection. An expired entry is a hard error, not a warning. The test was unfrozen by the calendar, not by a review.

That single rule changes the incentive. Freezing buys thirty days of quiet. It does not buy permanent silence, and it does not let a patch hide behind a stale entry.

Budget the spend

The patch does not get an unlimited suite. Give it a time limit, a test-count limit, and a skip limit of zero. If the suite exceeds any of them, the patch is unverified, not failed.

# tools/budget.py
import json, sys

def main(path: str, max_seconds: float, max_tests: int, max_frozen: int) -> int:
    report = json.load(open(path))
    summary = report["summary"]
    checks = {
        "seconds": report["duration"] <= max_seconds,
        "tests": report["total"] <= max_tests,
        "frozen": summary.get("deselected", 0) <= max_frozen,
        "skips": summary.get("skipped", 0) == 0,
    }
    breached = [name for name, ok in checks.items() if not ok]
    if breached:
        print("budget breach: " + ", ".join(breached))
        return 1
    print("budget ok")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], 240, 180, 0))

The numbers are local and should stay local. A 240 second ceiling fits a small service. A monorepo needs a different figure, and inventing one here would be theater.

Wire the gate as three commands. The sequence is boring on purpose.

python tools/pin_fixtures.py --check
pytest -q --json-report --json-report-file=report.json
python tools/budget.py report.json 240 180 0

Property checks scoped to the diff

Full property suites are slow. Scoped ones are honest. Generate the inputs, keep the oracle fixed, and only run properties for modules the patch touched.

# tests/properties/test_retry_policy.py
from hypothesis import given, settings, strategies as st
from payments.retry import backoff

@given(st.integers(min_value=1, max_value=8),
       st.floats(min_value=0, max_value=30, allow_nan=False))
@settings(max_examples=200, deadline=None)
def test_backoff_is_monotonic(attempt, base):
    assert backoff(attempt + 1, base) >= backoff(attempt, base)

Monotonic backoff is a property a reviewer can state in one sentence. That is the test. If the property cannot be stated without reading the implementation, it is not a property, it is a copy.

Selection stays mechanical.

git diff --name-only origin/main -- 'src/**' > changed.txt
python tools/select.py changed.txt > selected.txt
pytest -q $(cat selected.txt) tests/properties

select.py maps module paths to test files and is deliberately dumb. A clever selector is a new source of untested behavior.

Where a free runner fits

This loop needs two things: a model to draft property candidates from a diff, and a machine to run a budgeted suite. MonkeyCode's free model access and free server option cover both for a first pass, and the token allowance on the free tier is operator-supplied. Check the current limits on the project page before you build a pipeline around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My own use is narrow. The model proposes candidate properties in the docstring style above, and I keep or discard each one by hand. The server exists so the budget gate runs somewhere that is not a laptop. That is the whole integration. If you want to try the loop once end to end, the free tier is enough for a single patch.

None of the harness above depends on it. The scripts are plain Python and pytest.

What this does not fix

A property suite with no oracle is a tautology generator. If backoff is the only definition of correct backoff, the test proves nothing.

Small suites should skip the budget entirely. Below roughly twenty tests, a fixed ceiling mostly records noise, and you will spend more time tuning numbers than reading failures.

Teams with genuinely non-deterministic dependencies need to fix those first. A freeze list over a broken clock is a filing cabinet, not a gate.

And the numbers in budget.py are mine. They are a starting point with no benchmark behind them, so measure your own suite before trusting the ceiling.

The twelve-line patch still needs a human. The budget just stops it from grading its own homework.

A Quarantine Ledger for Agent Patches

Finley Sun — Fri, 16 Oct 2026 05:05:10 +0000

A CI suite runs 300 tests. Two of them fail on almost every diff. The failures land in files nobody touched.

Reviewers learn to scroll past red. An agent opens forty diffs a week. Each one looks green except the usual two. The suite stops being a gate and becomes wallpaper.

This post describes a workflow for that situation. It has three parts: a quarantine ledger with expiry dates, a property harness for agent patches, and a fixture budget that makes silent fixture edits visible. Every snippet below is runnable pytest against a small synthetic importer. The counts are illustrative, not measured.

The ledger

A ledger is one JSON file in the repo. Each entry names a test, an owner, a reason, and an expiry date.

{
  "tests": [
    {
      "id": "tests/test_billing.py::test_retry_after_webhook",
      "owner": "payments",
      "reason": "clock-dependent, fails ~3 of 20 runs on clean main",
      "opened_at": "2026-10-16",
      "expires_at": "2026-11-13"
    }
  ]
}

A conftest hook reads the ledger at collection time. Unexpired entries get skipped. Expired entries run again, which is the enforcement you want. The skip markers themselves are plain pytest, documented at https://docs.pytest.org/en/stable/how-to/skipping.html.

# conftest.py
import json
import time
from pathlib import Path
import pytest

LEDGER = Path("quarantine.json")
RUN_LOG = Path("artifacts/quarantine-run.json")


def pytest_collection_modifyitems(config, items):
    entries = json.loads(LEDGER.read_text())["tests"] if LEDGER.exists() else []
    today = time.strftime("%Y-%m-%d")
    by_id = {e["id"]: e for e in entries}
    report = {"date": today, "skipped": [], "expired": [], "runnable": 0}

    for item in items:
        entry = by_id.get(item.nodeid)
        if entry and entry["expires_at"] > today:
            item.add_marker(pytest.mark.skip(reason=f"quarantined: {entry['reason']}"))
            report["skipped"].append(item.nodeid)
        else:
            if entry:
                report["expired"].append(item.nodeid)
            report["runnable"] += 1

    RUN_LOG.parent.mkdir(parents=True, exist_ok=True)
    RUN_LOG.write_text(json.dumps(report, indent=2))

Expiry is the whole point. A skip without a date is a deletion in disguise. The run log also gives you one number to watch. The skipped count should fall over a quarter. If it rises, the suite is decaying faster than you repair it.

Properties instead of examples

Agent patches tend to preserve shapes and break rules. An example test asserts that row 42 credits ten dollars. A property test asserts that accepted plus rejected equals input, for any input.

Here is the code under test, small enough to reason about.

# app/importer.py
from dataclasses import dataclass


@dataclass
class ImportResult:
    accepted: int
    rejected: int
    credited_sum: int
    checksum: int


class Importer:
    def __init__(self, conn):
        self.conn = conn

    def run(self, rows):
        accepted = rejected = credited = 0
        for row in rows:
            if row["amount"] == 0:
                rejected += 1
                continue
            cur = self.conn.execute(
                "insert or ignore into ledger_row (id, amount, currency) values (?, ?, ?)",
                (row["id"], row["amount"], row["currency"]),
            )
            if cur.rowcount == 1:
                accepted += 1
                credited += row["amount"]
        self.conn.commit()
        checksum = self.conn.execute(
            "select coalesce(sum(id), 0) from ledger_row"
        ).fetchone()[0]
        return ImportResult(accepted, rejected, credited, checksum)

And the fixtures, with an in-memory database and a seeded generator.

# tests/conftest.py
import random
import sqlite3
import pytest
from app.importer import Importer


class Snapshot:
    def __init__(self, conn):
        self.conn = conn

    def count(self, table):
        return self.conn.execute(f"select count(*) from {table}").fetchone()[0]

    def sum(self, table, column):
        return self.conn.execute(
            f"select coalesce(sum({column}), 0) from {table}"
        ).fetchone()[0]


@pytest.fixture
def snapshot():
    conn = sqlite3.connect(":memory:")
    conn.execute(
        "create table ledger_row (id integer primary key, amount integer, currency text)"
    )
    yield Snapshot(conn)
    conn.close()


@pytest.fixture
def importer(snapshot):
    return Importer(snapshot.conn)


@pytest.fixture
def rows():
    def build(seed, n=40):
        rng = random.Random(seed)
        return [
            {
                "id": rng.randrange(10_000),
                "amount": rng.randrange(-500, 500),
                "currency": rng.choice(["USD", "EUR"]),
            }
            for _ in range(n)
        ]

    return build

Two properties carry most of the weight here: idempotency and conservation.

# tests/properties/test_import_properties.py
import pytest

pytestmark = pytest.mark.property

SEEDS = range(1, 26)


@pytest.mark.parametrize("seed", SEEDS)
def test_import_is_idempotent(seed, importer, snapshot, rows):
    batch = rows(seed)
    first = importer.run(batch)
    assert snapshot.count("ledger_row") == first.accepted

    second = importer.run(batch)
    assert second.accepted == 0
    assert snapshot.count("ledger_row") == first.accepted


@pytest.mark.parametrize("seed", SEEDS)
def test_rejection_never_breaks_totals(seed, importer, snapshot, rows):
    batch = rows(seed)
    result = importer.run(batch)
    assert result.accepted + result.rejected == len(batch)
    assert snapshot.sum("ledger_row", "amount") == result.credited_sum

Twenty-five seeds across two properties is fifty cheap runs. That is a friendlier signal than one hand-written example. It also survives refactors better, because the invariant does not move when the row format does. When a property fails on seed 17, you keep seed 17 forever as a regression.

A fixture budget

Fixtures are shared state that agents can quietly rewrite. Pin a fingerprint and fail the build when the fixture set drifts.

# tests/test_fixture_budget.py
import hashlib
from pathlib import Path

PINNED = "9f2c41ab77de0013"  # update only in the same diff as a ledger entry


def fixture_fingerprint(root="tests/fixtures"):
    digest = hashlib.sha256()
    for path in sorted(Path(root).rglob("*")):
        if path.is_file():
            digest.update(path.name.encode())
            digest.update(path.read_bytes())
    return digest.hexdigest()[:16]


def test_fixtures_did_not_drift():
    assert fixture_fingerprint() == PINNED, "fixture set changed; justify it in the diff"

This test is intentionally annoying. It forces the patch author to state why the fixture moved. Most agent diffs leave fixtures alone, so the cost is near zero. The benefit arrives the day a diff rewrites a fixture to match a bug.

Re-entry

Quarantine ends with evidence, not optimism. Run the test against clean main, twenty times, before you delete the entry.

#!/usr/bin/env bash
# scripts/reentry.sh <nodeid>
set -euo pipefail
TEST_ID="$1"
git switch main && git pull --ff-only
for i in $(seq 1 20); do
  pytest "$TEST_ID" -q || { echo "flake on run $i"; exit 1; }
done
python - "$TEST_ID" <<'PY'
import json, pathlib, sys
path = pathlib.Path("quarantine.json")
data = json.loads(path.read_text())
data["tests"] = [t for t in data["tests"] if t["id"] != sys.argv[1]]
path.write_text(json.dumps(data, indent=2))
print("removed", sys.argv[1])
PY

Twenty passing runs on clean main is a weak guarantee, and you should say so out loud. It still beats deleting a skip because the log annoyed someone.

Deciding what to do

What you observe	Ledger action	Effect on the merge gate
Fails 2 of 10 runs on clean main	Open entry with a 30-day expiry	Excluded from the verdict, still recorded
Fails only on the agent diff	No entry	Merge blocked, treat as a real regression
Passes locally, fails in CI	Freeze clock, timezone, locale	New entry for the environment diff
Diff edits fixtures	Fingerprint test fails	Author must justify the change
Property fails on one seed	Keep that seed as a regression	Merge blocked until fixed

The rule behind the table is simple. Unknown failures lose their vote. Known failures keep it.

Where MonkeyCode fits

MonkeyCode is the operator's open-source project, and the fit here is narrow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states that free model access and a free server option are available, with a free allowance of 10 million tokens.

I have not verified quotas, hardware, or how long the offer lasts. Treat every limit as changeable and check the current terms before designing a nightly job around it. If the numbers hold for you, the free model access is where patch generation happens, and the free server option is where the repeated sweeps run. Fifty property runs per diff adds up quickly, and that repeated compute is exactly what a metered runner pushes you to skip.

Nothing above depends on MonkeyCode. Swap in any model and any runner, and the ledger, the properties, and the fingerprint test behave the same way.

Limits and who should skip this

A ledger without expiry becomes a graveyard. Review the skipped count monthly and treat growth as a bug. Property tests only work when you can state the invariant in one sentence. If you cannot, example tests are honest enough.

This approach fits three groups poorly. Teams with a deterministic suite should fix the flake and move on. Teams that need per-test evidence for compliance should not hide tests behind a skip marker. Teams without a named ledger owner should not start, because the file will rot within a quarter.

Start with one entry and one property. The ledger is not a place to hide failures. It is a record of what you do not know yet, with a date attached.

Kill Mutants in the Agent's Diff

Finley Sun — Thu, 15 Oct 2026 11:17:13 +0000

Kill Mutants in the Agent's Diff

Picture a Tuesday queue with fourteen agent patches waiting. Each one touches three files and arrives with a green suite beside it. Two tests in that suite fail at random, and the dashboard never says which two.

Reviewers read diffs slower than agents produce them. A full green run answers one weak question: did anything break? It never answers the useful one: can this patch's new logic fail at all?

I want a gate that answers the second question in minutes. It needs three mechanisms, and all three are boring on purpose. Patch-local mutation, coverage contexts, and a quarantine ledger.

Why the whole suite is the wrong instrument

Running everything is easy to build and hard to trust. The suite mixes evidence about the changed lines with evidence about code the patch never touched. A red test outside the diff blocks the merge anyway. A green run outside the diff proves nothing at all.

Mutation testing flips the problem around. Stop asking whether tests pass, and ask whether a deliberate defect survives. If you break the new comparison and every test still passes, the patch has no witness.

That is the fact I want before review, not after an incident. The catch is cost. Full-repo mutation on a large Python project is a weekend job, not a gate. So the mutation set must stay tiny and scoped to the diff.

Map the patch to the tests that actually ran

Start with changed line numbers, parsed from a zero-context diff so hunks stay small.

# diff_lines.py -- sketch, Python 3.11
import re, subprocess

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def changed_lines(base="origin/main"):
    diff = subprocess.run(
        ["git", "diff", "-U0", base, "--"],
        capture_output=True, text=True, check=True
    ).stdout
    files, current = {}, None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            current = line[6:]
            files.setdefault(current, set())
        elif line.startswith("@@"):
            m = HUNK.match(line)
            start, count = int(m.group(1)), int(m.group(2) or 1)
            files[current].update(range(start, start + count))
    return {f: sorted(n) for f, n in files.items() if n}

Now record which test executed each of those lines. coverage.py supports this with dynamic contexts, and the config change is one line.

# .coveragerc
[run]
dynamic_context = test_function

Run the suite once, then export JSON with the contexts included.

coverage run --rcfile=.coveragerc -m pytest -q
coverage json --show-contexts -o /tmp/cover.json

Intersecting the two files is short.

def tests_for_lines(cover_json, diff_map):
    data = json.load(open(cover_json))
    picks = set()
    for path, info in data["files"].items():
        targets = set(diff_map.get(path, []))
        for lineno, ctxs in info.get("contexts", {}).items():
            if int(lineno) in targets:
                picks.update(c for c in ctxs if c)
    return sorted(picks)

Two honest notes. A context appears only if that line executed during that test. Config branches, feature flags, and lazy imports can leave a changed line with no context at all. When that happens, the gate must return "no evidence" and never "pass."

Mutate only the lines the agent touched

Keep the operator set small and textual. Five replacements cover a lot of real logic: equality flips, boundary shifts, constant offsets, and boolean inversions.

MUTATIONS = [("==", "!="), ("<=", "<"), (">=", ">"), ("True", "False"), ("+ 1", "- 1")]

def propose_mutants(path, lines, limit=6):
    src = Path(path).read_text().splitlines()
    out = []
    for n in lines:
        for old, new in MUTATIONS:
            if old in src[n - 1]:
                out.append((path, n, old, new))
                break
        if len(out) == limit:
            break
    return out

Apply one mutant, run only the picked tests, then restore the file. A failing run means the mutant died, which is the result you want.

def run_mutant(root, mutant, picked):
    path, n, old, new = mutant
    original = Path(path).read_text()
    lines = original.splitlines()
    lines[n - 1] = lines[n - 1].replace(old, new, 1)
    Path(path).write_text("\n".join(lines) + "\n")
    try:
        r = subprocess.run(["pytest", "-q", "-x", *picked],
                           cwd=root, capture_output=True)
    finally:
        Path(path).write_text(original)
    return r.returncode != 0    # True == killed

Fixtures sit in the middle of this loop, and they must not drift between mutants. Hash the directory once, then record the digest next to the gate output.

def fixture_digest(root="tests/fixtures"):
    h = hashlib.sha256()
    for p in sorted(Path(root).rglob("*")):
        if p.is_file():
            h.update(p.read_bytes())
    return h.hexdigest()[:12]

If the digest changes mid-run, discard every result from that run. A mutant that dies because the input changed is not evidence of anything.

Quarantine beats deletion for flaky tests

A flaky test is not useless. It is simply untrustworthy as a witness. Deleting it hides a real race under the carpet. Leaving it in the gate makes every result random.

Keep a ledger instead, and let the gate read it. Each entry names the test, the observed failure mode, and the owner who agreed to fix it.

{
  "tests/pricing/test_round.py::test_half_up": {
    "reason": "shared clock, fails about 1 in 40 runs",
    "owner": "pricing",
    "added": "2026-10-06"
  }
}

One rule makes the ledger useful. A quarantined test can never be the witness that kills a mutant. If it is the only test covering a changed line, the gate returns "no evidence" and stays closed. The patch then waits for a test someone trusts.

The gate has three outcomes

Keeping the decision space small keeps the conversations short.

Outcome	Meaning	Action
`killed_all`	every diff mutant died	merge eligible for review
`survivors`	at least one mutant lived	request a test that fails
`no_evidence`	no trusted test covers the diff	gate stays closed

The gate prints the mutant list, the killed count, and the fixture digest. Reviewers read that log in twenty seconds and know exactly where the patch stands. Note that killed_all is not a badge of quality. It means the diff is pinned by something, which is the minimum bar for handing code to a human.

```bash # proposed run order, not a benchmark
python diff_lines.py > /tmp/diff.json
coverage run --rcfile=.coveragerc -m pytest -q
coverage json --show-contexts -o /tmp/cover.json
python gate.py --diff /tmp/diff.json --cover /tmp/cover.json --limit 6




## Limitations, and who should skip this

Equivalent mutants are the main noise source. Flipping a flag no test can observe produces survivors that mean nothing. Expect a nonzero equivalent rate, and review survivors by hand before blocking anyone.

Text-level mutation can also break syntax. Guard each mutation with a compile step, and skip it if the file stops parsing. Generated code with heavy metaprogramming tends to defeat this approach entirely.

Teams without per-test coverage contexts should not start here. Build the context data first, even if the gate arrives a quarter later. Repos with a suite already over thirty minutes should fix that first too.

The real cost is concrete. The gate adds one full coverage run per patch, plus a handful of short focused runs. On a quiet laptop that is fine. In a busy CI queue it is not, so scope it to risky patches only.

## Where MonkeyCode fits, briefly

Proposing candidate mutations on the diff lines is a narrow generation task, and it is the one place a model helps here. The operator states that MonkeyCode offers free model access and a free server option. The free server matters because mutation runs are bursty: idle most of the day, then six at once.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not claiming benchmarks in this piece, and any numbers from your own runs are yours to produce. Use the free access to propose candidate mutations, keep your thresholds yours, and treat a survivor as a question rather than a verdict. If you want to run this gate somewhere other than your laptop, the project is open source and the free server option is a low-risk place to start.

Stamp the Verdict Card on the First Run

Finley Sun — Tue, 13 Oct 2026 00:43:55 +0000

A composite checkout failed on Friday and passed on the retry. The invoice property missed the total by one cent. The second run saw a newer wall clock.

This note reconstructs that failure mode for review. This note is not a personal incident log. No customer metric is claimed in this draft.

Agent patches move faster than most review notes. Test oracles drift when the same patch may edit them. A quiet retry can hide the first miss.

A retry can help a dead socket recover. It does not repair a moved test oracle. Those two failures need two different kinds of gates.

The useful split stays small, strict, and ordered. Seal the fixture before any property is allowed to speak. Run the suite once, then freeze every flake.

That order matters more than another green badge. A second run is always a different experiment. It must not erase the first recorded miss.

The fixture is a sealed box

Think of the fixture as a packed lunch for the trip. You weigh the box before you leave the kitchen. You do not swap the apple after the scale.

A sealed fixture carries three locks on purpose. Input bytes stay fixed for the whole run. The clock stays fixed, and the network stays closed.

Hash the input tree before tests start on the patch. Store that digest beside the proposed agent change. Refuse the run when the digest drifts at all.

find fixtures/invoices -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum \
  | sha256sum \
  | awk '{print $1}' > .fixture-seal

The command above is a local recipe for a clean worktree. Skip editor swap files and any local cache files. Commit the seal only when the inputs truly change.

# proposal: not an executed benchmark
import hashlib
from pathlib import Path

def seal_tree(root: Path) -> str:
    digest = hashlib.sha256()
    files = sorted(p for p in root.rglob("*") if p.is_file())
    for path in files:
        rel = path.relative_to(root).as_posix().encode()
        digest.update(rel)
        digest.update(b"\0")
        digest.update(path.read_bytes())
        digest.update(b"\0")
    return digest.hexdigest()

Compare this digest with the verdict card later. A mismatch means someone opened the sealed box. Opened boxes do not cast a merge vote.

Label the clock inside the fixture file too. A file named clock.json can hold one frozen instant. Tests read that instant, not the host time.

{"frozen_at": "2026-03-01T00:00:00Z", "zone": "UTC"}

The timestamp is sample data for the recipe. It is not a report of any live outage. Swap it for the instant your fixture owner chose.

Properties speak a single time

A property check is a pure question over bytes. The same fixture should yield the same answer. Wall time and network stay outside the function.

Here is a small invoice property for totals. Line amounts sum to a net figure first. Tax uses the rate stored in the fixture file.

# proposal: illustrative property, not a production suite
from decimal import Decimal, ROUND_HALF_EVEN

def line_total(lines, tax_rate: str) -> Decimal:
    net = sum((Decimal(row["amount"]) for row in lines), Decimal("0"))
    rate = Decimal(tax_rate)
    tax = (net * rate).quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
    return net + tax

def property_invoice_total(fixture: dict) -> None:
    got = line_total(fixture["lines"], fixture["tax_rate"])
    expect = Decimal(fixture["expected_total"])
    if got != expect:
        raise AssertionError(f"total drift {got} != {expect}")

Run that function against the sealed invoice file. Do not let the same patch rewrite expected_total. Oracle edits still need a later human review.

python -m agent_gate.run \
  --fixture fixtures/invoices \
  --seal .fixture-seal \
  --retries 0 \
  --card artifacts/verdict-card.json

The zero-retry flag is the whole point of this gate. A red property stays red until someone explains it. You do not buy a greener clock with a second attempt.

If the module path does not exist yet, treat the command as a contract. Your runner must honor the same four flags. The names matter less than the refusal to retry.

Flakes freeze instead of voting

A flake is a test that changed color without a code change. It behaves like bad weather over the suite. That weather is not evidence for a merge.

When the single run marks a test flaky, write a freeze file. Frozen tests stay out of the merge vote. They also stay off the retry path for this patch.

{
  "freeze_id": "invoice-clock-sample",
  "tests": ["test_invoice_total_wall_clock"],
  "reason": "color changed across identical fixture bytes",
  "retry_allowed": false
}

Replace the sample id with your own run id. The reason string should name the fixture seal. A freeze without a seal is just a shrug.

The freeze file is a lease on silence, not a pass. A frozen test is debt on the board. Someone must fix the oracle or delete the test.

Do not auto-retry a frozen name in the same patch. A retry would launder red into green over time. That laundering is how the Friday miss survived review.

Keep the freeze set stable across the vote. A patch that adds a freeze and a feature together needs two commits. The first commit only records the testing debt.

The next commit may change the product code. Reviewers can then see the debt apart from the feature. Mixed commits hide which change actually earned the green.

The verdict card binds the run

The card binds four facts from the single run. It stores the fixture hash and the property rows. It stores the freeze set and the retry count.

# proposal: gate sketch, unexecuted
def accept(card: dict, expected_seal: str) -> bool:
    if card["retry_count"] != 0:
        return False
    if card["fixture_hash"] != expected_seal:
        return False
    if card["new_oracle_edits"]:
        return False
    frozen = set(card["flake_freeze"])
    for row in card["properties"]:
        if row["name"] in frozen:
            continue
        if row["status"] != "pass":
            return False
    return True

Read that function as policy, not as a benchmark. This draft does not report timings or pass rates. No suite size is claimed for any real repository.

A pass requires every voting property to pass cleanly. Frozen names are skipped, not counted as green. Retry count must stay zero, and the seal must match.

If the agent edited an expected file, mark oracle edits true. The card then rejects the patch without debate. A human must bless the new oracle in a later change.

That rule stops a quiet cheat during review. The model cannot move the goalposts and then grade itself. The expected file is evidence, not a scratch pad.

Store the card under artifacts/ and keep it with the patch. Reviewers should diff the card, not only the source. A missing card is the same as a failed gate.

python -m agent_gate.accept \
  --card artifacts/verdict-card.json \
  --seal .fixture-seal

Treat a non-zero exit as a blocked merge. Fill the seal from the seal file on the same host. Do not hand-copy a hash from a different tree.

A hand copy is how seals start to lie. The accept command should read both files itself. Humans compare the diff, not the raw digest.

Where a clean runner fits

You can draft these properties on a daily laptop. You still want a clean machine for the sealed run. Laptops leak clocks, agents, and half-saved files into the tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source project for this workflow. The supplied notes mention free model access and a free server option. This draft does not state quotas, hardware, duration, or permanence.

Use the free model access to sketch property stubs from fixture shapes. Keep secrets and customer rows out of the prompt. Paste field names and ranges, not live invoices.

Use the free server option for the one sealed run. The point is a clean clock and a closed network. It is a convenience, not a permanent budget for CI.

If that offer changes, the gate still stands on its own. Run the same card on any clean host you already trust. The product does not replace the hash or the freeze file.

One practical next step stays narrow on purpose. Sketch the property stubs with that free access. Then commit the verdict card from a sealed run you can rerun.

What this gate will not save

Skip this gate when tests must call a live partner. A closed network will fail those checks every time. That suite needs a contract test, not a sealed box.

Skip it when nobody owns the fixture format. A seal without an owner becomes stale folklore fast. Stale seals block good patches and wave bad ones through.

Skip it as a substitute for security review. A green card says the named properties held once. It does not say the tool surface stayed narrow.

Oracle changes still need a person in the loop. The card can only refuse to vote on them. It cannot decide the new business truth by itself.

Free model output can be wrong in quiet ways. Treat every model draft as an untrusted patch. The sealed run is the check, not the model's confidence line.

Do not widen fixtures to make a red property pass. That move is an oracle edit with extra steps. The card should see it and return false.

A short operating rhythm

Refresh the seal in the morning only when inputs change. Commit the new digest in the same change as the fixture. Leave the seal untouched for a code-only patch.

For each agent patch, run once with retries forced off. Write the verdict card next to the diff. If a voting property fails, stop and inspect the box.

If a test flickers, add it to the freeze file now. Leave it out of the vote for this merge. File a repair task that names the run id and the seal.

Merge only when accept returns true on the card. Revert if production later disagrees with that card. The card is a gate, not a pardon for the release.

This rhythm is slower than a blind retry loop. It is cheaper than a Monday ledger chase after a false green. One sealed miss is a fact you can still read.

Freeze Flakes Before Properties Vote

Finley Sun — Sun, 11 Oct 2026 21:13:56 +0000

The Friday patch looked small, tidy, and safe. An agent renamed a retry helper without touching public signatures. The first CI run returned a clean green badge.

The second run failed one idempotence property on identical inputs. The third run passed that same property again without a code change. The merge still landed before the Friday lunch break.

Monday's ledger showed duplicate charges on one customer account. The badge had treated unstable noise as proof of safety. This opening is a composite incident, not a measured outage recorded by this account.

A nearby testing question is whether a retry should count as proof. Better model drafts do not repair an oracle that flips on identical inputs. The gate has to separate those two failures before anyone merges.

Who may vote

Treat every automated check as a voter with one ballot. A property may block merge after two identical failures. A property may allow merge only after a stable pass.

A frozen flake may not cast either kind of vote. Its old failures stay filed as history, not as a fresh block. Its old passes also stay silent until a human repairs the oracle.

Example tests still help when you debug a single path. They do not certify behavior across generated inputs and boundaries. That split stays useful when an agent patch is wide.

Pin fixture bytes before you trust any oracle result. Hash the fixture pack and store the digest beside the patch. Commit that digest with the fixtures, outside the agent's unreviewed diff.

If that digest drifts, stop the run and refuse every vote. A moving fixture can counterfeit stability across two repeats. Drift is a stop condition, not a warning to click past.

Do not reuse a flake lease as this control. A lease expires and then invites another retry. A freeze removes the voter until a human repairs the oracle.

Those controls fail in different ways and should stay separate. Write the choice down so the next reviewer does not invent a third rule. The scorecard below is the written form of that choice.

Run the scorecard

The script is an unexecuted proposal, not a production result from this account. Copy it into a scratch repository and point it at your fixtures. Expect to replace the property names with invariants from your own service.

# proposal: unexecuted property gate
import hashlib, json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).resolve().parent
FIX = ROOT / 'fixtures'
FREEZE = ROOT / 'freeze_set.json'
PROPS = ['idempotent_write', 'bounded_retry', 'no_secret_in_log']

def digest(folder: pathlib.Path) -> str:
    h = hashlib.sha256()
    for p in sorted(folder.rglob('*')):
        if p.is_file():
            h.update(p.relative_to(folder).as_posix().encode())
            h.update(p.read_bytes())
    return h.hexdigest()

def load_freeze() -> dict:
    if not FREEZE.exists():
        return {'frozen': []}
    return json.loads(FREEZE.read_text())

def run_once(prop: str) -> int:
    cmd = [sys.executable, '-m', 'pytest', f'properties/test_{prop}.py', '-q']
    return subprocess.run(cmd, cwd=ROOT).returncode

def vote(prop: str, runs: int = 2) -> str:
    frozen = set(load_freeze()['frozen'])
    if prop in frozen:
        return 'frozen'
    codes = [run_once(prop) for _ in range(runs)]
    if any(c != codes[0] for c in codes):
        return 'freeze'
    return 'pass' if codes[0] == 0 else 'fail'

def main() -> int:
    expected = (ROOT / 'fixture_digest.txt').read_text().strip()
    if digest(FIX) != expected:
        print('fixture drift; no vote')
        return 2
    ballots = {p: vote(p) for p in PROPS}
    freeze = load_freeze()
    for name, ballot in ballots.items():
        if ballot == 'freeze' and name not in freeze['frozen']:
            freeze['frozen'].append(name)
    FREEZE.write_text(json.dumps(freeze, indent=2) + '\n')
    print(json.dumps(ballots, indent=2))
    if any(v == 'fail' for v in ballots.values()):
        return 1
    if any(v in {'freeze', 'frozen'} for v in ballots.values()):
        print('frozen voters abstain; human repair required')
        return 3
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

Pair the gate with one property that states a numeric cap. The example below is a proposal and has not been executed here. Swap the billing import before you treat a failure as real.

# proposal: unexecuted property, not a measured result
# import billing  # replace with the module under test

def test_bounded_retry(monkeypatch):
    calls = []

    def send():
        calls.append(1)
        raise TimeoutError('synthetic')

    monkeypatch.setattr('billing.transport.send', send)
    try:
        billing.charge(amount=10, retries=3)
    except TimeoutError:
        pass
    assert len(calls) == 3

Two repeats are a starting probe, not a statistical certificate. Raise the repeat count when the property touches clocks, networks, or files. Record that count in the scorecard so reviewers can see the budget.

A stable failure blocks the merge with exit code one. A mixed result freezes that property and writes its name. A frozen name abstains on every later run of the gate.

Abstention is not a pass and must not hide inside zero. The script returns three when any voter is frozen or newly frozen. CI should fail closed until that name is repaired in review.

Pin the digest before the agent starts editing the tree. Run the hasher once on a clean fixture directory. Store the printed line in fixture_digest.txt and commit it with the pack.

python - <<'PY'
import hashlib, pathlib
root = pathlib.Path('fixtures')
h = hashlib.sha256()
for p in sorted(root.rglob('*')):
    if p.is_file():
        h.update(p.relative_to(root).as_posix().encode())
        h.update(p.read_bytes())
print(h.hexdigest())
PY

Then invoke the gate on the agent patch and print the exit status. Let the script own the repeat count on ordinary patch runs. Run an extra manual pass only when you are testing the gate itself.

python property_gate.py
echo exit:$?

The process speaks in four codes, like a traffic light with a closed lane. Code two means the fixture bytes moved, so nobody votes. Code one means a stable property failed, so the merge stops.

Code three means a voter abstained and still needs a human owner. Code zero means every voting property passed with no abstention left. Wire CI to accept only zero if the badge must mean clean.

Inspect the freeze file after a noisy night of reruns. Each name in that file needs a human owner and a repair note. Do not delete a name merely to paint the CI badge green.

python -m json.tool freeze_set.json

The script has no waiver flag and no hidden retry button. Removal of a frozen name belongs in a reviewed repair commit. That commit should show new stable runs beside the removal.

How the patch enters

Give the agent a sealed task, not a blank repository. State the invariants that must remain true after the edit. Forbid edits under properties/ and fixtures/ inside the same patch.

A patch that rewrites its own judge is not evidence of safety. If the agent touched a property, stop and split the change. Land the oracle fix with human tests before the behavior patch faces this gate.

A coding assistant can draft text without becoming a voter. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source assistant with free model access.

A free server option is the other availability claim supplied for this draft. This article does not state quotas, model names, hardware, or duration. It also states no benchmark numbers and no promise that the offer remains.

Use free model access as a drafter, not as a judge of the patch. Paste a stable failure log and ask for three hypotheses only. Pin any new property yourself before it joins the voter list.

The free server option can host the repeat probe away from a laptop. If that server is unavailable, the local scorecard still stands alone. Availability is a convenience claim, not a correctness proof.

Read a stable miss

A stable failure is a product signal, not an inconvenience to rerun away. Open the property file and name the broken invariant in one sentence. Compare the agent diff with that sentence before you blame the test.

Many agent patches pass narrow examples and still break a boundary. Suppose bounded_retry fails twice on the same fixture pack. The helper now retries four times instead of the pinned cap of three.

The example test used a stub that stopped early and hid the cap. The property called the real limit and failed on both repeats. Keep the property, reject the patch, and request a smaller diff.

Suppose that same property flips between pass and fail. Freeze it and stop counting it as a voter. The cap may be fine while the clock inside the oracle is not.

A sleep inside the oracle is a common source of mixed ballots. Replace the sleep with a fake clock and rerun the same probe. Unfreeze the name only after identical inputs agree twice.

Who should skip it

Skip the gate when you have no pinned fixtures to hash. A property without fixed inputs is a story, not a vote. Skip it when every check needs a live vendor network.

Jitter will fill the freeze file and hide real regressions underneath. Skip it for prototypes that will be deleted within the week. The ceremony then costs more than the patch itself is worth.

Teams with a mature mutation suite should not treat this gate as a replacement. It does not replace mutation work or production monitoring either. It answers whether this patch broke a named invariant or the oracle wobbled.

Limits to keep visible

Two repeats will miss rare flakes that appear only weekly. A freeze file can grow until almost nothing still votes. Review that file each week and assign a named owner.

A green exit with many abstentions is a warning, not a release certificate. Hashing fixtures will not catch a property that encodes the same bug. If the agent and the oracle share a wrong assumption, both runs pass.

Human review of every new property remains mandatory before it votes. Generated cases also need a seed logged beside the ballot. Without that seed, you cannot replay the failure on demand.

Install pytest before you trust any ballot from this script. A missing runner looks like a stable failure, not like noise. Fix the environment first, then rerun the same pinned fixtures.

Free model drafts can look precise and still be wrong. Free server runs can differ from production clocks and filesystems. Treat both as optional tools around the scorecard, not above it.

Need a second draft for one frozen property name. MonkeyCode's free model access can propose wording, not a ballot. Keep the vote on your tree, and show stable runs before unfreezing.

Hold Merge Until the Trace Score Clears

Finley Sun — Sat, 10 Oct 2026 17:41:31 +0000

A payment pull request arrived just after two. The coding agent rewrote four payment assertions overnight. Shared CI painted the build a clean green.

Quiet fixture drift hid inside that green build. The same idempotency key could then pass twice. A green badge is not a money verdict.

Agent patches often fail in a quiet way. They edit the test until the test agrees. The production rule never gets a vote.

This gate treats the agent diff as a candidate only. A frozen request trace stays the merge authority. A green CI color never outranks that trace.

Three gates, one score

The first gate seals the fixture before scoring starts. A SHA-256 digest pins the fixture file bytes. A moved clock or edited row fails closed.

The second gate checks properties, not snapshots of prose. Negative amounts must stay rejected by the oracle. Currency and idempotency keys must survive the patch.

The sample property does not bless every accepted status. It only forces rejection for a negative amount. Extend the oracle before you trust richer flows.

The third gate keeps known flaky tests frozen. An agent commit cannot thaw a frozen flake. Thaw belongs in a separate human change.

Those three gates combine into one trace score. Any failed gate blocks the merge command. Partial credit is how quiet bugs ship.

A local score you can run

The harness below is a proposal, not a recorded run. Copy it into a disposable checkout before you trust it. Run it only on data you already own.

"""Proposal harness. Not executed for this article. Run it locally."""

import hashlib
import json
import pathlib
import sys

ROOT = pathlib.Path("agent_gate")
TRACE = ROOT / "frozen_trace.json"
SEAL = ROOT / "fixture_seal.json"
FLAKES = ROOT / "flake_quarantine.json"
PATCH = ROOT / "agent_patch.json"


def digest(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load(path: pathlib.Path) -> dict:
    return json.loads(path.read_text(encoding="utf-8"))


def inside_repo(path: pathlib.Path) -> bool:
    if path.is_absolute() or ".." in path.parts:
        return False
    return path.is_file()


def seal_matches(record: dict) -> bool:
    fixture_path = pathlib.Path(record["fixture_path"])
    if not inside_repo(fixture_path):
        return False
    return digest(fixture_path) == record["sha256"]


def property_holds(trace: dict, patch: dict) -> bool:
    cases = trace.get("cases") or []
    if not cases:
        return False
    if patch.get("oracle_sha256") != trace.get("oracle_sha256"):
        return False
    results = patch.get("results") or {}
    for case in cases:
        got = results.get(case["id"])
        if not isinstance(got, dict):
            return False
        if got.get("currency") != case["currency"]:
            return False
        if got.get("idempotency_key") != case["idempotency_key"]:
            return False
        if case["amount_cents"] < 0 and got.get("status") != "rejected":
            return False
    return True


def flakes_still_frozen(record: dict) -> bool:
    items = record.get("tests") or []
    if not items:
        return False
    for item in items:
        if item.get("state") != "frozen":
            return False
        if item.get("thaw_allowed") is not False:
            return False
        if not item.get("owner") or not item.get("review_on"):
            return False
    return True


def score() -> int:
    gates = {
        "seal": seal_matches(load(SEAL)),
        "property": property_holds(load(TRACE), load(PATCH)),
        "flakes": flakes_still_frozen(load(FLAKES)),
    }
    failed = [name for name, ok in gates.items() if not ok]
    print(json.dumps({"gates": gates, "failed": failed}))
    return 0 if not failed else 1


if __name__ == "__main__":
    sys.exit(score())

Create the four JSON files beside the script. Point the seal record at a real fixture file. Record the fixture digest before the agent runs.

{
  "fixture_path": "fixtures/payment.json",
  "sha256": "replace-with-sha256sum-output"
}

{
  "clock": "frozen-sample-clock",
  "rows": [
    {"id": "pay-1", "amount_cents": 1200, "currency": "USD"}
  ]
}

{
  "oracle_sha256": "replace-with-reviewed-oracle-digest",
  "cases": [
    {
      "id": "pay-1",
      "amount_cents": 1200,
      "currency": "USD",
      "idempotency_key": "key-1"
    },
    {
      "id": "pay-2",
      "amount_cents": -40,
      "currency": "USD",
      "idempotency_key": "key-2"
    }
  ]
}

{
  "oracle_sha256": "replace-with-reviewed-oracle-digest",
  "results": {
    "pay-1": {
      "status": "accepted",
      "currency": "USD",
      "idempotency_key": "key-1"
    },
    "pay-2": {
      "status": "rejected",
      "currency": "USD",
      "idempotency_key": "key-2"
    }
  }
}

{
  "tests": [
    {
      "name": "test_refund_retry_window",
      "state": "frozen",
      "thaw_allowed": false,
      "owner": "payments-review",
      "review_on": "set-by-owner"
    }
  ]
}

The sample digests are placeholders, not measured values. Replace them before you trust a green score. A copied placeholder should fail the human review.

The trace stores the oracle hash and the cases. The patch file stores results, not new rules. If the oracle hash moves, the property gate fails.

Negative amounts are the smallest useful property here. A richer payment domain needs its own invariants. Do not pretend this sample covers live refunds.

Flake records must say frozen and deny thaw. A missing flag is a failed gate. Silence from a missing file is not a pass.

Store that digest in the seal file by hand. Do not ask the agent to fill the digest. The agent is the party being scored.

mkdir -p agent_gate fixtures
sha256sum fixtures/payment.json | awk '{print $1}'
python3 agent_gate/score_trace.py
test $? -eq 0

Run the scorer from the repository root directory. A zero exit code means every gate held. Any other exit should stop the merge.

Wire that command into the protected merge job. Do not let the agent edit the scorer in the same diff. Review scorer edits as policy changes, not chores.

Name the merge job so humans can find the brake. Call the required status check trace-score, not tests. Vague names let a green badge sneak through.

Fill the results map from a runner you start. A short adapter can read a local JSON log. That adapter must live outside the agent diff.

The adapter only copies fields from the runner. It must not invent status values for a case. A missing case stays missing, and the score fails.

The results map is filled by your local test runner. The model does not get to write that map. Otherwise the candidate grades its own exam.

Rules the badge cannot overrule

Snapshot tests lock incidental strings and stray timestamps. Property checks lock rules the business cannot drop. Prefer the business rule when the two disagree.

Each case carries an idempotency key from the trace. The patch result must echo that same key. A new key is a new charge, not a retry.

Currency is checked as an exact field match. A patch that converts cents in secret fails. Any conversion belongs in an explicit reviewed change.

Suppose the agent changes a rejected status to ok. The sample case amount is negative forty cents. The property gate fails before any merge starts.

Suppose a test edits the fixture clock during setup. The seal digest no longer matches the record. Scoring stops even when every assertion still passes.

Suppose a flaky retry test is marked thawed. The agent did it to make CI quiet. The flake gate fails on that flag alone.

The reviewer reads the printed JSON before the diff. Failed gate names should match the code story. A green story with a failed seal is a lie.

A real contract change needs a new trace commit first. Land the oracle update only with human review. Only then may a later agent patch follow.

The score script uses only the standard library. No network call is allowed inside the score. A gate that phones home is no longer local.

Paths stay relative so the checkout remains portable. Absolute paths hide whose machine produced the seal. Refuse a seal that points outside the repo.

This score does not parse the git diff. A rewritten assertion can still sit beside a passing map. Read the test diff even when the score is clean.

Rehearse away from the oracle

You can draft the candidate patch on cheaper infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access can draft the candidate diff.

The free server option can rehearse this harness. The trace score still belongs on a host you control. The rehearsal host is not the merge oracle.

Clocks, data, and packages can differ on that host. Bring the diff home before you score it. This article claims no quota, model name, or duration.

Treat current availability as something you verify yourself. A free option is a drafting desk, not a promise. The local score file remains the release rule.

Where this brake fails

The trace score is narrow by design here. It catches oracle edits, seal drift, and silent flake thaw. It does not prove the patch is wise.

A correct property can still allow a bad product change. A human still must read each money movement. Automation should only refuse clear gate breaks here.

Frozen traces rot when the contract truly changes. Update the trace in its own reviewed commit. Do not let the agent refresh the oracle.

Flake quarantine can hide real regressions if it grows forever. Give each frozen test an owner and a review date. A quarantine without an owner becomes a junk drawer.

Keep the quarantine file small enough to read aloud. If you cannot explain a frozen test, thaw it or delete it. Mystery flakes should not ride along forever after that.

The free server rehearsal can pass while local scoring fails. That gap is useful information, not a reason to skip the score. Environment luck is not evidence of a safe patch.

Skip this approach if you have no stable oracle. Skip it if every test is already flaky and frozen. A gate with no healthy suite only blocks work.

Skip it for unattended merges of payment code. The score is a brake, not a driver. An agent should not approve its own witness.

Skip it if you cannot keep fixtures small and owned. Huge fixtures hide drift inside unused fixture fields. Seal only the bytes that the property reads.

Do not publish score output that contains customer data. Keep every trace synthetic until counsel says otherwise. A useful gate can still leak live secrets.

Start with one payment path and three cases. Seal the fixture and freeze the known flakes. Score the next agent diff before anyone merges.

A low-cost draft desk is optional, not the oracle. MonkeyCode's free model access and free server can rehearse the harness. Then run the trace score on a machine you control.

Destroy the Host After the Agent Verdict

Finley Sun — Thu, 08 Oct 2026 09:38:37 +0000

A checkout job failed at 02:11 on a shared runner. The agent patch was green in the log. An earlier job had rewritten the fixture file.

The suite never judged that patch on its own. It judged leftover files from a neighbor job. Shared hosts fail this way more often than logic.

The machine still remembers yesterday's files and caches. A clean replay is the only honest witness. This note is a proposed workflow, not a measured report.

No production metrics are claimed in this draft. Every command below is an unexecuted local example. Adapt paths, hosts, and secrets before you run them.

Residue beats the diff

Think of a kitchen sponge shared by every cook. It still looks ready for the next plate. It still carries the spill from last night.

A shared CI runner behaves like that sponge. Caches, env files, and half-written fixtures survive the job. An agent patch then passes against residue, not against code.

A longer suite does not fix that lie. More retries only stir the same dirty sponge. The useful move is a disposable host and a frozen flake list.

Generation and judgment must not share one budget. The agent may spend tokens to draft a diff. The verdict must spend a separate, smaller budget on replay.

Freeze the ledger before the diff

Flaky tests act like fog on the road. They hide a real miss behind random noise. They also invite a retry loop that spends the window.

Freeze them before the agent diff is judged. A frozen id is skipped, not deleted from the tree. The skip is recorded so the debt stays visible.

The script name below is a proposal, not a shipped tool. Point it at a ledger you already review by hand. Do not treat the path as a public package.

# Unexecuted example. Adapt paths before any real run.
git diff --name-only origin/main...HEAD > /tmp/agent.touched
python scripts/flake_freeze.py \
  --ledger tests/flake_ledger.json \
  --touched /tmp/agent.touched \
  --out /tmp/flake.skip

The ledger is a small contract, not a junk drawer. Each row names a test id, a reason, and an expiry date. An expired row cannot hide a new failure.

# Proposal only. This draft did not execute it.
import json
from datetime import date

def active_skips(ledger_path: str, today: date) -> list[str]:
    rows = json.loads(open(ledger_path, encoding="utf-8").read())
    kept = []
    for row in rows:
        exp = date.fromisoformat(row["expires"])
        if exp >= today and row.get("state") == "frozen":
            kept.append(row["nodeid"])
    return kept

def write_skip(ledger_path: str, today: date, out_path: str) -> int:
    skips = active_skips(ledger_path, today)
    with open(out_path, "w", encoding="utf-8") as handle:
        handle.write("\n".join(skips))
    return len(skips)

If the touched diff includes a frozen test, stop the gate. Do not let the agent weaken that assert. A quieter assert is not a flake fix.

Write the stop into the job log in one line. Name the node id and the ledger expiry. A reviewer should see the halt without opening a chat.

Pin the service, not the patch

Property checks should describe the service, not the patch text. A cart total stays non-negative after a refund. A retry must not double-book the same seat.

Those sentences form the contract under test. The agent may change handlers and their helpers. It may not edit the contract in that same diff.

Split a contract edit into a human review. Land it before the agent patch, or after, never mixed. Mixed diffs make the replay argue with itself.

# Proposal. Hypothesis-style check, not executed here.
from hypothesis import given, strategies as st

@given(
    st.integers(min_value=0, max_value=10_000),
    st.integers(min_value=0, max_value=10_000),
)
def test_refund_never_exceeds_charge(charge, refund):
    result = apply_refund(charge_cents=charge, refund_cents=refund)
    assert result.refunded <= charge
    assert result.balance >= 0

Seed the generator and record the seed in the artifact. A later replay must see the same examples. Otherwise two passes can describe two different worlds.

Reject a run that collects zero property examples. An empty pass is a missing witness, not a success. File that as a setup fault and discard the host.

Bin the host after the copy

Replay the touched properties on a clean machine. A laptop cache is not clean enough for a verdict. A local volume often keeps the last container's files.

A disposable server is the closer analogy here. It is a paper plate used once, then binned. The next patch must not taste the last patch.

MonkeyCode is an open-source tool with a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that server as a throwaway box, not as production.

Confirm current limits in the project docs that same day. This draft states no quotas, hardware, or duration. A number you have not opened is only a rumor.

Free model access can sit beside the host, with a fence. Use it to label an unknown replay row. Do not use it to rewrite a property or a fixture.

# Unexecuted sketch. Point this at your disposable host.
rsync -a --delete ./tests ./src replay@replay-host:/opt/replay/
ssh replay@replay-host 'cd /opt/replay && python -m pytest -q --tb=line -k property --junitxml=/tmp/replay.xml'
scp replay@replay-host:/tmp/replay.xml ./artifacts/replay.xml
ssh replay@replay-host 'rm -rf /opt/replay'

The pytest call expects a conftest that honors the skip file. That hook is also a proposal, shown next, and it was not run. Copy the XML home before you delete the tree.

# Proposal. Unexecuted pytest hook for a frozen skip file.
from pathlib import Path

def pytest_collection_modifyitems(config, items):
    skip_file = Path("/tmp/flake.skip")
    if not skip_file.exists():
        return
    frozen = {
        line.strip()
        for line in skip_file.read_text(encoding="utf-8").splitlines()
        if line.strip()
    }
    items[:] = [item for item in items if item.nodeid not in frozen]

The last remote command is the method in one line. Destroy the workspace after the verdict is copied home. A second patch must inherit nothing but the git tree.

If the copy fails, do not trust a verbal green. The JUnit file is the witness, not the agent's summary. Missing XML means the replay did not happen.

Spend the model only on unknown rows

Keep the model inside a closed label set. It may read a single failure message only. It may not emit a patch, a fixture, or a new assert.

# Proposal. Closed labels for one JUnit failure message.
LABELS = ("assert", "timeout", "setup", "skip_expired", "unknown")

def label_failure(message: str) -> str:
    text = message.lower()
    if "assertionerror" in text:
        return "assert"
    if "timeout" in text or "timed out" in text:
        return "timeout"
    if "fixture" in text or "setup" in text:
        return "setup"
    return "unknown"

Call the free model only when the label is unknown. Ask for one sentence of hypothesis, not a diff. Drop any reply that contains a code fence or a file path.

If free model access is unavailable, stop at the local label. Do not block the merge on a missing clerk. Block it on the property result from the clean host.

An assert label rejects the agent patch outright. A timeout label discards the host and allows one replay. A second timeout rejects the patch as not replay-stable.

A setup label also discards the host before any retry. Two setup faults mean the patch is not ready. Do not burn a third cycle to be hopeful.

Walk the cases without a shortcut

Start with the touched file list from git. If that list hits a frozen node id, halt. Open a human ticket and leave the agent diff unmerged.

If the clean host reports a property assert, reject. Do not ask the model for a softer contract. The contract was pinned before this diff existed.

If teardown did not run, the verdict is incomplete. Rerun teardown, or mark the host tainted, before any next job. A tainted host is the shared sponge again.

An expired flake that this diff did not touch stays frozen. It does not enter today's verdict at all. Mixing old debt into a new patch hides both problems.

Store three files beside the pull request record. Store the skip list, the JUnit XML, and the teardown log. A reviewer can audit the judgment without trusting chat.

Leave these jobs outside the gate

This path will not prove concurrency safety under load. It will not replace a staging soak or a canary. It only answers whether the pinned properties held on a clean box.

Do not use it for payment code alone. Add a reconciliation check that this sketch does not include. Synthetic carts are not a real settlement guarantee.

Do not put customer data on the free host. The fixture must be generated, not copied from production. A paper plate is not a vault for customer rows.

Skip this path if you cannot destroy the machine. A free server you must keep warm is another shared runner. Residue returns, and the verdict rots with it.

Skip it if you have no flake ledger yet. Build the ledger before you automate the replay. A cap spent on fog is not a testing strategy.

Skip it if the agent diff edits fixtures and code together. Split that change, then replay only the code half. Otherwise you are grading a test the author rewrote.

Check the docs on the day you run

No speed benchmark is claimed in this article. No model name is required for the label fence. No token quota is stated, because an unchecked quota is not evidence.

Read the current MonkeyCode docs on the day you run. Free model access and the free server option can change. Your gate should depend on the docs, not on a post.

If you try the disposable host, start from those current notes. Verify the access rules yourself before a nightly job. Then keep the fence and never mint the contract.