DEV Community: Dakota Wu

Declare a Blast Radius Before You Refactor: Keep the Smallest Safe Change Actually Small

Dakota Wu — Tue, 27 Oct 2026 12:56:49 +0000

Consider a 2,300-line pricing module that three teams call through a single public function, and an assistant that returns a 340-line diff touching eleven files. The diff may even be correct, but nobody on the review call can tell you which of those eleven files the change was supposed to touch. "Smallest safe change" is a nice principle and a terrible gate, because nothing in the pipeline defines what "smallest" means for this commit.

The practical fix is to write the intended scope down as data before you edit anything. Below is a workflow that freezes behavior at the module boundary, records an allowed blast radius in a file, and fails CI when the diff escapes it. It also adds a revert drill, because a change you cannot cleanly undo was never small.

Three ways "smallest safe change" quietly fails

Most refactor regressions I have seen reviewed share one of these shapes, and none of them are caught by reading the diff.

Scope creep by convenience. The change touches a helper two directories away because the import was already there, and that helper has its own callers.
Behavior drift at the edges. Output looks right for the fixture in the PR description but changes for None, empty lists, or rounding at the fourth decimal.
Unprovable revert. The commit mixes a rename, a moved function, and a logic tweak, so git revert conflicts and the rollback becomes its own project.

Each failure is a measurement problem rather than a taste problem, which means each one can be turned into a check.

Step 1: Freeze behavior at the boundary, not inside the module

Characterization tests are usually written against internals, which makes them break during exactly the refactor they were meant to protect. Put the corpus at the seam instead: call the module the way production calls it, capture the returned value and the raised exception, and normalize both into canonical JSON.

# refactor/corpus/runner.py  (Python 3.11, stdlib only)
import hashlib
import importlib
import json
import sys
from decimal import Decimal
from pathlib import Path


def normalize(value):
    """Reduce a return value to a stable, comparable structure."""
    if isinstance(value, Decimal):
        return {"__decimal__": format(value, "f")}
    if isinstance(value, float):          # keep repr, never round silently
        return {"__float__": repr(value)}
    if isinstance(value, dict):
        return {str(k): normalize(v) for k, v in sorted(value.items(), key=lambda kv: str(kv[0]))}
    if isinstance(value, (list, tuple)):
        return [normalize(v) for v in value]
    if isinstance(value, (str, int, bool)) or value is None:
        return value
    return {"__type__": type(value).__name__}


def run(module_path, cases_path):
    module = importlib.import_module(module_path)
    cases = json.loads(Path(cases_path).read_text())
    records = []
    for case in cases:
        entry = {"id": case["id"]}
        try:
            result = module.price_order(**case["args"])
            entry["ok"] = normalize(result)
        except Exception as exc:           # exceptions are behavior too
            entry["err"] = {"type": type(exc).__name__, "msg": str(exc)}
        records.append(entry)
    return records


if __name__ == "__main__":
    records = run(sys.argv[1], sys.argv[2])
    blob = json.dumps(records, sort_keys=True, indent=2)
    Path(sys.argv[3]).write_text(blob + "\n")
    print("snapshot sha256:", hashlib.sha256(blob.encode()).hexdigest()[:16])
    print("cases:", len(records))

Run it on the untouched code and commit the snapshot as the golden file. From then on, the refactor must reproduce that file byte for byte, and any intentional difference becomes a reviewable line in the diff instead of a surprise in production.

Two rules keep this corpus honest over time:

Include the ugly inputs you already know about — empty collections, zero quantity, duplicate line items, unknown currency codes.
Regenerate the snapshot only in a separate commit with a second reviewer, so "update golden" never hides a behavior change inside a refactor.

Step 2: Write the blast radius down as a file

A scope file turns an intention into an artifact that CI can read. Keep it small, keep it next to the refactor, and treat edits to it as review-worthy events.

{
  "base": "9f3c2ab",
  "max_changed_lines": 180,
  "allowed_globs": ["src/pricing/**", "refactor/corpus/**"],
  "forbidden_globs": ["src/billing/**", "migrations/**", "pyproject.toml"]
}

The checker below is deliberately blunt: it lists changed files, compares them against the globs, and sums the diff size. Blunt is a feature here, because a reviewer can read the whole rule in under a minute.

# scripts/check_scope.py
import fnmatch
import json
import subprocess
import sys

scope = json.load(open("refactor/scope.json"))
base = scope["base"]

changed = subprocess.run(
    ["git", "diff", "--name-only", f"{base}...HEAD"],
    capture_output=True, text=True, check=True,
).stdout.split()

numstat = subprocess.run(
    ["git", "diff", "--numstat", f"{base}...HEAD"],
    capture_output=True, text=True, check=True,
).stdout.strip().splitlines()

added = sum(int(l.split("\t")[0]) for l in numstat if l.split("\t")[0].isdigit())
removed = sum(int(l.split("\t")[1]) for l in numstat if l.split("\t")[1].isdigit())

problems = []
for path in changed:
    if any(fnmatch.fnmatch(path, p) for p in scope["forbidden_globs"]):
        problems.append(f"forbidden path touched: {path}")
    elif not any(fnmatch.fnmatch(path, p) for p in scope["allowed_globs"]):
        problems.append(f"outside declared scope: {path}")

if added + removed > scope["max_changed_lines"]:
    problems.append(f"diff size {added + removed} exceeds budget {scope['max_changed_lines']}")

for problem in problems:
    print("FAIL", problem)

sys.exit(1 if problems else 0)

Wire it into the CI job that already runs your tests, and the conversation in review changes from "is this diff reasonable" to "does this diff match the declared intent".

Step 3: Prove the revert before you merge

The revert drill takes about two minutes and settles arguments that otherwise run for days. On the refactor branch, run these in order:

python -m refactor.corpus.runner src.pricing refactor/corpus/cases.json /tmp/after.json
git diff --no-index refactor/corpus/golden.json /tmp/after.json and confirm it is empty.
git revert --no-commit HEAD and confirm the revert applies without conflicts.
Re-run the runner and confirm the snapshot matches golden again.
git revert --abort and continue with the original commit.

If step 3 conflicts, the commit is doing too many things at once, and splitting it is cheaper now than during an incident. If step 4 mismatches, the revert is incomplete, which means the commit carried state changes outside the files it edited.

Where free tooling fits without owning the decision

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The corpus is the part of this workflow that people skip, because writing thirty awkward inputs by hand is tedious. MonkeyCode's free model access is useful there as a candidate generator: paste the public function signature and docstring, ask for edge-case inputs, then keep only the ones you can justify against real callers. The model never writes the golden snapshot, and a human reviews every case before it lands in cases.json.

The free server option is a reasonable home for the runner and its report, so a reviewer on another machine hits the same endpoint instead of rebuilding a local environment for a twenty-minute check. Treat availability and terms as things to confirm on the project page before you plan around them, since free-tier offers change and this article should not be your source of truth for a quota.

When this workflow is the wrong tool

Be honest about the boundary conditions; forcing this pattern everywhere produces ceremony with no signal.

Situation	Use this workflow?	Reason
Large module with stable public entry points	Yes	Boundary is real and cheap to snapshot
Module whose behavior is intentionally changing	No	Golden file churn hides real regressions
Heavily concurrent or time-dependent code	Rarely	Non-deterministic output defeats the snapshot
One-file, one-caller script	No	Review effort exceeds the risk
Code with no test harness at all	After, not before	You need some harness to capture anything

The reviewer checklist

Before approval, four questions should have four answers, and none of them require reading every line of the diff.

Does the scope file name the base commit, and is the diff inside it?
Does the golden snapshot match, with no "regenerated" commit mixed into the refactor?
Did the revert drill pass, including the post-revert snapshot comparison?
Is there one stated intent for this commit, written in the PR body in a single sentence?

None of this makes a refactor safe by itself, and the corpus only knows the inputs somebody thought to include. What it does give you is a cheap, mechanical answer to the two questions that stall real reviews: how far was this allowed to reach, and can we put it back.

Count the Argument Shapes Your Fixtures Never See Before Refactoring a Messy Module

Dakota Wu — Mon, 26 Oct 2026 11:30:11 +0000

A shipping-cost module has twelve tests, all passing, and a refactor that looks mechanical. Someone extracts the rate calculation into a helper, the suite stays green, and the change ships. Three days later a batch job that passes an empty line-item list starts returning None instead of 0.0, because no test ever called that function with an empty collection. The interesting part is not the bug itself; it is that the suite had no way to tell anyone which input shapes it had never seen.

Coverage tools answer a different question than the one you are asking. A line-coverage report tells you which statements executed, but it says nothing about the diversity of values that reached them. A branch that has only ever received positive floats looks fully covered while being completely uncharacterized. This article describes a small, reproducible step you can run before touching a messy module: mine the call sites, list the argument shapes, and refuse to refactor shapes your fixtures never exercise.

Why coverage numbers lie about input shape

Characterization tests are only as good as their corpus, and most corpora are written from memory rather than from evidence. Engineers write the happy path, one failure path, and maybe a case from the bug tracker. The result is a suite that documents intent but not reality, and a refactor that preserves intent while quietly discarding reality.

The artifact below closes that gap in three steps, each of which is small enough to finish in an afternoon. First, extract every call site of the target function and bucket its arguments by shape. Second, compare those buckets against the shapes your fixtures already cover. Third, convert every unseen shape into a characterization row before you allow a single line to move.

Step 1 — Mine call sites into a shape manifest

The following script is plain Python with no dependencies beyond the standard library, and it works on any repository that parses as valid Python. It walks every .py file, finds calls whose name matches your target, and classifies each positional argument into a coarse shape bucket. Coarse is deliberate: StrLiteral and Number are enough to catch the empty-collection and None gaps that real refactors break on.

# tools/call_shapes.py
import ast, json, sys
from pathlib import Path
from collections import Counter

TARGET = "compute_quote"

def shape(node):
    if node is None:
        return "omitted"
    if isinstance(node, ast.Constant):
        if node.value is None:
            return "none"
        if node.value == "":
            return "empty_str"
        if isinstance(node.value, bool):
            return "bool"
        if isinstance(node.value, (int, float)):
            return "number"
        return "str_literal"
    if isinstance(node, (ast.List, ast.Tuple, ast.Set)):
        return "empty_collection" if not node.elts else "collection"
    if isinstance(node, ast.Dict):
        return "empty_collection" if not node.keys else "collection"
    if isinstance(node, ast.Name):
        return "name"
    if isinstance(node, ast.Call):
        return "call_result"
    if isinstance(node, ast.Attribute):
        return "attribute"
    return "other"

rows = []
for path in Path(".").rglob("*.py"):
    if ".venv" in path.parts or "tests" in path.parts:
        continue
    tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
    for node in ast.walk(tree):
        if not isinstance(node, ast.Call):
            continue
        fn = node.func
        name = fn.id if isinstance(fn, ast.Name) else getattr(fn, "attr", None)
        if name != TARGET:
            continue
        rows.append({
            "file": str(path),
            "line": node.lineno,
            "args": [shape(a) for a in node.args],
            "kwargs": sorted(k.arg for k in node.keywords if k.arg),
        })

json.dump(rows, open("shape_manifest.json", "w"), indent=2)
print(f"{len(rows)} call sites of {TARGET}", file=sys.stderr)

Run it from the repository root with python tools/call_shapes.py. The manifest is evidence: every entry is a real call site with a file and line you can open. Filtering tests/ out is intentional, because test calls describe intent while production calls describe load.

Reading the manifest honestly

The output below is illustrative rather than measured, but the pattern it shows is the one you should expect: a long tail of literals plus a cluster of name arguments you cannot resolve statically.

[
  {"file": "jobs/nightly.py", "line": 41, "args": ["collection", "number"], "kwargs": []},
  {"file": "api/checkout.py", "line": 88, "args": ["empty_collection", "name"], "kwargs": []},
  {"file": "etl/backfill.py", "line": 12, "args": ["name", "number"], "kwargs": ["currency"]}
]

Two buckets deserve attention immediately. empty_collection is the shape that most pure helpers silently mishandle, and name is the shape that static analysis cannot resolve. For name arguments, grep the enclosing function for the variable's assignment rather than guessing.

Step 2 — Turn unseen shapes into characterization rows

Now compare the manifest against your existing fixtures and write down the difference. A short comparison script is enough: collect the shapes your parametrized tests declare, subtract them from the manifest's shapes, and print the remainder. The remainder is your refactor backlog, ordered by how many production call sites depend on each shape.

Each remainder entry becomes one characterization row. Do not assert what the function should return; record what it does return today, even if today's behavior looks wrong.

# tests/test_quote_characterization.py
import json, pathlib, pytest
from pricing.quote import compute_quote

GOLDEN = pathlib.Path("tests/golden/quote.json")
CASES = json.loads(GOLDEN.read_text())

@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_characterization(case):
    assert compute_quote(*case["args"]) == case["expected"]

The golden file is generated once by a small recorder script and then reviewed by a human before it is committed. Reviewing the recorded values is the moment you learn whether the empty-list case really returns 0.0 or something uglier. If it returns something ugly, that is a finding, not a problem to fix during the refactor.

Step 3 — Make the smallest safe change

With the corpus in place, the refactor itself becomes almost boring, which is the point. Change one thing per commit, and let the characterization suite prove that every recorded shape still produces its recorded value.

Add the guard clause for the empty collection, guarded by the new row that pins today's behavior.
Run the full characterization suite and confirm the row still passes unchanged.
Move the rate calculation into a helper, touching nothing else in the same commit.
Re-run the suite; any diff here is a behavior change you did not intend.
Only after the suite is green for the whole corpus do you correct the ugly behavior, in a separate commit with its own test.

A useful decision table keeps the team honest about when a refactor is allowed to proceed.

Shape in production	Fixture covers it	Action
Yes	Yes, with recorded value	Safe to move code
Yes	No	Add a characterization row first
No	Yes	Keep the row, do not delete it
No	No	Leave it out; do not invent a case

Where free model access and a free server actually fit

Shape manifests generate tedious work: writing fixture rows, resolving name arguments, and writing the one-line commit messages that explain each rung. This is a reasonable place to use MonkeyCode, an open-source project that offers free model access and a free server option; per the project's own description, the free tier includes a large token allowance, and the server option lets you run the tooling without provisioning a box first.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A workflow that respects the evidence looks like this: run call_shapes.py locally, paste only the shape summary into the model, and ask it to propose fixture rows for the unseen buckets. Never paste proprietary source if your organization's policy forbids it, and treat every proposal as a draft to be verified against the actual function. The free server is useful for running the characterization suite in a clean environment where cached .pyc files and stale virtualenvs cannot hide a regression.

Limitations and who should skip this

This approach is static and shallow, so it will miss shapes that arrive through dynamic dispatch, *args unpacking, or data loaded at runtime. The shape buckets are intentionally coarse, and a number bucket hides every difference between -1, 0, and 10**9. Repositories with heavy metaprogramming will get a mostly-name manifest that provides little signal.

Skip this entirely if the module under refactor has no production call sites yet, or if a single well-understood caller exists that you can read in full. Skip it too if your test suite already runs against recorded production traffic, because that corpus is strictly better evidence than anything static analysis can produce.

A short checklist

Mine call sites before you open the file you intend to change.
Bucket arguments coarsely and count production sites per bucket.
Subtract fixture shapes from production shapes and print the remainder.
Record today's outputs verbatim; never assert intent during characterization.
Change one thing per commit and keep the corpus fixed between commits.

The corpus is the deliverable; the refactor is just the thing you were finally allowed to do. If you try the shape-manifest step on a module of your own, the interesting part is usually the third bucket you never expected to find.

Witness Maps: Never Refactor a Line That No Test Executes

Dakota Wu — Tue, 20 Oct 2026 09:37:03 +0000

A 41-file cleanup diff lands on a Tuesday morning, the CI suite is green, and the reviewer approves it in nine minutes. Two days later an invoicing edge case breaks in production, inside a function that no test has ever entered. The suite was not lying — it simply never executed the lines that changed.

That gap is the entire problem. A green run tells you the paths your tests cover still behave, and it stays silent about the paths they never touch. Aggregate coverage numbers hide this further, because a repository reported at 78% line coverage still leaves roughly a fifth of its lines anonymous to every assertion in the build.

This post describes a small gate you can build in an afternoon: a witness map. It answers one narrow question before you accept a refactor — for each line inside the spans you plan to touch, which existing test executes it?

Why suite-level green is the wrong unit of evidence

Coverage tools report per file, per function, or per branch, and reviewers read those aggregates as a proxy for safety. The unit that actually matters during a refactor is the individual line you are about to rename, extract, or reorder. Three properties of real code make the aggregate misleading:

A module can sit at 90% coverage while the exact ten lines you plan to extract sit in the remaining 10%.
Coverage counts execution, not assertion — a line can be executed by a test that never checks its result.
AI-generated cleanup diffs tend to touch many small spans at once, so the probability that at least one span is unwitnessed grows quickly with diff size.

If you have been feeling the review-queue pressure that shows up in so many recent posts about AI-assisted refactoring, this is the mechanical cause. The bottleneck is not generating the diff; it is localizing which lines of the diff are safe.

Step 1: Freeze the candidate diff against the current tree

A witness map only works if old-side line numbers match the code you measured. Produce the candidate diff against the branch you actually intend to modify, and verify it applies cleanly before doing anything else.

# candidate.diff must be relative to the current HEAD, not to a rewritten branch
git apply --check .evidence/candidate.diff

If a model or an agent produced the diff on a scratch branch, rebase that branch onto your target commit first, then regenerate with git diff --unified=3. Three lines of context are enough for the mapping and keep hunk spans small.

Step 2: Capture executed lines from the suite you already have

You do not need new tests yet. Run the existing suite under coverage and export machine-readable data.

mkdir -p .evidence
python -m coverage run --source=app -m pytest -q
python -m coverage json -o .evidence/cov-before.json
git diff --unified=3 > .evidence/candidate.diff

If your suite needs a database or a seeded fixture, wrap the whole command in whatever your project already uses for a clean environment. The only requirement is that cov-before.json describes the same commit that .evidence/candidate.diff was generated against.

Step 3: Intersect hunk spans with executed lines

The script below parses old-side hunk ranges out of the unified diff and intersects them with the executed_lines arrays in the coverage JSON. Lines that fall inside a hunk but never appear in executed_lines are reported as orphans, and any orphan fails the gate.

# tools/witness_map.py
import json, re, sys
from pathlib import Path

HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")

def old_side_spans(diff_text):
    spans, path = {}, None
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            spans.setdefault(path, [])
        elif line.startswith(("--- ", "diff ", "index ")):
            continue
        m = HUNK.match(line)
        if m and path:
            start, count = int(m.group(1)), int(m.group(2) or 1)
            spans[path].append((start, start + count - 1))
    return spans

def executed_lines(cov_path):
    data = json.loads(Path(cov_path).read_text())
    return {f: set(v["executed_lines"]) for f, v in data["files"].items()}

def main(diff_path, cov_path):
    spans = old_side_spans(Path(diff_path).read_text())
    covered = executed_lines(cov_path)
    orphans_total = 0
    for path in sorted(spans):
        hit = covered.get(path, set())
        for start, end in spans[path]:
            touched = set(range(start, end + 1))
            witness = touched & hit
            orphans = sorted(touched - hit)
            pct = 100 * len(witness) / len(touched)
            print(f"{path}:{start}-{end} {len(witness)}/{len(touched)} ({pct:.0f}%)")
            if orphans:
                orphans_total += len(orphans)
                print("  orphans:", ",".join(map(str, orphans)))
    print(f"\norphan lines in planned spans: {orphans_total}")
    return 1 if orphans_total else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it and treat exit code 1 as a stop sign, not a warning:

python tools/witness_map.py .evidence/candidate.diff .evidence/cov-before.json

Sample output from a small demonstration repository, included only to show the format (numbers are illustrative, not a benchmark):

app/billing/invoice.py:118-146 4/29 (14%)
  orphans: 121,122,123,127,131,133,134,135,136,138,141,142,143,144,145,146
app/billing/tax.py:12-40 28/28 (100%)

orphan lines in planned spans: 16

That output changes the plan immediately. The tax module is cleared for extraction because every line in the span already has a witness. The invoice span is not, and no amount of careful renaming fixes the missing evidence.

Step 4: Turn orphans into characterization tests, one rung at a time

Each orphan is a small assignment, and the smallest safe change means resolving them before editing rather than after. Work in this order:

Read the orphan line and write a test that executes it through the public entry point, not by importing a private helper.
Assert the current output verbatim, including awkward formatting, so a later behavior change fails loudly.
Re-run the harness; when a span reaches 100%, you may edit only the lines inside that span.
Commit the test and the refactor separately, so a revert of the change never removes the evidence.

Some orphan lines are genuinely unreachable: dead branches, defensive guards for states the parser cannot produce, or code behind a feature flag that no longer ships. Delete those in their own commit and record the reason in the commit message. A witness map that stays red because of dead code stops being trusted within a week.

Step 5: Detect witness drift after the change

The gate does not end at the diff. After applying the refactor, re-run coverage and compare executed lines against the baseline for the touched files. A line that used to run and now does not is witness drift, and drift usually means a condition or a call path silently disappeared.

# tools/witness_drift.py  ->  python tools/witness_drift.py before.json after.json app/billing/invoice.py
import json, sys
from pathlib import Path

def executed(path):
    data = json.loads(Path(path).read_text())
    return {f: set(v["executed_lines"]) for f, v in data["files"].items()}

before, after = executed(sys.argv[1]), executed(sys.argv[2])
failed = False
for p in sys.argv[3:]:
    lost = sorted(before.get(p, set()) - after.get(p, set()))
    if lost:
        failed = True
        print(f"witness drift {p}: {lost[:12]}{' ...' if len(lost) > 12 else ''}")
sys.exit(1 if failed else 0)

Renames and extractions shift line numbers, so expect a small amount of noise on the first run. Filter it by comparing the set of executed line contents rather than line numbers if your diff is mostly moves. The signal you care about is a behavior that stopped being reached, not a line that slid upward by four positions.

Decision table for a span under review

Span status	What it means	Action before editing
100% witnessed, assertions exist	Evidence matches the plan	Proceed with the smallest change in that span only
100% witnessed, assertions weak	Tests execute but never check the result	Strengthen one assertion or add a golden output fixture
Partial witness, live code	Real paths are untested	Write characterization tests until the span closes
Partial witness, dead code	Nothing can reach it	Delete in a separate commit with a recorded reason
Drift after the change	A path stopped running	Revert the rung and re-measure

Where a free model and a free server fit

The witness map is deliberately boring local tooling, but two costs around it are real: producing candidate diffs, and running full-suite coverage captures. MonkeyCode, an open-source coding assistant, offers free model access and a free server option that push both costs down — I generate the first batch of candidate diffs with it, then run the coverage capture on the server option when a full suite takes longer than a coffee break. Terms, quotas, and limits change, so confirm them in the project's own documentation before you plan around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important boundary is that the gate stays local and offline. A model can propose the diff, but only your repository's coverage data can witness it, and no external service gets to overrule a red span.

Limitations and who should skip this

Line coverage is not behavior coverage. Async continuations, threads, subprocess calls, and template rendering can execute without attributing lines to your source files, producing false orphans.
Moves break the mapping. A wholesale file move or a large extract-method produces spans too wide to reason about; split the change into smaller rungs first.
No harness means no gate. If the project has no runnable test entry point, build a smoke test before anything else; the witness map cannot substitute for the first test.
Execution is not correctness. Pair the map with an occasional deliberate fault injection so you know your witnesses can actually fail.
Skip this approach if your refactor is really a rewrite, if type checking already covers the surface you are changing, or if the team will not maintain the harness when it turns red.

A short checklist to run before the next cleanup diff

Regenerate the candidate diff against the exact commit you will modify.
Export executed lines from the unmodified suite.
Refuse any span with an orphan unless the line is proven dead.
Change only witnessed lines, in one rung per commit.
Re-measure after the change and revert on drift.

The suite going green is the start of the conversation, not the end of it. A witness map turns that vague green into per-line evidence, which is the only thing that makes the smallest safe change actually small and actually safe. If you try it on a messy module, generating a few candidate diffs through the free model access is a cheap way to see how fast orphan lines show up.

One Rung per Commit: A Refactor Ladder for Reviewing AI Cleanup Diffs

Dakota Wu — Mon, 19 Oct 2026 08:18:59 +0000

A 900-line diff lands in review with a one-line description: "pure cleanup, no behavior change." Nothing in the diff says which of the six things it changed altered the invoice total. By line 400 the reviewer is pattern-matching instead of reasoning, and the honest answer is that nobody knows. This post is about a small gate that keeps that diff from existing in the first place.

The context matters here. Reviewers now receive more machine-proposed cleanup patches than they can read carefully, and the cost has shifted from writing changes to proving them. So the useful question is no longer "is this refactor good?" but "which single step in it could be wrong, and can I test that step alone?"

Why a ranked ladder beats one big cleanup commit

A cleanup diff usually mixes changes with very different risk profiles: renames, extractions, module moves, and one quiet inlining that changes evaluation order. Each one needs a different kind of evidence, yet the diff presents them as one undifferentiated blob. When something breaks, git bisect cannot help you, because the whole blob is a single commit.

The ladder approach fixes that by making the rung the unit of review, not the file. You declare what kind of change you are making, the gate checks that the diff stays inside that budget, and the next rung starts from a green snapshot.

Step 1: freeze characterization snapshots before touching anything

The first rung is always the same: record what the module does today, keyed by call site rather than by test name. Call-site keys survive renames, which test names do not, so a rung-1 rename refactor produces a zero-line snapshot diff by construction.

# snapshot.py - record current behavior before any refactor
import hashlib
import importlib
import json
import pathlib
import sys

CASES = json.loads(pathlib.Path("fixtures/cases.json").read_text())


def stable(value: object) -> str:
    return json.dumps(value, sort_keys=True, default=repr, ensure_ascii=False)


def main(module_name: str, out_path: str) -> None:
    mod = importlib.import_module(module_name)
    snapshot: dict[str, dict[str, object]] = {}
    for case in CASES:
        key = f"{case['id']}@{case['entry']}"
        fn = getattr(mod, case["entry"])
        try:
            payload = {"ok": True, "value": stable(fn(*case["args"]))}
        except Exception as exc:  # behavior includes failures
            payload = {"ok": False, "error": type(exc).__name__}
        payload["digest"] = hashlib.sha256(stable(payload).encode()).hexdigest()[:12]
        snapshot[key] = payload
    pathlib.Path(out_path).write_text(
        json.dumps(snapshot, indent=2, sort_keys=True) + "\n"
    )
    print(f"{len(snapshot)} cases recorded -> {out_path}")


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

python snapshot.py billing.adjuster snapshots/before.json
git add snapshots/before.json && git commit -m "rung 0: freeze adjuster behavior"

A few details make this harness durable across rungs. Record exceptions as first-class results, because error paths are behavior too. Normalize anything nondeterministic (timestamps, UUIDs, floats) inside stable rather than in each case. And keep fixtures/cases.json in the repo so the evidence is reproducible on a clean checkout.

Step 2: rank candidate changes into rungs

Before the diff exists, write down the ladder. Rung numbers are a contract with the reviewer, not a maturity badge.

Rung	Allowed change	Required evidence	Rollback
0	Snapshots and fixtures only	Snapshot file committed alone	`git revert`
1	Rename, reorder, comments	Snapshot diff must be empty	revert
2	Extract a pure function, inline a local	Empty snapshot diff plus one new case per extracted branch	revert
3	Move code between modules	Empty snapshot diff plus import-graph check	revert
4	Change a data shape or default	Explicitly approved snapshot diff	revert + fixture review
5	Change behavior	New expectations, reviewed as a feature	feature flag

Two rules carry most of the value. First, one rung per commit, so a failing rung tells you exactly what to inspect. Second, a snapshot diff at rung 1, 2, or 3 is a failure, not a review item; if the bytes changed, the change was not mechanical.

Step 3: enforce the rung with a small gate

The gate does not judge code quality. It only checks that the diff matches the rung you declared, which is cheap and hard to argue with.

#!/usr/bin/env python3
# ladder_gate.py - fail if the diff exceeds the declared rung budget
# Requires Python 3.11+ (tomllib). Treat as a reference implementation.
import pathlib
import subprocess
import sys
import tomllib

BUDGET = {0: 1, 1: 6, 2: 4, 3: 8, 4: 3, 5: 2}  # max non-snapshot files per rung


def numstat(base: str) -> list[tuple[int, int, str]]:
    out = subprocess.run(
        ["git", "diff", "--numstat", base],
        capture_output=True, text=True, check=True,
    ).stdout
    rows = []
    for line in out.splitlines():
        added, removed, path = line.split("\t")
        rows.append((int(added or 0), int(removed or 0), path))
    return rows


def main() -> int:
    cfg = tomllib.loads(pathlib.Path(".ladder.toml").read_text())
    rung = int(cfg["rung"])
    rows = numstat(cfg.get("base", "HEAD~1"))
    snapshots = [r for r in rows if r[2].startswith("snapshots/")]
    code = [r for r in rows if not r[2].startswith("snapshots/")]
    problems = []
    if len(code) > BUDGET[rung]:
        problems.append(f"rung {rung} allows {BUDGET[rung]} files, diff touches {len(code)}")
    if snapshots and not cfg.get("approve_snapshot", False):
        problems.append("snapshot changed without approve_snapshot = true")
    if rung <= 3 and snapshots:
        problems.append(f"rung {rung} must keep the snapshot byte-identical")
    for problem in problems:
        print(f"FAIL: {problem}")
    if not problems:
        print(f"OK: rung {rung}, {len(code)} code files, {len(snapshots)} snapshot files")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

# .ladder.toml
rung = 2
base = "HEAD~1"
approve_snapshot = false

python ladder_gate.py && git commit -m "rung 2: extract prorate()"

Run it in a pre-commit hook and in CI on the pull request base, so a local bypass still gets caught. If your repo uses squash merges, point base at the merge-base with the target branch rather than HEAD~1.

Step 4: use model assistance inside the rung, not around it

This is where a free tier is actually relevant. Rung-1 and rung-2 edits are mechanical, well-bounded, and easy to verify with the snapshot, which makes them a good fit for a coding assistant that proposes candidate diffs. The gate stays in charge: a proposal that spills into rung 4 gets rejected by file count or by a non-empty snapshot diff, whichever trips first.

MonkeyCode is an operator-described open-source coding agent project whose free tier is stated to include 10 million tokens of model access and a free server option; verify current limits, licence, and terms in the project itself before planning around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A workflow that keeps the tool secondary:

Write the ladder entry and rung number first, then ask for a single mechanical step.
Run snapshot.py after the proposal, before reading the diff.
If the snapshot bytes moved at rung 1-3, discard the proposal instead of debugging it.
Run ladder_gate.py, then read only the files it lists.
Commit, and start the next rung from a green baseline.

The free server option is most useful for the snapshot job, not for the editing. Moving that job off a laptop means the same command runs on a clean machine, which is the whole point of reproducible evidence. Check the project's data-handling terms before pointing any private repository or credentials at a hosted tier.

Limits and who should skip this

This ladder is for behavior-preserving work only. It gives you nothing on a genuine feature change, where rung 5 expectations are the actual design conversation. It also assumes your snapshots are deterministic; if a module returns wall-clock times, random identifiers, or floats with unstable formatting, normalize them first or the empty-diff rule becomes noise.

Small, well-typed codebases with strong compiler guarantees may not need the ceremony, and a single-module script certainly does not. Teams working under strict data-handling rules should treat any hosted free tier as unavailable until their own review says otherwise. The scripts above are reference implementations: adapt them, and test the gate itself against a deliberately oversized diff before you trust it.

A short checklist

Rung 0 committed alone: fixtures plus snapshot, no code changes.
Rung written down before the diff exists.
Exactly one rung per commit, with a rollback that is a plain revert.
Empty snapshot diff enforced at rungs 1-3, approved at rung 4.
Gate running in pre-commit and in CI against the merge-base.

If you adopt one thing from this post, make it the empty snapshot diff at rung 1. It costs one script and one commit, and it turns "pure cleanup, no behavior change" from a claim into a check.

MonkeyCode provides free models that can run this workflow.

Prove Your Refactor Tests Can Fail: A Mutation-Calibrated Gate for Messy Repos

Dakota Wu — Sun, 18 Oct 2026 07:02:56 +0000

The refactor that passed every test and still broke billing

The scenario below is a composite I assembled from common review patterns, not a customer story. A payments-adjacent repo carried a roughly 900-line adjuster.py: one public adjust(), three private helpers, and a currency table pasted into two of them. A cleanup extracted one helper into its own module, the test suite went green, and the diff merged the same afternoon.

Six weeks later, finance flagged that a slice of credit adjustments were one cent low on EUR invoices. The suite had never once disagreed with the code: it asserted that adjust() returned a dict with the expected keys, not what the values were. Green tests had proved only that the suite could not see the difference it was supposed to guard.

That gap is the subject of this article. A characterization snapshot proves your refactor preserved the behavior you recorded; a calibrated harness proves the recording can detect a behavioral change at all. The first is cheap and common. The second is what most cleanups skip, and it is the one that catches the extraction which silently widens a boundary condition.

Why a passing snapshot is weak evidence on its own

Snapshots compare output against output produced by the same code, so any blind spot in the input corpus stays invisible.
Line coverage measures execution, not detection: a line can run ten thousand times without a single assertion caring about its result.
Agent-generated refactor diffs now land faster than a reviewer can reason about them line by line, which turns "read it carefully" into a hope.
The failure mode after a mechanical edit is usually a quiet branch change, not an exception, so nothing pages you.

The loop below adds one missing step and one guardrail. You freeze inputs, snapshot outputs, then deliberately break the code and require the suite to notice. Only after the alarm is calibrated do you accept the smallest change that keeps it ringing.

Step 1: Freeze the input corpus behind a seed

Random input generation is fine as long as it is reproducible. Pin the seed, commit the generator, and add boundary rows by hand so the corpus does not drift between runs.

# tests/corpus.py
import random

SEED = 20261018


def cases(n: int = 400) -> list[dict]:
    rng = random.Random(SEED)
    rows = []
    for _ in range(n):
        rows.append(
            {
                "amount": round(rng.uniform(-500, 500), 4),
                "kind": rng.choice(["credit", "debit", "fee"]),
                "currency": rng.choice(["USD", "EUR"]),
            }
        )
    # Boundary rows matter more than the random bulk.
    rows.append({"amount": -0.005, "kind": "credit", "currency": "USD"})
    rows.append({"amount": 0.0, "kind": "fee", "currency": "USD"})
    return rows

Run the suite with plugins that shuffle order or reseed data disabled, because a nondeterministic corpus makes every later comparison meaningless. In pytest that means -p no:randomly, and it means no datetime.now() inside the code path under test.

Step 2: Snapshot the messy behavior as it is today

This part is a prerequisite rather than the interesting half, so keep it boring: record the current outputs to a checked-in file and diff against it.

# tests/test_adjuster.py
import json
import pathlib

from billing.adjuster import adjust
from tests.corpus import cases

GOLDEN = pathlib.Path("tests/golden/adjuster.json")


def test_snapshot_matches():
    actual = [adjust(**c) for c in cases()]
    assert actual == json.loads(GOLDEN.read_text())

A snapshot written against buggy code is still useful, because you are not approving the behavior, you are pinning it. The next step tells you whether the pin has teeth.

Step 3: Calibrate by breaking the code on purpose

Write a handful of mutants by hand: small, plausible bugs a careless edit could introduce. Then run the suite against each one and require a red result. A mutant the suite does not catch is called a survivor, and every survivor is a hole in your safety net rather than a flaw in the mutant.

[
  {"id": "rounding-mode", "old": "return round(amount, 2)", "new": "return math.floor(amount * 100) / 100"},
  {"id": "zero-boundary", "old": "if amount < 0:", "new": "if amount <= 0:"}
]

# tools/mutation_gate.py
"""Inject known bugs; the suite must fail on each. Non-zero exit means survivors exist."""
import json
import pathlib
import subprocess
import sys

TARGET = pathlib.Path("billing/adjuster.py")
MUTANTS = json.loads(pathlib.Path("tools/mutants.json").read_text())
TEST_CMD = [sys.executable, "-m", "pytest", "-q", "-p", "no:randomly", "tests/test_adjuster.py"]


def suite_passes() -> bool:
    return subprocess.run(TEST_CMD, capture_output=True).returncode == 0


def main() -> int:
    original = TARGET.read_text()
    if not suite_passes():
        print("baseline is already red; fix that before calibrating")
        return 2
    survivors = []
    try:
        for m in MUTANTS:
            if original.count(m["old"]) != 1:
                print(f'{m["id"]}: anchor is not unique, rewrite the mutant')
                return 2
            TARGET.write_text(original.replace(m["old"], m["new"]))
            if suite_passes():
                survivors.append(m["id"])
    finally:
        TARGET.write_text(original)
    total = len(MUTANTS)
    print(f"killed {total - len(survivors)}/{total} mutants")
    for name in survivors:
        print(f"  survivor: {name}")
    return 0 if not survivors else 1

The baseline check is not decoration. Without it, a mutant that breaks an import produces a red suite and gets scored as killed, which inflates your score while proving nothing.

Where the mutants come from, and why a model helps here

Hand-written mutants encode the bugs you already fear, which is a biased sample. A useful complement is asking a model to propose domain mutations: swap round for floor, invert a comparison, drop a default currency, reorder a deduction. I use MonkeyCode for that candidate step, because proposing twenty plausible mutants from one function signature is exactly the kind of cheap generation work where a free model allowance is enough; the operator describes free model access and a free server option, and the outreach material currently cites an allowance on the order of ten million tokens. Verify the current terms on the project page before you plan around them, since free-tier details change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two rules keep the model in its lane. First, every candidate is filtered by a human who can answer "would this change a value a customer sees?" Second, a proposed mutant that survives gets inspected before it gets trusted: some are equivalence mutants, meaning the change cannot alter behavior for any input in the corpus. Swap payload.get("currency", "USD") for payload.get("currency") or "USD" and you will usually get a survivor that reflects an equivalent rewrite rather than weak tests. Delete those, and record why in a comment next to the mutant file.

Practical sizing: five to ten mutants take under a minute on a small module, and each mutant forces one full suite run. That is where the free server option fits honestly, because a matrix of twenty mutants across a slow suite is the kind of job you start on a machine that is not your laptop.

Step 4: Budget the diff so the change stays small

With the alarm calibrated, the refactor itself should be embarrassingly small: one symbol moved, one branch tidied, no new abstractions. Enforce that with a script rather than a code review comment, and exclude test files from the count so the harness does not trip its own gate.

# tools/diff_budget.py
import subprocess
import sys

MAX_FILES = 1
MAX_LINES = 40

out = subprocess.run(
    ["git", "diff", "--numstat", "origin/main...HEAD"],
    capture_output=True, text=True, check=True,
).stdout

files, changed = [], 0
for line in out.splitlines():
    added, removed, path = line.split("\t")
    if path.startswith("tests/") or added == "-":
        continue
    files.append(path)
    changed += int(added) + int(removed)

if len(files) > MAX_FILES or changed > MAX_LINES:
    sys.exit(f"diff budget exceeded: {len(files)} source file(s), {changed} changed lines")
print(f"diff budget ok: {len(files)} file(s), {changed} lines")

Wire it after the test job, and let it fail the build. When the budget trips, the answer is not a bigger budget: it is a second, separately reviewed change with its own snapshot and its own calibration run.

The re-calibration trap most teams miss

After the change merges, run the mutation gate again against the new code. This is the step that catches a harness losing coverage as the implementation moves, because a mutant the old suite killed will sometimes survive the new one.

Old code killed zero-boundary; new code survives it, which means the extraction moved the comparison behind a guard the corpus never exercises.
Old code killed rounding-mode; new code still kills it, which is the signal you actually want.
A mutant that starts failing for the wrong reason, such as an import error, is a broken mutant rather than a caught bug.

Track the survivor count in the pull request description the same way you would track a coverage delta. Two numbers, killed and total, tell a reviewer more about confidence than a paragraph of prose.

Decision table: when this loop is worth the hour

Situation	Run the loop	Skip it
Deterministic function handling money, dates, or permissions	Yes, every change
Module under ~100 lines with two call sites	Yes, it is minutes
Code scheduled for deletion this quarter		Yes, do not build a harness around a corpse
Behavior depends on wall clock, network, or device sensors	Only after pinning the clock and seed	Yes, until pinned
Suite already takes more than ten minutes	Yes, with a five-mutant sample	Full matrix on every commit
You are rewriting rather than refactoring		Yes, different problem entirely

Limitations and who should not use this

The approach assumes a deterministic code path, a suite that runs in isolation, and a git history clean enough to diff against a base branch. Teams whose "tests" touch a shared staging database will spend more time stabilizing the harness than refactoring, and they should fix that first. If your repo has no CI at all, the mutation gate becomes a script nobody runs, which is worse than no script because it looks like coverage.

Equivalence mutants are the other honest caveat: the gate measures detection by the corpus you wrote, so a survivor is a question, not a verdict. Budget reviewer time for judging each survivor rather than deleting the annoying ones. And on a large legacy module, ten mutants will not tell you the refactor is safe; they will tell you where your suite is blind, which is the more actionable result.

A small first move

Pick one function with a numeric boundary, write five mutants that you genuinely fear, and run the gate before touching the implementation. Report killed over total in the pull request, then make the smallest edit that keeps that ratio intact and rerun it afterward. If you would rather not tie up your laptop for the matrix, the free server option is the cheapest way to test-drive the loop on a throwaway branch, and the project page is the place to confirm what the free tier currently includes.

The Shape Census: Triage a Messy Module Before You Refactor It

Dakota Wu — Fri, 16 Oct 2026 05:03:28 +0000

Picture a billing module with a 900-line invoice function, four boolean flags threaded through it, and a test suite that has been skipped since a migration nobody remembers. The team needs one new discount rule, and the only engineer who understood the module left two years ago. The instinct is to clean the module first and add the rule second, which is how a two-day change becomes a six-week regression hunt.

There is a cheaper order of operations. Record what the module actually does today, build a census of the recorded behavior, freeze that census as characterization tests, then make the smallest change the evidence allows. Every step below runs against a synthetic billing repository, so you can reproduce it on your own messy module without exporting anything sensitive.

Why "read it and clean it up" fails on untested modules

Reading code tells you what a branch could do, not what it does on the inputs your users actually send. On an untested module those two things diverge fast, and the divergence is exactly where refactors break.

You cannot tell which branches execute, so deleting "dead" code removes behavior you never observed.
You cannot tell which output fields are load-bearing, because downstream consumers read dictionaries by key without any written contract.
A rename is a one-line diff for a reviewer and a breaking change for a regex-based consumer in another repository.
Refactor pull requests that mix structural moves with behavior tweaks force reviewers to approve both at once, which nobody can honestly do.

A call ledger replaces those guesses with counts, and counts survive code review arguments.

Step 1: Record a call ledger at the module boundary

Pick the seams you intend to touch, meaning the functions whose callers you control today. Wrap each one in a decorator that logs arguments, result, status, and latency to JSON Lines, and keep the recording scoped to a short window.

# ledger.py
import functools, json, pathlib, time, uuid

LEDGER = pathlib.Path("ledger/calls.jsonl")

def record(seam: str):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            started = time.perf_counter()
            status, result = "ok", None
            try:
                result = fn(*args, **kwargs)
            except Exception as exc:
                status = "error"
                result = {"__error__": type(exc).__name__, "message": str(exc)}
            payload = {
                "call_id": uuid.uuid4().hex[:12],
                "seam": seam,
                "status": status,
                "args": args,
                "kwargs": kwargs,
                "result": result,
                "elapsed_ms": round((time.perf_counter() - started) * 1000, 3),
            }
            LEDGER.parent.mkdir(parents=True, exist_ok=True)
            with LEDGER.open("a", encoding="utf-8") as fh:
                print(json.dumps(payload, default=str, sort_keys=True), file=fh)
            if status == "error":
                raise
            return result
        return wrapper
    return deco

Run the test suite and a staging job, then check what you captured. Redact or hash customer identifiers before the file leaves the machine, and never ship a raw ledger to a repository.

python -m pytest tests/ -q
wc -l ledger/calls.jsonl
python tools/triage.py ledger/calls.jsonl > ledger/triage.md

Step 2: Build a shape census instead of reading every output

Thousands of recorded results are unreadable, but their structure is small. Fingerprint each result by types and dictionary keys, ignoring values entirely, then count fingerprints per seam. The figures below come from a synthetic recording of roughly 4,000 calls across three seams, and you should read them as a format illustration rather than a benchmark.

# tools/triage.py
import collections, json, sys

def shape(value, depth=0):
    if depth > 4:
        return "..."
    if isinstance(value, dict):
        inner = ",".join(f"{k}:{shape(v, depth + 1)}" for k, v in sorted(value.items()))
        return "{" + inner + "}"
    if isinstance(value, (list, tuple)):
        return "[" + shape(value[0], depth + 1) + "]" if value else "[]"
    return type(value).__name__

rows = collections.defaultdict(lambda: collections.Counter())
statuses = collections.defaultdict(collections.Counter)
for line in open(sys.argv[1], encoding="utf-8"):
    call = json.loads(line)
    statuses[call["seam"]][call["status"]] += 1
    rows[call["seam"]][shape(call["result"])] += 1

print("| seam | calls | errors | distinct shapes | top shape share |")
print("| --- | --- | --- | --- | --- |")
for seam, counter in sorted(rows.items()):
    total = sum(counter.values())
    top = counter.most_common(1)[0][1] / total
    print(f"| {seam} | {total} | {statuses[seam]['error']} | {len(counter)} | {top:.0%} |")

seam	calls	errors	distinct shapes	top shape share
`compute_invoice`	2,140	31	3	96%
`apply_adjustments`	1,502	118	11	61%
`format_receipt`	388	0	7	43%

The third column is the one that changes your plan. A seam with three shapes is a small, bounded surface you can freeze quickly; a seam with eleven shapes is where the flags hide and where a rename will hurt. Use the census to choose the order of work rather than your intuition.

Calls	Distinct shapes	Top shape share	Verdict
High	1-3	Above 90%	Freeze and extract in the same day
High	4-8	60-90%	Freeze first, extract only one branch per PR
High	9+	Below 60%	Document behavior, do not restructure yet
Low	Any	Any	Collect more traffic before deciding

Step 3: Turn the recording into characterization tests

Characterization tests lock in current behavior, bugs included, so normalization has to remove the fields you genuinely cannot predict. Timestamps, generated identifiers, and trace headers get dropped; numeric money fields get rounded; everything else must match exactly.

# tests/test_characterization.py
import json, pathlib, pytest
from billing.legacy import compute_invoice

CASES = [json.loads(line) for line in pathlib.Path("ledger/calls.jsonl").read_text().splitlines()]
OK = [c for c in CASES if c["seam"] == "compute_invoice" and c["status"] == "ok"]

def normalize(result):
    result = dict(result)
    for volatile in ("invoice_id", "created_at", "trace_id"):
        result.pop(volatile, None)
    if "amounts" in result:
        result["amounts"] = {k: round(v, 2) for k, v in result["amounts"].items()}
    return result

@pytest.mark.parametrize("case", OK, ids=lambda c: c["call_id"])
def test_matches_recorded_behavior(case):
    actual = normalize(compute_invoice(*case["args"], **case["kwargs"]))
    assert actual == normalize(case["result"])

For the noisy seams, assert the shape rather than the values, and keep the census report under version control so drift is visible. A CI step that regenerates ledger/triage.md and fails on a byte difference is usually enough.

python tools/triage.py ledger/calls.jsonl > /tmp/triage.md
diff -u ledger/triage.md /tmp/triage.md

Step 4: Make the smallest change the census allows

Three rules keep the diff reviewable and the risk bounded. Apply them literally, because every shortcut here has a cost you will pay during an incident.

One structural move per commit; extraction and renaming never share a pull request.
Extract only behind a name that already appears in the census as a distinct shape.
If the regenerated census changes shape counts, revert instead of adjusting the tests to pass.

The actual change is usually smaller than the preparation suggests, which is the point.

-        if customer.get("plan") == "legacy" and order["total"] > 500:
-            discount = round(order["total"] * 0.07, 2)
-            apply_discount(order, discount)
+        if qualifies_for_legacy_discount(customer, order):
+            apply_discount(order, legacy_discount_amount(order))

The acceptance conditions are mechanical: the characterization suite is green, the census diff is empty, and the new rule has at least one recorded case of its own before you merge.

Where free model access and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The parts of this workflow that are genuinely tedious are also mechanical: clustering the shape fingerprints into a short list, drafting the parametrize scaffolding from a ledger sample, and writing the triage table you reviewed above. That is the kind of work where a model with tool access saves real time, and the operator describes both free model access and a free server option, with an allowance on the order of ten million tokens, so you can try the loop before committing a budget to it. Verify current terms on the project page, since availability and limits change and this article was not written to track them.

The free server option matters for one specific reason. A replay harness that only runs on a laptop records whenever someone remembers to run it, whereas moving the same container to a hosted instance gives you a stable endpoint and a repeatable recording window. Keep secrets out of the container, keep the ledger window short, and treat the hosted run as a staging job rather than a production dependency.

Limits, and who should skip this approach

Nondeterministic seams without a clean normalization function produce flaky characterization tests, and flaky tests get skipped.
Modules whose real behavior lives in database triggers or queues cannot be frozen at a Python function boundary alone.
Recording inputs may capture personal data, so legal review belongs before the first ledger file, not after.
Low-traffic seams need weeks to reach useful coverage, which may exceed your deadline.
Teams that cannot deploy a recording wrapper to staging have no cheap path here, and should write the missing tests by hand instead.

Checklist before you push

The ledger window is short, redacted, and excluded from the repository.
Every seam you plan to touch has at least one recorded error case.
Normalization removes only fields you can defend in review.
The census report is committed and the CI diff check passes.
The pull request contains one structural move and nothing else.
A reviewer who has never seen the module can explain the change after reading the filter to the third commit? they should not need that filter at all.

Start with the seam that has the fewest shapes, not the one that annoys you most, and let the census pick the order of work for you. If you run the harness on your own repo this week, the triage table will tell you more about your legacy module than a week of reading it will.

Pin the Fault Matrix Before You Refactor a Messy Module

Dakota Wu — Thu, 15 Oct 2026 11:16:22 +0000

Pin the Fault Matrix Before You Refactor a Messy Module

Consider an illustrative composite scenario rather than one specific incident, because the shape repeats across review threads. A service module grows past nine hundred lines, a coding agent returns a four hundred line refactor diff, and the existing suite turns green on the first run. The diff merges, and days later a partner integration fails, because one validation path now raises a bare ValueError where callers expected a typed error with a stable code.

Recent community threads argue loudly about how fast agent-authored diffs arrive, and much less about what those diffs quietly change. The piece that changes first is usually not the happy path but the error surface: exception types, error codes, and the order in which validation rejects a payload. This post pins that surface as data before a messy refactor starts, then constrains the change to the smallest seam available.

Why the Error Surface Decays First

Happy paths get covered because they are easy to observe, cheap to assert, and pleasant to demo in a pull request. Error paths stay thin because each one needs a deliberate fault injection, and their assertions tend to encode prose instead of structure. When a refactor moves validation across layers, the happy path usually survives the move and the exception taxonomy does not.

Three patterns account for most of the drift reviewers see:

Type flattening — specific errors collapse into ValueError or Exception once a helper is extracted.
Code loss — the class survives but stops carrying the code attribute that callers branch on.
Ordering drift — validation runs in a new sequence, so a payload that failed on currency now fails on weight.

All three pass a suite whose only error assertion is "something raised".

Step 1: Enumerate Every Raise Site as Data

Turn the current error surface into a list you can diff, using nothing beyond the standard library. ast.unparse requires Python 3.9 or newer.

# tools/raise_sites.py
"""Print every raise site in a package as file:line:expression."""
import ast
import sys
from pathlib import Path


def raise_sites(path: Path):
    tree = ast.parse(path.read_text(encoding='utf-8'), filename=str(path))
    for node in ast.walk(tree):
        if isinstance(node, ast.Raise):
            if node.exc is None:
                name = 'bare-raise'
            elif isinstance(node.exc, ast.Call):
                name = ast.unparse(node.exc.func)
            else:
                name = ast.unparse(node.exc)
            yield path, node.lineno, name


def main(root: str) -> int:
    rows = []
    for path in sorted(Path(root).rglob('*.py')):
        if '.venv' in path.parts:
            continue
        rows.extend(raise_sites(path))
    for path, line, name in rows:
        print(f'{path}:{line}:{name}')
    print(f'\n{len(rows)} raise sites')
    return 0


if __name__ == '__main__':
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else 'src'))

Run it once and commit the output as a baseline artifact, then run it again after the refactor.

python tools/raise_sites.py src > .refactor/raise_before.txt
grep -c . .refactor/raise_before.txt   # example baseline: 47

The raw count is not a quality score, and it should never be optimized for its own sake. What matters is that the diff tells you exactly which raises moved, appeared, or disappeared during the change.

Step 2: Reduce the Raise Sites to a Fault Matrix

Forty-seven raise sites normally collapse into a handful of distinct faults, and the faults are what tests should pin. A shipping rate card gives a compact example.

ID	Injected fault	Expected exception	Stable code	Prose fragment
Z1	zone missing from the rate card	`UnknownZoneError`	`unknown_zone`	"unknown zone"
W1	weight absent or `None`	`MissingWeightError`	`missing_weight`	"weight"
C1	parcel currency differs from card	`CurrencyMismatchError`	`currency_mismatch`	"currency"
O1	weight above the carrier maximum	`OverweightError`	`overweight`	"maximum"

Four rules keep the matrix worth maintaining over several refactors:

One row per fault class, not per raise site, since two sites can guard the same contract.
Assert on the code column and treat prose as a secondary, replaceable signal.
Record which layer raises, because the whole point is detecting a layer move.
Keep the matrix in version control beside the tests, so a behavior change is a reviewable diff.

Step 3: Write Characterization Tests That Pin Codes, Not Prose

Keep the tests shaped like the table, so adding one fault stays a one-line change.

# tests/test_error_surface.py
import pytest
from pricing.rate_card import (
    CurrencyMismatchError,
    MissingWeightError,
    OverweightError,
    RateCard,
    UnknownZoneError,
)

# (id, payload, exception class, stable code)
FAULTS = [
    ('Z1', {'zone': 'ZZ', 'weight': 1.0, 'currency': 'USD'}, UnknownZoneError, 'unknown_zone'),
    ('W1', {'zone': 'US', 'weight': None, 'currency': 'USD'}, MissingWeightError, 'missing_weight'),
    ('C1', {'zone': 'US', 'weight': 1.0, 'currency': 'EUR'}, CurrencyMismatchError, 'currency_mismatch'),
    ('O1', {'zone': 'US', 'weight': 99.0, 'currency': 'USD'}, OverweightError, 'overweight'),
]


@pytest.mark.parametrize('case_id,payload,exc,code', FAULTS, ids=[f[0] for f in FAULTS])
def test_error_surface(case_id, payload, exc, code):
    with pytest.raises(exc) as caught:
        RateCard.load().quote(payload)
    assert caught.value.code == code

Parametrized ids keep the output readable, so pytest -q tests/test_error_surface.py reports Z1, W1, C1, and O1 as separate lines. Add one structural guard as well: assert that every exception class in the matrix exposes a non-empty code attribute, which stops a new raise site from shipping without a machine-readable identifier.

Step 4: Make the Smallest Change the Only Allowed Change

With the matrix frozen, give the refactor an explicit churn budget and pick exactly one seam inside it.

git diff --numstat origin/main...HEAD | awk '{a+=$1; d+=$2} END {print a+d " changed lines"}'

Candidate seam	Call sites	In/out types	Side effects	Error paths	Verdict
`quote()` body	3	dict to Money	none	4	extract one helper
currency formatting	11	str to str	none	1	defer
carrier client wrapper	6	network	I/O	6	do not touch
rate cache	2	dict	TTL clock	2	defer

A seam worth taking has few call sites, no I/O, and error paths already represented in the matrix. Everything else waits for a second pass with its own baseline.

The Verification Gate

Wrap the workflow in one script that CI or a human can run against the branch.

#!/usr/bin/env bash
set -euo pipefail
BASE=${1:-origin/main}
BUDGET=${2:-40}

python tools/raise_sites.py src > /tmp/raise_after.txt
diff -u .refactor/raise_before.txt /tmp/raise_after.txt || true

pytest -q tests/test_error_surface.py tests/test_value_characterization.py

CHURN=$(git diff --numstat "$BASE"...HEAD | awk '{a+=$1; d+=$2} END {print a+d}')
echo churn=$CHURN budget=$BUDGET
test "$CHURN" -le "$BUDGET"

The raise-site diff stays informational, while the two hard gates are the error-surface tests and the churn limit. Treat any new raise site as a contract change: either add a matrix row in the same commit, or move the raise instead of introducing it.

Where an Agent Fits, and Where It Does Not

Drafting matrix rows across dozens of raise sites is mechanical work, which is the kind of task a coding agent can genuinely accelerate. This workflow uses MonkeyCode for that step, where the free model access covers the drafting and the free server option gives the gate somewhere to run without local provisioning. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Scope the prompt tightly, and ask the agent to read .refactor/raise_before.txt, cluster the raise sites by exception name and payload condition, propose one matrix row per cluster with a candidate code, and then stop. Verify every proposed code against one real runtime call before freezing it, because the agent cannot know your taxonomy intent.

Three things the agent should never be trusted with:

Inventing codes that the exception classes do not actually expose.
Merging two faults into one row because their payloads look superficially similar.
Editing the expectations after a failing run, which inverts the purpose of the matrix.

If you want to try the gate end to end without provisioning a machine, the free model access and free server option are enough to run the script, the tests, and the churn check.

Limitations and Who Should Skip This

Code assertions require errors that actually carry codes; prose matching is wording- and locale-dependent and breaks for the wrong reasons.
Coverage here means "raise sites referenced by at least one matrix row", which is a proxy for completeness rather than proof of safety.
Concurrency, timing, and performance behavior sit outside this workflow, and refactors that move work across threads need separate evidence.
The churn budget is a smell detector, not a correctness check, so a small diff can still break a contract and a large legitimate move needs a deliberate budget raise.
Skip this entirely if your errors are user-facing prose only, or if the module has no taxonomy worth preserving, since the ceremony would protect nothing.

Messy refactors rarely fail because the happy path changed, and they fail quietly because the error contract moved. Freeze the fault matrix first, take one seam with a churn budget, and let the gate make "the tests still pass" mean something specific about behavior.

Require an Empty Fixture Diff Before Extracting a Batch Loop

Dakota Wu — Sun, 11 Oct 2026 21:09:09 +0000

A late cleanup request arrived with a four-hundred-line diff that rewrote validation, deduplication, and persistence in one pass. The existing tests covered only the happy path, so a green CI run could not explain the legacy rows. The reviewer needed a smaller change that preserved every observed output before any structural boundary was allowed to move. This composite scenario shows how to freeze those outputs, then accept only the smallest safe edit.

The module under review is a batch upload normalizer that grew by successive tickets rather than by design. It reads a list of records, fills missing defaults, drops exact repeats, and writes survivors to storage. Those four jobs share one function, so a cleanup that touches all of them cannot be reviewed as one claim. The method below separates observation from edit, and it does not depend on any particular assistant.

Name the behavior you will freeze

Before changing code, write down the inputs that already matter and the outputs you are willing to treat as current truth. Include awkward rows from production samples, not only the tidy examples attached to the latest ticket. A characterization file is a pinned record of those pairs, not a statement that the behavior is desirable. If a row is known to be wrong, mark it as accepted debt instead of silently correcting it during the same change.

Three observations keep the freeze honest when the repository is already messy.

Record the returned list and the store payload, because a matching return can still hide a second write.
Preserve the original row order, because a sort that looks cleaner can change which duplicate survives.
Keep skipped rows visible in the fixture notes, because a deleted branch is still a behavior change.

That list is the review contract for this module, and it should stay visible in the pull request description. It is narrower than a repository-wide characterization suite, and it is not a license to rewrite neighboring helpers. This pass does not hash a repository-wide side-effect ledger, and it does not ask you to freeze every neighboring seam. The next section turns the contract into a local command you can rerun after every proposed diff.

Pin a fixture before the extraction

The following Python example is a proposed harness, not a log from a measured production run. It uses a memory store so the test process does not touch a real database or a shared queue. The cases cover a blank SKU, a negative quantity, an exact repeat, and a lowercase SKU that the current code uppercases. Run the capture against the untouched function, then commit the fixture before you accept any cleanup.

import json
from pathlib import Path

class MemoryStore:
    def __init__(self):
        self.batches = []

    def write(self, rows):
        self.batches.append([dict(row) for row in rows])

def normalize_upload_batch(records, store):
    seen = set()
    kept = []
    for record in records:
        sku = (record.get('sku') or '').strip().upper()
        qty = record.get('qty', 0) or 0
        if not sku or qty < 0:
            continue
        key = (sku, int(qty))
        if key in seen:
            continue
        seen.add(key)
        kept.append({'sku': sku, 'qty': int(qty)})
    store.write(kept)
    return kept

CASES = [
    {
        'name': 'blank-sku',
        'records': [{'sku': '  ', 'qty': 2}],
        'debt': False,
    },
    {
        'name': 'negative-qty',
        'records': [{'sku': 'ab1', 'qty': -1}],
        'debt': False,
    },
    {
        'name': 'case-and-repeat',
        'records': [
            {'sku': 'ab1', 'qty': 2},
            {'sku': ' AB1 ', 'qty': 2},
            {'sku': 'ab1', 'qty': 3},
        ],
        'debt': False,
    },
]

def capture(cases):
    report = []
    for case in cases:
        store = MemoryStore()
        returned = normalize_upload_batch(case['records'], store)
        report.append({
            'name': case['name'],
            'debt': case['debt'],
            'returned': returned,
            'written': store.batches,
        })
    return report

def main():
    path = Path('batch_characterization.json')
    payload = json.dumps(capture(CASES), indent=2) + '\n'
    path.write_text(payload, encoding='utf-8')
    print('wrote', path)

if __name__ == '__main__':
    main()

Compare the fixture, do not skim the diff

A second, ordinary command compares a fresh capture with the fixture you already committed to the branch. The comparison is a textual check of the JSON, not a visual skim of a large assistant diff. If you later correct a debt row, change the fixture in a separate commit and state the product reason in that commit message. Until you trust the harness, treat a non-empty fixture diff as a failed review rather than as a suggestion.

Save the harness as capture_batch.py beside the fixture so the commands below match the file you run.

python capture_batch.py
git diff -- batch_characterization.json

The sample function above is embedded so a reader can follow the article without cloning a repository. In a real repository, import the existing function instead of pasting a simplified copy into the test file. A pasted copy can drift from the production function and then bless the wrong behavior during review. Keep the fixture next to the module so a reviewer can open both without searching the whole tree.

Score the smallest safe change

After the fixture is committed, the only edit this pass allows is an extraction of the pure selection loop. The wrapper may call the new function and then perform exactly one write to the existing store. It may not add a sort, a new skip rule, a retry, or a second store call. The table below is the acceptance gate for the diff, and a single failed row is enough to reject the change.

Check	Accept this pass	Send it back
Files touched	The one module that owns the batch function	Any helper outside that file
Fixture JSON	Unchanged after a local rerun	Any added, dropped, or reordered field
Store writes	One write, matching the returned rows	An extra write, a cleared batch, or a write before selection
New branches	None	A fallback, a retry, or a temporary compatibility flag
Debt rows	Still marked debt in the fixture	Quietly corrected inside the cleanup

The extraction that the gate allows

The extracted shape that satisfies every row of the table looks like the unexecuted sketch below. It remains a proposal until you rerun the capture and the fixture diff is completely empty. Notice that the skip rules are copied, not improved, because an improvement is a behavior change and belongs in a later commit. The wrapper stays responsible for the write, so persistence bugs are not mixed into the pure move.

def select_upload_rows(records):
    seen = set()
    kept = []
    for record in records:
        sku = (record.get('sku') or '').strip().upper()
        qty = record.get('qty', 0) or 0
        if not sku or qty < 0:
            continue
        key = (sku, int(qty))
        if key in seen:
            continue
        seen.add(key)
        kept.append({'sku': sku, 'qty': int(qty)})
    return kept

def normalize_upload_batch(records, store):
    kept = select_upload_rows(records)
    store.write(kept)
    return kept

Follow the same four steps

Commit the characterization fixture that you generated from the untouched function before any extraction work begins.
Extract only the selection loop into a pure helper, and leave the storage write inside the original wrapper.
Rerun the local capture after that extraction and require an empty diff on the committed fixture file.
Schedule debt corrections, retries, and storage changes as later commits that carry their own expected results.

That order matters more than the size or brand of the assistant that happened to propose the cleanup. A small extraction can still be wrong if you regenerate the fixture after the edit and then treat the new file as the baseline. Commit the baseline first, or the acceptance gate has nothing stable left to protect later. If step three fails, restore the extraction and inspect the case, rather than widening the diff to force a match.

Let an assistant draft, then keep the gate local

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A coding assistant can draft fixture cases or the extraction sketch after you have already named the rows you care about. For this draft, the operator supplied two availability claims: free model access and a free server option. This article does not state model names, token quotas, hardware size, duration, or benchmark scores, because those details were not provided as primary sources. Check the project documentation on the day you start, because availability terms can change and should not be copied from an older post.

The useful split should stay small even when the assistant is allowed to see the whole repository. Ask it to propose cases that match the contract, then paste those cases into the harness and run the capture yourself. Ask for the extraction only after the fixture is committed, and reject any diff that fails a row in the table. A free server can be a convenient place to run that capture when your laptop cannot match the language version. The committed fixture still belongs in the repository you review, not only on the temporary machine.

If that gate matches how you already review cleanups, read the current access notes before you try it. Then run the same fixture workflow on one module you can execute on a machine you control. Keep the product optional: the harness, the table, and the empty fixture diff are the parts a teammate can reuse without an account. A reviewer should never treat a model-written explanation as proof that the fixture file remained stable.

Limitations and who should skip this

This approach freezes current behavior, including bugs that nobody has yet scheduled for a product fix. It is a poor fit when the ticket requires a visible behavior change inside the same pull request. It is also a poor fit when you cannot obtain representative rows, because an empty fixture will bless whatever the first draft happens to emit. A memory store does not prove transaction order, locking, or partial failure, so do not use it to clear a data-migration change.

Skip the gate as your only review when the diff touches authentication, billing totals, or deletion. Those areas need explicit expected outcomes, not a snapshot of whatever the old code already did. Do not plan a team rollout around an unverified free quota or an assumed server lifetime from an old post. If today's documentation does not confirm the access terms you need, treat the assistant as optional. Keep the harness on a machine you control rather than on a shared environment you have not reviewed.

The closing claim stays modest, and a reviewer can audit it from the fixture diff alone. A four-hundred-line cleanup becomes reviewable when one characterized seam moves and the fixture file stays still. Everything else can wait for a later commit that states the behavior it intends to change. That later commit should carry its own cases, rather than borrowing the freeze you used to keep this extraction honest.

Freeze a Golden Trace Matrix Before the One-Seam Cleanup

Dakota Wu — Sat, 10 Oct 2026 17:39:14 +0000

This opening is an illustrative composite, not a report of a named production incident or a measured outage. A billing helper in a five-year-old repository still writes an audit line while it computes a customer discount. Three optional flags arrived through separate pull requests, and none of those flags share one documented contract. An unbudgeted agent rewrite then dropped that audit line for empty customer codes, even though the returned discount stayed numeric.

Why the large cleanup failed

The cleaner diff was accepted because review focused on names and nesting instead of the branch production actually hit. That pattern is common in messy modules, where important behavior lives in side effects rather than in the returned number. A characterization pass has to record both, or the next cleanup will look tidy while a nightly job changes. Any later agent draft stays downstream of that recorded contract, and it does not choose the row.

A messy module usually breaks during a refactor at one narrow seam, not across every line of the file. The risky place is often an uncovered branch, a silent write, or a default that treats missing input as zero. A broad rewrite hides that drift among renames, so reviewers spend their time on style rather than behavior. The safer sequence is to pin current results first, then permit only the smallest edit those pins already cover.

That sequence costs more attention on the first afternoon, and it usually costs less time on the first regression. You record what the function does today, including results that a later ticket may classify as bugs. You then allow one named seam to move, and you reject any diff that changes a row outside that budget. An agent may draft the edit, but the budget should come from cases you already stored and reviewed.

What the seam budget counts

A seam, in this workflow, is one decision the function already makes for a recognizable class of inputs. Examples include a missing customer code, an empty customer code, a gold tier, or a write onto the audit log. The budget is a small table you can recompute from the same cases whenever a new diff appears. A seam with no pinned case is not eligible for an agent edit during the current pass.

The score stays crude on purpose, because a crude score is easier to audit than a narrative summary of the file. Count pinned cases, count distinct returned values, and count side-effect events that those cases actually recorded. Do not treat the score as profiler coverage, and do not treat it as permission to redesign the surrounding module. It only answers which single seam is eligible to change next, given evidence you already checked in.

Apply these eligibility rules before you open a remote drafting session:

Every eligible seam needs at least two pinned cases with meaningfully different inputs.
A side effect belongs to the contract when any pinned case records that event.
Seams with zero pinned cases stay frozen, even when the implementation looks obviously wrong.
The budget names one seam in the diff message, and every other seam remains out of scope.

A local harness you can adapt

The Python below is a local gate proposal, not a claim that this exact file ran in a production repository. It executes a function against a fixed matrix, stores outputs beside side effects, and rejects an unpinned target seam. You can replace the sample function with the one you own, then commit the matrix beside the existing tests. The sample treats missing and empty codes as different seams, because production defects often hide in that split.

from dataclasses import dataclass, field
from typing import Callable

@dataclass(frozen=True)
class Case:
    name: str
    customer_code: str | None
    tier: str | None

@dataclass(frozen=True)
class Trace:
    name: str
    amount: int
    events: tuple[str, ...] = field(default_factory=tuple)

def apply_discount(code, tier, audit):
    # Illustrative baseline, not a recommended pricing implementation.
    if code is None:
        return 0
    if code == "":
        audit.append(f"empty:{tier}")
        return 0
    if tier == "gold":
        audit.append(f"gold:{code}")
        return 80
    audit.append(f"default:{code}")
    return 95

def seam_for(case: Case) -> str:
    if case.customer_code is None:
        return "missing_code"
    if case.customer_code == "":
        return "empty_code"
    if case.tier == "gold":
        return "gold_tier"
    return "default_tier"

def run_matrix(fn: Callable, cases: list[Case]) -> list[Trace]:
    traces = []
    for case in cases:
        audit: list[str] = []
        amount = fn(case.customer_code, case.tier, audit)
        traces.append(Trace(case.name, amount, tuple(audit)))
    return traces

def budget(cases: list[Case], traces: list[Trace], target: str) -> dict:
    pinned = [case for case in cases if seam_for(case) == target]
    if len(pinned) < 2:
        return {"target": target, "eligible": False, "reason": "need two pins"}
    shapes = []
    for case, trace in zip(cases, traces):
        if seam_for(case) == target:
            shapes.append(str(trace.events))
    return {
        "target": target,
        "eligible": True,
        "pinned_cases": [case.name for case in pinned],
        "side_effect_shapes": sorted(set(shapes)),
        "allowed_change": "one seam only",
    }

def same_contract(golden: list[Trace], fresh: list[Trace]) -> bool:
    left = [(item.name, item.amount, item.events) for item in golden]
    right = [(item.name, item.amount, item.events) for item in fresh]
    return left == right

Commands that keep the gate local

Store the golden traces beside the module, and compare those traces again after every proposed edit lands. A short command wrapper around the two functions above is enough for the first version of this local gate. The shell lines below assume a tiny argument parser that calls the matrix runner and the comparison helper. They illustrate the intended workflow, and they are not evidence that a particular repository already ships this file.

python3 seam_budget.py --matrix cases.json --write-golden golden.json
python3 seam_budget.py --matrix cases.json --check golden.json
python3 seam_budget.py --matrix cases.json --target empty_code --budget

The first command records the current contract, including audit events, before anyone starts to rewrite the function. The second command should exit nonzero when a fresh run disagrees on amount or on the recorded events. The third command prints whether the named seam has two pins and may enter the one-seam budget. If that check fails, stop and read the drifted row before you discuss naming, formatting, or structure.

A starter matrix

The matrix below is enough to expose the missing-code, empty-code, gold, and default seams in the sample. Notice that empty input and null input are separate rows, because the sample writes an audit event for only one of them. Gold has two rows so that seam can become eligible, while a single gold row would fail the two-pin rule. Replace the values with fixtures from your module, and keep customer identifiers synthetic rather than copied from production.

[
  {"name": "missing", "customer_code": null, "tier": "gold"},
  {"name": "empty_gold", "customer_code": "", "tier": "gold"},
  {"name": "empty_std", "customer_code": "", "tier": "std"},
  {"name": "gold_ok", "customer_code": "c-1", "tier": "gold"},
  {"name": "gold_other", "customer_code": "c-2", "tier": "gold"},
  {"name": "default_ok", "customer_code": "c-3", "tier": "std"}
]

With that file, budget(..., target="empty_code") is eligible, and budget(..., target="missing_code") is not. The missing-code seam has only one row, so the rule keeps it frozen until you add a second pin. That split is the practical difference between a cleanup you can review and a rewrite that drifts an untested branch. Run the check locally before you ask any editor, human or remote, for a patch.

When a free drafting session is in scope

Only a green local check should precede a remote draft, and the draft should name the budgeted seam in the prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability claims for this draft: free model access, and a free server option. The operator also describes MonkeyCode as an open-source project that can host this kind of drafting session.

This article does not name models, token quotas, hardware sizes, or a duration, because those details were not verified from a primary source here. Paste the frozen matrix, the chosen seam, and a clear instruction to leave every other branch untouched. Ask for a patch that preserves every golden event, except where you already edited the golden file yourself. Re-run the local check on your own machine after the diff returns, and discard the draft if any unbudgeted row moves.

The remote session can make that bounded edit easier to try, but it does not become the acceptance gate. Keep the remote prompt limited to the function text, the matrix, and the seam name from the budget output. A free server session is a reasonable place to draft that single patch when the matrix already lives in the repository. The same local check still has to pass on your machine before anyone merges the returned diff.

Choosing the next hour of work

The table below is a workflow control for this cleanup, not a performance comparison and not a vendor benchmark. It separates work you must finish locally from work a remote draft may attempt after the gate is green. Read the row that matches the repository you have today, not the row you wish the repository had. If two rows apply at once, use the stricter row and keep the agent session closed until the gap is fixed.

Situation	Local action	Remote draft
Target seam has two or more pins	Freeze the golden file	Allowed for that seam only
Target seam has only one pin	Add a second case first	Not yet
Any pin records a side effect	Treat the event as contract	Do not delete it in this pass
Desired work spans three seams	Split the work into passes	Refuse a combined diff
Exercise needs a live payment API	Stub the call or stop	Do not point a session at production

A remote session remains optional even on the eligible row, because a local editor can make the same narrow change. The value of the session is convenience when you want a draft against the pinned matrix, not authority over the contract. Keep secrets, customer dumps, and production credentials out of the prompt regardless of which row you select. If the remote view cannot see the repository, copy only the function and the cases you already pinned.

Limitations you should state in the pull request

Characterization pins preserve current behavior, including defects you have not yet decided to fix in this pass. If the empty-code branch should stop writing an audit line, change the golden row in a separate commit first. Otherwise the gate will correctly reject the behavior change you actually intended to ship this week. That extra commit makes the intended drift reviewable without mixing it into a rename or a formatting sweep.

The scorer does not observe concurrency, time zones, or input combinations that the matrix never listed. Two pinned cases can still miss a third flag combination that production sends on a particular weekday. Add a case when a real input appears, instead of widening the agent prompt to cover unknown branches. A wider prompt is how a one-seam budget quietly becomes a whole-file rewrite with no new evidence.

Free model access and a free server option are availability claims for this draft, not a promise about next quarter. Do not copy a quota, a machine size, or a permanent allowance into the team runbook from these paragraphs. If either option is unavailable when you sit down, the local gate still stands without any remote session. The comparison and the budget do the real work, and they do not depend on a particular hosted editor.

Skip this approach when nobody on the team can review the resulting diff carefully, line by line. Skip it when the requested change is a redesign across several modules rather than one local seam. Skip it when the code cannot be exercised without live customer data or credentials you must not paste. An agent draft of an unpinned module repeats the failure already described in the composite opening above.

Checklist before you merge

Confirm the budget names one seam and lists at least two pinned cases.
Confirm the golden file changed only where you intended a behavior change.
Confirm side-effect events still match on every seam the budget did not name.
Confirm the remote draft was not accepted without a fresh local check.

That sequence lets a messy repository move without treating a shorter function as proof of behavioral safety alone. The original artifact is the seam budget together with the golden comparison, and any agent session stays downstream of both. Review the returned diff against the table, then merge only when every unbudgeted row is still identical.

Limit a Messy Adjuster Cleanup to Calls You Have Already Recorded

Dakota Wu — Fri, 09 Oct 2026 13:40:57 +0000

Consider a checkout service that accumulated one function applying coupons, tax rounding, and partial refunds in a single pass. That function returned a payable total, yet it also inserted an audit row and mutated the cart in place. An assistant then proposed a broad cleanup that split the file into six new modules and renamed several helpers. The review would stall if the team could not name which outputs were already guaranteed by an executable check.

Start from a call set, not a target layout

The useful starting point is a small call set whose outputs a current caller already depends on today. Each recorded call should carry the inputs that change money and the fields a downstream job already reads. Side effects belong in that record when a missing write would become a production incident for the on-call reader. If a path has no current caller and no incident history, leave that path outside the first change budget.

This worked example uses a fictional apply_adjustment module so the method can be copied without borrowing a private repository. The numbers below are illustrative fixtures, not production metrics collected from a named company or incident. Treat every expected value as something you must regenerate on the current code before you trust it.

What the messy function is allowed to promise

The function currently mixes three decisions that tend to fail in different ways during a structural cleanup. Coupon application can drop the subtotal to zero before any tax amount is computed on the remainder. Tax rounding can change a single cent when the order of arithmetic operations shifts inside the function. A partial refund can subtract money that the coupon had already reduced from the same payable total.

Those three paths are enough to fund a first characterization budget without inviting a wider rewrite. A fourth path, such as a loyalty multiplier, stays unrecorded until some caller proves that it matters. Expanding the budget before the first edit usually recreates the broad rewrite you were trying to avoid.

Three calls that pin the current contract

A standard cart applies a ten percent coupon, then rounds tax on the reduced subtotal, and writes one audit row.
A zero subtotal after the coupon must return a zero total and must not insert a negative tax line.
A partial refund must reduce the payable total without applying the same coupon a second time afterward.

Record outputs before anyone edits

The runner below is a proposed example, and it has not been executed against a production repository for this article. It loads a JSON call list, invokes the current function, and writes one golden file per call. A later check fails when the payable total, the audit payload, or the mutated cart fields differ from that file.

# proposal: not executed for this article
import json
from pathlib import Path

def record(calls_path: Path, golden_dir: Path, apply_adjustment) -> None:
    golden_dir.mkdir(parents=True, exist_ok=True)
    calls = json.loads(calls_path.read_text())
    for call in calls:
        result = apply_adjustment(call["cart"], call["coupon"], call["refund"])
        payload = {
            "total": result.total,
            "audit": result.audit,
            "cart": result.cart,
        }
        target = golden_dir / f"{call['id']}.json"
        target.write_text(json.dumps(payload, sort_keys=True, indent=2) + "\n")

def check(calls_path: Path, golden_dir: Path, apply_adjustment) -> list[str]:
    failures = []
    calls = json.loads(calls_path.read_text())
    for call in calls:
        result = apply_adjustment(call["cart"], call["coupon"], call["refund"])
        actual = {"total": result.total, "audit": result.audit, "cart": result.cart}
        expected = json.loads((golden_dir / f"{call['id']}.json").read_text())
        if actual != expected:
            failures.append(call["id"])
    return failures

A companion call file keeps the budget visible in review, which matters more than a clever assertion library. Three records are easier to dispute than a generated suite that nobody can read in one sitting. Keep each call identifier stable so a reviewer can map one golden file to one row in the later decision table.

[
  {"id": "coupon-then-tax", "cart": {"subtotal": "20.00"}, "coupon": "0.10", "refund": "0.00"},
  {"id": "zero-after-coupon", "cart": {"subtotal": "0.00"}, "coupon": "0.10", "refund": "0.00"},
  {"id": "refund-once", "cart": {"subtotal": "20.00"}, "coupon": "0.10", "refund": "5.00"}
]

Run the record step on the unchanged function, and commit the golden files before you open a cleanup branch. The check step then becomes the only executable gate that can approve a later structural edit safely. Store both commands in the module README only after a teammate can run them from a clean checkout. Parse money strings to Decimal inside the test adapter, or the golden files will pin binary float noise instead of cents.

python -m adjuster_characterize record --calls calls.json --out golden/
python -m adjuster_characterize check --calls calls.json --golden golden/
git diff --name-only main...HEAD

Keep the first edit inside the recorded lines

The smallest safe change, in this example, is extracting tax rounding into a pure helper while coupon and refund logic stay put. That extraction is safe only when the three golden files still match and the diff does not touch an unrecorded module. A rename of the audit writer, or a move of refund math, is a second change and waits for its own record.

# proposal: smallest safe extraction, not executed for this article
from decimal import Decimal, ROUND_HALF_UP

def round_tax(taxable: Decimal) -> Decimal:
    return taxable.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

The helper accepts a decimal taxable amount and returns a half-up cent, with no cart mutation inside it. Coupon percentages and refund subtraction remain in the original function until a later budget records them alone. That split is the whole structural change the first patch is allowed to contain under this budget.

Reviewers can apply a simple blast-radius rule without opening a style argument about names or file layout. If git diff --name-only lists a file that no recorded call executes, the patch is too wide for this budget. If the check command prints a call id, the edit changed behavior and is no longer a cleanup. The shell fragment below is a proposal, and it treats any path besides the characterized file as a failed budget.

# proposal: fail review when a changed path sits outside the characterized file
git diff --name-only main...HEAD > changed.txt
grep -vx 'billing/adjuster.py' changed.txt && echo 'patch too wide' && exit 1
python -m adjuster_characterize check --calls calls.json --golden golden/

The grep step prints every changed path that is not the characterized file, and a nonempty result means the patch is too wide. Run the characterization check only after that path list is empty, so a behavior failure is not confused with an oversized diff. Label this script as a review aid, because a rename of the characterized file would otherwise look like an oversized patch.

Review decision table

Observed signal	Review action
Golden files match, and the diff stays inside the characterized function	Accept the single extraction
Golden files match, but the diff edits an unrecorded helper	Reject the patch and ask for a smaller diff
Any recorded call fails	Stop and restore behavior before discussing structure
A draft adds calls that nobody has executed on current code	Label them as proposals and exclude them from the gate

Where a free assistant fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is useful here as a drafting aid for candidate call descriptions, not as a source of expected totals. You can paste the current function and ask for three input combinations that exercise coupon, zero, and refund branches. You still generate expected JSON by running the unchanged function, because a model can invent a cent that the production code does not return. A free server option fits the check command, since a clean environment avoids pinning a laptop path, a local time zone, or an editor-installed package.

Neither option replaces the golden-file gate, and this article does not claim a named model, a token quota, a hardware size, or a lasting availability window. Readers who lack a clean checkout can try that free server option while the golden-file check remains the only merge gate. Skip the assistant entirely when the function is short enough to record by hand in a single review sitting.

A failure you should expect on the first check

The first check often fails because the golden file captured a datetime string or a random audit identifier. Strip those fields in the recorder, or replace them with a fixed clock and a fixed identifier before you commit the files. A failure that mentions only the payable total is a behavior change, and it should block the extraction rather than refresh the golden file. Updating the golden file in the same commit as the extraction hides the drift this budget was built to catch.

Limitations and who should skip this

This approach pins observed behavior, and it does not prove that the observed behavior is legally or financially correct. A tax bug that already ships will be preserved on purpose until a separate, reviewed correction changes the golden file. Teams that need an audited tax opinion should not treat these three golden files as formal compliance evidence.

The runner also assumes you can call the function without sending a real payment or writing to a shared production table. If the audit insert cannot be intercepted, stop and wrap that write before you record any golden file. People who need a one-pull-request rewrite of the whole checkout flow should not use a three-call budget. The budget will reject that rewrite by design, which is the intended outcome rather than a tooling failure.

Model-drafted expected values are the wrong input for this gate, even when the draft looks precise. A hand-written expected total is also wrong unless it was produced by the unchanged function on the same inputs. If you cannot rerun the current function, you do not yet have a characterization record and should not edit.

What to merge first

Merge the golden files and the unchanged function in one commit, with no structural edit attached to that commit. Open a second commit that extracts only tax rounding, and require a clean check plus a name-only diff that stays inside the characterized file. Leave coupon stacking, refund ordering, and module splits for later budgets that carry their own recorded calls. That sequence is slower than accepting a six-file cleanup, and it still has an executable answer when the assistant layout is wrong.

Pin Duplicate-Delivery Receipts Before You Extract a Webhook Alias

Dakota Wu — Thu, 08 Oct 2026 09:37:57 +0000

Consider a billing module that verified webhook signatures, parsed event JSON, and inserted a ledger row for every accepted delivery. During a routine cleanup, the verifier moved into a new file while duplicate detection stayed in the old one. A replayed event then inserted a second ledger row, because the new boundary never received the prior event id. Treat that replay as an illustrative composite for the workflow below, not as a measured incident from this account.

What actually broke

The failure was not a weak signature algorithm, and it was not a missing unit test for a happy path. Callers already relied on a quieter duplicate contract whenever the provider retried the same event id. An extract can keep every crypto check green and still break that contract at the new function boundary. That is why the first artifact is a receipt row, not a fresh package layout or a renamed client class.

Three rules worth pinning

Production logs for this kind of module often show three rules that no README states in a checkable form. A repeated event id must return the first outcome and must skip a second ledger write. Signature header names can arrive under more than one alias, and only a known alias may pass. Raw body bytes are hashed before JSON parsing, so key order must not change the stored receipt.

Those rules describe current behavior, not a claim that the behavior is the desired long-term design. A characterization pin freezes those rules so a later cleanup can change exactly one rule on purpose. If finance still sends an older alias, record that alias in the pin before anyone deletes the accepting branch.

Receipt rows before any move

Keep the first suite to four fixtures, and give each fixture one receipt row with four fields. Store the event id, the alias that matched, the SHA-256 of the raw body, and an outcome label. Use the labels accepted, duplicate, and rejected, and do not invent a fourth label until a log line requires it. Four rows are enough to reject an extract that drifts, and a larger harness can wait for a real fifth case.

Use the fixture set below for this pass, and do not widen that set while the alias extract is still open.

A valid delivery with the primary signature alias and an event id the process has not seen.
The same raw bytes and the same event id sent again, which must remain a duplicate.
An unknown alias that must be rejected without inserting a ledger row or growing the seen-set.
A known alias whose signature fragment does not match the raw body hash prefix.

Add a fifth row only after an incident shows a behavior these four fixtures do not already cover. Until that incident exists, extra rows mostly add review noise and hide the contract you meant to protect. If two incidents describe the same alias mismatch, extend the existing row instead of opening a second fixture file. The goal is a stable pin for this extract, not a growing archive of one-off provider quirks.

Proposed artifact

The Python below is a proposed, unexecuted example for a fictional ledger webhook, not a log of a real outage. It is not a measured run, not a production verifier, and not a report of customer traffic. The signature compare is a toy prefix check so the example can run without a vendor secret. Replace that compare with your real check only after the receipt fields are already pinned on the old path.

# proposed and unexecuted: pin receipts before moving code
import hashlib
from dataclasses import dataclass

ALIASES = ("x-signature", "x-ledger-signature")

@dataclass(frozen=True)
class Receipt:
    event_id: str
    alias_used: str
    body_sha256: str
    outcome: str

def receipt_for(headers, body: bytes, seen: set[str]) -> Receipt:
    alias = next((name for name in ALIASES if name in headers), "")
    digest = hashlib.sha256(body).hexdigest()
    event_id = headers.get("x-event-id", "")
    fragment = headers.get(alias, "") if alias else ""
    if not alias or fragment != digest[:16]:
        outcome = "rejected"
    elif event_id in seen:
        outcome = "duplicate"
    else:
        seen.add(event_id)
        outcome = "accepted"
    return Receipt(event_id, alias, digest, outcome)

def test_duplicate_preserves_first_outcome():
    seen: set[str] = set()
    body = b'{"amount":10,"currency":"USD"}'
    headers = {
        "x-event-id": "evt-7",
        "x-signature": hashlib.sha256(body).hexdigest()[:16],
    }
    first = receipt_for(headers, body, seen)
    second = receipt_for(dict(headers), body, seen)
    assert (first.outcome, second.outcome) == ("accepted", "duplicate")
    assert first.body_sha256 == second.body_sha256
    assert first.alias_used == "x-signature"

Store expectations outside the assertion

Keep fixtures beside the test so a reviewer can open one directory and see every pinned byte. Store raw bodies as files, store headers as JSON, and keep expected receipts in a table the test loads. Do not recompute the expected hash inside the assertion when a stored digest can sit beside the body file. A stored digest makes a parser that reorders keys fail the pin, which is the drift you want to notice.

# proposed and unexecuted: load stored receipts, do not rebuild them in the assert
import json
from pathlib import Path

def load_cases(root: Path):
    rows = json.loads((root / "expected_receipts.json").read_text())
    for row in rows:
        body = (root / row["body_file"]).read_bytes()
        headers = json.loads((root / row["headers_file"]).read_text())
        yield headers, body, row["receipt"]

{
  "body_file": "evt-7.body",
  "headers_file": "evt-7.headers.json",
  "receipt": {
    "event_id": "evt-7",
    "alias_used": "x-signature",
    "outcome": "accepted"
  }
}

How to run the pin

Run one quiet pytest command, and treat any red assertion as a contract question rather than a prompt to edit the expected tuple. If the current module accepts an alias the test rejects, update the fixture from a captured request before you judge the extract. Reviewers should diff receipt fields, not the whole module, when they decide whether this pass may merge. A green run means these four observations still hold, and it says nothing about SQL, retries, or dashboard copy.

mkdir -p proposed/fixtures
python -m pytest proposed/test_delivery_receipts.py -q --tb=short

How to read a failing receipt

A failing body hash means the bytes changed, or the hash moved onto parsed JSON instead of the raw input. A failing alias means the extracted function consulted a different header map than the fixture recorded. A failing outcome, with hash and alias still equal, usually means duplicate memory never crossed the new boundary. Fix the wiring that dropped the seen-set before you touch backoff numbers, SQL, or operator-facing error text.

Smallest safe extract

After the four receipts pass against the old module, move only the alias tuple into a new file. Leave hashing, the seen-set, and the ledger write in the original function for this pass. The new module exports the tuple and does not import the database, the JSON parser, or the logger. Name the new file for the table it holds, so reviewers do not expect a full client move.

# proposed and unexecuted: smallest extract is the alias table
SIGNATURE_ALIASES = ("x-signature", "x-ledger-signature")

Point the old function at that tuple, then rerun the same four receipts without editing assertions. If the receipts match, stop the pull request there and write the follow-up as a separate change. A second pass may move the hash compare, but only after a mismatched-digest row is already in the pin. One exported tuple is a reviewable diff, while a relocated verifier plus a new retry loop is not.

This sequencing is deliberate scope control, not a claim that alias tables are the hardest part of webhook code. The hard part is knowing which behavior you promised not to change while the file is still messy. Keep the promise in the receipt table, and let the diff stay smaller than the module you are trying to leave behind. A reviewer should be able to explain this pass in one sentence without mentioning the database at all.

Where a separate host fits

Teams sometimes want a second machine for the first characterization run so the laptop that holds deploy keys is not also the test host. MonkeyCode offers free model access and a free server option for drafting redacted fixture rows and running this pytest file. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This article does not state a token quota, a model name, a hardware shape, or a benchmark.

Those details belong in the current product documentation, and they can change without notice to this post. Paste any model draft beside the receipt table, and do not treat the draft as a captured request. Accept a new row only when a human can match its alias and outcome to a log line or a saved request. If those fixtures already live in git, the same free access and free server option can host this first off-laptop run.

Check the current limits in the product documentation before you depend on that run for a merge. Do not send signing secrets, live customer bodies, or unredacted payment identifiers to a hosted runner or a model prompt. The example above uses a fake event id, and a real verifier should stay on systems your policy already allows. A free server is a place to run the pin, not a place to terminate live provider webhooks.

Scope lock

Observation	Pin in this pass	Leave for a later pass
Duplicate event id keeps the first outcome	Yes	No
Which signature alias matched	Yes	No
Raw body hash recorded before JSON parsing	Yes	No
Ledger SQL and account mapping	No	Yes, after receipts stay green
Retry delays and backoff numbers	No	Yes, unless a fixture already encodes them
Operator-facing text for rejected events	No	Yes

If a requested edit is not in the pin column, it does not belong beside the alias extract. Split that edit, or the receipt suite can no longer tell you which change broke the row. The table is a review tool, not a ranking of which subsystem is more important to the business. Use it in the pull request description so the next reviewer does not reopen ledger SQL in this same diff.

Limitations and who should skip this

A pin preserves current behavior, including behavior that a later ticket may correctly call a defect. If the seen-set is only process memory, a duplicate after restart can still write a second row. This four-row test will not catch that gap, because both calls share one in-memory set. Fixing persistence is a separate change with its own fixture, not a drive-by edit inside the alias extract.

Skip the workflow when you have no captured request and would be inventing the contract from memory. Skip it when an external contract suite is already authoritative and your team treats that suite as the merge gate. Also skip a hosted runner when policy forbids third-party execution of repository code, even for a four-row file.

Model output is only a hypothesis generator, not a witness of what production actually sent last week. A draft can add a header that never arrived, or it can repair a duplicate path you meant only to record. The receipt comparison decides whether this pass may merge, and a fluent model explanation does not.

Review checklist

Confirm each fixture came from a captured request or a log line, not from a model guess alone.
Confirm the duplicate fixture reuses the same event id and the same raw bytes as the first delivery.
Confirm the unknown-alias fixture expects rejected and asserts that the seen-set did not gain an id.
Confirm the pull request diff touches the alias tuple and its import, not the ledger write.
Confirm the pytest command in the change note is the same file that holds the four receipts.

What to merge this week

Write down the event id, the matched alias, the raw body hash, and the outcome label for four fixtures. Run that pin on the untouched module, extract only the alias tuple, and rerun the same assertions unchanged. Open the next cleanup only after that second run matches, even if the diff looks too small to matter. A small diff is the intended result when characterization comes first and the extract stays inside one seam.

Pin Absent, Empty, and Null Before You Extract a Settings Overlay

Dakota Wu — Thu, 08 Oct 2026 07:02:39 +0000

Picture a billing worker that skipped invoice generation after a weekend deploy because a blank region string replaced a valid environment value. In that scenario, the on-call engineer expected missing keys and blank values to behave the same way under overlay. They did not, and the same function treated a JSON null as an explicit delete of that key. File reads, environment scans, and CLI parsing lived in one module, so a broad cleanup looked useful and unsafe.

The notes below describe a characterization-first path for that class of settings merger, using a labeled composite module. Nothing in this draft reports a private employer incident, a measured outage length, or any customer count. The tests pin today's behavior before anyone moves code, including behavior that a later change may deliberately reject. Product assistance appears only after that contract exists, and the outreach disclosure sits beside the first mention.

What must stay stable during the extract

A safe extract preserves outputs for every input class you can name, even when those outputs look wrong. The merger currently walks sources in file, environment, then CLI order, and later writes replace earlier writes. A null value deletes the key, while an empty string, zero, and an empty list remain stored values. Moving I/O, renaming keys, and fixing null handling in the same diff would hide which edit changed behavior.

Decision table before any code move

Lock the contract in a table before you touch the module, because prose arguments drift during review. Each row names three source states and the exact merged result you observed, not the result you prefer. You should record absent keys, empty strings, JSON null, numeric zero, and empty collections as distinct states. If two reviewers disagree on a row, rerun the current function and paste the observed output into the table.

Case	File value	Env value	CLI value	Observed merge
later value wins	region=us	region=eu	absent	region=eu
empty string wins	region=us	region=empty string	absent	region=empty string
null deletes	region=us	region=null	absent	region absent
zero is kept	retries=3	retries=0	absent	retries=0
empty list kept	tags=[a]	tags=[]	absent	tags=[]
cli null deletes	region=us	region=eu	region=null	region absent
delete then set	region=null	region=eu	absent	region=eu

Composite module under test

Label this module as an unexecuted teaching example until you paste it into a scratch repository and run it. It intentionally keeps the awkward rules so the tests describe the code you have, not the code you want. Do not import production secrets, live endpoints, or customer fixtures into this small characterization harness at all. The function accepts plain dictionaries so characterization does not depend on disk layout or the process environment.

Messy function to pin

# scratch/overlay.py
# Labeled composite. Run it before you treat any row as evidence.

def overlay_settings(file_cfg, env_cfg, cli_cfg):
    merged = {}
    for source in (file_cfg, env_cfg, cli_cfg):
        for key, value in source.items():
            if value is None:
                merged.pop(key, None)
            else:
                merged[key] = value
    return merged

Characterization tests that refuse silent drift

These tests call the current function and compare full dictionaries, because a partial assert can miss deleted keys. A missing key and a stored null are different outcomes, so the expected object must not contain a placeholder. Keep one assertion per case name, and let that case name match the corresponding decision table row. If a test fails on the first run, fix the expectation to match observed output before you edit production code.

# tests/test_overlay_characterize.py
import pytest

from overlay import overlay_settings

CASES = [
    ("later_value_wins", {"region": "us"}, {"region": "eu"}, {}, {"region": "eu"}),
    ("empty_string_wins", {"region": "us"}, {"region": ""}, {}, {"region": ""}),
    ("null_deletes", {"region": "us"}, {"region": None}, {}, {}),
    ("zero_is_kept", {"retries": 3}, {"retries": 0}, {}, {"retries": 0}),
    ("empty_list_kept", {"tags": ["a"]}, {"tags": []}, {}, {"tags": []}),
    (
        "cli_null_deletes",
        {"region": "us"},
        {"region": "eu"},
        {"region": None},
        {},
    ),
    ("delete_then_set", {"region": None}, {"region": "eu"}, {}, {"region": "eu"}),
]


@pytest.mark.parametrize(
    "name,file_cfg,env_cfg,cli_cfg,expected",
    CASES,
    ids=[row[0] for row in CASES],
)
def test_overlay_characterizes_current_contract(
    name, file_cfg, env_cfg, cli_cfg, expected
):
    del name  # parametrize ids already label the failure
    assert overlay_settings(file_cfg, env_cfg, cli_cfg) == expected

Commands for a local proof

Run the suite from a clean shell so inherited variables do not masquerade as deliberate environment input. The sample function does not read the process environment, but the next extract might, and the habit should start now. Save the command output with the commit that introduces the tests, because a later green run is not the original evidence. If your laptop shell exports REGION or RETRIES, the clean-server rerun described later is the stricter check.

python -m venv .venv
. .venv/bin/activate
python -m pip install pytest
env -u REGION -u RETRIES python -m pytest -q tests/test_overlay_characterize.py

python -c 'from overlay import overlay_settings; print(overlay_settings({"region": "us"}, {"region": None}, {}))'
python -c 'from overlay import overlay_settings; print(overlay_settings({"region": "us"}, {"region": ""}, {}))'

git diff --stat -- overlay.py tests/test_overlay_characterize.py
git diff -- overlay.py

Smallest safe change after the pin

The smallest safe change extracts the loop body into a pure helper and leaves file, environment, and CLI loading untouched. Callers still pass three dictionaries in the same order, and null still deletes while empty strings still overwrite. That boundary is one function move plus an import update, which a reviewer can check against the unchanged test file. A second change may later redefine null, but only after the table marks those rows as intended breaks.

def apply_source(merged, source):
    for key, value in source.items():
        if value is None:
            merged.pop(key, None)
        else:
            merged[key] = value
    return merged


def overlay_settings(file_cfg, env_cfg, cli_cfg):
    merged = {}
    for source in (file_cfg, env_cfg, cli_cfg):
        apply_source(merged, source)
    return merged

What not to smuggle into the move

Adding deepcopy, sorting keys, or coercing empty strings to null would change observable results for some rows. Shared mutable values remain part of the current contract until a test proves callers rely on isolation. Sorting keys would change JSON dump order if a later caller serializes the dict without an explicit sort. Leave those ideas in a follow-up list, and keep their code out of this characterization-backed diff.

Review checklist before merge

Confirm every decision-table row has a passing test whose expected value came from a run, not from a design discussion.
Confirm the diff does not add file reads, environment reads, or argument parsing inside the new helper.
Confirm falsy values other than null, including zero and empty collections, still survive a later source write.
Confirm a planned behavior change is absent from this commit, or is listed as an explicit failing-test follow-up.

Where free model access and a free server fit

Once the table and tests exist, MonkeyCode's free model access can propose extra rows that your first pass missed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability claims for this draft: free model access, and a free server option for running work away from a laptop. This article does not name models, token quotas, hardware sizes, time limits, or benchmark scores, because those details were not verified here.

How to use the assistant without letting it rewrite the contract

Ask for candidate rows only, and require each proposal to cite a source cell and an expected dictionary. Reject any suggestion that corrects null into a stored value, unless you opened a separate behavior-change task. Paste accepted rows into the test file yourself, then run the suite, because an unrun suggestion is not evidence. Treat the open-source project as a helper for drafting and execution, not as a source of production truth.

Why a free server rerun matters for this bug

Environment leaks are part of the original failure mode, so a second run on a machine you did not customize is useful. A free server option lets you repeat the same pytest command outside a shell that already exports billing variables. Record the image or setup notes you can actually verify, and do not assume the remote box matches production libc, locale, or timezone.

If the remote result differs from the laptop result, stop the extract and reconcile the inputs before you trust either green log. If the characterization table is already green locally, one remote rerun is enough of a second opinion before you merge the helper. Keep that remote log next to the test file so a reviewer can see which command produced the green result.

Limitations and who should not use this path

This path assumes you can execute the current function on synthetic dictionaries and read stable, comparable outputs. Skip it when the merger calls a live network, a paid vendor, or a database you cannot stub without changing behavior. Also skip it when product policy already demands a null-semantics change in the same release, because characterization-then-extract would delay a required break. Free model access can invent plausible rows that never occur in your payloads, so every accepted row still needs a local run.

A free server option is a cleaner shell, not a production replica, and unverified hardware claims would only add false confidence. Do not use the remote run as evidence if you cannot record which command you executed and which test file you copied. Teams without permission to upload even synthetic fixtures should keep the suite on an approved internal runner instead. If you cannot explain absent, empty, and null to a reviewer in one table, you are not ready to extract the helper.

What this workflow deliberately leaves undone

The extract does not choose a new precedence policy, and it does not migrate callers to a typed settings object. It also does not prove thread safety, file encoding, or comment-preserving YAML edits, because those risks sat outside the pinned contract. A later change can replace null-delete with store-null, but it should flip specific table rows and show the red tests first. Until that change exists, the honest description of the system is the table you ran, not the overlay you wish you had.